학술논문

StreamFlow: Cross-Breeding Cloud With HPC
Document Type
Periodical
Source
IEEE Transactions on Emerging Topics in Computing IEEE Trans. Emerg. Topics Comput. Emerging Topics in Computing, IEEE Transactions on. 9(4):1723-1737 Jan, 2021
Subject
Computing and Processing
Cloud computing
Task analysis
High performance computing
Data transfer
Computer architecture
DSL
Workflow management software
containers
data transfer
high performance computing (HPC)
kubernetes
streamflow
workflows
workflow management system (WMS)
Language
ISSN
2168-6750
2376-4562
Abstract
Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g., Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.