학술논문

Adaptive Placement of Data Analysis Tasks For Staging Based In-Situ Processing
Document Type
Conference
Source
2021 IEEE 28th International Conference on High Performance Computing, Data, and Analytics (HiPC) HIPC High Performance Computing, Data, and Analytics (HiPC), 2021 IEEE 28th International Conference on. :242-251 Dec, 2021
Subject
Bioengineering
Communication, Networking and Broadcast Technologies
Computing and Processing
Data analysis
Runtime
Soft sensors
Scalability
High performance computing
Conferences
Computer architecture
in-situ
in-transit
data-driven
adaptive workflow
monitor
near-real-time decision
Language
ISSN
2640-0316
Abstract
In-situ processing addresses the gap between speeds of computing and I/O capabilities by processing data close to the data source, i.e., on the same system as the data source (e.g., a simulation). However, the effective implementation of in-situ processing workflows requires the optimization of several design parameters such as where on the system workflow data analysis/visualization (ana/vis) as placed and how execution as well as the interaction and data exchanges between ana/vis are coordinated. For example, in the case of hybrid in-situ processing, interacting ana/vis may be tightly or loosely coupled depending on their placement, and this can lead to very different performance and scalability. A key challenge is deciding the most appropriate ana/vis placement, which depends on dynamic applications, workflow, and system characteristics that might change at runtime. In this paper, we present a framework to support online adaptive data analysis placement during the execution of an in-situ workflow. Specifically, the paper presents a model and architecture, and explores several data analysis placement strategies. Evaluation results show that dynamically choosing appropriate data analysis placement strategies can balance the benefits and overhead of different data analysis placement patterns to reduce in-situ processing time.