Skip to Main content Skip to Navigation
Conference papers

PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows

Abstract : This paper presents PaDaWAn, an infrastructure written in Python to provide loosely coupled in situ capabilities to accelerate file-based simulation workflows. It provides services for in-memory data exchange between applications and a simple configuration model to switch from a file-based workflow to a loosely coupled in situ workflow. The infrastructure is currently based on CEA-DAM Hercule parallel I/O library by providing an ABI-compatible library to intercept simulation data in a transparent way and to facilitate integration into existing simulation codes and tools. PaDaWAn implements a producer-consumer pattern with buffering of data in an in-memory staging service with automatic memory management and running on dedicated resources. We describe the key design decisions and main architectural features, and share the lessons learned from the development of the infrastructure and from setting up test runs on two production-like workflow cases. We conclude on the perspectives for our infrastructure.
Complete list of metadatas

Cited literature [17 references]  Display  Hide  Download

https://hal-cea.archives-ouvertes.fr/cea-02490376
Contributor : Julien Capul <>
Submitted on : Tuesday, February 25, 2020 - 10:43:05 AM
Last modification on : Wednesday, February 26, 2020 - 1:41:56 AM
Long-term archiving on: : Tuesday, May 26, 2020 - 1:57:06 PM

File

ws_isav116s3-file1(1).pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Julien Capul, Sébastien Morais, Jacques-Bernard Lekien. PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows. ISAV '18: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, Nov 2018, Dallas, United States. pp.7-12, ⟨10.1145/3281464.3281470⟩. ⟨cea-02490376⟩

Share

Metrics

Record views

72

Files downloads

136