PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows

Julien Capul (1, 2) , Sébastien Morais (1, 2) , Jacques-Bernard Lekien (2, 1)

1 DAM/DIF - DAM Île-de-France
2 CEA - Commissariat à l'énergie atomique et aux énergies alternatives

This paper presents PaDaWAn, an infrastructure written in Python to provide loosely coupled in situ capabilities to accelerate file-based simulation workflows. It provides services for in-memory data exchange between applications and a simple configuration model to switch from a file-based workflow to a loosely coupled in situ workflow. The infrastructure is currently based on CEA-DAM Hercule parallel I/O library by providing an ABI-compatible library to intercept simulation data in a transparent way and to facilitate integration into existing simulation codes and tools. PaDaWAn implements a producer-consumer pattern with buffering of data in an in-memory staging service with automatic memory management and running on dedicated resources. We describe the key design decisions and main architectural features, and share the lessons learned from the development of the infrastructure and from setting up test runs on two production-like workflow cases. We conclude on the perspectives for our infrastructure.

Mots clés: In situ; In transit; I/O; Workflow; Python; HPC; Simulation; Coupling; PaDaWAn

Domaines: Informatique [cs]; Calcul parallèle, distribué et partagé [cs.DC]; Modélisation et simulation; Génie logiciel [cs.SE]; Performance et fiabilité [cs.PF]

Julien Capul, Sébastien Morais, Jacques-Bernard Lekien. PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows. ISAV '18: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, Nov 2018, Dallas, United States. pp.7-12, ⟨10.1145/3281464.3281470⟩. ⟨cea-02490376⟩