Can I read End-to-End Analysis Automation over Distributed Resources with Luigi Analysis Workflows on EtoBox?

End-to-End Analysis Automation over Distributed Resources with Luigi Analysis Workflows by Rieger, Marcel is a scholarly article available to read on EtoBox.

What is End-to-End Analysis Automation over Distributed Resources with Luigi Analysis Workflows about?

In particle physics, workflow management systems are primarily used as tailored solutions in dedicated areas such as Monte Carlo production. However, physicists performing data analyses are usually required to steer their individual, complex workflows manually, frequently involving job submission in several stages and interaction with distributed storage systems by hand. This process is not only time-consuming and error-prone, but also leads to undocumented relations between particular workloads, rendering the steering of an analysis a serious challenge. This article presents the Luigi Analysis Workflow (Law) Python package which is based on the open-source pipelining tool Luigi, originally developed by Spotify. It establishes a generic design pattern for analyses of arbitrary scale and complexity, and shifts the focus from executing to defining the analysis logic. Law provides the building blocks to seamlessly integrate with interchangeable remote resources without, however, limiting itself to a specific choice of infrastructure. In particular, it introduces the concept of complete separation between analysis algorithms on the one hand, and run locations, storage locations, and so

Author

Rieger, Marcel

Published

2024

Language

EN