Can I read DPA Load Balancer: Load balancing for Data Parallel Actor-based systems on EtoBox?

DPA Load Balancer: Load balancing for Data Parallel Actor-based systems by Wang, Ziheng; Aguer, Atem; Ziai, Amir is a scholarly article available to read on EtoBox.

What is DPA Load Balancer: Load balancing for Data Parallel Actor-based systems about?

In this project we explore ways to dynamically load balance actors in a streaming framework. This is used to address input data skew that might lead to stragglers. We continuously monitor actors' input queue lengths for load, and redistribute inputs among reducers using consistent hashing if we detect stragglers. To ensure consistent processing post-redistribution, we adopt an approach that uses input forwarding combined with a state merge step at the end of the processing. We show that this approach can greatly alleviate stragglers for skewed data.

Author

Wang, Ziheng; Aguer, Atem; Ziai, Amir

Published

2023

Language

EN