Welcome to the ADMT Publication Server

A Holistic View of Stream Partitioning Costs

DocUID: 2017-007 Full Text: PDF

Author: Nikos R. Katsipoulakis, Alexandros Labrinidis, Panos K. Chrysanthis

Abstract: Stream processing has become the dominant processing model for monitoring and real-time analytics. Modern Parallel Stream Processing Engines (pSPEs) have made it feasible to increase the performance in both monitoring and analytical queries by parallelizing a query’s execution and distributing the load on multiple workers. A determining factor for the performance of a pSPE is the partitioning algorithm used to disseminate tuples to workers. Until now, partitioning methods in pSPEs have been similar to the ones used in parallel databases and only recently load-aware algorithms have been employed to improve the effectiveness of parallel execution. We identify and demonstrate the need to incorporate aggregation costs in the partitioning model when executing stateful operations in parallel, in order to minimize the overall latency and/or throughput. Towards this, we propose new stream partitioning algorithms, that consider both tuple imbalance and aggregation cost. We evaluate our proposed algorithms and show that they can achieve up to an order of magnitude better performance, compared to the current state of the art.

Published In: Proceedings of the VLDB Endowment

Volume: 10Number: 11Pages: 1286-1297

Place Published: http://www.vldb.org/pvldb/vol10.html

Year Published: 2017

Project: Others Subject Area: Data Streams

Publication Type: Conference Paper

Sponsor: Others

Citation:Text Latex BibTex XML Nikos R. Katsipoulakis, Alexandros Labrinidis, and Panos K. Chrysanthis. A Holistic View of Stream Partitioning Costs. Proceedings of the VLDB Endowment. 10(11):1286-1297. 2017. http://www.vldb.org/pvldb/vol10.html.