Real-time Data Streaming using Kafka, Kinesis, and RabbitMQ
Vladyslav Vodopianov , Senior Software Engineer, Wirex Kyiv, UkraineAbstract
In the present work a comprehensive comparative analysis of the three leading platforms for organizing message streaming — Apache Kafka, Amazon Kinesis and RabbitMQ — is performed with the aim of identifying their architectural features, operational strengths and limitations under conditions of peak loads and stringent latency requirements. The study relies on a comprehensive methodological approach, including a systematic review of current scientific publications, the conduct of comparative performance measurements in laboratory settings and the synthesis of practical case studies of integrating the systems under consideration into real IT landscapes. The obtained results demonstrate that a reasoned choice of platform for stream processing depends on a multitude of interrelated factors: the volume of messages processed, the required throughput metrics and maximum response time, the preferred deployment model (on-premises solution, cloud service or their hybrid), the capabilities for seamless integration with existing services and infrastructure, as well as the project’s budgetary constraints. On the basis of the conducted analysis a unified decision-making methodology is proposed for selecting tools for streaming data processing, adapted to the tasks of data engineers, distributed systems architects and researchers of high-performance information platforms. The material is of practical interest to specialists designing fault-tolerant and scalable distributed message queues, as well as to experts in real-time analytics and cloud solution developers seeking to gain a deeper understanding of the architectural schemes and methods for optimizing throughput applied in Kafka, Kinesis and RabbitMQ. In addition, the research results may be useful to scientists in the field of distributed computing and the Internet of Things, focusing on the theoretical foundations and practical aspects of constructing reliable event-data pipelines.
Keywords
streaming data processing, Apache Kafka, Amazon Kinesis, RabbitMQ, big data
References
Research and Markets. (n.d.). Streaming analytics market by technology (real-time data processing, complex event processing, data visualization & reporting, event stream processing), application (fraud detection, predictive asset management, risk management) - Global forecast to 2029. https://www.researchandmarkets.com/report/streaming-analytics (accessed June 6, 2025)
Amilineni, K., Krishnan, R., Goyal, S., & Rao, S. V. N. (2022). Optimizing data stream throughput for real-time applications. In S. K. Bhoi, S. Patnaik, S. P. Mohanty, & B. K. Tripathy (Eds.), International conference on big data intelligence and computing, 410-417.
Padmanaban, K., Balaji, R. V., Baskar, S., & Sharma, V. (2024). Apache Kafka on big data event streaming for enhanced data flows. In 2024 8th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), 977-983. https://doi.org/10.1109/I-SMAC61858.2024.10714884
Velickovska, M., & Gusev, M. (2022). Comparing AWS streaming services: A use case on ECG data streams. In 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO),1387-1392. https://doi.org/10.23919/MIPRO55190.2022.9803359
George, J. (2024). Build a realtime data pipeline: Scalable application data analytics on Amazon Web Services (AWS). SSRN, 1-9. http://dx.doi.org/10.2139/ssrn.4963387
Bux, R., & Shenoy, G. S. (2024). Performance analysis of RESTful web services and RabbitMQ for microservices based systems on cloud environment. In 2024 3rd International Conference for Innovation in Technology (INOCON),1-6. https://doi.org/10.1109/INOCON60754.2024.10511747
Vyas, S., Jain, P., Sharma, S., & Soni, P. (2021). Literature review: A comparative study of real time streaming technologies and Apache Kafka. In 2021 Fourth International Conference on Computational Intelligence and Communication Technologies (CCICT), 146-153. https://doi.org/10.1109/CCICT53244.2021.00038
Chen, F., Yan, Z., & Gu, L. (2022). Towards low-latency big data infrastructure at Sangfor. In A. K. Das, P. K. Singh, & H. Ghayvat (Eds.), International symposium on emerging information security and applications, 37-54.
Dingorkar, S., Singh, S., Ghosh, S., & Roy, R. (2024). Real-time data processing architectures for IoT applications: A comprehensive review. In 2024 First International Conference on Technological Innovations and Advance Computing (TIACOMP), 507-513. https://doi.org/10.1109/TIACOMP64125.2024.00090
Confluent. (n.d.). Apache Kafka® performance. https://developer.confluent.io/learn/kafka-performance/ (accessed May 22, 2025)
Microsoft. (n.d.). Choose a stream processing technology in Azure. https://learn.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing (accessed June 07, 2025)
Amazon Web Services. (n.d.). Choosing between messaging services for serverless applications.https://aws.amazon.com/ru/blogs/compute/choosing-between-messaging-services-for-serverless-applications/ (accessed June 07, 2025)
Article Statistics
Copyright License
Copyright (c) 2025 Vladyslav Vodopianov

This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain the copyright of their manuscripts, and all Open Access articles are disseminated under the terms of the Creative Commons Attribution License 4.0 (CC-BY), which licenses unrestricted use, distribution, and reproduction in any medium, provided that the original work is appropriately cited. The use of general descriptive names, trade names, trademarks, and so forth in this publication, even if not specifically identified, does not imply that these names are not protected by the relevant laws and regulations.