Real-Time Log Analytics in Distributed Systems: Minimal-Latency Detection of Critical Events for Cloud-Native Back-End Platforms
Abstract
The paper examines real-time log analytics for distributed, cloud-native back-end systems, where operational decisions depend on the rapid recognition of critical runtime conditions. The relevance follows from the latency sensitivity of microservice-based finance and trading workloads, where propagation of failures, retries, and cascading timeouts rapidly degrades user-facing and internal processing. The novelty lies in an integrated analytical synthesis that ties stream-processing scalability evidence, tracing-tool capabilities, monitoring-tool taxonomies, instrumentation overhead studies, and modern log-anomaly detection research into one consistent engineering narrative. The study aims to develop a low-latency detection approach based on peer-reviewed findings. To achieve this goal, the work employs a systematic selection of recent literature, structured extraction of architectural patterns, and comparative reasoning across the ingestion, correlation, detection, and alerting stages. The analysis encompasses distributed stream processing benchmarks, near-real-time processing in practical architectures, runtime verification for streaming systems, and state-of-the-art log anomaly detection methods. The closing part derives design implications for practitioners building observability and incident-response pipelines.
Keywords
Downloads
References
- Aladib, L., Su, G., & Yang, J. (2025). Real-Time Monitoring for Distributed Stream Processing Systems Using Linear Temporal Logic. Electronics, 14(7), 1448. https://doi.org/10.3390/electronics14071448
- Daksa, D., & Kemala, E. (2025). Comparative Analysis of Apache Flink and Apache Spark for Near Real-Time Fraud Detection in Stream Processing Architecture. Procedia Computer Science, 242, 4691–4698. https://doi.org/10.1016/j.procs.2024.11.247
- Duan, X., Du, D., Liu, Z., Zhu, H., & Liang, C. (2024). LogEDL: Log Anomaly Detection by Evidential Deep Learning. Applied Sciences, 14(16), 7055. https://doi.org/10.3390/app14167055
- Giamattei, L., Guerriero, A., Pietrantuono, R., Russo, S., Malavolta, I., Islam, T., … Simon Panojo, F. (2024). Monitoring tools for DevOps and microservices: A systematic grey literature review. Journal of Systems and Software, 208, 111906. https://doi.org/10.1016/j.jss.2023.111906
- Hammad, M., Ahmad, A. & Andras, P. (2025). An empirical study on the performance overhead of code instrumentation in containerised microservices. Journal of Systems and Software, 230, 112573. https://doi.org/10.1016/j.jss.2025.112573.
- Henning, S., & Hasselbring, W. (2024). Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud. Journal of Systems and Software, 208, 111879. https://doi.org/10.1016/j.jss.2023.111879
- Janes, A., Li, X., & Lenarduzzi, V. (2023). Open tracing tools: Overview and critical comparison. Journal of Systems and Software, 204, 111793. https://doi.org/10.1016/j.jss.2023.111793
- Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024). The Impact of Log Parsing on Deep Learning-based Anomaly Detection in System Logs. Empirical Software Engineering, 29, 10. https://doi.org/10.1007/s10664-023-10440-5
- Landauer, M., & Skopik, F. (2023). Deep learning for anomaly detection in log data: A survey. Internet of Things and Cyber-Physical Systems, 5, 100071. https://doi.org/10.1016/j.iotcps.2023.100071
- Li, W., Wu, Y., Huang, W., Ou, W., Wang, H., Zhou, F., & Deng, L. (2025). System log anomaly detection based on contrastive learning and retrieval augmented. Scientific Reports, 15, 38370. https://doi.org/10.1038/s41598-025-22208-7