Methodology for Designing Event-Driven High-Throughput Systems in the .NET Environment: Ingestion Pipeline Protocol and Hybrid Storage

The methodology examines a design approach for event-driven High-Throughput systems in the .NET environment, aimed at resilient processing of heterogeneous inbound traffic and suppression of degradation under stochastic load bursts. The relevance of the work stems from the limitations of synchronous REST-to-DB patterns and direct transactional writes to OLTP DBMSs, particularly in the context of the growth of Velocity–Variety–Volume, where latencies and locks increase nonlinearly, and scaling ceases to be quasi-linear. The goal of the study is to formalize an ingestion pipeline and a hybrid storage protocol that ensures the determinism of the hot path and the predictability of SLO. The novelty of the methodology lies in combining Canonical Domain Streams (ACL and Protobuf) and the Fan-In principle with asynchronous fire-and-forget ingestion on bounded channels (backpressure), as well as in reinterpreting ClickHouse as an active Calculation Node with a Stream-Back loop and a Unified Source of Truth, complemented by CQRS, hybrid caching, and an optimistic transactional outbox. Key conclusions: unification of inbound streams and decoupling of intake/processing increases resilience to spikes; migrating incremental aggregates into ClickHouse reduces load on the transactional contour; No-Magic and zero-allocation practices improve resource density and tail latencies. The methodology will be helpful to architects and engineers designing high-load streaming systems on .NET and Kafka.

High-Throughput, .NET, event-driven architecture, ingestion, Canonical Domain Streams, Protobuf, Fan-In, System, Threading, Channels

Çiftçi, B., & Çiloğlugil, B. (2025). A Review of Comparative Studies on Performance Evaluation of Communication Mechanisms for Microservices. Lecture Notes in Computer Science, 15650, 98–113. https://doi.org/10.1007/978-3-031-96962-1_7

Dai, F., Hossain, M. A., & Wang, Y. (2025). State of the Art in Parallel and Distributed Systems: Emerging Trends and Challenges. Electronics, 14(4), 677. https://doi.org/10.3390/electronics14040677

Goyal, M., & Bhasin, P. (2025). Moving from monolithic to microservices architecture for multi-agent systems. World Journal of Advanced Engineering Technology and Sciences, 15(1), 2119–2124. https://doi.org/10.30574/wjaets.2025.15.1.0480

Landry, M., Basile, E., Velempini, M., Talla, B. F., & Bomgni, A. B. (2024). The Impact of Data Ingestion Layer in an Improved Lambda Architecture. Lecture Notes in Networks and Systems, 824, 325–333. https://doi.org/10.1007/978-3-031-47715-7_22

Ma, R., Zhang, K., He, Z., Jing, Y., Sean, W. X., & Chen, Z. (2025). CHASE: A Native Relational Database for Hybrid Queries on Structured and Unstructured Data. ArXiv. https://doi.org/10.48550/arxiv.2501.05006

Putrama, I. M., & Martinek, P. (2024). Heterogeneous data integration: Challenges and opportunities. Data in Brief, 56, 110853. https://doi.org/10.1016/j.dib.2024.110853

Saleh, A., Morabito, R., Dustdar, S., Tarkoma, S., Pirttikangas, S., & Lovén, L. (2025). Towards Message Brokers for Generative AI: Survey, Challenges, and Opportunities. ACM Computing Surveys, 58(1), 1–37. https://doi.org/10.1145/3742891

Viotti, J. C., & Kinderkhedia, M. (2022). A Benchmark of JSON-compatible Binary Serialization Specifications. ArXiv. https://doi.org/10.48550/arxiv.2201.03051

Zandberg, K., Gulati, M., Wunder, G., & Baccelli, E. (2024). Model CBOR Serialization for Federated Learning. ArXiv. https://doi.org/10.48550/arxiv.2401.14056

Zhang, C., Li, Y., Zhang, R., Qian, W., & Zhou, A. (2022). Scalable and quantitative contention generation for performance evaluation on OLTP databases. Frontiers of Computer Science, 17(2), 172202. https://doi.org/10.1007/s11704-022-1056-2