★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★Open research. Global perspectives.
editor@theamericanjournals.comPublisher Home ↗
Engineering and Technology OPEN ACCESS

AutoCortex: Autonomous Job Orchestration Framework for Intelligent ETL Scheduling Using Linux and AutoSys

Tirumalavenkata Naga Lakshmi Jammula
Independent Researcher, Charlotte, NC, United States of America
DOI Crossref DOI 10.37547/tajet/Volume08Issue05-02
tajet 2026
VOL. 8 / NO. 05 MAY
FULL TEXT PP. 10-17
DOWNLOAD PDF

View all articles in this issue ↗

VOLUME 8
ISSUE 05
YEAR 2026
PAGES 10-17
RESEARCH OVERVIEW

Abstract

The current data warehousing infrastructure heavily depends on the ETL (Extract, Transform, Load) process to deliver timely and accurate information to support analytics and decision-making processes. Nevertheless, conventional ETL scheduling methodologies, such as AutoSys and Linux-based scripting, tend to be rigid, rule-based, and inflexible to handle dynamic workload management, thereby leading to wastage of resources, delays, and inefficiencies. This paper novel study of AutoCortex, an autonomous job orchestration framework which improves ETL scheduling efficiency through intelligent, adaptive and self-optimizing mechanisms.The AutoCortex framework combines Linux shell scripting, AutoSys job scheduling, decision models to facilitate real-time monitoring, predictive scheduling and automated dependency of complex ETL workflows. The AutoCortex framework will utilize historical ETL execution information, system resource utilization and job dependencies to dynamically manage ETL job priorities, optimize execution sequences, and eliminate performance bottlenecks. AutoCortex will also include fault tolerance mechanisms such as automated recovery, alerting and self-healing to enhance system reliability.

RESEARCH TOPICS

Keywords

AutoCortex ETL Scheduling AutoSys

Downloads

Download data is not yet available.
SOURCES & SCHOLARSHIP

References

  1. Adhikari, M., Amgoth, T., & Srirama, S. N. (2019). A survey on scheduling strategies for workflows in cloud environment and emerging trends. ACM Computing Surveys, 52(4), 1–36. https://doi.org/10.1145/3325097
  2. Ali, S. M. F., & Wrembel, R. (2017). From conceptual design to performance optimization of ETL workflows: Current state of research and open problems. The VLDB Journal, 26(6), 777–801. https://doi.org/10.1007/s00778-017-0477-2
  3. Dayal, U., Castellanos, M., Simitsis, A., & Wilkinson, K. (2009). Data integration flows for business intelligence. In Proceedings of the VLDB Endowment (pp. 1–11). https://doi.org/10.1145/1516360.1516362
  4. Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P., Mayani, R., Chen, W., Ferreira da Silva, R., Livny, M., & Wenger, K. (2015). Pegasus: A workflow management system for science automation. Future Generation Computer Systems, 46, 17–35. https://doi.org/10.1016/j.future.2014.10.008
  5. Karagiannis, A., Vassiliadis, P., & Simitsis, A. (2013). Scheduling strategies for efficient ETL execution. Information Systems, 38(6), 927–945. https://doi.org/10.1016/j.is.2012.12.001
  6. Kephart, J. O., & Chess, D. M. (2003). The vision of autonomic computing. Computer, 36(1), 41–50. https://doi.org/10.1109/MC.2003.1160055
  7. Liu, J., Pacitti, E., Valduriez, P., & Mattoso, M. (2015). A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13(4), 457–493. https://doi.org/10.1007/s10723-015-9329-8
  8. Prodan, R., & Fahringer, T. (2005). Dynamic scheduling of scientific workflow applications on the grid: A case study. In Proceedings of the ACM Symposium on Applied Computing (pp. 687–694). https://doi.org/10.1145/1066677.1066835
  9. Thiele, M., Fischer, U., & Lehner, W. (2009). Partition-based workload scheduling in living data warehouse environments. Information Systems, 34(4–5), 382–399. https://doi.org/10.1016/j.is.2008.06.001
  10. Topcuoglu, H., Hariri, S., & Wu, M. Y. (2002). Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems, 13(3), 260–274. https://doi.org/10.1109/71.993206
  11. Tziovara, V., Vassiliadis, P., & Simitsis, A. (2007). Deciding the physical implementation of ETL workflows. In Proceedings of the ACM International Workshop on Data Warehousing and OLAP (pp. 49–56). https://doi.org/10.1145/1317331.1317341
  12. Zarate, G., Ritter, D., Falcone, M., et al. (2024). Evolution of ETL processes towards modern data pipeline technologies. Proceedings of the ACM. https://doi.org/10.1145/3685651.3686662
  13. Sanjalawe, Y., et al. (2025). AI-driven job scheduling in cloud computing. Artificial Intelligence Review. https://doi.org/10.1007/s10462-025-11208-8