★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★Open research. Global perspectives.
editor@theamericanjournals.com Publisher Home ↗
Engineering and Technology OPEN ACCESS

Conformal Gating of Extracted Factor Vectors in LLM Signal Pipelines under Delayed Labels and Regime Shift: A Simulation Study

Wrick Talukdar
AWS AI & ML, IEEE CIS, USA
Gowtham Shankar
AWS AI, IEEE, USA
Prasad Prabhu
IEEE, USA
DOI Crossref DOI 10.37547/tajet/Volume08Issue10-07
tajet 2026
VOL. 8 / NO. 10 OCTOBER
FULL TEXT PP. 102-124
DOWNLOAD PDF

View all articles in this issue ↗

VOLUME 8
ISSUE 10
YEAR 2026
PAGES 102-124
RESEARCH OVERVIEW

Abstract

Language model pipelines extract multi-dimensional factor vectors from financial text, but a trading desk needs to know when an extracted vector can be trusted. Audit labels arrive late, errors cluster in stress regimes, and error scale shifts after model or prompt updates. We evaluate a conformal gate on the extraction layer in a seeded simulation (300 firms, 60 quarters, 200 replications) with heavy-tailed errors, regime stress, a structural break and a two-quarter audit lag. Per-dimension 90% intervals cover the full eight-dimension vector only 43 to 48% of the time, while a max-score construction reaches 89.8% before the break with intervals 20 to 26% narrower than Bonferroni. Calibration frozen at go-live falls to 79.9% joint coverage after the break. Rolling, weighted and delayed-feedback adaptive calibration recover 88.2 to 89.8% on average, but no label-driven method protects the quarters just after a shift. As a risk control, the gate holds the error rate among accepted signals near 10% after the break, the adaptive protocols accept only 25 to 35% of signals, and the gate lowers risk-adjusted return in this design. As a risk control, the adaptive gates reject most signals after the break, accepting only 25 to 35% in the main study, and gating lowers risk-adjusted return in this design. A label-free drift monitor flags 83.5% of abrupt breaks in the quarter they occur, against 2.5% for label-based monitoring, but under gradual drift it flags only 3.5%, so its advantage but in the supplementary simulation code, where it flags 78.5% of abrupt breaks, it flags only 3.5% under gradual drift, so its advantage is specific to abrupt breaks. To test whether these mechanisms appear with a real extractor, we run a pilot that extracts seven financial fields from the earnings press releases of 492 S&P 500 companies the earnings press releases of 492 large US companies from an S&P 500 earnings-call dataset with an open-weight model and scores each figure against the SEC XBRL-reported value. Extraction is accurate for most figures, 94.9% within 1% once reference-label errors in the XBRL tags are corrected, and most of the figures first scored as errors turned out to be label errors rather than misreading. Extraction is accurate for most figures, 95.5% within 1% once an error in our choice of revenue tag for the reference labels is corrected, and about a quarter of the figures first scored as large errors turned out to be label errors rather than misreading. The model's own GAAP flag is constant and detects none of the errors, while cross-model disagreement detects a minority of them, which supports the paper's central claim that a usable confidence signal must come from an external check rather than the model's own output. The one self-reported signal collected, the model's GAAP flag, is constant and uninformative, while disagreement across three models detects 45% of the large errors at the cost of flagging 8.6% of correct figures, which is consistent with the view that a usable confidence signal must come from a check outside a single model's output, although richer self-reported signals were not tested. A Monte Carlo simulation calibrated to the pilot reproduces these rates at scale. A Monte Carlo resampling calibrated to the pilot, with the detector's operating point held fixed, gives sampling intervals for the accuracy and large-error rates. The results support three design rules: The results suggest three design rules: report joint reliability with a max-score rectangle, gate on the composite the operator consumes, and size positions by a smoothed scale rather than by calibrated width. We place the gate in a deployment framework and state its limits.

RESEARCH TOPICS

Keywords

Conformal prediction large language models signal verification distribution shift

Downloads

Download data is not yet available.
SOURCES & SCHOLARSHIP

References

  1. Angelopoulos, A. N., & Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4), 494–591.
  2. Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., & Schuster, T. (2024). Conformal risk control. In Proceedings of the International Conference on Learning Representations (ICLR 2024).
  3. Angelopoulos, A. N., Candès, E. J., & Tibshirani, R. J. (2023). Conformal PID control for time series prediction. In Advances in Neural Information Processing Systems (Vol. 36).
  4. Barber, R. F., Candès, E. J., Ramdas, A., & Tibshirani, R. J. (2023). Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2), 816–845.
  5. Bates, S., Angelopoulos, A., Lei, L., Malik, J., & Jordan, M. (2021). Distribution-free, risk-controlling prediction sets. Journal of the ACM, 68(6), Article 43.
  6. Board of Governors of the Federal Reserve System & Office of the Comptroller of the Currency. (2011). Supervisory guidance on model risk management (SR 11-7).
  7. Commission Delegated Regulation (EU) 2017/589 supplementing Directive 2014/65/EU of the European Parliament and of the Council with regard to regulatory technical standards specifying the organisational requirements of investment firms engaged in algorithmic trading.
  8. Directive 2014/65/EU of the European Parliament and of the Council on markets in financial instruments (MiFID II).
  9. El Halabi, L., & Brandt, A. (2026). Adaptive conformal inference under delayed feedback: Coverage guarantees and a delay-to-memory diagnostic (arXiv:2609.07251). arXiv. https://arxiv.org/abs/2609.07251
  10. Gibbs, I., & Candès, E. J. (2021). Adaptive conformal inference under distribution shift. In Advances in Neural Information Processing Systems (Vol. 34).
  11. Gibbs, I., & Candès, E. J. (2024). Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25. https://www.jmlr.org/papers/v25/22-1218.html
  12. Kurry. (2025). S&P 500 earnings transcripts [Data set]. Hugging Face. MIT License. https://huggingface.co/datasets/kurry/sp500_earnings_transcripts
  13. Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094–1111.
  14. Messoudi, S., Destercke, S., & Rousseau, S. (2021). Copula-based conformal prediction for multi-target regression. Pattern Recognition, 120, 108101.
  15. OpenAI. (2025). gpt-oss-120b & gpt-oss-20b model card (arXiv:2508.10925). arXiv. https://arxiv.org/abs/2508.10925
  16. Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T., & Barzilay, R. (2024). Conformal language modeling. In Proceedings of the International Conference on Learning Representations (ICLR 2024).
  17. Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).
  18. Securities and Exchange Commission. Risk management controls for brokers or dealers with market access, 17 C.F.R. § 240.15c3-5.
  19. Securities and Exchange Commission. (n.d.-a). Financial Statement Data Sets. https://www.sec.gov/dera/data/financial-statement-data-sets
  20. Securities and Exchange Commission. (n.d.-b). Company tickers [Data file]. https://www.sec.gov/files/company_tickers.json
  21. Singh, R., & Rezaeimanesh, S. (2026). Ordinal gates, cardinal bets: Matching LLM confidence to the financial decision operator (arXiv:2609.00187). arXiv. https://arxiv.org/abs/2609.00187
  22. Tibshirani, R. J., Barber, R. F., Candès, E. J., & Ramdas, A. (2019). Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems (Vol. 32).
  23. Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic learning in a random world. Springer.
  24. Wang, Y., Kuang, P., Han, X., Xu, K., & Wang, H. (2026). Beyond surface statistics: Robust conformal prediction for LLMs via internal representations (arXiv:2604.16217). arXiv. https://arxiv.org/abs/2604.16217
  25. Xu, C., & Xie, Y. (2021). Conformal prediction interval for dynamic time-series. In Proceedings of the 38th International Conference on Machine Learning.
  26. Ye, Y., Han, J., Hu, A., Bu, J., Chen, Y., Wen, L., Mandic, D., Sun, D. D., Yinghui, X., & Xu, Z. (2026). The alpha illusion: Reported alpha from LLM trading agents should not be treated as deployment evidence (arXiv:2605.16895). arXiv. https://arxiv.org/abs/2605.16895
  27. Zaffran, M., Féron, O., Goude, Y., Josse, J., & Dieuleveut, A. (2022). Adaptive conformal predictions for time series. In Proceedings of the 39th International Conference on Machine Learning.