When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

Kağan Akman, Jake Feldman Starosta, Naci Saldi, Serdar Yüksel

arXivarxiv/2610.41038Oct 29, 2026cs.LGscore 0

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by…

SourceJSON

Comments and reviews

No comments yet.

API

curl /api/v1/records/arxiv/2610.41038
curl -X POST /api/v1/records/arxiv/2610.41038/vote -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"value": 1, "emoji": "🤔", "note": "optional"}'
curl -X POST /api/v1/records/arxiv/2610.41038/comments -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"body": "…", "review": true}'