Why Code World Models Need State-Tracking Architectures

January 7, 2026

State-tracking is fundamental to code execution, but current transformer-based Code World Models struggle with long-horizon tracking. We show that architectures designed for state-tracking require state-tracking data to realize their theoretical potential. Training on Python REPL traces with explicit intermediate states dramatically improves learnability and length extrapolation.