Open- and closed-model LLMs struggle to reason about runtime behavior in real code
A real-world benchmark finds current AI code helpers aren’t reliably tracing how software actually runs.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the paper
- Authors
- Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
What they did and found
A repository-level benchmark tested multiple LLMs on dynamic code reasoning. The best model scored about 37% accuracy, with strong local reasoning but poor handling of dataflow and cross-file execution. Creating fresh, harder variants via input changes further challenges models.
Why it matters
Practitioners shouldn’t rely on AI to fully explain runtime behavior; results show need for careful checks and hybrid tools when debugging or auditing code.
Two real-world findings matter for teams: AI tends to handle simple, local code logic better than it tracks how values move across many files. This means you’ll still need traditional debugging and test traces for complex behavior.
A practical takeaway is to pair AI prompts with execution traces or static analysis tools to guard against missing or misinterpreting runtime states during code reviews or fault finding.
Original sources · 1
Check the original paper for its authors, methods, version and access terms.
What remains uncertain
Results are from a specific, Python repository benchmark; generalization to all languages or environments is not proven.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.