Shared AI endpoints give different results across days and providers
Two preregistered checks found same inputs yielded different readings over time and across providers, with logs not explaining the variation.
- Publication
- arXiv
- Stage
- Preprint
- What we read
- Summary of the paper
- Authors
- Haoyaun Zhu, Jie Zhang
- Universities and research institutions
- Not yet supplied in verified metadata; the Brief does not guess.
Institution metadata: OpenAlex record ↗
What they did and found
Two preregistered tests checked a language-model observer on shared endpoints. The same input and timing yielded different rankings on different days and from different providers; logs offered no clear cause for the variation.
Why it matters
Relying on a single readout from a shared AI endpoint can mislead; consistency is not guaranteed across time or providers.
Context matters: even with careful setup, a judge-like AI can produce different rankings for the same task at different times or under different workloads.
Example: in a customer-support test, run the same check several times in one day and report the range of rankings.
Original sources · 1
- Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints ↗arXiv · 2026-09-03
Check the original paper for its authors, methods, version and access terms.
What remains uncertain
Findings come from a small set of providers and tasks; results may not apply to other shared endpoints.
What is your take?
Ask a question, add useful context or share a different perspective. Keep the conversation respectful and grounded.