Quantifying uncertainty in LLM is an important research direction to address hallucinations and other issues that limit reliable deployment. In this study, we show that inference trace length is a simple and useful confidence estimator in large-scale inference models. Through comprehensive experiments across multiple models, datasets, and prompts, we show that trace length performs in a comparable but complementary manner to other zero-shot confidence estimators, such as verbalized confidence. Our study reveals that post-training inference fundamentally alters the relationship between trace length and accuracy, going beyond previous work showing that post-training inference generally causes longer traces (e.g., “overthinking”). We investigated the mechanisms behind the performance of trace length as a reliability signal and observed that the effect remained even after controlling for confounding factors such as problem difficulty and GRPO-induced length bias. We identify high-entropy or “fork” tokens as playing an important role in the mechanism. Our findings show that post-training inference enhances the quantification of uncertainty beyond linguistic representations and establishes trace length as a practical confidence measure for large-scale inference models.
- † University of Southern California
- ‡ Stanford University
