Speculative decoding accelerates autoregressive speech generation by allowing a fast draft model to suggest tokens that can be verified by a larger target model. However, for voice LLMs that generate acoustic tokens, exact token matching is overly limited. Many discrete tokens are acoustically or semantically interchangeable, reducing acceptance rates and limiting speedups. We introduce principal coarse-graining (PCG), which validates the proposal at the level of acoustic similarity groups (ASGs) derived from the embedding space of the target model. We define a coarse-grained distribution that accounts for overlap by dividing each token’s probability mass across the overlapping groups it contains, and perform rejection sampling on the resulting group variable. This ensures accuracy at the group level while allowing accepted draft tokens to actually stand in for members of the group. In LibriTTS, PCG improves acceptance and throughput compared to standard speculative decoding and previous speech-specific relaxations while preserving intelligibility and speaker similarity. These results suggest acoustically aware group-level acceptance as a simple and general method to accelerate speech token production while preserving speech quality.
