In the industry recommendation system, Generative search (GR) We replace traditional embedding-based nearest neighbor search with large-scale language models (LLMs). These models represent items as Semantic ID (SID)— a discrete token sequence — and treats acquisition as an autoregressive decoding task. However, industrial applications often need to strictly follow business logic, such as enforcing content freshness or inventory availability. Standard autoregressive decoding cannot natively enforce these constraints, so the model often “hallucinates” invalid or out-of-stock product identifiers.
Accelerator Bottlenecks: Attempts vs. TPU/GPU
To guarantee valid output, developers typically use prefix trees (tries) to mask invalid tokens during each decoding step. Although traditional trie implementations are conceptually simple, they are fundamentally inefficient on hardware accelerators such as TPUs and GPUs.
The efficiency gap stems from two main issues:
- memory delay: Pointer tracking structures result in discontinuous random memory access patterns. This prevents memory coalescing and prevents you from taking advantage of the high-bandwidth memory (HBM) bursting capabilities of modern accelerators.
- Compilation incompatibilities: Accelerators rely on static computational graphs (such as Google’s XLA) for machine learning compilation. Standard attempts use data-dependent control flow and recursive branching, but these are incompatible with this paradigm and often force costly round trips between host devices.
STATIC: Sparse transition matrix fast trie index
Google DeepMind and Youtube Introduced by researchers static (Sparse Transition Matrix-Accelerated Trie Index for Constrained Decoding) to resolve these bottlenecks. Instead of treating the trie as a graph to be traversed, STATIC flattens it into a static graph. Compressed sparse rows (CSR) matrix. This transformation allows irregular tree traversals to be performed as fully vectorized sparse matrix operations.
Hybrid decoding architecture
STATIC employs a two-phase lookup strategy to balance memory usage and speed.
- High-density masking (t-1 d): first things first dFor =2 layers (highest branching coefficient), STATIC uses a bit-packed dense Boolean tensor. This results in ○(1) Lookup is the most computationally expensive initial step.
- Vectorized Node Transition Kernel (VNTK): For deeper layers (I ≥ 3), STATIC uses a branch-free kernel. This kernel performs “speculative slicing” of a fixed number of entries (Bt), corresponds to the maximum branching coefficient at that level. By using fixed-size slices regardless of the actual number of children, the entire decoding process remains a single static computational graph.
With this approach, I/O complexity ○(1) It scales relative to the size of the constraint set, whereas traditional hardware-accelerated binary search methods scale logarithmically (○(log|C|)).
Performance and scalability
batch size 2, beam size (M)/70, STATIC showed significant performance improvement compared to existing methods.
| method | Latency overhead per step (ms) | Percentage of total inference time |
| STATIC (ours) | +0.033 | 0.25% |
| Approximate PPV | +1.56 | 11.9% |
| hash bitmap | +12.3 | 94.0% |
| CPU try | +31.3 | 239% |
| PPV accurate | +34.1 | 260% |
STATIC is 948x speedup It outperformed CPU offload attempts and outperformed a precise binary search baseline (PPV). 1033x. Semantic ID vocabulary size (|V|) increases.
For a vocabulary of 20 million items, the HBM usage limit for STATIC is approximately 1.5GB. In practice, due to the uneven distribution and clustering of semantic IDs, the actual utilization is usually ≤75% of this limit. A good rule of thumb for capacity planning is approximately: 90 MB HBM for every 1 million constraints.
Introduction results
STATIC was introduced on YouTube to enforce a “last 7 days” freshness constraint on video recommendations. The system provided a vocabulary of 20 million perishables with 100% compliance.
Online A/B testing revealed the following:
- a +5.1% increase Fresh video views in 7 days.
- a +2.9% increase Fresh video views in 3 days.
- a +0.15% increase By click-through rate (CTR).
Cold start performance
This framework also addresses the “cold start” limitation of generative search, i.e. recommending items that were not seen during training. By constraining the model to the cold-start itemset of the Amazon reviews dataset, STATIC significantly improved performance compared to the unconstrained baseline, recording 0.00% Recall@1. These tests used the 1 billion parameter Gemma architecture. L = 4 tokens and vocabulary size |V|=256.
Important points
- vectorized efficiency: STATIC recasts constrained decoding from graph traversal problems into hardware-friendly vectorized sparse matrix operations by flattening the prefix tree into a static compressed sparse row (CSR) matrix.
- Significant speedup: The system achieves a latency of 0.033 ms per step, representing a 948x speedup compared to CPU offload attempts and a 47x to 1033x speedup compared to the hardware-accelerated binary search baseline. +1
- scalable ○(1) Complexity: By achieving ○(1) I/O complexity relative to constraint set size. STATIC maintains high performance with a low memory footprint of approximately 90 MB per million items.
- Proven results in production: Deployment on YouTube demonstrated 100% compliance with business logic constraints, increasing new video views by 5.1% and click-through rate by 0.15%.
- cold start solution: This framework enables generative search models to better recommend cold-start items, increasing the performance of Recall@1 from 0.00% to non-trivial levels on the Amazon Reviews benchmark.
Please check paper and code. Also, feel free to follow us Twitter Don’t forget to join us 120,000+ ML subreddits and subscribe our newsletter. hang on! Are you on telegram? You can now also participate by telegram.
