Chinese artificial intelligence company DeepSeek has released its latest, large-scale language model, DeepSeek-V3.2-Exp. This is the “experimental version” that includes the company's “sparse attention” mechanism, which can improve performance when given long inputs.
“We look forward to announcing the official release of our experimental version of the model, DeepSeek-V3.2-Exp,” the company said of the latest release. “As an intermediate step into next-generation architecture, V3.2-EXP is based on the v3.1 end by introducing the sparse attention of DeepSeek. It is a sparse attention mechanism designed to explore and validate optimizations for training and inference efficiency in long-view scenarios.
Deepseek has some interesting selling points for the latest LLM. This should improve performance for longer token streams. (📷: deepseek)
Large-scale language models are the lifeblood of the current “artificial intelligence” boom, despite their own absence. Trained with vast, generally malicious data without copyright or permission considerations, it converts user-supplied input into a stream of “tokens” and then returns the most statistically fitting tokens needed to continue the stream.
Deepseek has risen to fame First open model releaseCritics quickly pointed out that there are smaller “distilled” variants, especially smaller “distilled” variants, thanks to the claim that Deepseek-R1 is reaching standard with comparable proprietary models like Openai and Meta on a “mere” $10 million budget.
However, like all LLMS, DeepSeek-R1 had a limitation. It puts aside the fundamental problem that LLM cannot “understand” in a meaningful way, leading to a response known as “hatography” of being completely divorced from reality. The key issue is the size of the “context window” or the number of tokens the model can keep in memory at any time. If you have a large number of inputs, such as long enough conversations or inappropriate requests to summarise a long document, there is a LLM-friendly task that can produce results carefully when it is not a careful summary compared to the person who actually reads the document in question.
Deepseek argues that the new model has the advantage of dealing with longer context windows better, which means that it has the performance of its “PAR” with its predecessor. (📷: deepseek)
As a band-aid for this issue, DeepSeek-V3.2-Exp includes a company implementation of a “sparse attention” system called Deepseek Sparks Attention (DSA). Called a “prototype,” it is designed to prune tokens in a way that maximizes the useful context provided to the model, while minimizing the overall length of the token stream. In many benchmarks, especially benchmarks that use tools to create “agent” models that allow you to perform actions on your behalf, this provides a small performance gain. There is a small performance loss as well for others, including benchmarks that do not exceed the context window and thus cannot benefit from sparse attention.
More information available In the GitHub repository for your projectwith a demo and a link to the kernel. The model weights, along with the contents of the repository itself, are released under the acceptable MIT license. Additional information is available Hugging my face.
