Large-scale language models such as Masked LM, Autoregressive LM, and Encoder/Decoder LM (BART) show state-of-the-art results for various NLP problems. Among these, autoregressive LMs such as GPT3 and GPT-4 show remarkable in-context learning ability and excellent long text production performance. Due to its importance, the community has made numerous attempts to scale up such autoregressive generated LMs with more data and parameters. The result has been significant achievements in real-world applications such as open-ended text generation and numerous downstream tasks.
Successful instances in the public domain include GPT-3, Gopher, Megatron-Turing, and PaLM. Large-scale autoregressive LM has been very successful, but has some drawbacks.
- Costly to implement due to the large number of model parameters required to store global information.
- Maintaining factual accuracy can be difficult and may misinform users.
- Updating the model knowledge obtained in pre-training with current information is expensive and results in stale responses.
Certain lines of research suggest using search to enhance language models to address the above issues. Searches can be included in LM during the pre-training or fine-tuning stages.
Most previous studies acquire and enhance BERT or encoder/decoder LMs with fine-tuning steps and show results for knowledge-intensive NLP applications. However, pre-training of autoregressive LM with rescue remains largely unexplored, especially given the remarkable performance of ChatGPT, highlighting the important role of autoregressive LM. RETRO recently demonstrated a substantially scalable search module from scratch by recovering billions of tokens and significantly reducing model parameters, all with less complexity than traditional GPT. proposed pre-training of the autoregressive LM using You can also change the knowledge held in the LM by changing the search database without retraining the LM.
To address the previous question and fill in the gaps, NVIDIA researchers have done extensive research on RETRO. To their knowledge, RETRO is the only search augmented autoregressive LM that supports large pretraining involving large pretraining corpora. Hundreds of billions or trillions of tokens. Because their exhaustive research outperforms standard GPT models in terms of perplexity, text generation quality, and downstream task performance, especially for knowledge-intensive tasks such as open domains, We highlight promising directions for autoregressive LM using search as a future underlying model. QA.
In this paper, they do a detailed study of search-extended LM and answer the following questions: Especially in knowledge-intensive work like open domain QA, there are sustained improvements in the quality of text production, factual accuracy, reduced toxicity, and accuracy of downstream tasks. Given the 25% increase in GPU time for pre-training, they believe pre-training a generative language model with search is a viable avenue. The full codebase and data are open sourced on GitHub.
check out paper and githubdon’t forget to join Our 19k+ ML SubReddit, cacophony channeland email newsletterWe share the latest AI research news, cool AI projects, and more. If you have any questions about the article above or missed something, feel free to email me. Asif@marktechpost.com
🚀 Check out 100 AI Tools in the AI Tools Club
Aneesh Tickoo is a consulting intern at MarktechPost. He is currently pursuing a Bachelor’s Degree in Data Science and Artificial Intelligence from the Indian Institute of Technology (IIT), Bhilai. He spends most of his time on projects aimed at harnessing the power of machine learning. His research interest is image processing and his passion is building solutions around it. He loves connecting with people and collaborating on interesting projects.
