The creative application and management of pre-trained language models has significantly improved the quality of information retrieval (IR). Existing IR models are typically trained on large datasets that contain hundreds of thousands or even millions of queries and relevance decisions, especially those that can generalize to new and uncommon topics.
The usefulness and necessity of such large-scale data in optimizing language models for information retrieval tasks has been called into question, raising scientific and engineering problems: in particular, from a scientific perspective, it is not clear that this vast amount of data is necessary, and from an engineering perspective, it is not clear how to train IR models for languages or niche domains for which there is little or no labeled IR data.
In a recent study, a team of researchers from the University of Waterloo, Stanford University, and IBM Research AI presented an approach to training small neural information retrieval models using just 10 gold relevance labels, i.e. a model with fewer than 100 million parameters. The approach is named PATH (Prompts as Automatically Optimizing Training Hyperparameters).
The basis of our method is to create fictitious document queries via a Language Model (LM). A key innovation is that the Language Model automatically optimizes the prompts it uses to create these fictitious queries, ensuring that training quality is optimized.
The steps the team shared are: start with a text corpus and a very small number of relevant labels. Then use LM to create potential search queries that may be relevant to documents in the corpus. To create training data, you need to create query-sentence pairs. Optimizing the LM prompts that guide you in creating queries is a key step to improve the quality of your synthetic data in response to inputs from the training step.
The team conducted trials using the BIRCO benchmark, which consists of difficult and unusual IR tasks, and found that this approach significantly improved the performance of trained models. In particular, small-scale models trained with minimal labeled data and optimized prompts outperform RankZephyr and are competitive with RankLLama. These later models, trained on datasets containing 7 billion parameters and over 100,000 labels, are significantly larger.
These results demonstrate how automated fast optimization can generate artificial datasets of superior quality. Not only does this approach show that effective IR models can be trained with fewer resources, but it also shows that with proper tuning of the data creation process, smaller models can outperform much larger ones.
Please check paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us. twitter.
participate Telegram Channel and LinkedIn GroupsUp.
If you like our work, you will love our Newsletter..
Please join us 45,000+ ML subreddits
🚀 Create, edit, and enhance tabular data with Gretel Navigator, the first complex AI system now generally available. [Advertisement]
Tanya Malhotra is a final year undergraduate student from the University of Petroleum and Energy Studies, Dehradun, doing a BTech in Computer Science Engineering with specialisation in Artificial Intelligence and Machine Learning.
She is an avid fan of Data Science and has strong analytical and critical thinking skills with a keen interest in learning new skills, group leadership and managing organized work.
