The eternal question: Where is the knowledge?
Very few people think about the fundamental existence of knowledge. Try this experiment with me and ask yourself: Where does knowledge exist? Where did it exist before we saw it, read it, and understood it from a book, article, image, or video?
The answer to this question is that knowledge, at its most fundamental level, resides in the collective human consciousness. In today's AI era, knowledge is beginning to reside in systems called Large Scale Language Models (LLMs).
1. How does an LLM work?
LLM works by generating text based on patterns learned from vast amounts of training data. It uses a technique called “next word prediction” to predict the next word in a text sequence, given the previous word. By repeating this process on millions of examples, the model learns to accurately predict the next word based on the previous context. Next word prediction is a fundamental feature that enables many language model applications, including:
- Autocomplete and predictive text (See this in search engines)
- Text Generation and Creative Writing Support (ChatGPT, Claude)
- Machine Translation (Google or Apple Translate, Duolingo)
- Conversational AI and Chatbots (ChatGPT, Claude)
- Text Summary (to summarize a long passage or an entire book)
- Sentiment Analysis and Natural Language Understanding (Classifying emails as spam or not based on the sentiment of the email body)
The ability to understand and predict natural language flow is a key strength of large-scale language models, enabling them to generate coherent, context-appropriate text for a wide range of tasks.
LLM uses a technique called “next word prediction” to predict the next word in a sequence of text given the previous word.
2. LLM Restrictions
However, LLM has limitations: you may “hallucinate” information, lack up-to-date knowledge, or struggle with tasks that require external information. Hallucinations occur because LLM has no reliable basis and selects each word purely based on statistical calculations.
OpenAI's ChatGPT Hallucination:
When asked to provide a source for its answer, ChatGPT corrected itself.
Google's Gemini Hallucination:
Google's AI Summary feature, which provides concise summaries at the top of search results, has recently come under fire for generating inaccurate, misleading and even dangerous answers — in one example, it recommended adding extra glue when making pizza.
3. Size of LLM
Despite these illusions, what makes LLM so great is the amount of data it's trained on – specifically, the number of parameters in the model. LLM parameters are numbers learned during the training process on a massive dataset. Parameters determine how the model processes information and makes predictions.
Important points about LLM parameters:
- Parameters act like adjustable dials that fine-tune the model's language understanding and production.
- The number of parameters in an LLM is often used as a proxy for model size and complexity, with larger models having more parameters: for example, GPT-4 is estimated to have 1.7 trillion parameters, while the largest Llama-2 model has 70 billion parameters.
- In general, more parameters mean more complex representations and may improve performance on language tasks, but larger models require more computational resources to train and deploy, making them more expensive and less accessible.
- The size of an LLM in memory is directly proportional to the number of parameters it contains: for example, the 70 billion parameter Llama-2 model requires at least two A100 GPUs (80 GB) for inference or fine-tuning.
4. Reducing LLM hallucinations using RAG and vector search:
Hallucinations are a feature of LLM, and as each model gets exponentially larger, it becomes harder to completely prevent such hallucinations, but that would be the ideal state.
Two ways AI companies are trying to minimize hallucinations are something called search augmentation generation (RAG) and vector searching in vector databases. These are key concepts for understanding how LLM works today and how it will continue to work in the future.
5. Search Extension Generation (RAG):
An LLM trained only on the original training data may hallucinate or generate plausible but incorrect or unverified content, especially when asked about a topic that is not well covered in the training data.
RAG addresses this issue by allowing the LLM to retrieve relevant information from external data sources, such as vector databases, and return that information to the user in the form of a response. (See figure below). RAG provides unique, contextually relevant, private data to LLM, improving performance and accuracy.
A vector database is a special type of database that stores information in a way that makes it easy to find similar things even if you don't have the exact same words or numbers.
The main benefits of using RAG to reduce LLM hallucinations include:
- Access to external up-to-date information: RAG allows the LLM to retrieve up-to-date facts and details from the vector database that may not have been included in the original training data, allowing the LLM's responses to be based on the most current available information.
- Improving factual accuracy: By incorporating retrieved information, RAG-enhanced LLMs can provide more accurate and detailed responses, reducing the chance of hallucinations and inaccuracies.
- Ability to leverage private data: RAG enables LLM to leverage private company data stored in the Vector database, further improving the relevance and accuracy of responses to specific domains or use cases.
- Efficient search with vector search: The vector search feature of the underlying database allows you to quickly retrieve the most relevant information to include in your LLM prompts.
6. Vector Search:
Vector search is a way to find things that are similar to each other, even if they don't have the exact same words. It's like when you're looking for a book but can't remember the title. You might remember that it had a blue cover and was about dogs. With vector search, a computer can look at words in the book's description and find things that are similar to “blue” and “dog”, even if they don't have the exact same words. All of this happens within a vector database.
Here's an example: Say you have a bunch of books and you want to find books similar to “The Very Hungry Caterpillar.” With vector search, the computer looks at the words in that book and finds other books with similar words and ideas, even if they have different titles. So you might find books like this:
- “Busy spider” (meaning that the spider is busy tending insects, and resembles a caterpillar or hungry)
- “Goodnight Moon” (because it's a classic children's book, just like “The Very Hungry Caterpillar”)
- “Where the Wild Things Are” (It has animals and imagination as its theme, so it has a similar theme to “The Very Hungry Caterpillar”)
Computers do this by converting the words in a book into special codes called vectors. These vectors contain numbers that represent the meaning or idea of the book. Books with similar meanings have similar vectors.
When you search for a book, the computer also converts your search into a vector. It then looks at all the book vectors to find the ones that are closest to your search vector. Those are the books that are closest to what you're looking for, even if the words aren't exactly the same. This is super helpful in finding related things, even if they're not an exact match. It's like having a super smart friend who can understand what you're looking for without you having to explain it perfectly.
Vector search is a way to find things that are similar to each other, even if they don't have the exact same words. It's like when you're looking for a book but can't remember the title. You might remember that it had a blue cover and was about dogs.
7. Open Source and Closed Source LLM:
Open source and closed source refer to the licensing and accessibility of the LLM.
Open source LLMs have publicly available source code, model architectures, and pre-trained weights. This ensures transparency by allowing researchers to access the underlying models, inspect the training data, and customize the code. Examples of open source LLMs include LLaMA, Mistral, and SDXL.
The main benefits of the Open Source LLM are:
- High degree of flexibility to customize to your specific needs
- The possibility to build unique solutions on open technologies and gain a competitive advantage
- Cost Savings and Optimized Hardware Infrastructure
Closed-source LLMs have proprietary source code and model weights that are not publicly available, which limits the possibilities for customization and adaptation. Examples include OpenAI's GPT-3.5, GPT-4, and GPT-4o, Anthropic's Claude, and Google's Gemini.
The benefits of a closed-source LLM include:
- Extensive research, development and continuous improvement backed by extensive resources
- Enthusiastic support from developers
- Intellectual property rights are clearly defined and companies do not usually own the underlying technology
The choice between open source and closed source LLM depends on factors such as your budget, specific requirements, the level of customization you want, and whether you prioritize cost savings or cutting edge performance. Open source offers greater flexibility and innovation, while closed source offers greater control, support, and ease of use.
Understanding how LLM works will enable non-technical people to converse with AI/ML engineers and data scientists.
In the next post, we will discuss Generative AI, diving deeper into aspects like pre-training, fine-tuning, and RAG. Until next time!
