In this series, deep learning.
And when people talk about deep learning, we immediately conjure up images of deep neural network architectures with many layers, neurons, and parameters.
In reality, the real change brought about by deep learning lies elsewhere.
it is about Learning data representation.
In this article we will focus on Embedding textwe explain their role in the field of machine learning and show how they can be realized. Understand and explore with Excel.
1. Comparison of classical machine returns and deep learning
This part explains why embeddings are introduced.
1.1 Where does deep learning fit in?
To understand embedding, we first need to clarify the place of deep learning.
use the term classic machine learning Describes deep architecture-independent methods.
All previous articles deal with classical machine learning, which can be explained in two complementary ways.
learning paradigm
- supervised learning
- unsupervised learning
model family
- distance-based model
- tree-based model
- weight-based model
Throughout this series, we have already studied the learning algorithms behind these models. In particular, we have seen gradient descent Applies to all weight-based models, from linear regression to neural networks.
Deep learning often reduces to neural networks with many layers.
However, this explanation is incomplete.
From an optimization perspective, deep learning does not introduce new learning rules.
So what does it introduce?
1.2 Deep learning as data representation learning
What is deep learning? How functionality is created.
Deep learning instead of manually designing features Automatically learns expressionsThis is often done through multiple successive transformations.
This also raises important conceptual questions.
Where is the boundary between feature engineering and model learning?
Let me clarify this with some examples.
- Polynomial regression is still a linear model, but the features are polynomial
- Kernel methods project data into a high-dimensional feature space.
- Density-based methods implicitly transform the data before training.
Deep learning inherits this idea, but at a larger scale.
From this perspective, deep learning belongs to:
- of Philosophy of feature engineeringas a representative
- of Weight-based model familyfor learning
1.3 Images and convolutional neural networks
The image is represented as pixel.
From a technical perspective, image data is already quantified and structured as a grid of numbers. however, information The data contained in these pixels is not structured in a way that classical models can easily exploit.
Pixels do not explicitly encode edges, shapes, textures, objects, etc.
Convolutional neural networks (CNNs) are designed as follows. Create information from pixels. Apply filters to discover local patterns and gradually combine them to create higher-level representations.
We published an article showing how to implement a CNN in Excel to make this process explicit.
In the case of images, the challenge is do not have Convert the data into numbers, Extract meaningful expressions From data that has already been quantified.
1.4 Text data: another problem
The texts present fundamentally different challenges.
Unlike images, text not numerical in nature.
Before modeling context and order, the first problem is more fundamental.
How can we represent words as numbers?
The first step is to create a numerical representation of the text.
In text deep learning, this step is handled as follows. embedded.
Embeddings convert discrete symbols (words) into vectors that the model can manipulate. The presence of embeddings allows us to model context, order, and relationships between words.
This article will focus on this first important step.
How embeddings create a numerical representation of textand how you can explore this process in Excel.
2. Two ways to learn text embeddings
In this article: IMDB Movie Review Dataset I'll explain both approaches. Datasets are distributed under the Apache License 2.0.
There are two main ways to learn text embedding, and we will use this dataset to perform both methods.
- Supervised: Create embeddings to predict emotions
- Unsupervised or self-supervised: uses the word2vec algorithm
In either case, the goal is the same.
Convert words to numeric vectors that can be used in machine learning models.
Before comparing these two approaches, we first need to clarify what embeddings are and how they relate to traditional machine learning.
2.1 Embeddings and classical machine learning
In classical machine learning, categorical data is typically handled as follows:
- label encodingassigns fixed integers but introduces an artificial order.
- One hot encodingthe order is removed but a high-dimensional sparse vector is generated.
How you can use them depends on the nature of your model.
distance-based model One-hot encoding cannot be used effectively because all categories will be equally distant from each other. Label encoding only works if you can attribute meaningful numbers to categories, which is usually not the case in traditional models.
weight-based model One-hot encoding can be used because the model learns weights for each category. In contrast, label encodings have fixed numbers that cannot be adjusted to represent meaningful relationships.
tree-based model Label encoding is actually acceptable because it treats all variables as categorical splits rather than numerical magnitudes. However, most implementations, including scikit-learn, still require numeric input. As a result, categories must be converted to numbers through label encoding or one-hot encoding. This would also be useful if the numbers had semantic meaning.
Overall, this highlights the limitations of traditional approaches.
Category values are fixed and not learned.
Embedding extends this idea. Learn the expression itself.
Each word is associated with a trainable vector, turning the representation of categories into a learning problem rather than a preprocessing step.
2.2 Supervised embedding
In supervised learning, embeddings are learned as part of a prediction task.
For example, the IMDB dataset has labels about sentiment analysis. Therefore, you can create very simple architectures.
In our case we can use a very simple architecture. That is, each word one dimensional embedding
This is possible because our goal is binary emotion classification.
Once training is complete, you will be able to: Export the embed Explore them in Excel.
Plotting embedding on the x-axis and word frequency on the y-axis shows a clear pattern.
- Positive values are associated with words such as: wonderful or wonderful,
- Negative values are associated with words such as: worst or waste
Logistic regression layers also have parameters that affect the final prediction, so they can reverse sign depending on initialization.
Finally, in Excel, rebuild the complete pipeline corresponding to the architecture you defined earlier.
Input field
The input text (review) is cut into words, with each line corresponding to one word.
Embedded search
Using a lookup function, the embedding values associated with each word are retrieved from the embedding table learned during training.
world average
The global average embedding is computed by averaging the embeddings of all words seen so far. This corresponds to a very simple sentence representation: the average of word vectors.
probability prediction
The averaged embeddings are then passed to a logistic function to generate sentiment probabilities.
what we observe
- strong words positive embedding (for example wonderful, love, fun) boosts the average.
- strong words negative embedding (for example worst, worst, waste) pulls the average downward.
- Neutral or low-weight words have little impact.
As more words are added, the overall average embedding becomes more stable and sentiment prediction becomes more reliable.
2.3 Word2Vec: Embedding by co-occurrence
In Word2Vec, Similarity does not mean that two words have the same meaning.
That is, they appear in similar contexts.
Word2Vec learns word embeddings by looking at: Which words tend to co-occur? In a fixed window within the text. Two words are considered similar if they occur frequently Around the same adjacent wordeven if their meanings are opposite.
Calculate the cosine similarity of the words as shown in the Excel sheet below. good Find the most similar words.
From the model's perspective, the surrounding words are almost the same. Only the adjective itself changes.
As a result, Word2Vec learns: “Good” and “bad” play similar roles in languageIn fact, they have opposite meanings, so please be careful not to confuse them.
That is, Word2Vec captures distribution similaritynot semantic polarity.
Here's a helpful way to think about this:
Words used in the same place are similar.
2.4 How to use embedding
In modern systems, RAG (Search Extension Generation)Embedding is often used to retrieve documents or passages to answer questions.
However, this approach has limitations.
The most commonly used embeddings are self-monitoring Methods based on co-occurrence or contextual predictive goals. As a result, general language similarities are captured rather than task-specific meanings.
This means:
- Although embeddings may retrieve linguistically similar text, it doesn't matter
- Semantic proximity is not guaranteed correctness of answer
Other embedding strategies can be used, such as task-adaptive embeddings and supervised embeddings, but the core often remains self-supervised.
Therefore, before using embeddings in downstream systems such as RAG, it is important to understand how they are created and what is and is not encoded.
conclusion
Embeddings are numerical representations of learned words that allow similarity to be measured.
Embeddings, whether learned through observation or co-occurrence, map words to vectors based on how they are used in the data. By exporting these to Excel, you can directly inspect these representations, calculate similarities, and understand what is captured and what is not.
This makes embeddings less of a mystery and clarifies their role as the basis for more complex systems such as search and RAGs.
