An intuitive overview of the weak director | Essam Witham | June 2024

In this story, we introduce and broadly explore the topic of weakly supervised learning in machine learning. Weakly supervised learning is one of the learning paradigms in machine learning that has started to gain attention in recent years. In a nutshell, fully supervised learning requires a training set. (x,y) where Yeah The correct label is XOn the other hand, weak supervision assumes a general setting. (x, y') where Yo It is not necessarily correct (i.e. it is a potentially incorrect, weak label). Furthermore, with weak supervision, there may be multiple weak supervisors, (x, y'1, y'2, …, y'F) For each example each Yeah It has been taken from another source and may be incorrect.

table of contents

∘ Problem description
∘ General Framework
∘ General Architecture
∘ Snorkel
∘ Examples of Weak Supervision

Problem statement

In more practical terms, weakly supervised learning solves what I call the supervised learning dilemma: If you're a company or someone with a new idea for machine learning, you need data. It's not that hard to collect a lot of examples. (x1, x2, …, xm) In some cases, this can be done programmatically, but the real dilemma is that you would need to hire human annotators to label this data, paying them around $Z per label. The problem is not only that you're not sure if your project is worth it, but that you might not be able to afford to hire annotators, as this process can be quite costly, especially in fields like law and medicine.

So, you may wonder, how do we solve this problem with weak supervision? Simply put, instead of paying annotators to label you, we ask them to provide us with some general, possibly inaccurate, rules for labeling your data (which takes much less time and money). In some cases, it may be easy for your development team to figure out these rules themselves (for example, if the task does not require expert annotators).

Now consider an example use case: You are trying to build an NLP system to mask words that correspond to sensitive information like phone numbers, names, addresses, etc. Instead of hiring people to label words in a corpus of sentences that you collect, you write some functions that automatically label all the data based on whether the word is all digits (likely but not certain it's a phone number), whether the word starts with a capital letter but isn't at the beginning of a sentence (likely but not certain it's a name), etc., and then train the system on the weakly labeled data. You might think that a trained model would not be better than such a labeling source, but that's a mistake. Weakly supervised models are designed to generalize beyond the labeling source by recognizing that there is uncertainty and often accounting for it in some way.

General Framework

Let us now take a formal look at the weak supervision framework employed in natural language processing.

✦ Given the

A series of debt Labeling feature {L1 L2, …, LF} where L.J. Assigning weak (i.e. potentially incorrect) labels to inputs X Labeling Functions L.J. This can be one of the following:

1. Crowdsourced annotators (may not be accurate)
2. Remotely supervised labels (i.e., extracted from a separate knowledge base)
3. Weak models (e.g., inherently weak or trained on a different task)
4. Heuristic features (e.g., label observations based on the presence of keywords or patterns, or defined by domain experts)
5. Gazetteers (e.g., label observations based on their occurrence in a particular list)
6. LLM call at specific prompt P (recent work)
7. A general function that will (hopefully) perform better than random guessing at guessing the label of x.

In general Li In some cases, no label is assigned (for example, in the following heuristic function): “If the word contains numbers, label it as a phone number, if not, don't label it.”).

Assuming there are N examples in the training set, this corresponds to an (N,F) weak label matrix for sequence classification. For token classification with sequences of length T, this is an (N,T,F) matrix of weak labels.

✦ Recruitment

We train a model,M,that effectively exploits weakly labeled data and,when present, strongly labeled data.

✦ Common NLP tasks

• Sequence Classification (e.g. sentiment analysis) or Token Classification (e.g. named entity recognition) Here, the labeling function is typically a heuristic function or a gazetteer.
• Low Resource Translation (x→y) Here, the labeling function is usually a weaker translation model (eg, a reverse translation model). (y→x) Add more (x,y) Translation pair.

General Architecture

For sequence or token classification tasks, the most common architectures in the literature appear to take the following form:

of Label Model It learns to map the output from the label function to probabilistic or deterministic labels that are used to train the final model. In other words, it takes the (N,F) or (N,T,F) label matrix described above and returns a (N) or (N,T) matrix of labels (often these are probabilistic (i.e. soft) labels).

of End Model After this step they are used separately and are just regular classifiers operating on the soft labels (made possible by cross-entropy loss) generated by the label model. Some architectures use deep learning to merge the label model and the end model.

Note that once we train the label model, we use it to generate labels for the final model and then we no longer use the label model. In this sense, this is completely different from staking, even if the label function is another machine learning model.

Alternative Architectureis the default in the case of translation (less common in sequence/token classification) and weights weak example (src, trg) pairs based on their quality (usually there is only one labeling function for translation, which is the reverse weak model as mentioned above). Such weights can be used in the loss function so that the model learns more from higher quality examples and less from lower quality examples. Approaches in this case try to devise ways to assess the quality of a given example. For example, one approach uses round-trip BLEU scores (i.e. translating a sentence to the target and then back to the source) to estimate such weights.

snorkel

To see an example of how the label model works, snorkel This is arguably the most fundamental work in weakly supervised learning for sequence classification.

In snorkeling, the authors P(yi|Λ(xi)) where Λ(xi) is the weak label vector for the i-th example. Once we know this probability, we can use it as the soft label for the final model (because, as we said before, cross-entropy loss can handle soft labels). Also, P(y, Λ(x)) That way you can easily find P(y|Λ(x)).

From the above equation, we can see that we are modeling using the same assumptions as logistic regression. P(y, Λ(x)) (Z means sigmoid/softmax regularization). The difference is, wax we have w.φ(Λ(xi),yi). especially, φ(Λ(xi),yi) is a dimensional vector 2F+|C|. F is the number of labeling functions as before, while C is the set of pairs of correlated labeling functions (so |C| is the number of correlated pairs). The authors refer to a method in another paper to automate the construction of C, but for brevity we will not go into detail here.

vector φ(Λ(xi),yi) What's Included:

• debt A binary element specifying whether each labeling function abstained for a given example.
• debt A binary element that specifies whether each label function is equal to the true label y (where y is left as a variable and serves as an input to the distribution)
• C In this example, a binary element that specifies whether each correlated pair voted the same.

We then train this label model (i.e. estimate the length weight vector). 2F+|C|) by solving the following objective: minimize the negative log marginal likelihood.

Note that we don't need any information about y, since this objective is solved regardless of the particular value represented by the sum. If we look closely (undoing the negative and logarithms), we can see that this is equivalent to finding the weights that maximize the probability of one of the true labels.

Once the label model is trained, we use it to no Soft Label P(y1|Λ(x1)), P(y2|Λ(x2)), …, P(yN|Λ(xN)) This is then used to train a discriminative model, typically of some kind (i.e., a classifier).

Examples of Weak Supervision

Snorkel has a great tutorial on spam classification. Skweak is another package (and paper) that provides the basis for weakly supervised token classification. Here is an example of how to get started with Skweak, shown on Github:

First, define the labeling function:

import spacy, re
from skweak import heuristics, gazetteers, generative, utils
### LF 1: heuristic to detect occurrences of MONEY entities
def money_detector(doc):
for tok in doc[1:]:
if tok.text[0].isdigit() and tok.nbor(-1).is_currency:
yield tok.i-1, tok.i+1, "MONEY"
lf1 = heuristics.FunctionAnnotator("money", money_detector)
### LF 2: detection of years with a regex
lf2= heuristics.TokenConstraintAnnotator("years", lambda tok: re.match("(19|20)\d{2}$", 
tok.text), "DATE")
### LF 3: a gazetteer with a few names
NAMES = [("Barack", "Obama"), ("Donald", "Trump"), ("Joe", "Biden")]
trie = gazetteers.Trie(NAMES)
lf3 = gazetteers.GazetteerAnnotator("presidents", {"PERSON":trie})

Applying to a corpus

# We create a corpus (here with a single text)
nlp = spacy.load("en_core_web_sm")
doc = nlp("Donald Trump paid $750 in federal income taxes in 2016")
# apply the labelling functions
doc = lf3(lf2(lf1(doc)))

Create and fit a label model

# create and fit the HMM aggregation model
hmm = generative.HMM("hmm", ["PERSON", "DATE", "MONEY"])
hmm.fit([doc]*10)
# once fitted, we simply apply the model to aggregate all functions
doc = hmm(doc)
# we can then visualise the final result (in Jupyter)
utils.display_entities(doc, "hmm")

Of course, you can also use the estimated soft labels to train a classifier based on this.

In this article, we have reviewed the problems addressed by weak supervision, provided a formal definition, outlined a general architecture commonly used in this context, detailed one of the foundational models for weak supervision – Snorkel – and concluded with a practical example showing how to apply weak supervision.

I hope this article was helpful, see you next time!

References

[1] Chan, J. etc (2021) Wrench: A comprehensive benchmark for weak supervision, arXiv.orgAvailable at: https://arxiv.org/abs/2109.11377 .

[2] Ratner, A. etc (2017) Snorkel: Rapid Training Data Creation with Weak Supervision, arXiv.orgAvailable at: https://arxiv.org/abs/1711.10160.

[3] Norsk Regnecentral (2021) NorskRegnesentral/skweak: Skweak: a software toolkit for weak supervision applied to NLP tasks., GitHubAvailable from: https://github.com/NorskRegnesentral/skweak.