Natural language processing (NLP) is a subfield of artificial intelligence that allows computers to understand, process, and manipulate human language.
Every tool in this course so far needs a vector: \(k\)-means (Lecture 3), \(k\)-nearest neighbours (Lecture 9), regression, SVD and PCA (Lectures 10–13). Text is not a vector. Today is about how text becomes one: text → tokens → vectors.
Learning outcomes. After this lecture you should be able to:
1950s-1970s: Rule-Based Era
1980s-2000s: Statistical Revolution
2006-2017: Deep Learning
2017-Present: Transformer Era
For the rest of this lecture we will cover:
There are a number of ways to do this. These include
However, prior to creating a numerical representation of text, we need to tokenize the text.
Tokenization is the process of splitting raw text into smaller pieces, called tokens.
Tokens can be individual characters, words, subwords, or sentences.
Examples of character and word tokenization:
Show me the money
Character tokenization:
['S', 'h', 'o', 'w', ' ', 'm', 'e', ' ', 't', 'h', 'e', ' ', 'm', 'o', 'n', 'e', 'y']
Word tokenization:
['Show', 'me', 'the', 'money']
Simple python implementation:
# Character and word tokenization
sentence = "Show me the money"
word_tokens = sentence.split()
print(word_tokens)
character_tokens = list(sentence)
print(character_tokens)['Show', 'me', 'the', 'money']
['S', 'h', 'o', 'w', ' ', 'm', 'e', ' ', 't', 'h', 'e', ' ', 'm', 'o', 'n', 'e', 'y']A character tokenizer keeps the spaces: they are what mark the word boundaries.
However, there are other strategies, such as subword and sentence tokenization, see for example:
See Andrej Karpathy’s Let’s build a GPT tokenizer video for a deep dive.
Here is a demo of how to tokenize using the transformers package from Huggingface.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")
tokens = tokenizer.tokenize(sentence)
print(tokens)
# Try a more advanced sentence
sentence2 = "Let's try to see if we can get this transformer to tokenize."
tokens2 = tokenizer.tokenize(sentence2)
print(tokens2)
print("vocabulary size:", tokenizer.vocab_size)['Show', 'me', 'the', 'money']
['Let', "'", 's', 'try', 'to', 'see', 'if', 'we', 'can', 'get', 'this', 'transform', '##er', 'to', 'token', '##ize', '.']
vocabulary size: 28996## means “continues the previous token”: token + ##ize = tokenize. BERT’s vocabulary has about 29,000 subwords, and any word can be spelled from them, so the model never meets an unknown word.
Associated to each token is a unique token ID.
The total number of unique tokens that a model can recognize and process is the vocabulary size.
The tokens (and token ids) alone hold no (semantic) information. What is needed is a numerical representation that encodes this information.
There are different ways to achieve this:
One encoding technique that we already considered is one-hot encodings.
Another more powerful encoding method, is the creation of word embeddings.
Before counting words you decide what counts as “the same word”. Each decision changes the vocabulary, and so every column of the matrix you build next.
from nltk.stem import PorterStemmer, WordNetLemmatizer
stemmer, lemmatizer = PorterStemmer(), WordNetLemmatizer()
print(f"{'word':10} {'stem':10} {'lemma (verb)':12}")
for w in ["running", "runs", "ran", "easily", "studies"]:
print(f"{w:10} {stemmer.stem(w):10} {lemmatizer.lemmatize(w, pos='v'):12}")word stem lemma (verb)
running run run
runs run run
ran ran run
easily easili easily
studies studi study easili, studi) and it misses irregular forms (ran).running and runs (and with the lemmatizer ran) collapse into one column.You met one-hot encodings for categorical features in Lecture 1.
Example
Given the words cat, dog, and emu here are sample one-hot encodings
\[ \begin{align*} \text{cat} &= [1, 0, 0]^{T}, \\ \text{dog} &= [0, 1, 0]^{T}, \\ \text{emu} &= [0, 0, 1]^{T}. \\ \end{align*} \]
A document becomes the count of each vocabulary word, ignoring grammar and word order. Three sentences, one row each:
import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
docs = ["The cat sat on the mat.",
"The dog sat on the log.",
"The emu sat on the mat."]
count_vec = CountVectorizer() # lower-cases, drops punctuation
counts = count_vec.fit_transform(docs).toarray()
vocab = count_vec.get_feature_names_out()
pd.DataFrame(counts, columns=vocab, index=["S1", "S2", "S3"])| cat | dog | emu | log | mat | on | sat | the | |
|---|---|---|---|---|---|---|---|---|
| S1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 2 |
| S2 | 0 | 1 | 0 | 1 | 0 | 1 | 1 | 2 |
| S3 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 2 |
for options in [{}, {"lowercase": False}, {"stop_words": "english"}]:
names = CountVectorizer(**options).fit(docs).get_feature_names_out()
print(f"{str(options):28} {len(names)} terms: {list(names)}"){} 8 terms: ['cat', 'dog', 'emu', 'log', 'mat', 'on', 'sat', 'the']
{'lowercase': False} 9 terms: ['The', 'cat', 'dog', 'emu', 'log', 'mat', 'on', 'sat', 'the']
{'stop_words': 'english'} 6 terms: ['cat', 'dog', 'emu', 'log', 'mat', 'sat']the, which says nothing about which sentence this is.For a term \(t\), a document \(d\), and a corpus of \(N\) documents:
\[ \mathrm{tf}(t,d) = \#\{\text{occurrences of } t \text{ in } d\}, \qquad \mathrm{df}(t) = \#\{\text{documents containing } t\}, \]
\[ \mathrm{idf}(t) = \ln\frac{N}{\mathrm{df}(t)}, \qquad \text{tf-idf}(t,d) = \mathrm{tf}(t,d)\times\mathrm{idf}(t). \]
With \(N = 3\):
| term | df | idf \(= \ln(3/\mathrm{df})\) |
|---|---|---|
the, sat, on |
3 | \(\ln 1 = 0\) |
mat |
2 | \(\ln 1.5 = 0.405\) |
cat, dog, emu, log |
1 | \(\ln 3 = 1.099\) |
Sentence 1: the has tf \(= 2\) but weight \(2 \times 0 = 0\). What is left is cat \(= 1.099\) and mat \(= 0.405\). The words that tell the sentences apart now carry the whole vector.
The same computation, with counts from the bag-of-words slide:
N = len(docs)
df = (counts > 0).sum(axis=0)
idf = np.log(N / df)
tfidf_textbook = counts * idf
pd.DataFrame(tfidf_textbook, columns=vocab, index=["S1", "S2", "S3"]).round(3)| cat | dog | emu | log | mat | on | sat | the | |
|---|---|---|---|---|---|---|---|---|
| S1 | 1.099 | 0.000 | 0.000 | 0.000 | 0.405 | 0.0 | 0.0 | 0.0 |
| S2 | 0.000 | 1.099 | 0.000 | 1.099 | 0.000 | 0.0 | 0.0 | 0.0 |
| S3 | 0.000 | 0.000 | 1.099 | 0.000 | 0.405 | 0.0 | 0.0 | 0.0 |
the, sat and on are zero in every row; mat is shared by S1 and S3, so it gets the smaller weight 0.405.
TfidfVectorizer actually computesscikit-learn does not use the textbook formula. Its defaults are
\[ \mathrm{idf}_{\text{sk}}(t) = \ln\frac{1+N}{1+\mathrm{df}(t)} + 1, \qquad\text{then each row is divided by its } \ell_2 \text{ norm.} \]
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vec = TfidfVectorizer()
tfidf_sk = tfidf_vec.fit_transform(docs).toarray()
pd.DataFrame(tfidf_sk, columns=tfidf_vec.get_feature_names_out(),
index=["S1", "S2", "S3"]).round(4)| cat | dog | emu | log | mat | on | sat | the | |
|---|---|---|---|---|---|---|---|---|
| S1 | 0.5219 | 0.0000 | 0.0000 | 0.0000 | 0.3969 | 0.3082 | 0.3082 | 0.6165 |
| S2 | 0.0000 | 0.4943 | 0.0000 | 0.4943 | 0.0000 | 0.2919 | 0.2919 | 0.5839 |
| S3 | 0.0000 | 0.0000 | 0.5219 | 0.0000 | 0.3969 | 0.3082 | 0.3082 | 0.6165 |
the is the largest entrythe gets \(\mathrm{idf}_{\text{sk}} = \ln(4/4) + 1 = 1\), not 0. With tf \(= 2\) it outweighs cat (\(\ln 2 + 1 = 1.69\)). No term is ever weighted exactly 0, and a word that is new at query time cannot cause a division by zero.smooth_idf=False removes the \(+1\)s inside the log but keeps the \(+1\) outside, so it is still not the textbook idf.stop_words="english", min_df / max_df to prune the vocabulary, sublinear_tf=True for \(1 + \ln \mathrm{tf}\).\[ \cos{(\theta)} = \frac{v_1 \cdot v_2}{\Vert v_1 \Vert_2 \Vert v_2 \Vert_2}. \]
It measures direction, not length. For counts and TF-IDF (no negative entries) it lies in \([0, 1]\), and 0 means no shared term.
from sklearn.metrics.pairwise import cosine_similarity
for name, X in [("raw counts", counts),
("textbook TF-IDF", tfidf_textbook),
("sklearn TF-IDF", tfidf_sk)]:
print(f"cos(S1, S2) on {name:16}: {cosine_similarity(X[[0]], X[[1]])[0, 0]:.2f}")cos(S1, S2) on raw counts : 0.75
cos(S1, S2) on textbook TF-IDF : 0.00
cos(S1, S2) on sklearn TF-IDF : 0.54the, sat, on.the, sat, on: 0.54.The most relevant document for a query \(q\) is \(\arg\max_j \cos(q, d_j)\).
query = ["the dog on the mat"]
for name, vec in [("raw counts", count_vec), ("sklearn TF-IDF", tfidf_vec)]:
sims = cosine_similarity(vec.transform(query), vec.transform(docs))[0]
print(f"{name:15} similarities {sims.round(3)} -> best: S{sims.argmax() + 1}")raw counts similarities [0.802 0.802 0.802] -> best: S1
sklearn TF-IDF similarities [0.665 0.744 0.665] -> best: S2On raw counts all three sentences tie: the and on decide everything. TF-IDF breaks the tie on the informative word, dog. (np.argmax returns the first of a tie, so “S1” on the first line is an accident.)
Word embeddings represent words as dense vectors in high-dimensional spaces.
The individual values of the vector may be difficult to interpret, but the overall pattern is that words with similar meanings are close to each other, in the sense that their vectors have small angles with each other.
The same holds for bag of words and TF-IDF: every word is its own axis, so “doctor” and “physician” are orthogonal and two documents look similar only if they share literal words.
The similarity of two word embeddings is again the cosine of the angle between them.
Word embeddings can be static or contextual.
Static:
A static embedding is when each word has a single embedding, e.g., Word2Vec.
Contextual:
A contextual embedding (used by more complex language model embedding algorithms) allows the embedding for a word to change depending on its context in a sentence.
Word2Vec (Mikolov et al. 2013) is a technique to learn static word embeddings from large text corpora.
Key characteristics:
Advantages:
Disadvantages:
| Feature | Static (Word2Vec, GloVe) | Contextual (BERT, GPT) |
|---|---|---|
| Representation | One vector per word | Different vectors per context |
| Example | “bank” always same vector | “bank” differs in “river bank” vs “bank account” |
| Model Type | Shallow model | Deep transformer model |
| Training | Fast, lightweight | Slow, resource-intensive |
| Best For | Similarity, clustering, simple tasks | Complex understanding, ambiguity resolution |
| Dimensionality | 100-300 dimensions | 768-1024+ dimensions |
When to use Word2Vec:
When to use Contextual Embeddings:
A language model is a statistical tool that predicts the probability of a sequence of words. It helps in understanding and generating human language by learning patterns and structures from large text corpora.
See RNNs and LSTMs for more details.
We’ll skip N-grams and focus on Transformers.
Transformers (Vaswani et al. 2017) are the foundation of modern NLP systems.
Key innovations:
Components:
Other variants of the transformer architecture include the encoder-only and decoder-only architectures.
Can grow the model size by increasing the number of layers and the number of attention heads.
Components:
Attention allows models to understand which words are most relevant to each other.
Example sentence:
“The elephant didn’t cross the river because it was tired.”
Question: What does “it” refer to?
For each word in a sentence:
Multi-head attention: Run multiple attention operations in parallel to capture different types of relationships
It’s a form of adaptive network where some weights are derived from inputs.
The attention mechanism uses three types of vectors for each word:
Attention formula: \[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]
Darker connections show stronger attention.
When processing “it”, the model attends most to “elephant”.
This is how transformers handle long-range dependencies and ambiguity.
See also BertViz for a visual exploration of the attention mechanism.
BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al. 2019)
Key features:
Common uses: Text classification, NER, question answering, sentiment analysis
GPT (Generative Pre-trained Transformer) (Brown et al. 2020)
Key features:
Common uses: Text completion, creative writing, chatbots, code generation
Transformers enable powerful NLP applications:
NLTK is one of the most comprehensive Python libraries for NLP, created for teaching and research.
Key Features:
Best For:
Limitations:
spaCy is a modern, industrial-strength NLP library designed for production use.
Key Features:
Best For:
LangChain (Chase 2022) is a framework for building applications on top of large language models.
Retrieval-augmented generation retrieves the documents nearest the question, as on the retrieval slide above but usually with dense embeddings, and hands them to the language model.
Runnable examples of every package on these slides (NLTK preprocessing and sentiment, the spaCy pipeline and NER, BERTopic with its visualizations, and a LangChain pointer) are in Appendix A5: NLP Packages, which also has a Colab notebook.
The rest of this lecture looks at two applications: named entity recognition and topic modeling.
Named Entity Recognition (NER) is the task of identifying and classifying named entities in text into predefined categories.
Common entity types:
Applications: Information extraction, content classification, question answering, knowledge graphs
spaCy’s small English pipeline (en_core_web_sm) tags one sentence pair:
Topic modeling is an unsupervised learning technique that discovers abstract “topics” in a collection of documents.
Key concepts:
Applications:
Classical approaches (LDA, NMF) use bag-of-words representations and require specifying the number of topics.
Modern approaches (BERTopic) cluster transformer embeddings; the clustering step decides how many topics there are, given a minimum topic size.
BERTopic (Grootendorst 2022) is a modern topic modeling technique using transformer embeddings.
Key advantages:
min_topic_sizeWhat it still needs from you:
min_topic_size, which sets how fine the topics areBERTopic uses a modular pipeline with four main steps:
1. Document Embeddings
all-MiniLM-L6-v2, 768+ for larger models)2. Dimensionality Reduction (UMAP)
3. Clustering (HDBSCAN)
4. Topic Representation (c-TF-IDF)
The same three newsgroups we clustered in Lecture 4:
from bertopic import BERTopic
from sentence_transformers import SentenceTransformer
from sklearn.datasets import fetch_20newsgroups
from umap import UMAP
categories = ['comp.os.ms-windows.misc', 'sci.space', 'rec.sport.baseball']
posts = fetch_20newsgroups(subset='all', categories=categories,
remove=('headers', 'footers', 'quotes'))['data']
posts = [p for p in posts if len(p.split()) >= 5] # drop empty posts
embedder = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embedder.encode(posts, show_progress_bar=False) # step 1, computed once
def fit_bertopic(min_topic_size):
model = BERTopic(
embedding_model=embedder,
umap_model=UMAP(n_neighbors=15, n_components=5, min_dist=0.0,
metric="cosine", random_state=42),
vectorizer_model=CountVectorizer(stop_words="english",
token_pattern=r"(?u)\b[a-zA-Z]{3,15}\b"),
min_topic_size=min_topic_size)
topics, _ = model.fit_transform(posts, embeddings)
return model, topics
for size in (20, 10):
model, topics = fit_bertopic(size)
print(f"min_topic_size={size}: {len(set(topics) - {-1})} topics, "
f"{topics.count(-1)} outliers out of {len(posts)} posts")min_topic_size=20: 3 topics, 0 outliers out of 2823 posts
min_topic_size=10: 21 topics, 285 outliers out of 2823 postsWith min_topic_size=20 HDBSCAN finds exactly the three newsgroups, without ever seeing a label. Halve it and the same embeddings give 21 topics. The number of topics is a consequence of the minimum size you ask for, not a property of the data alone.
With min_topic_size=10 we get many small topics; reduce_topics merges the most similar ones until 10 remain (the outlier row −1 counts as one).
topic_model = model
topic_model.reduce_topics(posts, nr_topics=10)
info = topic_model.get_topic_info()
info["Top words"] = [", ".join(w for w, _ in topic_model.get_topic(t)[:5]) for t in info.Topic]
print(info[["Topic", "Count", "Top words"]].to_string(index=False)) Topic Count Top words
-1 285 windows, file, dos, files, use
0 993 space, nasa, earth, launch, orbit
1 925 year, game, team, games, baseball
2 351 windows, file, dos, use, files
3 123 card, mouse, drivers, driver, video
4 54 printer, print, deskjet, canon, ink
5 39 font, fonts, characters, truetype, character
6 21 sound, midi, driver, play, soundblaster
7 20 max, bhj, giz, bxn, chz
8 12 disks, mac, read, disk, hardwareSpace and baseball stay whole. Windows splits into drivers, printers, fonts, sound and disks. Topic 7 is junk (posts of encoded binary files), and the 285 outliers are mostly Windows posts. Always read your topics.
fig = topic_model.visualize_topics()
fig.update_layout(width=600, height=450)
fig.show()Each circle is a topic, sized by its number of posts, placed by the similarity of the topics’ c-TF-IDF vectors.
the = 0.6165 survives in its table.