Natural language processing (NLP) is a subfield of artificial intelligence that allows computers to understand, process, and manipulate human language.
1950s-1970s: Rule-Based Era
1980s-2000s: Statistical Revolution
2006-2017: Deep Learning
2017-Present: Transformer Era
For the rest of this lecture we will cover:
There are a number of ways to do this. These include
However, prior to creating a numerical representation of text, we need to tokenize the text.
Tokenization is the process of splitting raw text into smaller pieces, called tokens.
Tokens can be individual characters, words, subwords, or sentences.
Examples of character and word tokenization:
Show me the money
Character tokenization:
['S', 'h', 'o', 'w', 'm', 'e', 't', 'h', 'e', 'm', 'o', 'n', 'e', 'y'].
Word tokenization:
['Show', 'me', 'the', 'money']
Simple python implementation:
# Character and word tokenization
sentence = "Show me the money"
word_tokens = sentence.split()
print(word_tokens)
character_tokens = [char for char in sentence if char != ' ']
print(character_tokens)['Show', 'me', 'the', 'money']
['S', 'h', 'o', 'w', 'm', 'e', 't', 'h', 'e', 'm', 'o', 'n', 'e', 'y']However, there are other strategies, such as subword and sentence tokenization, see for example:
See Andrej Karpathy’s Let’s build a GPT tokenizer video for a deep dive.
Here is a demo of how to tokenize using the transformers package from Huggingface.
from transformers import AutoTokenizer, logging
logging.set_verbosity_warning()
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokens = tokenizer.tokenize(sentence)
print(tokens)
# Try a more advanced sentence
sentence2 = "Let's try to see if we can get this transformer to tokenize."
tokens2 = tokenizer.tokenize(sentence2)
print(tokens2)['Show', 'me', 'the', 'money']
['Let', "'", 's', 'try', 'to', 'see', 'if', 'we', 'can', 'get', 'this', 'transform', '##er', 'to', 'token', '##ize', '.']Associated to each token is a unique token ID.
The total number of unique tokens that a model can recognize and process is the vocabulary size.
The tokens (and token ids) alone hold no (semantic) information. What is needed is a numerical representation that encodes this information.
There are different ways to achieve this:
One encoding technique that we already considered is one-hot encodings.
Another more powerful encoding method, is the creation of word embeddings.
We have previously considered the following sparse representations of textual data.
Example
Given the words cat, dog, and emu here are sample one-hot encodings
\[ \begin{align*} \text{cat} &= [1, 0, 0]^{T}, \\ \text{dog} &= [0, 1, 0]^{T}, \\ \text{emu} &= [0, 0, 1]^{T}. \\ \end{align*} \]
Example
Suppose we have the following sentences
| Sentence | the | cat | sat | on | mat | dog | log | emu |
|---|---|---|---|---|---|---|---|---|
| “The cat sat on the mat.” | 2 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
| “The dog sat on the log.” | 2 | 0 | 1 | 1 | 0 | 1 | 1 | 0 |
| “The emu sat on the mat.” | 2 | 0 | 1 | 1 | 1 | 0 | 0 | 1 |
Example
The TF-IDF representations corresponding to the previous sentences.
| cat | dog | log | mat | on | emu | sat | the | |
|---|---|---|---|---|---|---|---|---|
| Sentence 1 | 0.4698 | 0.0000 | 0.0000 | 0.4698 | 0.3546 | 0.0000 | 0.3546 | 0.7093 |
| Sentence 2 | 0.0000 | 0.4698 | 0.4698 | 0.0000 | 0.3546 | 0.0000 | 0.3546 | 0.7093 |
| Sentence 3 | 0.0000 | 0.0000 | 0.0000 | 0.4698 | 0.3546 | 0.4698 | 0.3546 | 0.7093 |
Word embeddings represent words as dense vectors in high-dimensional spaces.
The individual values of the vector may be difficult to interpret, but the overall pattern is that words with similar meanings are close to each other, in the sense that their vectors have small angles with each other.
The similarity of two word embeddings is the cosine of the angle between the two vectors. Recall that for two vectors \(v_1, v_2\in\mathbb{R}^{n}\), the formula for the cosine of the angle between them is
\[ \cos{(\theta)} = \frac{v_1 \cdot v_2}{\Vert v_1 \Vert_2 \Vert v_2 \Vert_2}. \]
Word embeddings can be static or contextual.
Static:
A static embedding is when each word has a single embedding, e.g., Word2Vec.
Contextual:
A contextual embedding (used by more complex language model embedding algorithms) allows the embedding for a word to change depending on its context in a sentence.
Word2Vec (Mikolov et al. 2013) is a technique to learn static word embeddings from large text corpora.
Key characteristics:
Advantages:
Disadvantages:
| Feature | Static (Word2Vec, GloVe) | Contextual (BERT, GPT) |
|---|---|---|
| Representation | One vector per word | Different vectors per context |
| Example | “bank” always same vector | “bank” differs in “river bank” vs “bank account” |
| Model Type | Shallow neural network | Deep transformer model |
| Training | Fast, lightweight | Slow, resource-intensive |
| Best For | Similarity, clustering, simple tasks | Complex understanding, ambiguity resolution |
| Dimensionality | 100-300 dimensions | 768-1024+ dimensions |
When to use Word2Vec:
When to use Contextual Embeddings:
A language model is a statistical tool that predicts the probability of a sequence of words. It helps in understanding and generating human language by learning patterns and structures from large text corpora.
See RNNs and LSTMs for more details.
We’ll skip N-grams and focus on Transformers.
Transformers (Vaswani et al. 2017) are the foundation of modern NLP systems.
Key innovations:
Components:
Other variants of the transformer architecture include the encoder-only and decoder-only architectures.
Can grow the model size by increasing the number of layers and the number of attention heads.
Components:
Attention allows models to understand which words are most relevant to each other.
Example sentence:
“The elephant didn’t cross the river because it was tired.”
Question: What does “it” refer to?
For each word in a sentence:
Multi-head attention: Run multiple attention operations in parallel to capture different types of relationships
It’s a form of adaptive network where some weights are derived from inputs.
The attention mechanism uses three types of vectors for each word:
Attention formula: \[\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V\]
Darker connections show stronger attention.
When processing “it”, the model attends most to “elephant”.
This is how transformers handle long-range dependencies and ambiguity.
See also BertViz for a visual exploration of the attention mechanism.
BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al. 2019)
Key features:
Common uses: Text classification, NER, question answering, sentiment analysis
GPT (Generative Pre-trained Transformer) (Brown et al. 2020)
Key features:
Common uses: Text completion, creative writing, chatbots, code generation
Transformers enable powerful NLP applications:
NLTK is one of the most comprehensive Python libraries for NLP, created for teaching and research.
Key Features:
Best For:
Limitations:
spaCy is a modern, industrial-strength NLP library designed for production use.
Key Features:
Best For:
BERTopic (Grootendorst 2022) is a modern topic modeling technique using transformer embeddings.
More later…
LangChain (langchain2023?) is a framework for building applications that use language models.
Key advantages:
Common uses:
We’ll dive a bit deeper into two of the most common NLP applications:
Named Entity Recognition (NER) is the task of identifying and classifying named entities in text into predefined categories.
Common entity types:
Applications: Information extraction, content classification, question answering, knowledge graphs
You need to download the model you want to use.
Installation:
pip install spacy
python -m spacy download en_core_web_sm # Small modelAvailable models:
en_core_web_sm: Small (12 MB) - fast, good accuracyen_core_web_md: Medium (40 MB) - word vectors includeden_core_web_lg: Large (560 MB) - best accuracy, full vectorsimport spacy
# Load pre-trained model
nlp = spacy.load("en_core_web_sm")
text = "Apple Inc. was founded by Steve Jobs in Cupertino, California. In 2024, the company is worth over $3 trillion dollars."
# Process text - creates Doc object
doc = nlp(text)
# Extract named entities
print("Entities found:")
for ent in doc.ents:
print(f" {ent.text:20} -> {ent.label_:15} ({spacy.explain(ent.label_)})")Entities found:
Apple Inc. -> ORG (Companies, agencies, institutions, etc.)
Steve Jobs -> PERSON (People, including fictional)
Cupertino -> GPE (Countries, cities, states)
California -> GPE (Countries, cities, states)
2024 -> DATE (Absolute or relative dates or periods)
over $3 trillion dollars -> MONEY (Monetary values, including unit)Built-in visualization with displacy:
from spacy import displacy
# Visualize entities inline
displacy.render(doc, style="ent", jupyter=True)Key features:
# Access entity properties
for ent in doc.ents:
print(f"{ent.text:20} | Label: {ent.label_:10} | Start: {ent.start_char:3} | End: {ent.end_char:3}")
# Filter by entity type
orgs = [ent.text for ent in doc.ents if ent.label_ == "ORG"]
print(f"\nOrganizations: {orgs}")
# Entity spans and context
for ent in doc.ents:
print(f"{ent.text} → Sentence: {ent.sent}")Apple Inc. | Label: ORG | Start: 0 | End: 10
Steve Jobs | Label: PERSON | Start: 26 | End: 36
Cupertino | Label: GPE | Start: 40 | End: 49
California | Label: GPE | Start: 51 | End: 61
2024 | Label: DATE | Start: 66 | End: 70
over $3 trillion dollars | Label: MONEY | Start: 93 | End: 117
Organizations: ['Apple Inc.']
Apple Inc. → Sentence: Apple Inc. was founded by Steve Jobs in Cupertino, California.
Steve Jobs → Sentence: Apple Inc. was founded by Steve Jobs in Cupertino, California.
Cupertino → Sentence: Apple Inc. was founded by Steve Jobs in Cupertino, California.
California → Sentence: Apple Inc. was founded by Steve Jobs in Cupertino, California.
2024 → Sentence: In 2024, the company is worth over $3 trillion dollars.
over $3 trillion dollars → Sentence: In 2024, the company is worth over $3 trillion dollars.# View pipeline components
print("Pipeline components:")
for name, component in nlp.pipeline:
print(f" - {name}")Pipeline components:
- tok2vec
- tagger
- parser
- attribute_ruler
- lemmatizer
- nerProcessing steps:
Topic modeling is an unsupervised learning technique that discovers abstract “topics” in a collection of documents.
Key concepts:
Applications:
Classical approaches (LDA, NMF) use bag-of-words representations and require specifying the number of topics.
Modern approaches (BERTopic) leverage transformer embeddings and can automatically determine topics.
BERTopic (Grootendorst 2022) is a modern topic modeling technique using transformer embeddings.
Key advantages:
When to use BERTopic:
BERTopic uses a modular pipeline with four main steps:
1. Document Embeddings
2. Dimensionality Reduction (UMAP)
Installation:
pip install bertopic
pip install umap-learn hdbscan # clustering dependenciesQuick start:
from bertopic import BERTopic
# Initialize with default settings
topic_model = BERTopic(language="english", calculate_probabilities=True)
# Fit and transform
topics, probs = topic_model.fit_transform(documents)from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups
categories = ['sci.space', 'rec.sport.baseball', 'comp.graphics']
docs = fetch_20newsgroups(subset='all', categories=categories, remove=('headers', 'footers', 'quotes'))['data']
topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)
print(f"Number of topics found: {len(set(topics)) - (1 if -1 in topics else 0)}")
print(f"Outlier documents: {sum(1 for t in topics if t == -1)}")Number of topics found: 41
Outlier documents: 702fig = topic_model.visualize_topics()
fig.update_layout(width=600, height=400)
fig.show()fig = topic_model.visualize_hierarchy()
fig.update_layout(width=800, height=550)
fig.show()fig = topic_model.visualize_barchart(top_n_topics=8)
fig.show()fig = topic_model.visualize_heatmap()
fig.update_layout(width=800, height=600)
fig.show()similar_topics, similarity = topic_model.find_topics("space exploration and satellites", top_n=3)
print(f"Topics similar to 'space exploration and satellites': {similar_topics}")Topics similar to 'space exploration and satellites': [22, 26, 1]# Get topic information
topic_info = topic_model.get_topic_info()
print("Topic Overview:")
print(topic_info[['Topic', 'Count', 'Name']].head(10))
# Get representative documents for a topic
topic_id = 0
rep_docs = topic_model.get_representative_docs(topic_id)
print(f"\nRepresentative documents for Topic {topic_id}:")
for i, doc in enumerate(rep_docs[:2], 1):
print(f"{i}. {doc[:100]}...")Topic Overview:
Topic Count Name
0 -1 702 -1_the_of_to_and
1 0 912 0_he_the_in_to
2 1 206 1_the_of_to_space
3 2 90 2_the_points_den_is
4 3 79 3_card_vesa_mode_video
5 4 74 4____
6 5 74 5_format_files_convert_to
7 6 68 6_image_and_3d_for
8 7 57 7_sky_the_advertising_that
9 8 54 8_for_and_data_ftp
Representative documents for Topic 0:
1.
Oh, yeah. Dave Winfield--marginal player. Guy didn't hit a lick, had
negligible power, was a cra...
2.
I am trying to think how to respond to this without involving personal feeling
or perceptions and I...# Show top words for each topic
print("Top words per topic:\n")
for topic_num in sorted(set(topics)):
if topic_num == -1: # Skip outlier topic
continue
words = topic_model.get_topic(topic_num)
if words:
# Get word and score
top_words = ', '.join([f"{word}({score:.2f})" for word, score in words[:8]])
print(f"Topic {topic_num}: {top_words}")Top words per topic:
Topic 0: he(0.02), the(0.02), in(0.02), to(0.02), that(0.02), and(0.01), his(0.01), was(0.01)
Topic 1: the(0.02), of(0.02), to(0.02), space(0.02), and(0.02), that(0.01), in(0.01), is(0.01)
Topic 2: the(0.02), points(0.02), den(0.02), is(0.02), problem(0.02), of(0.02), this(0.02), algorithm(0.02)
Topic 3: card(0.04), vesa(0.03), mode(0.03), video(0.02), it(0.02), driver(0.02), vga(0.02), to(0.02)
Topic 4: (0.00), (0.00), (0.00), (0.00), (0.00), (0.00), (0.00), (0.00)
Topic 5: format(0.04), files(0.03), convert(0.03), to(0.03), gif(0.02), file(0.02), me(0.02), it(0.02)
Topic 6: image(0.02), and(0.02), 3d(0.02), for(0.02), of(0.02), it(0.02), or(0.02), the(0.01)
Topic 7: sky(0.03), the(0.02), advertising(0.02), that(0.02), of(0.02), to(0.02), rights(0.02), it(0.02)
Topic 8: for(0.02), and(0.02), data(0.02), ftp(0.02), available(0.02), the(0.01), in(0.01), from(0.01)
Topic 9: hst(0.05), reboost(0.03), the(0.03), mission(0.02), shuttle(0.02), to(0.02), mass(0.02), is(0.02)
Topic 10: conference(0.04), int(0.03), nok(0.03), oprows(0.02), opcols(0.02), for(0.02), sas(0.02), on(0.02)
Topic 11: oort(0.04), cloud(0.03), the(0.03), grbs(0.03), distribution(0.03), of(0.03), burst(0.02), detectors(0.02)
Topic 12: siggraph(0.05), membership(0.03), me(0.03), to(0.02), my(0.02), send(0.02), you(0.02), address(0.02)
Topic 13: oxygen(0.03), of(0.02), the(0.02), to(0.02), in(0.02), is(0.02), pressure(0.02), it(0.02)
Topic 14: propulsion(0.03), space(0.02), of(0.02), and(0.02), the(0.02), fusion(0.02), lunar(0.02), was(0.02)
Topic 15: colour(0.04), rgb(0.03), luminosity(0.03), colours(0.02), color(0.02), bits(0.02), green(0.02), bit(0.02)
Topic 16: xv(0.08), bit(0.04), 24bit(0.04), image(0.04), 24(0.03), you(0.03), it(0.03), images(0.03)
Topic 17: group(0.05), groups(0.04), newsgroup(0.04), aspects(0.03), liefting(0.03), split(0.03), this(0.03), graphics(0.03)
Topic 18: space(0.02), venus(0.02), gopher(0.02), the(0.02), of(0.02), and(0.02), to(0.02), search(0.02)
Topic 19: joke(0.07), arbitron(0.04), was(0.03), deleted(0.03), flame(0.03), it(0.03), humour(0.03), macelwaines(0.03)
Topic 20: that(0.02), was(0.02), and(0.02), but(0.02), upgrade(0.02), sgi(0.02), lcd(0.02), screen(0.02)
Topic 21: sail(0.04), solar(0.04), pluto(0.03), mission(0.03), be(0.02), would(0.02), to(0.02), ship(0.02)
Topic 22: satellites(0.02), fc(0.02), satellite(0.02), the(0.02), of(0.02), to(0.02), are(0.02), optical(0.02)
Topic 23: question(0.07), 42(0.07), tea(0.05), number(0.05), answer(0.04), two(0.03), peter(0.03), said(0.03)
Topic 24: wings(0.04), aircraft(0.04), aviation(0.04), moscow(0.03), the(0.03), of(0.02), supersonic(0.02), have(0.02)
Topic 25: menu(0.04), pressing(0.04), program(0.04), bits(0.03), image(0.03), display(0.03), you(0.03), read(0.03)
Topic 26: space(0.05), nasa(0.03), astronaut(0.03), and(0.03), center(0.02), for(0.02), of(0.02), candidates(0.02)
Topic 27: why(1.72), hello(1.36), of(0.18), (0.00), (0.00), (0.00), (0.00), (0.00)
Topic 28: spacecraft(0.06), command(0.06), noop(0.04), timer(0.03), mode(0.03), loss(0.03), hga(0.03), that(0.02)
Topic 29: phigs(0.06), graphics(0.03), visualization(0.03), computer(0.03), and(0.03), will(0.02), of(0.02), research(0.02)
Topic 30: software(0.08), process(0.06), level(0.05), shuttle(0.03), that(0.03), wingert(0.02), warning(0.02), maturity(0.02)
Topic 31: constant(0.06), mass(0.05), km(0.04), velocity(0.04), radius(0.04), of(0.03), times(0.03), is(0.03)
Topic 32: hacker(0.08), hackers(0.04), who(0.03), ethic(0.03), to(0.02), computer(0.02), the(0.02), of(0.02)
Topic 33: probe(0.03), april(0.03), the(0.02), mars(0.02), spacecraft(0.02), was(0.02), on(0.02), mission(0.02)
Topic 34: sam(0.10), ls(0.08), name(0.07), my(0.07), telling(0.07), mom(0.06), yeah(0.06), lemur(0.06)
Topic 35: circumference(0.08), wk(0.08), grows(0.08), 5173552178(0.08), 18084tmibmclmsuedu(0.08), mcwilliams(0.08), 3369591(0.08), mystery(0.08)
Topic 36: comet(0.03), gehrels(0.03), spherical(0.02), for(0.02), and(0.02), of(0.02), projections(0.02), asteroids(0.02)
Topic 37: tiff(0.15), complexity(0.04), spec(0.03), that(0.03), to(0.03), read(0.02), is(0.02), it(0.02)
Topic 38: cview(0.16), temp(0.09), file(0.04), it(0.04), floppy(0.04), disk(0.04), files(0.04), directory(0.04)
Topic 39: adobe(0.11), sgi(0.09), illustrator(0.07), sun(0.06), photoshop(0.05), announced(0.05), for(0.04), wayne(0.04)
Topic 40: davewoodcscoloradoedu(0.36), rex(0.35), wood(0.34), boulder(0.28), colorado(0.24), david(0.22), blasphemy(0.21), perijovesim(0.20)# Find topics similar to a search query
query = "space exploration and satellites"
similar_topics, similarity = topic_model.find_topics(query, top_n=3)
print(f"Topics similar to '{query}':")
for topic_id, score in zip(similar_topics, similarity):
if topic_id != -1:
words = topic_model.get_topic(topic_id)
top_words = ', '.join([word for word, _ in words[:5]])
print(f" Topic {topic_id} (similarity: {score:.3f}): {top_words}")Topics similar to 'space exploration and satellites':
Topic 22 (similarity: 0.567): satellites, fc, satellite, the, of
Topic 26 (similarity: 0.520): space, nasa, astronaut, and, center
Topic 1 (similarity: 0.502): the, of, to, space, andBERTopic can track how topics evolve over time:
# Prepare timestamps
import pandas as pd
timestamps = pd.date_range('2020-01-01', periods=len(docs), freq='D')
# Track topics over time
topics_over_time = topic_model.topics_over_time(
docs=docs,
timestamps=timestamps,
nr_bins=10
)
# Visualize evolution
fig = topic_model.visualize_topics_over_time(topics_over_time)
fig.show()Use cases:
BERTopic is highly customizable:
from sentence_transformers import SentenceTransformer
from umap import UMAP
from hdbscan import HDBSCAN
# Custom embedding model
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
# Custom UMAP settings
umap_model = UMAP(n_neighbors=15, n_components=5, min_dist=0.0)
# Custom HDBSCAN settings
hdbscan_model = HDBSCAN(min_cluster_size=15, min_samples=10)
# Create custom BERTopic model
topic_model = BERTopic(
embedding_model=embedding_model,
umap_model=umap_model,
hdbscan_model=hdbscan_model,
verbose=False
)Best practices:
min_topic_size (5-10)min_topic_size (20-50)topic_model.save("my_model")Common issues:
min_cluster_sizemin_topic_sizeWhat we covered: