Word Embeddings: Word2Vec, GloVe & FastText

The Distributional Hypothesis

In 2013, Tomas Mikolov and colleagues at Google published a paper that would reshape all of NLP. The idea was elegantly simple: instead of defining word representations by hand, learn them from data. The result was Word2Vec: dense vectors where semantically similar words are geometrically close.

"You shall know a word by the company it keeps." J.R. Firth, 1957 (the distributional hypothesis)

This idea (the distributional hypothesis) says that words appearing in similar contexts tend to have similar meanings. Why does this work? Consider: the words "cat" and "dog" both appear near "pet," "veterinarian," "cute," "fed," and "walked." A word you have never seen, like "wug," that also appears near "pet" and "fed" is probably an animal too. Context is a remarkably reliable proxy for meaning. This principle is the foundation of all modern word representations, including the embeddings inside GPT-4 and Claude.

Word2Vec: How It Works

Word2Vec comes in two flavors. We will focus on Skip-gram, which is simpler to understand and more widely used.

The idea: given a center word, predict the surrounding context words. Let us trace through a concrete example.

Take the sentence: "the cat sat on the mat" with a window size of 2. The model slides a window across the sentence, and at each position, creates training pairs:

Center Word	Context Words (window=2)	Training Pairs Generated
the	cat, sat	(the→cat), (the→sat)
cat	the, sat, on	(cat→the), (cat→sat), (cat→on)
sat	the, cat, on, the	(sat→the), (sat→cat), (sat→on), (sat→the)
on	cat, sat, the, mat	(on→cat), (on→sat), (on→the), (on→mat)
...	...	...

After processing billions of such pairs, words that frequently appear in similar contexts (like "cat" and "dog," which both appear near "the," "sat," "chased") end up with similar vectors. Words that never share context (like "cat" and "economics") end up far apart.

The Architecture (It Is Surprisingly Simple)

Skip-gram is a shallow neural network with just one hidden layer:

1. Input: One-hot vector for the center word (dimension = vocabulary size V)
2. Hidden layer: Multiply by weight matrix W (V × d): this produces a d-dimensional vector. This IS the word embedding.
3. Output: Multiply by another matrix W' (d × V), apply softmax: this gives a probability distribution over all words in the vocabulary

Negative Sampling: Making Training Tractable

The naive softmax over a vocabulary of 100,000+ words is extremely expensive. Negative sampling simplifies this: instead of updating all 100K output weights, we only update the weights for the correct context word (positive) and a small random sample of "negative" words (typically 5 to 20).

In plain English: make the dot product between the center word and the real context word large (positive), and make the dot product with random words small (negative).

Training Word2Vec from Scratch

Let us train a Word2Vec model using Gensim on a real corpus:

from gensim.models import Word2Vec
from gensim.utils import simple_preprocess

# Sample corpus (in practice, use millions of sentences)
corpus = [
    "the king ruled the kingdom with wisdom",
    "the queen ruled the kingdom with grace",
    "the prince and princess lived in the castle",
    "the man worked in the field every day",
    "the woman worked in the market every day",
    "a dog chased a cat across the garden",
    "the cat sat on the mat near the dog",
    "paris is the capital of france",
    "berlin is the capital of germany",
    "tokyo is the capital of japan",
]

# Tokenize
sentences = [simple_preprocess(s) for s in corpus]

# Train Word2Vec (Skip-gram with negative sampling)
model = Word2Vec(
    sentences,
    vector_size=50,    # embedding dimensions
    window=3,          # context window size
    min_count=1,       # minimum word frequency
    sg=1,              # 1 = Skip-gram, 0 = CBOW
    negative=5,        # number of negative samples
    epochs=100,        # training epochs
)

# Explore the learned embeddings
print("Vector for 'king':", model.wv['king'][:5], "...")
print("Most similar to 'king':", model.wv.most_similar('king', topn=3))
print("Most similar to 'cat':", model.wv.most_similar('cat', topn=3))

# Peek inside: the embedding matrix is just a numpy array
print(f"\nEmbedding matrix shape: {model.wv.vectors.shape}")
# Output: (num_words, 50): each row is one word's vector

Measuring Similarity: Cosine Similarity

Before we explore analogies, we need to understand how similarity is measured between word vectors. The standard metric is cosine similarity: the cosine of the angle between two vectors.

# Measuring cosine similarity between word vectors
from numpy import dot
from numpy.linalg import norm

def cosine_sim(a, b):
    return dot(a, b) / (norm(a) * norm(b))

# Using pre-trained Word2Vec vectors
print(cosine_sim(wv['cat'], wv['dog']))      # ~0.76 (both are pets)
print(cosine_sim(wv['cat'], wv['king']))     # ~0.13 (unrelated)
print(cosine_sim(wv['king'], wv['queen']))   # ~0.65 (both are royalty)
print(cosine_sim(wv['paris'], wv['france'])) # ~0.77 (capital-country)

The Magic of Word Analogies

The most striking property of word embeddings is that they capture relationships as vector arithmetic:

# Word analogy: king - man + woman = ?
# (Using pre-trained vectors for reliable results)
import gensim.downloader as api

# Download pre-trained Word2Vec (trained on Google News, 3M words)
wv = api.load('word2vec-google-news-300')

# The famous analogy
result = wv.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
print(result)  # [('queen', 0.7118)]

# More analogies
print(wv.most_similar(positive=['paris', 'germany'], negative=['france'], topn=1))
# [('berlin', 0.7327)]  Paris is to France as Berlin is to Germany

print(wv.most_similar(positive=['walking', 'swam'], negative=['swimming'], topn=1))
# [('walked', 0.7458)]  captures verb tense relationships too!

GloVe: Global Vectors for Word Representation

GloVe (Pennington et al., 2014, Stanford) takes a fundamentally different approach from Word2Vec. Instead of learning from individual (center, context) pairs one at a time, GloVe first builds a global co-occurrence matrix (a giant table counting how often every word appears near every other word across the entire corpus) and then factorizes this matrix into low-dimensional vectors.

Think of it this way: Word2Vec learns by reading one sentence at a time (local context). GloVe first compiles all the statistics, then learns from the complete picture (global statistics). Neither approach is strictly better; they tend to produce similar-quality embeddings, but the mathematical foundations are quite different.

The real power of GloVe comes from the insight that ratios of co-occurrence probabilities encode meaning:

	P(w | ice)	P(w | steam)	Ratio
solid	high	low	>> 1 (related to ice, not steam)
gas	low	high	<< 1 (related to steam, not ice)
water	high	high	≈ 1 (related to both)
fashion	low	low	≈ 1 (related to neither)

# Loading pre-trained GloVe vectors
import gensim.downloader as api

# Download GloVe (trained on Wikipedia + Gigaword, 400K vocab, 100d)
glove = api.load('glove-wiki-gigaword-100')

# Same interface as Word2Vec
print("GloVe similarity cat/dog:", glove.similarity('cat', 'dog'))
print("GloVe analogy king-man+woman:",
      glove.most_similar(positive=['king', 'woman'], negative=['man'], topn=1))

FastText: Subword Embeddings

FastText (Facebook/Meta, 2016) extends Word2Vec with a critical improvement: it represents each word as a bag of character n-grams. The word "running" with n=3 becomes: <ru, run, unn, nni, nin, ing, ng>

Why this matters:

• Handles unseen words: Even if "unfriend" never appeared in training data, FastText can compose its vector from subwords like "un-", "friend", "-end" etc.
• Morphologically rich languages: Turkish, Finnish, Arabic, where a single word can have many inflected forms, benefit enormously from subword sharing.
• Typos and misspellings: "runnng" is still close to "running" because they share most subword n-grams.

# FastText handles out-of-vocabulary words
from gensim.models import FastText

# Train on same corpus
ft_model = FastText(
    sentences,
    vector_size=50,
    window=3,
    min_count=1,
    sg=1,
    epochs=100,
)

# FastText can produce vectors for UNSEEN words!
print("Vector for 'kingdoms' (never in training data):")
print(ft_model.wv['kingdoms'][:5])  # Works! Uses subword info from 'kingdom'

# Word2Vec would crash: KeyError: "word 'kingdoms' not in vocabulary"

Comparing the Three Approaches

Let us put Word2Vec, GloVe, and FastText side by side:

Property	Word2Vec	GloVe	FastText
Training approach	Predict context from center word (local)	Factorize co-occurrence matrix (global)	Same as Word2Vec but with subwords
Handles unseen words?	No: crashes on OOV words	No: crashes on OOV words	Yes: composes from subword n-grams
Handles morphology?	No: "run"/"running"/"ran" are unrelated	No	Yes: shared subwords connect inflections
Training speed	Fast (negative sampling)	Fast (matrix operations)	Slower (more parameters per word)
Best for	General English NLP	General English NLP	Morphologically rich languages, noisy text
Context-aware?	None of them: all produce static, context-independent vectors

Visualizing Embeddings

High-dimensional embeddings can be projected to 2D for visualization using t-SNE (preserves local structure) or UMAP (preserves both local and global structure, and is much faster).

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

# Get vectors for a subset of words
words = ['king', 'queen', 'man', 'woman', 'prince', 'princess',
         'cat', 'dog', 'paris', 'france', 'berlin', 'germany', 'tokyo', 'japan']

vectors = [wv[w] for w in words]

# Project to 2D with t-SNE
tsne = TSNE(n_components=2, random_state=42, perplexity=5)
vectors_2d = tsne.fit_transform(vectors)

# Plot
plt.figure(figsize=(10, 8))
for i, word in enumerate(words):
    plt.scatter(vectors_2d[i, 0], vectors_2d[i, 1])
    plt.annotate(word, (vectors_2d[i, 0]+0.5, vectors_2d[i, 1]+0.5), fontsize=12)
plt.title("Word Embeddings Projected to 2D with t-SNE")
plt.show()

import seaborn as sns
import numpy as np

words = ['king', 'queen', 'man', 'woman', 'cat', 'dog', 'paris', 'france']
vectors = np.array([wv[w] for w in words])

# Compute cosine similarity matrix
from sklearn.metrics.pairwise import cosine_similarity
sim_matrix = cosine_similarity(vectors)

# Plot heatmap
plt.figure(figsize=(8, 6))
sns.heatmap(sim_matrix, xticklabels=words, yticklabels=words,
            annot=True, fmt=".2f", cmap="RdYlGn", vmin=-0.2, vmax=1)
plt.title("Word Similarity Matrix (Cosine Similarity)")
plt.tight_layout()
plt.show()
# You'll see bright blocks where royalty words cluster together,
# and where animals cluster together, confirming the embedding structure