code/utils/word-embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [One-Hot Encoding](#toc2_)    \n",
    "    - [Generating One-Hot Vectors for a Vocabulary](#toc2_1_1_)    \n",
    "    - [Using One-Hot Vectors in a Simple Context](#toc2_1_2_)    \n",
    "- [Frequency-Based Word Representations](#toc3_)    \n",
    "  - [Count Vectorization](#toc3_1_)    \n",
    "  - [TF-IDF (Term Frequency-Inverse Document Frequency)](#toc3_2_)    \n",
    "- [Word Embedding](#toc4_)    \n",
    "  - [Random Initialization](#toc4_1_)    \n",
    "  - [Word2Vec](#toc4_2_)    \n",
    "  - [GloVe (Global Vectors for Word Representation)](#toc4_3_)    \n",
    "  - [FastText (Subword Information)](#toc4_4_)    \n",
    "  - [ELMo (Embeddings from Language Models)](#toc4_5_)    \n",
    "  - [BERT (Bidirectional Encoder Representations from Transformers)](#toc4_6_)    \n",
    "- [Document Embedding](#toc5_)    \n",
    "  - [Matrix Factorization (Latent Semantic Analysis - LSA)](#toc5_1_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import torch\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.manifold import TSNE\n",
    "from torch import nn, optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set a seed for deterministic results\n",
    "seed = 42\n",
    "torch.manual_seed(seed)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set print options to increase line width\n",
    "torch.set_printoptions(linewidth=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[One-Hot Encoding](#toc0_)\n",
    "\n",
    "- To plug words into a Neural Network, we need a way to turn the **words** into **numbers**.\n",
    "- One-hot encoding is a simple representation of **categorical** data in **binary vectors**.\n",
    "- Each unique **word** (or **token**) in a vocabulary is assigned a **unique index**.\n",
    "- The vector representation consists of **all zeros** except for a **`1`** at the position of the **word's index**.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../../assets/images/original/we/one-hot-embedding.svg\" alt=\"one-hot-embedding.svg\" style=\"width: 80%;\">\n",
    "  <figcaption style=\"text-align: center;\">One-Hot Embedding</figcaption>\n",
    "</figure>\n",
    "\n",
    "📜 **Properties**:\n",
    "\n",
    "- **Dimensionality**\n",
    "  - The dimension of the vector equals the size of the vocabulary.\n",
    "  - For example, if there are 10,000 words, each vector is 10,000-dimensional.\n",
    "- **Sparsity**\n",
    "  - Most values in the vector are 0, leading to high memory consumption and computational inefficiency.\n",
    "- **No Semantic Relationships**\n",
    "  - Vectors for words like **\"cat\"** and **\"dog\"** are as dissimilar as **\"cat\"** and **\"table\"**, even though **\"cat\"** and **\"dog\"** are **semantically related**.\n",
    "\n",
    "📉 **Limitations**:\n",
    "\n",
    "- **Inefficient** for large vocabularies due to high dimensionality.\n",
    "- Encodes words **independently** without considering their **meaning** or **relationships**.\n",
    "- **Sparse** vectors can lead to **poor** performance in machine learning models, especially for **large datasets**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc2_1_1_'></a>[Generating One-Hot Vectors for a Vocabulary](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the vocabulary\n",
    "vocabulary = [\"apple\", \"banana\", \"school\", \"date\"]\n",
    "vocab_size = len(vocabulary)\n",
    "\n",
    "# create and store one-hot vectors in a dictionary\n",
    "one_hot_vectors = {vocabulary[idx]: one_hot_vector for idx, one_hot_vector in enumerate(torch.eye(vocab_size))}\n",
    "\n",
    "# display the one-hot vectors\n",
    "for word, vector in one_hot_vectors.items():\n",
    "    print(f\"{word}: {vector.tolist()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc2_1_2_'></a>[Using One-Hot Vectors in a Simple Context](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to compute similarity (cosine similarity)\n",
    "def calculate_similarity(vec1: torch.Tensor, vec2: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.cosine_similarity(vec1.unsqueeze(dim=0), vec2.unsqueeze(dim=0))\n",
    "\n",
    "\n",
    "# create a 2D tensor for similarity values directly\n",
    "similarity_values = torch.zeros((vocab_size, vocab_size))\n",
    "\n",
    "for i in range(vocab_size):\n",
    "    for j in range(vocab_size):\n",
    "        similarity_values[i, j] = calculate_similarity(one_hot_vectors[vocabulary[i]], one_hot_vectors[vocabulary[j]])\n",
    "\n",
    "# plot the heatmap\n",
    "plt.figure()\n",
    "sns.heatmap(similarity_values, annot=True, fmt=\".1f\", xticklabels=vocabulary, yticklabels=vocabulary, cmap=\"Blues\")\n",
    "plt.title(\"Word Similarity Heatmap\")\n",
    "plt.xlabel(\"Words\")\n",
    "plt.ylabel(\"Words\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[Frequency-Based Word Representations](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_1_'></a>[Count Vectorization](#toc0_)\n",
    "\n",
    "- Represents each word as a **vector** with a **dimensionality** equal to the **vocabulary size**.\n",
    "- Each entry in the vector corresponds to the **number of times** the word appears in a **document** or **corpus**.\n",
    "- For the sentence **\"I love AI, AI loves me\"** with a vocabulary of `{I, love, AI, loves, me}`, the count vectors are:\n",
    "  - `I : [1, 0, 0, 0, 0]`\n",
    "  - `AI: [0, 0, 2, 0, 0]`\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../../assets/images/original/we/frequency-count-vectorization.svg\" alt=\"frequency-count-vectorization.svg\" style=\"width: 100%;\">\n",
    "  <figcaption style=\"text-align: center;\">Frequency-Based: Count Vectorization</figcaption>\n",
    "</figure>\n",
    "\n",
    "📈 **Advantages**:\n",
    "\n",
    "- Simple and easy to implement\n",
    "\n",
    "📉 **Disadvantages**:\n",
    "\n",
    "- Most entries are **zero** for large vocabularies.\n",
    "- Fails to capture **relationships** between words.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"The cat sat on the mat\",  # Document 1\n",
    "    \"The dog sat on the mat\",  # Document 2\n",
    "    \"The cat and the dog played outside\",  # Document 3\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize and convert to lowercase\n",
    "def preprocess(corpus: list[str]) -> list:\n",
    "    return [doc.lower().split() for doc in corpus]\n",
    "\n",
    "\n",
    "# count vectorization\n",
    "def count_vectorization(corpus: list[str]) -> dict[str, list[int]]:\n",
    "    docs = preprocess(corpus)\n",
    "    unique_vocabs = set(word for doc in docs for word in doc)\n",
    "    word_counts = {word: [doc.count(word) for doc in docs] for word in unique_vocabs}\n",
    "    return word_counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pre-process the raw corpus\n",
    "pp_corpus = preprocess(corpus)\n",
    "len_max_word = len(max(pp_corpus, key=len))\n",
    "\n",
    "# calculate Count Vectorization for the entire corpus\n",
    "word_counts = count_vectorization(corpus)\n",
    "\n",
    "# log\n",
    "print(\"Count Vectorization values:\")\n",
    "for term, counts in word_counts.items():\n",
    "    print(f\"  {term:{len_max_word + 1}}: {counts}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_2_'></a>[TF-IDF (Term Frequency-Inverse Document Frequency)](#toc0_)\n",
    "\n",
    "- Improves on **raw counts** by assigning **weights** to words based on their **importance** in the document relative to the entire corpus.\n",
    "- **TF (Term Frequency)**: Frequency of word $t$ in document $d$.\n",
    "- **IDF (Inverse Document Frequency)**: Importance of word $t$ across all documents.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../../assets/images/original/we/frequency-tf-idf.svg\" alt=\"frequency-tf-idf.svg\" style=\"width: 100%;\">\n",
    "  <figcaption style=\"text-align: center;\">Frequency-Based: TF-IDF</figcaption>\n",
    "</figure>\n",
    "\n",
    "🔬 **Formula**:\n",
    "\n",
    "1. **Term Frequency (TF)**\n",
    "    $$\\text{TF}(t, d) = \\frac{\\text{Count of term t in document d}}{\\text{Total terms in document d}}$$\n",
    "1. **Inverse Document Frequency (IDF)**\n",
    "    $$\\text{IDF}(t) = \\log \\left( \\frac{N + 1}{\\text{DF}(t) + 1} \\right)$$\n",
    "1. **Term Frequency-Inverse Document Frequency (TF-IDF)**\n",
    "    $$\\text{TF-IDF}(t, d) = \\text{TF}(t, d) \\times \\text{IDF}(t)$$\n",
    "\n",
    "where $N$ is the total number of documents, and $\\text{DF}(t)$ is the number of documents containing $t$.\n",
    "\n",
    "📝 **Papers**:\n",
    "\n",
    "- [**A Statistical Interpretation of Term Specificity and Its Application in Retrieval**](https://www.emerald.com/insight/content/doi/10.1108/eb026526/full/html) by [*Karen Spärck Jones*](https://scholar.google.com/citations?user=HzDsaGYAAAAJ&hl=en) in 1972.\n",
    "- [**A vector space model for automatic indexing**](https://dl.acm.org/doi/abs/10.1145/361219.361220) by [*Gerard Salton*](https://scholar.google.com/citations?user=PmJ2544AAAAJ&hl=en) et al. in 1975.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"The cat sat on the mat\",  # Document 1\n",
    "    \"The dog sat on the mat\",  # Document 2\n",
    "    \"The cat and the dog played outside\",  # Document 3\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize and convert to lowercase\n",
    "def preprocess(corpus: list[str]) -> list:\n",
    "    return [doc.lower().split() for doc in corpus]\n",
    "\n",
    "\n",
    "# calculate Term Frequency (TF)\n",
    "def term_frequency(doc: str, term: str) -> float:\n",
    "    term_count = doc.count(term)\n",
    "    return term_count / len(doc)\n",
    "\n",
    "\n",
    "# calculate Inverse Document Frequency (IDF)\n",
    "def inverse_document_frequency(corpus: list[str], term: str) -> float:\n",
    "    num_docs_with_term = sum(1 for doc in corpus if term in doc)\n",
    "    return math.log((len(corpus) + 1) / (num_docs_with_term + 1))\n",
    "\n",
    "\n",
    "# calculate TF-IDF for each term in the corpus\n",
    "def calculate_tf_idf(\n",
    "    corpus: list[str],\n",
    ") -> tuple[dict[str, list[float]], dict[str, float], dict[str, torch.Tensor]]:\n",
    "    terms = set(word for doc in corpus for word in doc)  # get unique terms from all documents\n",
    "    tf_values = {}\n",
    "    idf_values = {}\n",
    "    tf_idf_values = {}\n",
    "\n",
    "    # calculate TF and IDF for all terms\n",
    "    for term in terms:\n",
    "        tf_values[term] = [term_frequency(doc, term) for doc in corpus]\n",
    "        idf_values[term] = inverse_document_frequency(corpus, term)\n",
    "        tf_idf_values[term] = torch.tensor(tf_values[term]) * idf_values[term]\n",
    "\n",
    "    return tf_values, idf_values, tf_idf_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pre-process the raw corpus\n",
    "pp_corpus = preprocess(corpus)\n",
    "len_max_word = len(max(pp_corpus, key=len))\n",
    "\n",
    "# calculate TF, IDF, and TF-IDF for the entire corpus\n",
    "tf_values, idf_values, tf_idf_values = calculate_tf_idf(pp_corpus)\n",
    "\n",
    "# log\n",
    "print(\"Term Frequency (TF) values:\")\n",
    "for term, tf in tf_values.items():\n",
    "    print(f\"  {term:{len_max_word + 1}}: {[f'{t:.4f}' for t in tf]}\")\n",
    "\n",
    "print(\"\\nInverse Document Frequency (IDF) values:\")\n",
    "for term, idf in idf_values.items():\n",
    "    print(f\"  {term:{len_max_word + 1}}: {idf:.4f}\")\n",
    "\n",
    "print(\"\\nTF-IDF values:\")\n",
    "for term, tf_idf in tf_idf_values.items():\n",
    "    print(f\"  {term:{len_max_word + 1}}: {[f'{t:.4f}' for t in tf_idf.tolist()]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc4_'></a>[Word Embedding](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_1_'></a>[Random Initialization](#toc0_)\n",
    "\n",
    "- It maps **words** into **dense vectors of fixed size** (initialized with random values), capturing **semantic** meanings and **relationships** in **various contexts**.\n",
    "- Unlike one-hot encoding, embeddings are **continuous-valued** and **compact representations**.\n",
    "- The vectors are updated using some downstream task, like a classification task, so that words with similar meanings would be closer together in the vector space.\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../../assets/images/original/we/word-embedding.svg\" alt=\"word-embedding.svg\" style=\"width: 100%;\">\n",
    "  <figcaption style=\"text-align: center;\">Traditional Word Embedding using Neural Networks</figcaption>\n",
    "</figure>\n",
    "\n",
    "📜 **Properties**:\n",
    "\n",
    "- **Low Dimensionality**\n",
    "  - Vectors typically have dimensions like **50**, **100**, or **300**, **regardless of vocabulary size**.\n",
    "- **Semantic Relationships**\n",
    "  - Words with similar **meanings** or **contexts** have similar **vector representations**.\n",
    "  - Example: **\"king\"** and **\"queen\"** might have vectors that are close in the **embedding space**.\n",
    "- **Learned Representations**\n",
    "  - Embeddings are **learned from data**, capturing nuanced meanings based on word co-occurrences.\n",
    "\n",
    "📈 **Advantages**:\n",
    "\n",
    "- **Lower** **memory** and **computational** requirements compared to one-hot encoding.\n",
    "- Captures **semantic** relationships and contextual nuances.\n",
    "- Pre-trained embeddings (like [**GloVe**](https://nlp.stanford.edu/projects/glove/) or [**FastText**](https://fasttext.cc/)) can be used across multiple tasks.\n",
    "\n",
    "📉 **Limitations**:\n",
    "\n",
    "- Embeddings may not generalize well if the training data is biased or limited\n",
    "- Pre-trained embeddings struggle with unseen words, though methods like FastText address this by considering subword information.\n",
    "  - **GloVe & Word2Vec**\n",
    "    - These embeddings are **static** and **tied** directly to the words in the training corpus.\n",
    "    - If a word **does not appear** in the training corpus, the model **cannot** generate an **embedding** for it.\n",
    "    - Because they rely on **co-occurrence statistics** (or local context), and if a word is absent from the training data, the model has no data to create a meaningful representation.\n",
    "    - This results in a failure to handle **out-of-vocabulary (OOV)** words.\n",
    "  - **FastText**\n",
    "    - It represents each word as a bag of character n-grams (subword units).\n",
    "    - For example, the word \"playing\" could be broken down into subword units like:\n",
    "      - \"pla\", \"lay\", \"ayi\", \"yin\", \"ing\" (and also prefixes and suffixes like \"pl\", \"ing\", etc.)\n",
    "      - This is subword-level information that captures word morphology.\n",
    "\n",
    "📝 **Papers**:\n",
    "\n",
    "- [**A Neural Probabilistic Language Model**](https://proceedings.neurips.cc/paper_files/paper/2000/hash/728f206c2a01bf572b5940d7d9a8fa4c-Abstract.html) by [Yoshua Bengio](https://scholar.google.com/citations?user=kukA0LcAAAAJ&hl=en&oi=sra) et al. in 2000.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"cats like to chase mice\",  # Document 1\n",
    "    \"dogs like to chase cats\",  # Document 2\n",
    "    \"mice like to chase dogs\",  # Document 3\n",
    "    \"cats and dogs are pets\",  # Document 4\n",
    "    \"mice are small and quick\",  # Document 5\n",
    "    \"dogs are loyal and friendly\",  # Document 6\n",
    "    \"cats are independent and curious\",  # Document 7\n",
    "    \"mice are often found in fields\",  # Document 8\n",
    "    \"pets bring joy to their owners\",  # Document 9\n",
    "    \"dogs and cats can be friends\",  # Document 10\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize the corpus and build a vocabulary\n",
    "vocabulary = {word for sentence in corpus for word in sentence.split()}\n",
    "vocab_size = len(vocabulary)\n",
    "word_to_idx = {word: idx for idx, word in enumerate(vocabulary)}\n",
    "idx_to_word = {idx: word for word, idx in word_to_idx.items()}\n",
    "\n",
    "# hyperparameters\n",
    "embedding_dim = 10  # each word will be represented by 10 numbers\n",
    "context_size = 2  # number of context words to predict the next word\n",
    "learning_rate = 0.001\n",
    "epochs = 120\n",
    "\n",
    "# prepare training data (pairs of context words and target word)\n",
    "context_words = []\n",
    "target_words = []\n",
    "for sentence in corpus:\n",
    "    words = sentence.split()\n",
    "    for i in range(context_size, len(words)):\n",
    "\n",
    "        # context = previous 'context_size' words\n",
    "        context = words[i - context_size : i]\n",
    "        target = words[i]\n",
    "\n",
    "        # convert words to indices\n",
    "        context_indices = [word_to_idx[word] for word in context]\n",
    "        target_index = word_to_idx[target]\n",
    "\n",
    "        context_words.append(context_indices)\n",
    "        target_words.append(target_index)\n",
    "\n",
    "# convert to tensors\n",
    "context_tensor = torch.tensor(context_words, dtype=torch.long)\n",
    "target_tensor = torch.tensor(target_words, dtype=torch.long)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NextWordPredictionModel(nn.Module):\n",
    "    def __init__(self, vocab_size: int, embedding_dim: int):\n",
    "        super().__init__()\n",
    "        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.fc = nn.Linear(embedding_dim * context_size, vocab_size)\n",
    "\n",
    "    def forward(self, context: torch.Tensor) -> torch.Tensor:\n",
    "        embedded = self.embeddings(context)  # embedding layer [lookup table]\n",
    "        embedded = embedded.view(1, -1)  # flatten the embeddings\n",
    "        output = self.fc(embedded)  # fully connected layer\n",
    "        return output\n",
    "\n",
    "\n",
    "# instantiate the model, loss function, and optimizer\n",
    "model = NextWordPredictionModel(vocab_size, embedding_dim)\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training loop\n",
    "for epoch in range(epochs):\n",
    "    total_loss = 0\n",
    "    for i in range(len(context_tensor)):\n",
    "        context = context_tensor[i]  # x\n",
    "        target = target_tensor[i]  # y_true\n",
    "\n",
    "        # zero the gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward\n",
    "        output = model(context)\n",
    "\n",
    "        # backward\n",
    "        loss = loss_fn(output, target.unsqueeze(dim=0))\n",
    "        loss.backward()\n",
    "\n",
    "        # update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # store loss\n",
    "        total_loss += loss.item()\n",
    "\n",
    "    # log\n",
    "    if epoch % 10 == 0 or (epoch + 1) == epochs:\n",
    "        print(f\"epoch {epoch+1:0{len(str(epochs))}}/{epochs} -> loss: {total_loss}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len_max_word = len(max(vocabulary, key=len))\n",
    "\n",
    "# extract the learned word embeddings\n",
    "word_embeddings = model.embeddings.weight.data\n",
    "\n",
    "# display word embeddings for each word in the vocabulary\n",
    "for idx, word in idx_to_word.items():\n",
    "    print(f\"Word: {word:{len_max_word + 1}}, Embedding: {word_embeddings[idx]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to compute similarity (cosine similarity)\n",
    "def calculate_similarity(vec1: torch.Tensor, vec2: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.cosine_similarity(vec1.unsqueeze(dim=0), vec2.unsqueeze(dim=0))\n",
    "\n",
    "\n",
    "# create a 2D tensor for similarity values directly\n",
    "similarity_values = torch.zeros((vocab_size, vocab_size))\n",
    "\n",
    "for i in range(vocab_size):\n",
    "    for j in range(vocab_size):\n",
    "        similarity_values[i, j] = calculate_similarity(word_embeddings[i], word_embeddings[j])\n",
    "\n",
    "# plot the heatmap\n",
    "plt.figure(figsize=(24, 18))\n",
    "sns.heatmap(similarity_values, annot=True, fmt=\".1f\", xticklabels=vocabulary, yticklabels=vocabulary, cmap=\"Blues\")\n",
    "plt.title(\"Word Similarity Heatmap\")\n",
    "plt.xlabel(\"Words\")\n",
    "plt.ylabel(\"Words\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reduce dimensionality of word embeddings\n",
    "tsne = TSNE(n_components=2, perplexity=5, random_state=seed)\n",
    "reduced_embeddings = tsne.fit_transform(word_embeddings)\n",
    "\n",
    "# plot the embeddings\n",
    "plt.figure(figsize=(8, 8))\n",
    "for i, word in enumerate(idx_to_word.values()):\n",
    "    plt.scatter(reduced_embeddings[i, 0], reduced_embeddings[i, 1])\n",
    "    plt.annotate(word, (reduced_embeddings[i, 0], reduced_embeddings[i, 1]))\n",
    "\n",
    "plt.title(\"Word Embeddings Visualization\")\n",
    "plt.xlabel(\"Dimension 1\")\n",
    "plt.ylabel(\"Dimension 2\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_2_'></a>[Word2Vec](#toc0_)\n",
    "\n",
    "- A popular method for learning word embeddings.\n",
    "\n",
    "🏛️ **Two Main Architectures of Word2Vec**:\n",
    "\n",
    "- **Skip-gram**\n",
    "  - Predicts the **context words** (surrounding words) given a **target word**.\n",
    "  - Objective: Maximize the probability of context words appearing given the target word.\n",
    "    <figure style=\"text-align: center;\">\n",
    "      <img src=\"../../assets/images/original/we/word2vec-skipgram.svg\" alt=\"word2vec-skipgram.svg\" style=\"width: 100%;\">\n",
    "      <figcaption style=\"text-align: center;\">Word2Vec using Skip-Gram method</figcaption>\n",
    "    </figure>\n",
    "\n",
    "- **CBOW (Continuous Bag of Words)**\n",
    "  - Predicts a **target word** based on its surrounding **context words**.\n",
    "  - Objective: Maximize the probability of a target word given the context.\n",
    "    <figure style=\"text-align: center;\">\n",
    "      <img src=\"../../assets/images/original/we/word2vec-cbow.svg\" alt=\"word2vec-cbow.svg\" style=\"width: 100%;\">\n",
    "      <figcaption style=\"text-align: center;\">Word2Vec using CBOW method</figcaption>\n",
    "    </figure>\n",
    "\n",
    "📈 **Advantages**:\n",
    "\n",
    "- It provides better word embedding representations compared to traditional vanilla word embeddings\n",
    "- Optimized using techniques like **negative sampling** or **hierarchical softmax** to improve efficiency at training stage.\n",
    "  - **Negative Sampling**\n",
    "    - Focuses on distinguishing observed (**positive**) word-context pairs from randomly sampled (**negative**) pairs (**around 2-20 pairs**).\n",
    "\n",
    "📉 **Limitations**:\n",
    "\n",
    "- **Static Embeddings**\n",
    "  - Produces the **same** vector for a word, regardless of its **context**.\n",
    "  - For example, **\"bark\"** has the same embedding whether referring to the **outer covering of a tree** or the **sound a dog makes**.\n",
    "- **OOV Words**\n",
    "  - Struggles with **unseen** words in the test set unless extended methods like **FastText** are used.\n",
    "- These limitations are overcome by **contextual embeddings** (e.g., [**BERT**](https://github.com/google-research/bert), [**GPT-3**](https://github.com/openai/gpt-3)).\n",
    "\n",
    "📝 **Papers**:\n",
    "\n",
    "- [**Efficient Estimation of Word Representations in Vector Space**](https://www.khoury.northeastern.edu/home/vip/teach/DMcourse/4_TF_supervised/notes_slides/1301.3781.pdf) by [*Tomas Mikolov*](https://scholar.google.com/citations?user=oBu8kMMAAAAJ&hl=en&oi=sra) et al. in 2013.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_3_'></a>[GloVe (Global Vectors for Word Representation)](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_4_'></a>[FastText (Subword Information)](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_5_'></a>[ELMo (Embeddings from Language Models)](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_6_'></a>[BERT (Bidirectional Encoder Representations from Transformers)](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc5_'></a>[Document Embedding](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc5_1_'></a>[Matrix Factorization (Latent Semantic Analysis - LSA)](#toc0_)\n",
    "\n",
    "- **Matrix factorization** is a technique used in **Latent Semantic Analysis (LSA)** to uncover hidden structures in large text corpora.\n",
    "- It aims to capture the **semantic relationships** between words and documents by decomposing a large **term-document matrix** into a smaller **latent semantic space**.\n",
    "- It uses **singular value decomposition (SVD)** to reduce the dimensionality of the term-document matrix, revealing patterns and associations between terms and documents.\n",
    "\n",
    "📜 **Properties**:\n",
    "\n",
    "- **Dimensionality Reduction**\n",
    "  - The technique reduces the **high-dimensional** term-document matrix into a **lower-dimensional representation** while retaining key semantic information.\n",
    "- **Latent Semantic Structure**\n",
    "  - LSA uncovers **hidden relationships** and **topics** within the data that are not immediately visible in the raw term-document matrix.\n",
    "- **Singular Value Decomposition (SVD)**\n",
    "  - The matrix is factorized into three matrices: **U**, **Σ**, and **V**, where:\n",
    "    - **U**: Term matrix (words),\n",
    "    - **Σ**: Singular values (importance),\n",
    "    - **V**: Document matrix.\n",
    "\n",
    "📈 **Advantages**:\n",
    "\n",
    "- **Captures Synonymy**\n",
    "  - LSA can recognize words with similar meanings even if they don't appear together frequently in the documents.\n",
    "  - Example: **\"car\"** and **\"automobile\"** might be clustered together in the **latent space** despite not often appearing in the same context.\n",
    "- **Dimensionality Reduction**\n",
    "  - LSA simplifies the data while preserving key semantic information, reducing both **memory usage** and **computation time**.\n",
    "- **Discover Topics**\n",
    "  - It can uncover **latent topics** within the corpus, grouping similar documents and terms based on their underlying meaning.\n",
    "- **No Need for Labeling**\n",
    "  - Unlike supervised learning, LSA does not require labeled data to identify these patterns.\n",
    "\n",
    "📉 **Limitations**:\n",
    "\n",
    "- **Does Not Handle Polysemy Well**\n",
    "  - Words with multiple meanings may be grouped together despite having different contexts. For example, **\"bank\"** (riverbank vs. financial institution) could be treated the same.\n",
    "- **Sparse Matrices**\n",
    "  - LSA relies on large, sparse term-document matrices, which can be **computationally expensive** to create and process.\n",
    "- **Requires Sufficient Data**\n",
    "  - The quality of the latent semantic space depends heavily on the amount and diversity of the training data.\n",
    "- **Interpretability Issues**\n",
    "  - The topics or relationships discovered by LSA may be difficult to **interpret** due to the complexity of the latent space and the SVD transformation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"I love programming in Python\",  # Document 1\n",
    "    \"Python is a great programming language\",  # Document 2\n",
    "    \"I enjoy data science and machine learning\",  # Document 3\n",
    "    \"Data science is amazing with Python\",  # Document 4\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorize the corpus using TF-IDF (Term Frequency - Inverse Document Frequency)\n",
    "# removing stop words in English : \"the\", \"and\", \"is\", \"in\", \"at\", \"on\", \"of\", \"to\", \"a\", \"for\", etc.\n",
    "vectorizer = TfidfVectorizer(stop_words=\"english\")\n",
    "X = vectorizer.fit_transform(corpus)\n",
    "\n",
    "# apply Latent Semantic Analysis (LSA) using Truncated SVD\n",
    "# n_components is the number of latent semantic dimensions you want to reduce to\n",
    "lsa = TruncatedSVD(n_components=2)\n",
    "X_lsa = lsa.fit_transform(X)\n",
    "\n",
    "# log\n",
    "print(f\"Original Matrix Shape (term-document matrix): {X.shape}\")\n",
    "print(f\"Transformed Matrix Shape (after LSA): {X_lsa.shape}\")\n",
    "print(f\"\\nTransformed Matrix (LSA representation):\\n{X_lsa}\")\n",
    "\n",
    "# examine the topics (terms most related to each component)\n",
    "terms = vectorizer.get_feature_names_out()\n",
    "print(\"\\nTop terms for each LSA component:\")\n",
    "for i, topic in enumerate(lsa.components_):\n",
    "    print(f\"Component {i+1}:\")\n",
    "    terms_indices = topic.argsort()[:-4:-1]  # get the top 3 terms for this component\n",
    "    for index in terms_indices:\n",
    "        print(f\"  {terms[index]}\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the heatmap\n",
    "similarity_matrix = np.dot(X_lsa, X_lsa.T)\n",
    "plt.figure(figsize=(8, 6))\n",
    "sns.heatmap(\n",
    "    similarity_matrix,\n",
    "    annot=True,\n",
    "    cmap=\"YlGnBu\",\n",
    "    fmt=\".2f\",\n",
    "    linewidths=0.5,\n",
    "    xticklabels=[\"Doc 1\", \"Doc 2\", \"Doc 3\", \"Doc 4\"],\n",
    "    yticklabels=[\"Doc 1\", \"Doc 2\", \"Doc 3\", \"Doc 4\"],\n",
    ")\n",
    "plt.title(\"Document Similarity Matrix in LSA Space\")\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}