code/utils/hyperparameter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [Parameters](#toc2_)    \n",
    "- [Hyperparameters](#toc3_)    \n",
    "  - [A List of Hyperparameters](#toc3_1_)    \n",
    "  - [Train-Validation-Test Ratio](#toc3_2_)    \n",
    "  - [Data Augmentation](#toc3_3_)    \n",
    "  - [Batch Size](#toc3_4_)    \n",
    "  - [Weight Initialization](#toc3_5_)    \n",
    "  - [Number of Layers & Neurons](#toc3_6_)    \n",
    "  - [Normalizations](#toc3_7_)    \n",
    "  - [Activation Functions](#toc3_8_)    \n",
    "  - [Loss Function](#toc3_9_)    \n",
    "  - [Optimizer](#toc3_10_)    \n",
    "  - [Learning Rate](#toc3_11_)    \n",
    "  - [Momentum](#toc3_12_)    \n",
    "  - [Number of Epochs](#toc3_13_)    \n",
    "  - [Learning Rate Decay](#toc3_14_)    \n",
    "  - [Dropout Rate](#toc3_15_)    \n",
    "  - [Regularization](#toc3_16_)    \n",
    "  - [Gradient Clipping](#toc3_17_)    \n",
    "  - [Early Stopping](#toc3_18_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.optim import SGD, Adam\n",
    "from torch.optim.lr_scheduler import StepLR\n",
    "from torch.utils.data import DataLoader, TensorDataset, random_split\n",
    "from torchinfo import summary\n",
    "from torchvision.datasets import FakeData\n",
    "from torchvision.models import AlexNet\n",
    "from torchvision.transforms import v2 as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Parameters](#toc0_)\n",
    "\n",
    "- Parameters are the core elements that define the model's behavior and functionality.\n",
    "- These parameters are **learned** from the **training data** and are crucial for making accurate predictions.\n",
    "- The primary parameters in neural networks are **weights** and **biases**.\n",
    "  - **Weights** determine the strength of the connection between neurons\n",
    "  - **Biases** allow the model to shift the activation function to better fit the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize AlexNet with random weights and biases\n",
    "model = AlexNet(num_classes=1000)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(model=model, input_size=(1, 3, 227, 227), device=\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[Hyperparameters](#toc0_)\n",
    "\n",
    "- Hyperparameters in deep learning models are settings that you configure **before training** your model.\n",
    "- Hyperparameters are **not learned** from the **data** but are crucial for controlling the **training process** and **model architecture**.\n",
    "\n",
    "## <a id='toc3_1_'></a>[A List of Hyperparameters](#toc0_)\n",
    "\n",
    "<table style=\"margin: 0 auto;\">\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td>Train-Validation-Test Ratio</td>\n",
    "      <td>Data Augmentation</td>\n",
    "      <td>Normalizations</td>\n",
    "      <td>Weight Initialization</td>\n",
    "      <td>Number of Layers</td>\n",
    "      <td>Number of Neurons</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td>Activation Functions</td>\n",
    "      <td>Loss Function</td>\n",
    "      <td>Optimizer</td>\n",
    "      <td>Learning Rate</td>\n",
    "      <td>Learning Rate Decay</td>\n",
    "      <td>Momentum</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td>Batch Size</td>\n",
    "      <td>Number of Epochs</td>\n",
    "      <td>Dropout Rate</td>\n",
    "      <td>Regularization</td>\n",
    "      <td>Gradient Clipping</td>\n",
    "      <td>Early Stopping</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_2_'></a>[Train-Validation-Test Ratio](#toc0_)\n",
    "\n",
    "- The Train-Validation-Test Ratio is the proportion in which the dataset is split into three subsets:\n",
    "  - **Training Set**: Used to train the model.\n",
    "  - **Validation Set**: Used to tune hyperparameters and evaluate the model during training.\n",
    "  - **Test Set**: Used to evaluate the final model performance.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- A larger **training** set can help the model learn better, **but** it should not be so large that the **validation** and **test** sets are too small to provide reliable evaluations.\n",
    "- A properly sized **validation** set helps in **tuning hyperparameters** and **preventing overfitting**.\n",
    "- A sufficiently large **test** set ensures that the final evaluation of the model is **reliable** and **unbiased**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an artificial dataset\n",
    "dataset = FakeData(size=5000, image_size=(3, 32, 32), num_classes=3, transform=None)\n",
    "\n",
    "# define the train-validation-test split ratios\n",
    "train_ratio = 0.7\n",
    "val_ratio = 0.15\n",
    "test_ratio = 0.15\n",
    "\n",
    "# split the dataset\n",
    "train_dataset, val_dataset, test_dataset = random_split(dataset, [train_ratio, val_ratio, test_ratio])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract labels\n",
    "targets = torch.tensor([l for _, l in dataset])\n",
    "\n",
    "# calculate distribution of each set\n",
    "train_distribution = dict(zip(*[c.tolist() for c in torch.unique(targets[train_dataset.indices], return_counts=True)]))\n",
    "val_distribution = dict(zip(*[c.tolist() for c in torch.unique(targets[val_dataset.indices], return_counts=True)]))\n",
    "test_distribution = dict(zip(*[c.tolist() for c in torch.unique(targets[test_dataset.indices], return_counts=True)]))\n",
    "\n",
    "# log\n",
    "print(\"train_dataset:\")\n",
    "print(f\"\\t -> len(train_dataset) : {len(train_dataset)}\")\n",
    "print(f\"\\t -> distibution        : {train_distribution}\\n\")\n",
    "print(\"val_dataset:\")\n",
    "print(f\"\\t -> len(val_dataset)   : {len(val_dataset)}\")\n",
    "print(f\"\\t -> distibution        : {val_distribution}\\n\")\n",
    "print(\"test_dataset:\")\n",
    "print(f\"\\t -> len(test_dataset)  : {len(test_dataset)}\")\n",
    "print(f\"\\t -> distibution        : {test_distribution}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_3_'></a>[Data Augmentation](#toc0_)\n",
    "\n",
    "- Data augmentation is a technique used to **artificially increase** the size of a **training** dataset by creating **modified** versions of the data.\n",
    "- This helps improve the model's ability to **generalize** by providing more **varied** training examples.\n",
    "- Common data augmentation techniques include **rotations**, **translations**, **flips**, and **color adjustments**.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Improves Generalization**: By exposing the model to a wider **variety** of data, it can learn more **robust** features and perform better on **unseen data**.\n",
    "- **Reduces Overfitting**: Augmented data helps prevent the model from **memorizing the training data**, thus reducing **overfitting**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define transformations including data augmentation\n",
    "transform = transforms.Compose(\n",
    "    [\n",
    "        transforms.RandomHorizontalFlip(),  # randomly flips the image horizontally with a 50% chance\n",
    "        transforms.RandomRotation(degrees=10),  # randomly rotates the image by up to 10 degrees\n",
    "        transforms.ToImage(),  # convert the tensor to an image\n",
    "        transforms.ToDtype(dtype=torch.float32, scale=True),  # convert the image to float32 and scale it\n",
    "        transforms.Normalize(mean=(0.5,), std=(0.5,)),  # normalize the image\n",
    "    ]\n",
    ")\n",
    "\n",
    "# load the dataset with the defined transformations\n",
    "dataset = FakeData(size=5000, image_size=(3, 32, 32), num_classes=3, transform=transform)\n",
    "\n",
    "# log\n",
    "print(dataset.extra_repr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_4_'></a>[Batch Size](#toc0_)\n",
    "\n",
    "- Batch size is a hyperparameter that defines the **number of training examples** used in one **iteration**.\n",
    "- It determines how many samples are processed before the model's internal parameters are **updated**.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Small Batch Size**: Can lead to more **noisy** updates but can help the model **generalize** better. It also requires **less memory**.\n",
    "- **Large Batch Size**: Can speed up training by making more efficient use of hardware but may lead to **overfitting** and requires **more memory**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an artificial dataset\n",
    "data = torch.randn(1000, 10)  # 1000 samples, each with 10 features\n",
    "labels = torch.randint(0, 2, (1000,))  # binary labels (0 or 1) for each sample\n",
    "\n",
    "# combine data and labels into a TensorDataset\n",
    "dataset = TensorDataset(data, labels)\n",
    "\n",
    "# define different batch sizes\n",
    "batch_size = 128\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "for batch_idx, (inputs, targets) in enumerate(dataloader):\n",
    "    outputs = inputs.sum(dim=1)\n",
    "    print(outputs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_5_'></a>[Weight Initialization](#toc0_)\n",
    "\n",
    "- Weight initialization is the process of setting the initial values of the **weights** in a neural network **before** training begins.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Faster Convergence**: Proper initialization can lead to faster convergence by providing a **good starting point** for the optimization process.\n",
    "- **Stability**: Helps in stabilizing the training process by **preventing large updates** to the weights.\n",
    "- **Avoiding Vanishing/Exploding Gradients**: Proper initialization can prevent the gradients from becoming too small (**vanishing**) or too large (**exploding**) during **backpropagation**.\n",
    "\n",
    "**📈 Common Initialization Techniques for Weights**\n",
    "\n",
    "1. **Zero Initialization**\n",
    "    - All weights are initialized to zero.\n",
    "    - Not recommended for deep networks as it can lead to symmetry problems where all neurons in a layer learn the same features.\n",
    "\n",
    "1. **Random Initialization**\n",
    "    - Weights are initialized randomly, typically from a uniform or normal distribution.\n",
    "    - Provides a diverse set of starting points but can still lead to issues with vanishing or exploding gradients.\n",
    "\n",
    "1. **Xavier (Glorot) Initialization**\n",
    "    - Weights are initialized from:\n",
    "      - $W \\sim \\mathcal{N}\\left(0, \\frac{2}{n_{\\text{in}} + n_{\\text{out}}}\\right)$\n",
    "      - $W \\sim \\mathcal{U}\\left(-{gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}} + n_{\\text{out}}}}, {gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}} + n_{\\text{out}}}}\\right)$\n",
    "    - Helps in maintaining the variance of activations and gradients throughout the network\n",
    "\n",
    "1. **He (Kaiming) Initialization**\n",
    "    - Weights are initialized from:\n",
    "      - $W \\sim \\mathcal{N}\\left(0, \\frac{2}{n_{\\text{in}}}\\right)$\n",
    "      - $W \\sim \\mathcal{U}\\left(-{gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}}}}, {gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}}}}\\right)$\n",
    "    - Particularly useful for networks with ReLU activations as it helps in maintaining the variance of activations\n",
    "\n",
    "**📉 Common Initialization Techniques for Biases**\n",
    "\n",
    "1. **Zero Initialization**\n",
    "    - All biases are initialized to zero.\n",
    "    - Generally works well and is commonly used because it does not introduce any initial bias in the learning process.\n",
    "\n",
    "1. **Constant Initialization**\n",
    "    - All biases are initialized to a constant value, often a small positive value like 0.01.\n",
    "    - Can help in ensuring that all neurons in a layer start with a small positive bias, which can be beneficial in some cases.\n",
    "\n",
    "📝 **Papers**:\n",
    "\n",
    "- [**Understanding the difficulty of training deep feedforward neural networks**](https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) by [*Xavier Glorot*](https://scholar.google.com/citations?user=_WnkXlkAAAAJ&hl=en&oi=sra) and [*Yoshua Bengio*](https://scholar.google.com/citations?user=kukA0LcAAAAJ&hl=en&oi=sra) in 2010.\n",
    "- [**Delving deep into rectifiers: Surpassing human-level performance on imagenet classification**](https://openaccess.thecvf.com/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html) by [*Kaiming He*](https://scholar.google.com/citations?user=DhtAFkwAAAAJ&hl=en&oi=sra) et al. in 2015.\n",
    "\n",
    "📝 **Docs**:\n",
    "\n",
    "- `torch.nn.init.xavier_uniform_`: [pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_uniform_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_uniform_)\n",
    "- `torch.nn.init.xavier_normal_`: [pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_normal_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_normal_)\n",
    "- `torch.nn.init.kaiming_uniform_`: [pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_)\n",
    "- `torch.nn.init.kaiming_normal_`: [pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 50)\n",
    "        self.fc2 = nn.Linear(50, 2)\n",
    "        self._initialize_weights()\n",
    "\n",
    "    def _initialize_weights(self) -> None:\n",
    "        # initializing weights based on the xavier formula (normal distribution version)\n",
    "        nn.init.xavier_normal_(self.fc1.weight)\n",
    "        nn.init.xavier_normal_(self.fc2.weight)\n",
    "\n",
    "        # initializing biases with zero\n",
    "        nn.init.zeros_(self.fc1.bias)\n",
    "        nn.init.zeros_(self.fc2.bias)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.fc1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# other initializations\n",
    "# nn.init.uniform_\n",
    "# nn.init.zeros_\n",
    "# nn.init.normal_\n",
    "# nn.init.xavier_uniform_\n",
    "# nn.init.kaiming_normal_\n",
    "# nn.init.kaiming_uniform_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_6_'></a>[Number of Layers & Neurons](#toc0_)\n",
    "\n",
    "- The number of layers and neurons in a neural network defines its **architecture**.\n",
    "- **Number of Layers** Refers to the **depth** of the network.\n",
    "- **Number of Neurons** Refers to the **width** of each layer.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Model Capacity**: Increasing the number of layers and neurons increases the model's capacity to learn **complex patterns**.\n",
    "- **Overfitting**: Too many layers and neurons can lead to **overfitting**, where the model performs well on training data but poorly on **unseen data**.\n",
    "- **Computational Cost**: More layers and neurons increase the **computational cost** and **training time**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a complex neural network with more layers and neurons\n",
    "class ComplexNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 100)\n",
    "        self.fc2 = nn.Linear(100, 200)\n",
    "        self.fc3 = nn.Linear(200, 100)\n",
    "        self.fc4 = nn.Linear(100, 50)\n",
    "        self.fc5 = nn.Linear(50, 2)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = torch.relu(self.fc2(x))\n",
    "        x = torch.relu(self.fc3(x))\n",
    "        x = torch.relu(self.fc4(x))\n",
    "        x = self.fc5(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_7_'></a>[Normalizations](#toc0_)\n",
    "\n",
    "- Normalization techniques in neural networks are used to **standardize** the inputs to a layer, improving the **training speed** and **stability**.\n",
    "- Common normalization techniques include **Batch Normalization**, **Layer Normalization**, **Instance Normalization**, and **Group Normalization**.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Faster Convergence**: Normalization helps in **faster convergence** by reducing **internal covariate** shift.\n",
    "- **Improved Performance**: Ensures that the model learns more effectively by providing standardized inputs.\n",
    "- **Stability**: Helps in stabilizing the training process by preventing large gradient updates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a simple neural network with Batch Normalization\n",
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 50)\n",
    "        self.bn1 = nn.BatchNorm1d(50)\n",
    "        self.fc2 = nn.Linear(50, 2)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.fc1(x)\n",
    "        x = self.bn1(x)\n",
    "        x = torch.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_8_'></a>[Activation Functions](#toc0_)\n",
    "\n",
    "- Activation functions introduce **non-linearity** into the neural network, enabling it to learn and represent **complex patterns** in the data.\n",
    "- They determine the output of a neuron given an input or set of inputs.\n",
    "- Choosing the right activation function is crucial for the performance of the neural network.\n",
    "- **More info**: [activation-functions.ipynb](./activation-functions.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a custom neural network that uses several activation functions\n",
    "class CustomNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 50)\n",
    "        self.fc2 = nn.Linear(50, 30)\n",
    "        self.fc3 = nn.Linear(30, 20)\n",
    "        self.fc4 = nn.Linear(20, 10)\n",
    "        self.fc5 = nn.Linear(10, 5)\n",
    "        self.fc6 = nn.Linear(5, 3)\n",
    "\n",
    "        # define activation functions\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "        self.tanh = nn.Tanh()\n",
    "        self.relu = nn.ReLU()\n",
    "        self.leaky_relu = nn.LeakyReLU(0.01)\n",
    "        self.softmax = nn.Softmax(dim=1)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.sigmoid(self.fc1(x))\n",
    "        x = self.tanh(self.fc2(x))\n",
    "        x = self.relu(self.fc3(x))\n",
    "        x = self.leaky_relu(self.fc4(x))\n",
    "        x = self.relu(self.fc5(x))\n",
    "        x = self.softmax(self.fc6(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_9_'></a>[Loss Function](#toc0_)\n",
    "\n",
    "- Loss functions (aka **cost/objective** functions), measure how well a neural network's predictions match the actual target values.\n",
    "- They guide the optimization process by providing a measure of the model's performance.\n",
    "- The choice of loss function depends on the type of problem being solved (e.g., **regression**, **binary classification**, **multi-class classification**)\n",
    "- **More info**: [loss-functions.ipynb](./loss-functions.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# regression example\n",
    "\n",
    "# artificial true and predicted values\n",
    "y_true = torch.tensor([2.5, 0.0, 2.1, 7.8])\n",
    "y_pred = torch.tensor([3.0, -0.5, 2.0, 8.0])\n",
    "\n",
    "# define the loss function\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "# compute the loss\n",
    "loss = criterion(y_pred, y_true)\n",
    "print(f\"MSE: {loss.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# binary classification example\n",
    "\n",
    "# artificial true and predicted values\n",
    "y_true = torch.tensor([1, 0, 1, 0], dtype=torch.float32)\n",
    "y_pred = torch.tensor([0.9, 0.1, 0.8, 0.4], dtype=torch.float32)\n",
    "\n",
    "# define the loss function\n",
    "criterion = nn.BCELoss()\n",
    "\n",
    "# compute the loss\n",
    "loss = criterion(y_pred, y_true)\n",
    "print(f\"BCE: {loss.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# multi-class classification example\n",
    "\n",
    "# artificial true and predicted values\n",
    "y_true = torch.tensor([2, 0, 1])\n",
    "y_pred = torch.tensor([[0.1, 0.2, 0.7], [0.8, 0.1, 0.1], [0.2, 0.6, 0.2]])\n",
    "\n",
    "# define the loss function\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# compute the loss\n",
    "loss = criterion(y_pred, y_true)\n",
    "print(f\"CE: {loss.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_10_'></a>[Optimizer](#toc0_)\n",
    "\n",
    "- Optimizers are algorithms used to **update the weights** of a neural network to **minimize** the loss function.\n",
    "- They determine how the model's parameters are adjusted based on the gradients computed during **backpropagation**.\n",
    "- Different optimizers impact the **convergence speed** and **final performance** of the model.\n",
    "\n",
    "**Common Optimizers**:\n",
    "\n",
    "- Stochastic Gradient Descent (SGD)\n",
    "   $$ \\theta = \\theta - \\eta \\nabla J(\\theta) $$\n",
    "\n",
    "- Adam (Adaptive Moment Estimation)\n",
    "   $$ m_t = \\beta_1 m_{t-1} + (1 - \\beta_1) g_t $$\n",
    "   $$ v_t = \\beta_2 v_{t-1} + (1 - \\beta_2) g_t^2 $$\n",
    "   $$ \\hat{m}_t = \\frac{m_t}{1 - \\beta_1^t} $$\n",
    "   $$ \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^t} $$\n",
    "   $$ \\theta = \\theta - \\eta \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon} $$\n",
    "\n",
    "- RMSProp (Root Mean Square Propagation)\n",
    "   $$ E[g^2]t = \\gamma E[g^2]{t-1} + (1 - \\gamma) g_t^2 $$\n",
    "   $$ \\theta = \\theta - \\frac{\\eta}{\\sqrt{E[g^2]_t + \\epsilon}} g_t $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 50)\n",
    "        self.fc2 = nn.Linear(50, 1)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# create a model instance\n",
    "model = SimpleNN()\n",
    "\n",
    "# define the optimizer\n",
    "optimizer = Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "# example input and target\n",
    "input = torch.randn(5, 10)\n",
    "target = torch.randn(5, 1)\n",
    "\n",
    "# forward pass\n",
    "output = model(input)\n",
    "\n",
    "# compute the loss\n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(output, target)\n",
    "\n",
    "# backward pass and optimization\n",
    "optimizer.zero_grad()\n",
    "loss.backward()\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_11_'></a>[Learning Rate](#toc0_)\n",
    "\n",
    "- The learning rate is a hyperparameter that controls the step size at each iteration while moving toward a minimum of the loss function.\n",
    "- It determines how much to change the model's parameters in response to the estimated error each time the model weights are updated.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **High Learning Rate**: Can cause the model to **converge** too quickly to a **suboptimal solution** or even **diverge**.\n",
    "- **Low Learning Rate**: Can make the training process very **slow** and may get stuck in **local minima**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = AlexNet()\n",
    "optimizer_1 = SGD(params=model.parameters(), lr=0.1)\n",
    "optimizer_2 = Adam(params=model.parameters(), lr=0.001)\n",
    "\n",
    "# log\n",
    "print(optimizer_1)\n",
    "print(optimizer_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_12_'></a>[Momentum](#toc0_)\n",
    "\n",
    "- Momentum is a technique used to **accelerate** the convergence of the optimization process by **adding a fraction of the previous update to the current update**.\n",
    "- It helps in **smoothing** the optimization path and can **prevent** the model from getting stuck in **local minima**.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Faster Convergence**: Helps in accelerating the convergence by smoothing the optimization path.\n",
    "- **Stability**: Reduces oscillations and helps in stabilizing the training process.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = AlexNet()\n",
    "optimizer = SGD(params=model.parameters(), lr=0.1, momentum=0.5)\n",
    "\n",
    "# log\n",
    "print(optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_13_'></a>[Number of Epochs](#toc0_)\n",
    "\n",
    "- It defines how many times the learning algorithm will work through the **entire** training dataset.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- More epochs can lead to **better learning**, but too many can cause **overfitting**.\n",
    "- The right number of epochs helps the model **generalize well** to new data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 128)\n",
    "        self.fc2 = nn.Linear(128, 2)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# generate artificial data\n",
    "data = torch.randn(1000, 10)  # 1000 samples, 10 features each\n",
    "labels = torch.randint(0, 2, (1000,))  # Binary labels (0 or 1)\n",
    "\n",
    "# create DataLoader\n",
    "dataset = TensorDataset(data, labels)\n",
    "trainloader = DataLoader(dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "# initialize network, loss function, and optimizer\n",
    "model = SimpleNN()\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "epochs = 5\n",
    "\n",
    "# training loop\n",
    "for epoch in range(epochs):\n",
    "    running_loss = 0.0\n",
    "    for images, labels in trainloader:\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(images)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "    print(f\"epoch {epoch+1:0{len(str(epochs))}}/{epochs}, loss: {running_loss/len(trainloader):.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_14_'></a>[Learning Rate Decay](#toc0_)\n",
    "\n",
    "- Learning rate decay is a technique used to reduce the learning rate over time during training.\n",
    "- This helps the model converge more precisely by taking smaller steps as it approaches the minimum of the loss function.\n",
    "- Learning rate decay can be implemented in various ways, such as step decay, exponential decay, and adaptive learning rates.\n",
    "\n",
    "**✍️ Key Points**\n",
    "\n",
    "- **Improved Convergence**: Helps the model converge more precisely by reducing the learning rate over time.\n",
    "- **Stability**: Reduces the risk of overshooting the minimum of the loss function by taking smaller steps as training progresses.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 50)\n",
    "        self.fc2 = nn.Linear(50, 1)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# create a model instance\n",
    "model = SimpleNN()\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = SGD(model.parameters(), lr=0.1)\n",
    "\n",
    "# define a learning rate scheduler\n",
    "scheduler = StepLR(optimizer, step_size=3, gamma=0.1)\n",
    "\n",
    "# example input and target\n",
    "input = torch.randn(5, 10)\n",
    "target = torch.randn(5, 1)\n",
    "\n",
    "epochs = 10\n",
    "\n",
    "# mimic the training loop with learning rate decay\n",
    "for epoch in range(epochs):\n",
    "    # forward pass\n",
    "    output = model(input)\n",
    "\n",
    "    # compute the loss\n",
    "    loss = criterion(output, target)\n",
    "\n",
    "    # backward pass and optimization\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # step the learning rate scheduler\n",
    "    scheduler.step()\n",
    "\n",
    "    print(\n",
    "        f\"epoch {epoch+1:0{len(str(epochs))}}/{epochs}, loss: {loss.item():.5f}, learning rate: {scheduler.get_last_lr()[0]:.5f}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_15_'></a>[Dropout Rate](#toc0_)\n",
    "\n",
    "- It sets a **fraction of input units** to **zero** at each update during training time, which forces the network to learn more **robust features**.\n",
    "- Helps in regularizing the model and preventing **overfitting** by ensuring that the network does not rely too heavily on **any individual neuron**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self, dropout_rate=0.5):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(10, 128)\n",
    "        self.dropout = nn.Dropout(dropout_rate)\n",
    "        self.fc2 = nn.Linear(128, 2)\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.dropout(x)\n",
    "        x = self.fc2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_16_'></a>[Regularization](#toc0_)\n",
    "\n",
    "- Regularization is a technique used to prevent **overfitting** by adding **a penalty to the loss function**.\n",
    "- This penalty **discourages** the model from fitting **too closely** to the training data, which helps improve its **generalization** to new data.\n",
    "\n",
    "✍️ **Common Regularizations**\n",
    "\n",
    "- **L1 (Lasso) Regularization**:\n",
    "  - Adds the **absolute** value of the **coefficients** as a penalty term to the **loss function**.\n",
    "  - $ L1 = \\lambda \\sum_{i=1}^{n} |w_i| $\n",
    "- **L2 (Ridge) Regularization**:\n",
    "  - Adds the **squared** value of the **coefficients** as a penalty term to the **loss function**.\n",
    "  - $ L2 = \\lambda \\sum_{i=1}^{n} w_i^2 $\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training function + l1 regularization\n",
    "def train_model(epochs: int, l1_lambda: float = 0.01):\n",
    "    # initialize loss function, and optimizer\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        for inputs, targets in trainloader:\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(inputs)\n",
    "\n",
    "            # add L1 regularization to the loss\n",
    "            loss = criterion(outputs, targets)\n",
    "            l1_norm = sum(p.abs().sum() for p in model.parameters())\n",
    "            loss = loss + l1_lambda * l1_norm\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training function + l2 regularization\n",
    "def train_model(epochs: int, l2_lambda: float = 0.01):\n",
    "    # initialize loss function, and optimizer\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = SGD(model.parameters(), lr=0.01, weight_decay=l2_lambda)  # weight_decay is the coefficient for l2_norm\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        for inputs, targets in trainloader:\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(inputs)\n",
    "            loss = criterion(outputs, targets)\n",
    "            loss.backward()\n",
    "            optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_17_'></a>[Gradient Clipping](#toc0_)\n",
    "\n",
    "- It is a technique used to **prevent the exploding gradient** problem in neural networks, especially in **recurrent neural networks (RNNs)**.\n",
    "- It involves **capping the gradients** during backpropagation to a **maximum** value to ensure they don't become **too large**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training function + gradient clipping\n",
    "def train_model(epochs: int, clip_value: float = 1.0):\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        for inputs, targets in trainloader:\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(inputs)\n",
    "\n",
    "            loss = criterion(outputs, targets)\n",
    "            loss.backward()\n",
    "\n",
    "            # apply gradient clipping\n",
    "            nn.utils.clip_grad_norm_(model.parameters(), clip_value)\n",
    "\n",
    "            optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_18_'></a>[Early Stopping](#toc0_)\n",
    "\n",
    "- It is used to **prevent overfitting** by **stopping** the training process when the model's performance on a **validation set** starts to degrade.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(epochs: int, trainloader: DataLoader, valloader: DataLoader, patience: int = 3):\n",
    "    best_loss = float(\"inf\")\n",
    "    patience_counter = 0\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "\n",
    "        # train loop\n",
    "        model.train()\n",
    "        train_loss = 0.0\n",
    "        for inputs, targets in trainloader:\n",
    "            pass\n",
    "\n",
    "        # validation loop\n",
    "        model.eval()\n",
    "        val_loss = 0.0\n",
    "        with torch.no_grad():\n",
    "            for inputs, targets in valloader:\n",
    "                pass\n",
    "\n",
    "        # check for early stopping\n",
    "        if val_loss < best_loss:\n",
    "            best_loss = val_loss\n",
    "            patience_counter = 0\n",
    "        else:\n",
    "            patience_counter += 1\n",
    "            if patience_counter >= patience:\n",
    "                print(\"Early stopping triggered\")\n",
    "                break"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}