full network and hyperparam optimization with GA.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "8lqbeWAze76Q",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!pip install pygad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wu_yNfc2kEBP"
   },
   "source": [
    "Modified version of pygad samples. https://pygad.readthedocs.io/en/latest/README_pygad_ReadTheDocs.html#life-cycle-of-pygad <br>\n",
    "Model inspired and modified from http://uhurumkate.blogspot.com/2018/06/tiny-model-for-mnist-dataset.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a small project trying different methods to aid premature convergence. In my finding, loss functions such as cross-entropy loss, mean squared error and weighted f1 loss do terribly in GA. Only L1 loss can yield a good enough result. As for the network, contrary to my hypothesis, a small network (with ~2k param that achieves 96% acc with gradient descent) doesn't outperform a larger network. Or, a larger network like Resnet50 converges at 30% while a CNN with 8k param can go up to 72% with a decaying mutation rate. In my finding, epoch (repeat training on the same set of data), batch_size, mutation rate, num_parents_mating, keep_parents, num_solution all have played a role in aiding premature convergence. Of cause, the higher the num_solution the better as it increases diversity. Likewise, keep_parents is best to set to 0 for the same reason. The balance between epoch and batch_size is an art of computation time and result. Though, it can be addressed with a simple least square method to eval its oscillation and iteration vs result tradeoff. Finally, mutation rate and num_parents_mating have to be resolved by the GA. As when it first gein, we want large numbers, but as it continues we want it to be smaller (think of it as learning rate). It can be resolved with a decaying function (like decaying learning rate), but why not use GA?\n",
    "\n",
    "This code is using some properties of the TensorFlow and pygad library. Such like TensorFlow keeps its model weight and pygad re-initial its traits when called again after it's finished. This code also contains orphans as they may be the relic of the previous version or as a debugging tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "i5L3JMNgS-L0"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.keras\n",
    "from tensorflow.keras import losses, datasets, layers, optimizers, Sequential, metrics\n",
    "import pygad.kerasga\n",
    "import numpy\n",
    "import math\n",
    "import pygad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zdZ1owxPPsRn"
   },
   "source": [
    "# Hyper param"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "JGFxRvr3eWGX"
   },
   "outputs": [],
   "source": [
    "acc_meter = tf.keras.metrics.Accuracy()\n",
    "\n",
    "#Instead of training a entire dataset, do small batches.\n",
    "sgd_like = True\n",
    "\n",
    "#introduce new population when stucked. Often time it'll stuck in local optima.\n",
    "introduce_new_pop = False\n",
    "\n",
    "#increase batch size when stcuked.\n",
    "dynamic_batch_size = True\n",
    "\n",
    "#reshuffle\n",
    "reshuffle = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "hl1q_FZnfTG1"
   },
   "outputs": [],
   "source": [
    "#num cycles\n",
    "num_generations = 50\n",
    "\n",
    "#num soltions\n",
    "num_solution= 24\n",
    "\n",
    "#the following two param only works if introduce_new_pop = True\n",
    "#Maxium cycles stuck before introducing new population to the pool.\n",
    "max_cycles_stucked = 20\n",
    "\n",
    "#The follow two param have effect only when sgd_like = True\n",
    "#number of cycles you want on a single batch. \n",
    "epoch = 10\n",
    "#number of samples per batch\n",
    "batch_size = 500\n",
    "\n",
    "#number batch increase feed into prediction. only have effect if sgd_like = True\n",
    "#Ex. batch size 1500 -> 1500 + 500 = 2000. Note that some data will be lose during the process\n",
    "batch_increase = 1024\n",
    "\n",
    "#reshuffle every n cycles\n",
    "reshuffle_per = (int(60000/batch_size) -1) * epoch #so that it's insync with epoch\n",
    "\n",
    "#Typical pygad params\n",
    "num_parents_mating = int(num_solution * .5) #half parents mating\n",
    "parent_selection_type = \"sss\"\n",
    "mutation_type=\"adaptive\"\n",
    "mutation_higer = 90\n",
    "mutation_lower = 80\n",
    "mutation_num_genes=(mutation_higer, mutation_lower)\n",
    "keep_parents = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mm_Hl7BAgrzv"
   },
   "source": [
    "# dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8Rkq5w4dequA",
    "outputId": "fef47a88-cbbb-4e81-fde0-65bb028c7fec"
   },
   "outputs": [],
   "source": [
    "#load mnist dataset\n",
    "(X_train, y_train), (X_test, y_test) = datasets.mnist.load_data()\n",
    "#normalize data\n",
    "X_train = 2*tf.convert_to_tensor(X_train, tf.float32)/255.-1\n",
    "#[b, 28, 28] -> [b, 28, 28, 1]\n",
    "X_train = tf.reshape(X_train, (-1, 28,28,1))\n",
    "X_test = 2*tf.convert_to_tensor(X_test, tf.float32)/255.-1\n",
    "X_test = tf.reshape(X_test, (-1, 28,28,1))\n",
    "y_train = tf.convert_to_tensor(y_train, tf.int32)\n",
    "#[b] -> [b, 10]\n",
    "y_train = tf.one_hot(y_train, 10)\n",
    "y_test = tf.convert_to_tensor(y_test, tf.int32)\n",
    "y_test = tf.one_hot(y_test, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tGsLcsClp1P6"
   },
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "dW1AXe0zci0T",
    "outputId": "448e58d6-d081-4957-d126-ac6e056b422a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 26, 26, 15)        150       \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 26, 26, 15)        60        \n",
      "_________________________________________________________________\n",
      "average_pooling2d (AveragePo (None, 13, 13, 15)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 13, 13, 10)        160       \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 13, 13, 10)        40        \n",
      "_________________________________________________________________\n",
      "average_pooling2d_1 (Average (None, 6, 6, 10)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 4, 4, 5)           455       \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 4, 4, 5)           20        \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 4, 4, 128)         768       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 4, 4, 64)          8256      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 4, 4, 32)          2080      \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 17,119\n",
      "Trainable params: 17,059\n",
      "Non-trainable params: 60\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential([\n",
    "#input layer\n",
    "layers.Convolution2D(15, (3, 3), activation='relu', input_shape=(28,28,1)),\n",
    "layers.BatchNormalization(momentum=0.1),\n",
    "layers.AveragePooling2D(2),\n",
    "#convo layers1\n",
    "layers.Convolution2D(10,(1,1), activation='relu'),\n",
    "layers.BatchNormalization(momentum=0.1),\n",
    "#convo layers2\n",
    "layers.AveragePooling2D(2),\n",
    "layers.Convolution2D(5,(3,3), activation='relu'),\n",
    "layers.BatchNormalization(momentum=0.1),\n",
    "#fully connected layer1\n",
    "layers.Dense(128, activation='relu'),\n",
    "layers.Dense(64, activation='relu'),\n",
    "layers.Dense(32, activation='relu'),\n",
    "layers.Flatten(),\n",
    "#output layer\n",
    "layers.Dense(10, activation='softmax')\n",
    "])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "4NiU0RFOkL6r"
   },
   "outputs": [],
   "source": [
    "#l1 loss.\n",
    "def some_loss(y_true, y_pred):\n",
    "  return tf.reduce_mean(tf.math.abs(y_pred - y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vNFCsftXkL6r",
    "outputId": "c2ee218f-0e19-46d3-c45a-694e565abc7b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on testing set:9.200000017881393%\n",
      "Error rate on testing set:90.7999999821186%\n",
      "Accuracy on training set:9.549999982118607%\n",
      "Error rate on training set:90.4500000178814%\n"
     ]
    }
   ],
   "source": [
    "acc_meter.update_state(tf.argmax(model(X_test[:1000], training=False), axis=1), tf.argmax(y_test[:1000], axis=1))\n",
    "print(\"Accuracy on testing set:{acc}%\".format(acc=acc_meter.result().numpy()*100))\n",
    "print(\"Error rate on testing set:{err}%\".format(err=(1-acc_meter.result().numpy())*100))\n",
    "acc_meter.update_state(tf.argmax(model(X_train[:1000], training=False), axis=1), tf.argmax(y_train[:1000], axis=1))\n",
    "print(\"Accuracy on training set:{acc}%\".format(acc=acc_meter.result().numpy()*100))\n",
    "print(\"Error rate on training set:{err}%\".format(err=(1-acc_meter.result().numpy())*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ptOJ9miprDS"
   },
   "source": [
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "lMeNPbTi_E04"
   },
   "outputs": [],
   "source": [
    "#A method always modifing X_train_sgd and y_train_sgd\n",
    "def make_batch(batch_size):\n",
    "  idx = numpy.random.permutation(len(X_train))\n",
    "  global X_train_sgd, y_train_sgd\n",
    "  if len(X_train) % batch_size == 0:\n",
    "    X_train_sgd = tf.reshape(X_train.numpy()[idx], [-1, batch_size, 28, 28, 1])\n",
    "    y_train_sgd = tf.reshape(y_train.numpy()[idx], [-1, batch_size, 10])\n",
    "  else:\n",
    "    #Remove some samples if can't rehape \n",
    "    len_remove = len(X_train) % batch_size\n",
    "    X_train_sgd = tf.reshape(X_train.numpy()[idx][:-len_remove], [-1, batch_size, 28, 28, 1])\n",
    "    y_train_sgd = tf.reshape(y_train.numpy()[idx][:-len_remove], [-1, batch_size, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lsm can be use to eval the effectiness of the hyper param. If the fitness oscillate, it'll gain a low score. If it is not going up, also a low score. Only the ones are health are getting a positive high score. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "pifJ-HQ-kL6u"
   },
   "outputs": [],
   "source": [
    "def lsm(arr):\n",
    "    x = numpy.arange(len(arr))\n",
    "    n = len(arr)\n",
    "    pt1 = (n * numpy.sum(arr * x)) - (numpy.sum(x)*numpy.sum(arr))\n",
    "    pt2 = n * numpy.sum(x ** 2) - (numpy.sum(x)**2)\n",
    "    m = pt1 / pt2\n",
    "    return m\n",
    "    #b = (numpy.sum(arr) - (m*numpy.sum(x))) / n\n",
    "    #return m * x + b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "fgJVFs1nkL6u"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "_V7t28kieeVR"
   },
   "outputs": [],
   "source": [
    "#create inital training set\n",
    "X_train_sgd = []\n",
    "y_train_sgd = []\n",
    "make_batch(batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ROmBNKL2Pw2u"
   },
   "source": [
    "# Fitness "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "uAwfxH0VkL6w"
   },
   "outputs": [],
   "source": [
    "keras_ga = pygad.kerasga.KerasGA(model=model,\n",
    "                                 num_solutions=num_solution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "CuGqTK0NeeVR"
   },
   "outputs": [],
   "source": [
    "#Some helper varibles\n",
    "i = 0\n",
    "cycles_stucked = 0\n",
    "k=0\n",
    "tmp = 0\n",
    "\n",
    "#fitness_func uses acc matrix as fitness.\n",
    "def fitness_func(solution, sol_idx):\n",
    "\n",
    "    global model, i, cycles_stucked ,X_train, y_train, k, y_train_sgd, X_train_sgd\n",
    "    solution_fitness = 0\n",
    "    model_weights_matrix = pygad.kerasga.model_weights_as_matrix(model=model,\n",
    "                                                                 weights_vector=solution)\n",
    "\n",
    "    model.set_weights(weights=model_weights_matrix)\n",
    "\n",
    "    #if perfer to use loss instead\n",
    "    #solution_fitness = 1/cce(model.predict(X_train), y_train).numpy()\n",
    "\n",
    "    if sgd_like == True:\n",
    "      solution_fitness = some_loss(model(X_train_sgd[k], training=False), y_train_sgd[k]).numpy()\n",
    "    else:\n",
    "      solution_fitness = some_loss(model(X_train, training=False), y_train).numpy()\n",
    "    #print(solution_fitness)\n",
    "    #if result is invalid, return a very small number, where its trait will die in iterations\n",
    "    if numpy.isnan(solution_fitness):\n",
    "      return 1E-20\n",
    "    #inverse relationship with the loss.\n",
    "    else:\n",
    "      return 1/ solution_fitness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "ga_instance = []\n",
    "params = [epoch, batch_size, mutation_higer, mutation_lower, num_parents_mating]\n",
    "def param_fitness_func(solution, solution_idx):\n",
    "    \n",
    "    global epoch, batch_size, mutation_higer, mutation_lower, num_parents_mating, reshuffle_per, ga_instance\n",
    "\n",
    "    #set hyper param. Min, Max to make sure they're within range\n",
    "    epoch = max(min(300, solution[0]), 1)\n",
    "    batch_size = max(min(solution[1], 60000), 1)\n",
    "    mutation_higer = max(min(solution[2], 99), 2)\n",
    "    mutation_lower = max(min(solution[3], mutation_higer -1 ), 1)\n",
    "    num_parents_mating = max(min(solution[4], num_solution), 1)\n",
    "    print(solution)\n",
    "    reshuffle_per = (int(60000/batch_size) -1) * epoch\n",
    "    \n",
    "    #Flatten the model as a 1D numpy array and the create a 2D numpy array of n copy of model vector\n",
    "    initial_population = numpy.tile(pygad.kerasga.model_weights_as_vector(model), (num_solution, 1))\n",
    "    \n",
    "    ga_instance = pygad.GA(num_generations=epoch,\n",
    "                       num_parents_mating=num_parents_mating,\n",
    "                       initial_population=initial_population,\n",
    "                       fitness_func=fitness_func,\n",
    "                       keep_parents=keep_parents,\n",
    "                       mutation_type=mutation_type,\n",
    "                       parent_selection_type=parent_selection_type,\n",
    "                       crossover_type=crossover_func,\n",
    "                       mutation_num_genes=mutation_num_genes,\n",
    "                       on_generation=callback_generation)\n",
    "\n",
    "    ga_instance.run()\n",
    "    \n",
    "    net_solution, fitness, _ = ga_instance.best_solution()\n",
    "    best_solution_weights = pygad.kerasga.model_weights_as_matrix(model=model,\n",
    "                                                                  weights_vector=net_solution)\n",
    "    model.set_weights(best_solution_weights)\n",
    "    \n",
    "    slope = lsm(getattr(ga_instance, 'best_solutions_fitness'))\n",
    "    \n",
    "    #use least squared method to eval the effectiness of the hyper param\n",
    "    #fitness is the fitness of the network. Due to the nature of the tensorflow, the weight stays the same until next update\n",
    "    return fitness * slope"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N1eTVN72hDnR"
   },
   "source": [
    "# Crossover function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "52QeidT1oTdx"
   },
   "outputs": [],
   "source": [
    "def crossover_func(parents, offspring_size, ga_instance):\n",
    "  global cycles_stucked\n",
    "  offspring = []\n",
    "  idx = 0\n",
    "\n",
    "  #Standard offspring method from https://pygad.readthedocs.io/en/latest/README_pygad_ReadTheDocs.html#life-cycle-of-pygad\n",
    "  while len(offspring) != offspring_size[0]:\n",
    "    parent1 = parents[idx % parents.shape[0], :].copy()\n",
    "    parent2 = parents[(idx + 1) % parents.shape[0], :].copy()\n",
    "    random_split_point = numpy.random.choice(range(offspring_size[0]))\n",
    "    parent1[random_split_point:] = parent2[random_split_point:]\n",
    "    offspring.append(parent1)\n",
    "    idx += 1\n",
    "\n",
    "  offspring = numpy.array(offspring)\n",
    "  \n",
    "  if introduce_new_pop and cycles_stucked == max_cycles_stucked:\n",
    "    #craete new parents with new random variables\n",
    "    new_parents = tf.random.truncated_normal(offspring.shape).numpy()\n",
    "    #create an array that contain both offspring and new parents\n",
    "    new_parents = numpy.append(new_parents, offspring)\n",
    "    #shuffle\n",
    "    numpy.random.shuffle(new_parents)\n",
    "    #rehspae\n",
    "    new_parents = new_parents.reshape(-1, offspring.shape[1])\n",
    "    offspring = new_parents[:len(offspring)]\n",
    "    print('New gene added')\n",
    "    #Only want to do this once\n",
    "    cycles_stucked = 0\n",
    "\n",
    "  return numpy.array(offspring)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uMq0ymNksDWa"
   },
   "source": [
    "# callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "Ymw9h30fsBTr"
   },
   "outputs": [],
   "source": [
    "def callback_generation(ga_instance):\n",
    "    global i, k, cycles_stucked, tmp, X_train_sgd, mutation_lower, decay, epoch\n",
    "    \n",
    "    fitness = ga_instance.best_solution()[1]\n",
    "#    print(\"Generation = {generation}\".format(generation=ga_instance.generations_completed))\n",
    "#    print(\"Fitness    = {fitness}\".format(fitness=fitness))\n",
    "\n",
    "    if fitness == tmp:\n",
    "      cycles_stucked += 1\n",
    "      if dynamic_batch_size and cycles_stucked == max_cycles_stucked:\n",
    "        #increase batch size. Up until only 1 batch with all samples avaliable\n",
    "        make_batch(min(X_train_sgd.shape[1]+batch_increase, X_train_sgd.shape[0]*X_train_sgd.shape[1]))\n",
    "        print(\"Batch size increased -> new shape: \",X_train_sgd.shape)\n",
    "        k = numpy.random.randint(0, len(X_train_sgd)-1)\n",
    "        #Otherwise introduce_new_pop from cross over will never be True. \n",
    "        if not introduce_new_pop:\n",
    "          #keep track of cycles stucked\n",
    "          cycles_stucked = 0\n",
    "    else:\n",
    "      cycles_stucked = 0\n",
    "\n",
    "    #keep track of generation\n",
    "    i += 1\n",
    "    if sgd_like == True:\n",
    "      if i % (1 * epoch) == 0:\n",
    "        #select random data from reshaped dataset to train\n",
    "        k = numpy.random.randint(0, len(X_train_sgd)-1)\n",
    "#        print(\"Generation = {generation}\".format(generation=ga_instance.generations_completed))\n",
    "#        print('new set')\n",
    "    if reshuffle:\n",
    "      if i % (1 * reshuffle_per) == 0:\n",
    "        #select random data from reshaped dataset to train\n",
    "        make_batch(X_train_sgd.shape[1])\n",
    "#        print('reshuffled')\n",
    "    #update tmp\n",
    "    tmp = fitness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Call back function for param genetations\n",
    "def param_callback_generation(ga_instance):\n",
    "    print(\"Generation = {generation}\".format(generation=ga_instance.generations_completed))\n",
    "    print(\"Fitness    = {fitness}\".format(fitness=ga_instance.best_solution()[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8rfuRuDMuCfg"
   },
   "source": [
    "#  Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "u0e-_WFceeVS",
    "outputId": "e6b5d481-b1f8-4c20-ff18-79d1b2a296e3"
   },
   "outputs": [],
   "source": [
    "# Prepare the PyGAD parameters. Check the documentation for more information: https://pygad.readthedocs.io/en/latest/README_pygad_ReadTheDocs.html#pygad-ga-class\n",
    "# Use the pre-existing model weight as init population. \n",
    "initial_population = numpy.tile(pygad.kerasga.model_weights_as_vector(model), (num_solution, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/gsd/.local/lib/python3.9/site-packages/pygad/pygad.py:469: UserWarning: The percentage of genes to mutate (mutation_percent_genes=10) resutled in selecting (0) genes. The number of genes to mutate is set to 1 (mutation_num_genes=1).\n",
      "If you do not want to mutate any gene, please set mutation_type=None.\n",
      "  if not self.suppress_warnings: warnings.warn(\"The percentage of genes to mutate (mutation_percent_genes={mutation_percent}) resutled in selecting ({mutation_num}) genes. The number of genes to mutate is set to 1 (mutation_num_genes=1).\\nIf you do not want to mutate any gene, please set mutation_type=None.\".format(mutation_percent=mutation_percent_genes, mutation_num=mutation_num_genes))\n",
      "/home/gsd/.local/lib/python3.9/site-packages/pygad/pygad.py:621: UserWarning: The steady-state parent (sss) selection operator is used despite that no parents are kept in the next generation.\n",
      "  if not self.suppress_warnings: warnings.warn(\"The steady-state parent (sss) selection operator is used despite that no parents are kept in the next generation.\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  262 53702    30    44     5]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-19-80ea1efe9a88>:31: RuntimeWarning: divide by zero encountered in long_scalars\n",
      "  if i % (1 * reshuffle_per) == 0:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  281 16563    49    20     2]\n",
      "[  104 50332     2    93     8]\n",
      "[  131 31733    45    10    15]\n",
      "[  240 21563     6    55    17]\n",
      "[  262 53702    29    44     2]\n",
      "[   13 53702    30    44     5]\n",
      "[  262 53702    30     5     2]\n",
      "Generation = 1\n",
      "[  262 53702    29    44     2]\n",
      "[   13 53702    30    44     5]\n",
      "[  262 53702    30     5     2]\n",
      "Fitness    = 0.07548352094867761\n",
      "[   13 53702    30    18     5]\n",
      "[  262 53702    32    44     5]\n",
      "[   13 21197    30    44     5]\n",
      "Generation = 2\n",
      "[   13 53702    30    18     5]\n",
      "[  262 53702    32    44     5]\n",
      "[   13 21197    30    44     5]\n",
      "Fitness    = 0.3616519231815542\n",
      "[   13 17536    30    18     5]\n",
      "[   13 53702    44    44     5]\n",
      "[   92 53702    30    18     5]\n",
      "Generation = 3\n",
      "[   13 17536    30    18     5]\n",
      "[   13 53702    44    44     5]\n",
      "[   92 53702    30    18     5]\n",
      "Fitness    = 0.9667871484186336\n",
      "[  149 17536    30    18     5]\n",
      "[   13 21197    30    44     2]\n",
      "[   29 21197    30    18     5]\n",
      "Generation = 4\n",
      "[  149 17536    30    18     5]\n",
      "[   13 21197    30    44     2]\n",
      "[   29 21197    30    18     5]\n",
      "Fitness    = 0.4093932492433228\n",
      "[  296 21197    30    18     5]\n",
      "[   29 21197    30    55     5]\n",
      "[   29 21197    83    44     5]\n",
      "Generation = 5\n",
      "[  296 21197    30    18     5]\n",
      "[   29 21197    30    55     5]\n",
      "[   29 21197    83    44     5]\n",
      "Fitness    = 1.0079887970248051\n",
      "[   29 21197    94    18     5]\n",
      "[  261 21197    30    18     5]\n",
      "[   29 21197    30    70     5]\n",
      "Generation = 6\n",
      "[   29 21197    94    18     5]\n",
      "[  261 21197    30    18     5]\n",
      "[   29 21197    30    70     5]\n",
      "Fitness    = 1.0893398194536623\n",
      "[   29 21197    30    18     3]\n",
      "[   13 54516    30    18     5]\n",
      "[   29 21197    30    77     5]\n",
      "Generation = 7\n",
      "[   29 21197    30    18     3]\n",
      "[   13 54516    30    18     5]\n",
      "[   29 21197    30    77     5]\n",
      "Fitness    = 1.7883306296364152\n",
      "[   29 21197    30    18     7]\n",
      "[   29 21197    30    18    12]\n",
      "[   29 40285    30    18     5]\n",
      "Generation = 8\n",
      "[   29 21197    30    18     7]\n",
      "[   29 21197    30    18    12]\n",
      "[   29 40285    30    18     5]\n",
      "Fitness    = 1.4583806662043564\n",
      "[   29 57066    30    18     3]\n",
      "[   29 21197    30    18     9]\n",
      "Generation = 9\n",
      "[   29 57066    30    18     3]\n",
      "[   29 21197    30    18     9]\n",
      "Fitness    = 2.040995787240219\n",
      "[   29 21197    30    18     6]\n",
      "[   29 21197    30    51     9]\n",
      "[   29 21197    30    27     7]\n",
      "Generation = 10\n",
      "[   29 21197    30    18     6]\n",
      "[   29 21197    30    51     9]\n",
      "[   29 21197    30    27     7]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    25    18     7]\n",
      "[   29 21197    38    18     9]\n",
      "[  194 21197    30    18     7]\n",
      "Generation = 11\n",
      "[   29 21197    25    18     7]\n",
      "[   29 21197    38    18     9]\n",
      "[  194 21197    30    18     7]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    30    18    21]\n",
      "[   29 13491    30    18     9]\n",
      "[   29 33392    30    18     7]\n",
      "Generation = 12\n",
      "[   29 21197    30    18    21]\n",
      "[   29 13491    30    18     9]\n",
      "[   29 33392    30    18     7]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    30     8     9]\n",
      "[  204 13491    30    18     9]\n",
      "[   29 53671    30    18     9]\n",
      "Generation = 13\n",
      "[   29 21197    30     8     9]\n",
      "[  204 13491    30    18     9]\n",
      "[   29 53671    30    18     9]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    30     2     9]\n",
      "[   29 13491    30    45     9]\n",
      "[  204 13491    30    18     9]\n",
      "Generation = 14\n",
      "[   29 21197    30     2     9]\n",
      "[   29 13491    30    45     9]\n",
      "[  204 13491    30    18     9]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   63 21197    30    18     9]\n",
      "[   29 13491    56    18     9]\n",
      "[   29 22713    30    18     9]\n",
      "Generation = 15\n",
      "[   63 21197    30    18     9]\n",
      "[   29 13491    56    18     9]\n",
      "[   29 22713    30    18     9]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    79    18     9]\n",
      "[   29 13491    30    18    14]\n",
      "[   29 48973    30    18     9]\n",
      "Generation = 16\n",
      "[   29 21197    79    18     9]\n",
      "[   29 13491    30    18    14]\n",
      "[   29 48973    30    18     9]\n",
      "Fitness    = 3.0390210778117677\n",
      "[   29 21197    30    18    10]\n",
      "[   29 20495    30    18     9]\n",
      "[   29 13491    30    94     9]\n",
      "Generation = 17\n",
      "[   29 21197    30    18    10]\n",
      "[   29 20495    30    18     9]\n",
      "[   29 13491    30    94     9]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 13491    30    73     9]\n",
      "[   29 13491    30    18     8]\n",
      "[   29 13491    71    18     9]\n",
      "Generation = 18\n",
      "[   29 13491    30    73     9]\n",
      "[   29 13491    30    18     8]\n",
      "[   29 13491    71    18     9]\n",
      "Fitness    = 1.5573386901487185\n",
      "[   29 21197    31    18     9]\n",
      "[   29 13491    58    18     9]\n",
      "[   29 21197    80    18     9]\n",
      "Generation = 19\n",
      "[   29 21197    31    18     9]\n",
      "[   29 13491    58    18     9]\n",
      "[   29 21197    80    18     9]\n",
      "Fitness    = 1.8566759080715056\n",
      "[   29 13491    58    61     9]\n",
      "[   54 21197    30    18     9]\n",
      "[   29 21197    30    18    12]\n",
      "Generation = 20\n",
      "[   29 13491    58    61     9]\n",
      "[   54 21197    30    18     9]\n",
      "[   29 21197    30    18    12]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 21197    72    18     9]\n",
      "[   29 13491    58    18    23]\n",
      "[   29 37043    30    18     9]\n",
      "Generation = 21\n",
      "[   29 21197    72    18     9]\n",
      "[   29 13491    58    18    23]\n",
      "[   29 37043    30    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    31    18    23]\n",
      "[   29 13491    58    18    17]\n",
      "[   29 13491    58    53    23]\n",
      "Generation = 22\n",
      "[   29 13491    31    18    23]\n",
      "[   29 13491    58    18    17]\n",
      "[   29 13491    58    53    23]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  147 13491    58    18    23]\n",
      "[   29 13491    46    18    23]\n",
      "Generation = 23\n",
      "[  147 13491    58    18    23]\n",
      "[   29 13491    46    18    23]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 55869    58    18     9]\n",
      "[   29 13491    58    18    10]\n",
      "[  283 13491    58    18     9]\n",
      "Generation = 24\n",
      "[   29 55869    58    18     9]\n",
      "[   29 13491    58    18    10]\n",
      "[  283 13491    58    18     9]\n",
      "Fitness    = 0.5411578046040085\n",
      "[   29 13491    58    51     9]\n",
      "[   29 13491    60    18     9]\n",
      "[   15 13491    58    18     9]\n",
      "Generation = 25\n",
      "[   29 13491    58    51     9]\n",
      "[   29 13491    60    18     9]\n",
      "[   15 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 33076    58    18     9]\n",
      "[   29 13491    22    18     9]\n",
      "[   29 13491     9    18     9]\n",
      "Generation = 26\n",
      "[   29 33076    58    18     9]\n",
      "[   29 13491    22    18     9]\n",
      "[   29 13491     9    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    68     9]\n",
      "[   29 13491    59    18     9]\n",
      "[   29 13491    87    18     9]\n",
      "Generation = 27\n",
      "[   29 13491    58    68     9]\n",
      "[   29 13491    59    18     9]\n",
      "[   29 13491    87    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 36637    58    18     9]\n",
      "[   29 13491    58    18     4]\n",
      "[  29 8955   58   18    9]\n",
      "Generation = 28\n",
      "[   29 36637    58    18     9]\n",
      "[   29 13491    58    18     4]\n",
      "[  29 8955   58   18    9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    18    21]\n",
      "[   29 13491    58    57     9]\n",
      "[   53 13491    58    18     9]\n",
      "Generation = 29\n",
      "[   29 13491    58    18    21]\n",
      "[   29 13491    58    57     9]\n",
      "[   53 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  251 13491    58    18     9]\n",
      "[   29 13491    58    67     9]\n",
      "[   29 13491    32    18     9]\n",
      "Generation = 30\n",
      "[  251 13491    58    18     9]\n",
      "[   29 13491    58    67     9]\n",
      "[   29 13491    32    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    13    18     9]\n",
      "[   29 13491    31    18     9]\n",
      "[  29 7650   58   18    9]\n",
      "Generation = 31\n",
      "[   29 13491    13    18     9]\n",
      "[   29 13491    31    18     9]\n",
      "[  29 7650   58   18    9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 26477    58    18     9]\n",
      "[  288 13491    58    18     9]\n",
      "[   29 13491    10    18     9]\n",
      "Generation = 32\n",
      "[   29 26477    58    18     9]\n",
      "[  288 13491    58    18     9]\n",
      "[   29 13491    10    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 26295    58    18     9]\n",
      "[   29 13491    59    18     9]\n",
      "[   78 13491    58    18     9]\n",
      "Generation = 33\n",
      "[   29 26295    58    18     9]\n",
      "[   29 13491    59    18     9]\n",
      "[   78 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    42     9]\n",
      "[   29 13491    58    90     9]\n",
      "[   29 13491    58    18     5]\n",
      "Generation = 34\n",
      "[   29 13491    58    42     9]\n",
      "[   29 13491    58    90     9]\n",
      "[   29 13491    58    18     5]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  120 13491    58    18     9]\n",
      "[  178 13491    58    18     9]\n",
      "[  29 5216   58   18    9]\n",
      "Generation = 35\n",
      "[  120 13491    58    18     9]\n",
      "[  178 13491    58    18     9]\n",
      "[  29 5216   58   18    9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   80 13491    58    18     9]\n",
      "[   29 14761    58    18     9]\n",
      "[   29 13491    22    18     9]\n",
      "Generation = 36\n",
      "[   80 13491    58    18     9]\n",
      "[   29 14761    58    18     9]\n",
      "[   29 13491    22    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491     5    18     9]\n",
      "[   29 13491    46    18     9]\n",
      "[   29 56624    58    18     9]\n",
      "Generation = 37\n",
      "[   29 13491     5    18     9]\n",
      "[   29 13491    46    18     9]\n",
      "[   29 56624    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  117 13491    58    18     9]\n",
      "[   29 13491    58    18     2]\n",
      "[  217 13491    58    18     9]\n",
      "Generation = 38\n",
      "[  117 13491    58    18     9]\n",
      "[   29 13491    58    18     2]\n",
      "[  217 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 35735    58    18     9]\n",
      "[   29 13491    63    18     9]\n",
      "[   29 13491    58    18     8]\n",
      "Generation = 39\n",
      "[   29 35735    58    18     9]\n",
      "[   29 13491    63    18     9]\n",
      "[   29 13491    58    18     8]\n",
      "Fitness    = 5.873373433245966\n",
      "[  290 13491    58    18     9]\n",
      "[   29 13491    58    18    13]\n",
      "[  29 4072   58   18    9]\n",
      "Generation = 40\n",
      "[  290 13491    58    18     9]\n",
      "[   29 13491    58    18    13]\n",
      "[  29 4072   58   18    9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    23     9]\n",
      "[  199 13491    58    18     9]\n",
      "[  29 1683   58   18    9]\n",
      "Generation = 41\n",
      "[   29 13491    58    23     9]\n",
      "[  199 13491    58    18     9]\n",
      "[  29 1683   58   18    9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58     2     9]\n",
      "[    7 13491    58    18     9]\n",
      "[   29 13491    58    58     9]\n",
      "Generation = 42\n",
      "[   29 13491    58     2     9]\n",
      "[    7 13491    58    18     9]\n",
      "[   29 13491    58    58     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    18     3]\n",
      "[  192 13491    58    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "Generation = 43\n",
      "[   29 13491    58    18     3]\n",
      "[  192 13491    58    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 14700    58    18     9]\n",
      "[   29 13491    58    85     9]\n",
      "[   29 13491    58    18    20]\n",
      "Generation = 44\n",
      "[   29 14700    58    18     9]\n",
      "[   29 13491    58    85     9]\n",
      "[   29 13491    58    18    20]\n",
      "Fitness    = 3.8464859769463153\n",
      "[   29 13491    58    12     9]\n",
      "[   29 13491    22    18     9]\n",
      "[   29 13491    47    18     9]\n",
      "Generation = 45\n",
      "[   29 13491    58    12     9]\n",
      "[   29 13491    22    18     9]\n",
      "[   29 13491    47    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    68    18     9]\n",
      "[   29 13479    58    18     9]\n",
      "[  267 13491    58    18     9]\n",
      "Generation = 46\n",
      "[   29 13491    68    18     9]\n",
      "[   29 13479    58    18     9]\n",
      "[  267 13491    58    18     9]\n",
      "Fitness    = 4.2690837530730255\n",
      "[   29 13491    58    45     9]\n",
      "[   29 13491    58    18    21]\n",
      "[   29 13491    56    18     9]\n",
      "Generation = 47\n",
      "[   29 13491    58    45     9]\n",
      "[   29 13491    58    18    21]\n",
      "[   29 13491    56    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   39 13491    58    18     9]\n",
      "[   29 13491    58    18     2]\n",
      "[   91 13491    58    18     9]\n",
      "Generation = 48\n",
      "[   39 13491    58    18     9]\n",
      "[   29 13491    58    18     2]\n",
      "[   91 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 22395    58    18     9]\n",
      "[   29 13491    58    81     9]\n",
      "[  245 13491    58    18     9]\n",
      "Generation = 49\n",
      "[   29 22395    58    18     9]\n",
      "[   29 13491    58    81     9]\n",
      "[  245 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    63    18     9]\n",
      "[   29 43639    58    18     9]\n",
      "Generation = 50\n",
      "[   29 13491    63    18     9]\n",
      "[   29 43639    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  218 13491    58    18     9]\n",
      "[   29 24625    58    18     9]\n",
      "[  119 13491    58    18     9]\n",
      "Generation = 51\n",
      "[  218 13491    58    18     9]\n",
      "[   29 24625    58    18     9]\n",
      "[  119 13491    58    18     9]\n",
      "Fitness    = 1.7783607584592824\n",
      "[   29 22999    58    18     9]\n",
      "[   29 13491    58    75     9]\n",
      "[   29 13491    58    18    18]\n",
      "Generation = 52\n",
      "[   29 22999    58    18     9]\n",
      "[   29 13491    58    75     9]\n",
      "[   29 13491    58    18    18]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    79     9]\n",
      "[   29 13491    58    18    19]\n",
      "[  179 13491    58    18     9]\n",
      "Generation = 53\n",
      "[   29 13491    58    79     9]\n",
      "[   29 13491    58    18    19]\n",
      "[  179 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  124 13491    58    18     9]\n",
      "[   29 13491     9    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "Generation = 54\n",
      "[  124 13491    58    18     9]\n",
      "[   29 13491     9    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 10619    58    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "[   29 13491    58    93     9]\n",
      "Generation = 55\n",
      "[   29 10619    58    18     9]\n",
      "[   29 13491    58    18    20]\n",
      "[   29 13491    58    93     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  271 13491    58    18     9]\n",
      "[   29 39894    58    18     9]\n",
      "[  271 13491    58    18     9]\n",
      "Generation = 56\n",
      "[  271 13491    58    18     9]\n",
      "[   29 39894    58    18     9]\n",
      "[  271 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 30030    58    18     9]\n",
      "[   29 13491    58    97     9]\n",
      "[  146 13491    58    18     9]\n",
      "Generation = 57\n",
      "[   29 30030    58    18     9]\n",
      "[   29 13491    58    97     9]\n",
      "[  146 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  29 3402   58   18    9]\n",
      "[   10 13491    58    18     9]\n",
      "[  106 13491    58    18     9]\n",
      "Generation = 58\n",
      "[  29 3402   58   18    9]\n",
      "[   10 13491    58    18     9]\n",
      "[  106 13491    58    18     9]\n",
      "Fitness    = 4.409464831042124\n",
      "[  244 13491    58    18     9]\n",
      "[   29 13491    58    65     9]\n",
      "[   29 13491    58    18    11]\n",
      "Generation = 59\n",
      "[  244 13491    58    18     9]\n",
      "[   29 13491    58    65     9]\n",
      "[   29 13491    58    18    11]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  287 13491    58    18     9]\n",
      "[  246 13491    58    18     9]\n",
      "[   29 13491    58    39     9]\n",
      "Generation = 60\n",
      "[  287 13491    58    18     9]\n",
      "[  246 13491    58    18     9]\n",
      "[   29 13491    58    39     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 24775    58    18     9]\n",
      "[  276 13491    58    18     9]\n",
      "[  232 13491    58    18     9]\n",
      "Generation = 61\n",
      "[   29 24775    58    18     9]\n",
      "[  276 13491    58    18     9]\n",
      "[  232 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 23144    58    18     9]\n",
      "[   29 13491    58    18    13]\n",
      "[   29 13491    58    69     9]\n",
      "Generation = 62\n",
      "[   29 23144    58    18     9]\n",
      "[   29 13491    58    18    13]\n",
      "[   29 13491    58    69     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    18    18]\n",
      "[   29 13491    58    18    14]\n",
      "[   29 13491    63    18     9]\n",
      "Generation = 63\n",
      "[   29 13491    58    18    18]\n",
      "[   29 13491    58    18    14]\n",
      "[   29 13491    63    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  126 13491    58    18     9]\n",
      "[   29 13491    58    18    22]\n",
      "[   29 13491    58    18     1]\n",
      "Generation = 64\n",
      "[  126 13491    58    18     9]\n",
      "[   29 13491    58    18    22]\n",
      "[   29 13491    58    18     1]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   62 13491    58    18     9]\n",
      "[   29 13491    58    26     9]\n",
      "[   29 13491    58    15     9]\n",
      "Generation = 65\n",
      "[   62 13491    58    18     9]\n",
      "[   29 13491    58    26     9]\n",
      "[   29 13491    58    15     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    14     9]\n",
      "[   29 13491    58    18    13]\n",
      "[   29 13491    58    42     9]\n",
      "Generation = 66\n",
      "[   29 13491    58    14     9]\n",
      "[   29 13491    58    18    13]\n",
      "[   29 13491    58    42     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[  251 13491    58    18     9]\n",
      "[  226 13491    58    18     9]\n",
      "[   29 13491    58    18    11]\n",
      "Generation = 67\n",
      "[  251 13491    58    18     9]\n",
      "[  226 13491    58    18     9]\n",
      "[   29 13491    58    18    11]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    95    18     9]\n",
      "[   29 42551    58    18     9]\n",
      "[   29 38330    58    18     9]\n",
      "Generation = 68\n",
      "[   29 13491    95    18     9]\n",
      "[   29 42551    58    18     9]\n",
      "[   29 38330    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 49587    58    18     9]\n",
      "[   29 13491    58    76     9]\n",
      "[   29 13491    97    18     9]\n",
      "Generation = 69\n",
      "[   29 49587    58    18     9]\n",
      "[   29 13491    58    76     9]\n",
      "[   29 13491    97    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   45 13491    58    18     9]\n",
      "[   29 13491    58     2     9]\n",
      "[   29 13491    56    18     9]\n",
      "Generation = 70\n",
      "[   45 13491    58    18     9]\n",
      "[   29 13491    58     2     9]\n",
      "[   29 13491    56    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    52    18     9]\n",
      "[   29 13491    58    18     7]\n",
      "[   29 13491    58    18    18]\n",
      "Generation = 71\n",
      "[   29 13491    52    18     9]\n",
      "[   29 13491    58    18     7]\n",
      "[   29 13491    58    18    18]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    21    18     9]\n",
      "[   29 13491    36    18     9]\n",
      "[  206 13491    58    18     9]\n",
      "Generation = 72\n",
      "[   29 13491    21    18     9]\n",
      "[   29 13491    36    18     9]\n",
      "[  206 13491    58    18     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    70     9]\n",
      "[   29 20147    58    18     9]\n",
      "[   29 13491    58    36     9]\n",
      "Generation = 73\n",
      "[   29 13491    58    70     9]\n",
      "[   29 20147    58    18     9]\n",
      "[   29 13491    58    36     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   92 13491    58    18     9]\n",
      "[  29 3489   58   18    9]\n",
      "[   29 13491    58    27     9]\n",
      "Generation = 74\n",
      "[   92 13491    58    18     9]\n",
      "[  29 3489   58   18    9]\n",
      "[   29 13491    58    27     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    18    15]\n",
      "[  134 13491    58    18     9]\n",
      "[   61 13491    58    27     9]\n",
      "Generation = 75\n",
      "[   29 13491    58    18    15]\n",
      "[  134 13491    58    18     9]\n",
      "[   61 13491    58    27     9]\n",
      "Fitness    = 3.8297886967698735\n",
      "[   29 13491    58    31     9]\n",
      "[   29 13491    38    27     9]\n",
      "[   29 13491    27    18     9]\n",
      "Generation = 76\n",
      "[   29 13491    58    31     9]\n",
      "[   29 13491    38    27     9]\n",
      "[   29 13491    27    18     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    54    18     9]\n",
      "[   29 13491    58    48     9]\n",
      "[   29 13491    58    46     9]\n",
      "Generation = 77\n",
      "[   29 13491    54    18     9]\n",
      "[   29 13491    58    48     9]\n",
      "[   29 13491    58    46     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 48892    58    27     9]\n",
      "[   29 13491    14    27     9]\n",
      "[   29 13491    45    18     9]\n",
      "Generation = 78\n",
      "[   29 48892    58    27     9]\n",
      "[   29 13491    14    27     9]\n",
      "[   29 13491    45    18     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    18     2]\n",
      "[   29 13491    58    27    16]\n",
      "[   29 13491    58    27    12]\n",
      "Generation = 79\n",
      "[   29 13491    58    18     2]\n",
      "[   29 13491    58    27    16]\n",
      "[   29 13491    58    27    12]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    10     9]\n",
      "[   29 13491    26    27     9]\n",
      "[   29 58237    58    18     9]\n",
      "Generation = 80\n",
      "[   29 13491    58    10     9]\n",
      "[   29 13491    26    27     9]\n",
      "[   29 58237    58    18     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[  219 13491    58    18     9]\n",
      "[  218 13491    58    27     9]\n",
      "[  213 13491    58    18     9]\n",
      "Generation = 81\n",
      "[  219 13491    58    18     9]\n",
      "Batch size increased -> new shape:  (39, 1524, 28, 28, 1)\n",
      "[  218 13491    58    27     9]\n",
      "[  213 13491    58    18     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    96     9]\n",
      "[   29 49159    58    27     9]\n",
      "[   29 13491    58    20     9]\n",
      "Generation = 82\n",
      "[   29 13491    58    96     9]\n",
      "[   29 49159    58    27     9]\n",
      "[   29 13491    58    20     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    68     9]\n",
      "[   29 13491    58    27    22]\n",
      "[   29 13491    58    20     9]\n",
      "Generation = 83\n",
      "[   29 13491    58    68     9]\n",
      "[   29 13491    58    27    22]\n",
      "[   29 13491    58    20     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 32382    58    18     9]\n",
      "[   29 13491    83    27     9]\n",
      "[  290 13491    58    27     9]\n",
      "Generation = 84\n",
      "[   29 32382    58    18     9]\n",
      "[   29 13491    83    27     9]\n",
      "[  290 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    18     7]\n",
      "[  177 13491    58    27     9]\n",
      "[   29 13491    54    18     9]\n",
      "Generation = 85\n",
      "[   29 13491    58    18     7]\n",
      "[  177 13491    58    27     9]\n",
      "[   29 13491    54    18     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[  280 13491    58    27     9]\n",
      "[   29 15281    58    27     9]\n",
      "Generation = 86\n",
      "[  280 13491    58    27     9]\n",
      "[   29 15281    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   25 13491    58    27     9]\n",
      "[  29 9264   58   27    9]\n",
      "[   29 24089    58    27     9]\n",
      "Generation = 87\n",
      "[   25 13491    58    27     9]\n",
      "[  29 9264   58   27    9]\n",
      "[   29 24089    58    27     9]\n",
      "Fitness    = 2.1947218420464405\n",
      "[   29 55602    58    27     9]\n",
      "[  154 13491    58    27     9]\n",
      "[  257 13491    58    27     9]\n",
      "Generation = 88\n",
      "[   29 55602    58    27     9]\n",
      "[  154 13491    58    27     9]\n",
      "[  257 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    70     9]\n",
      "[   29 13491    58    97     9]\n",
      "[   29 45349    58    27     9]\n",
      "Generation = 89\n",
      "[   29 13491    58    70     9]\n",
      "[   29 13491    58    97     9]\n",
      "[   29 45349    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    46    27     9]\n",
      "[  173 13491    58    27     9]\n",
      "[  281 13491    58    27     9]\n",
      "Generation = 90\n",
      "[   29 13491    46    27     9]\n",
      "[  173 13491    58    27     9]\n",
      "[  281 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[  263 13491    58    27     9]\n",
      "[   29 13491    58    76     9]\n",
      "[   29 28208    58    27     9]\n",
      "Generation = 91\n",
      "[  263 13491    58    27     9]\n",
      "[   29 13491    58    76     9]\n",
      "[   29 28208    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    40     9]\n",
      "[   29 13491     4    27     9]\n",
      "[  198 13491    58    27     9]\n",
      "Generation = 92\n",
      "[   29 13491    58    40     9]\n",
      "[   29 13491     4    27     9]\n",
      "[  198 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   42 13491    58    27     9]\n",
      "[   29 13491    58    20     9]\n",
      "[   29 13624    58    27     9]\n",
      "Generation = 93\n",
      "[   42 13491    58    27     9]\n",
      "[   29 13491    58    20     9]\n",
      "[   29 13624    58    27     9]\n",
      "Fitness    = 5.994331410331815\n",
      "[   29 50446    58    27     9]\n",
      "[   29 22237    58    27     9]\n",
      "[   29 13491    58     5     9]\n",
      "Generation = 94\n",
      "[   29 50446    58    27     9]\n",
      "[   29 22237    58    27     9]\n",
      "[   29 13491    58     5     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[  29 1764   58   27    9]\n",
      "[  29 3147   58   27    9]\n",
      "[   29 13491    58    42     9]\n",
      "Generation = 95\n",
      "[  29 1764   58   27    9]\n",
      "[  29 3147   58   27    9]\n",
      "[   29 13491    58    42     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    10     9]\n",
      "[   29 13491    58    27    22]\n",
      "[   48 13491    58    27     9]\n",
      "Generation = 96\n",
      "[   29 13491    58    10     9]\n",
      "[   29 13491    58    27    22]\n",
      "[   48 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    25     9]\n",
      "[   83 13491    58    27     9]\n",
      "[  222 13491    58    27     9]\n",
      "Generation = 97\n",
      "[   29 13491    58    25     9]\n",
      "[   83 13491    58    27     9]\n",
      "[  222 13491    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    58    31     9]\n",
      "[   29 13491    91    27     9]\n",
      "Generation = 98\n",
      "[   29 13491    58    31     9]\n",
      "[   29 13491    91    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[   29 13491    11    27     9]\n",
      "[   29 59337    58    27     9]\n",
      "Generation = 99\n",
      "[   29 13491    11    27     9]\n",
      "[   29 59337    58    27     9]\n",
      "Fitness    = 5.988367798359984\n",
      "[  266 13491    58    27     9]\n",
      "[   29 13491    58    27     7]\n",
      "[   29 13491    46    27     9]\n",
      "Generation = 100\n",
      "[  266 13491    58    27     9]\n",
      "[   29 13491    58    27     7]\n",
      "[   29 13491    46    27     9]\n",
      "Fitness    = 0.3886546272744886\n"
     ]
    }
   ],
   "source": [
    "param_ga_instance = pygad.GA(num_generations=100,\n",
    "                       sol_per_pop=5,\n",
    "                       num_parents_mating=2,\n",
    "                       num_genes=5,\n",
    "                       fitness_func=param_fitness_func,\n",
    "                       #only wants integer\n",
    "                       gene_type=int,\n",
    "                       #top to bottom epoch, batch_size, mutation_higer, mutation_lower, num_parents_mating\n",
    "                       #to limit range\n",
    "                       gene_space = [{'low': 1, 'high': 300}, \n",
    "                                     {'low': 1, 'high': 60000}, #60000 is num training data of mnist\n",
    "                                     {'low': 2, 'high': 99},\n",
    "                                     {'low': 1, 'high': 98},\n",
    "                                     {'low': 1, 'high': num_solution}],\n",
    "                       on_generation=param_callback_generation\n",
    "                        )\n",
    "\n",
    "param_ga_instance.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KQYVcXPuuFMx"
   },
   "source": [
    "# Stat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 334
    },
    "id": "gKFLa_wLeeVT",
    "outputId": "9fdc01bb-d9d7-4fd5-85ad-b1f96844ab7a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEbCAYAAADZFj8oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjoElEQVR4nO3deZxcVZ338c83CVnYwTRbQMMmiCCLLYsggwoCigzM8IwiOohKxHEccHnU6ChRx3F5cBxRRycCggPigrjAOKhsg4qDEyJCTABRIkuCNAQISxKy/J4/zimovl3VXdVVXev3/XrdV9dd6p5z61b/6tTvnjpXEYGZmfW+Se2ugJmZtYYDvplZn3DANzPrEw74ZmZ9wgHfzKxPOOCbmfUJB3yzLiMpJJ3U7npMJEnzJC1qdz16jQP+OEm6MP/jhaS1kv4o6RxJm9S5n/0kXSppmaQ1ku6R9GNJJ0oacX4k/UjSeklHVVg3r6xO6yStkHSjpLmSNq2hLpMk/bOk+yWtkrRI0utqPI7rJX2psOx0SU9Lenst++gkkmbn13Gw0nyL6nChpCsrrNoeuKJV9Zgohfdr+XQCcA7wF2XbVnstrA5T2l2BLnc18CZgI+BlwHnAJsA7anmypOOA7wHXAKcBvwemAgcDHwb+F7ivbPvtgVcCnwfeBvyswm7vAI4ABGwNHAbMBd4i6WUR8cAoVXoj8H+BU4EbgV2A6bUcS4VjmwucDbwhIi4b5z6mRsTT43luJ2v0uMY4h92m9H4t90hErAGeaH11elxEeBrHBFwIXFlY9jVgOSnY3gW8r7B+dyCAA0gfDEPA5aOUocL8XNIHxPOAVcBzCuvnAYsq7Gd74GHgojGO6Y3An4vl1vh6XA98KR/754DHgSML27wWuBlYDdwNfBKYWrZ+aT6GC4BHge/m5Z8mBYZVeZvPAtPLnrcT8ENgBfAUcDvw+gbP7+x8rgbzfBSm68u2PQ1YnI/rTuDdwKSy9QG8E7gceJLUep0MnJ9fh1WkD/v3l56XX4dimUeU7e+ksv3vQ2p8rMqvwYXAFsX3KnAmcD/wCPB1YOMqxz4JuBd4V2H583PZB+T5t+fjXQ08BPwEmFLHa1zx/VpcV+21KDtHf01q/DyVz8NRhX3tBfwn6T35IHApsF3h9bsGWEn6kPkt8PK8biPgXGAZsCa/Lp+eiJjSiqntFejWicoB/1zgofx4LvC7wvpPAb/Jj0/Mb9aDayxPOSicmOevB84qbDPaP9C5wGOUBaIK22yX3/CfGsfrcT3w1fy6DAEvKaw/Ov9DnQbsCrycFMTPKdtmad7m/cBuwO55+UeAQ/M/+KuBe4BPlD3vivwPvy+wM3AMcEyD57cUTEoB/yV5/uj8Om2dl59O+pA/KZf9WuAB4O/L9hU50LyN9K1p5xxIPp73Oxv4G9KH3FvzczYFvp2Pa7s8TS3b30n58SY5GP2AFLj+ghSEv1d4rz5GapC8AHhVLmvuKMf/WeB/Css+BizOjweBdcAppAbIvqQPuokI+BVfi7JzdHt+3XcHLiI1bjbNz92e9GH0mXzsL8rvl5t49sP1NuBiYE/S++5E4JC87r2kIH848FzgpcBpExVXJnpqewW6daIQ8IED8xvr23l+O2AtOaCTWnT3lwIB8IH8Zt2qbB/7kAJuaTqlbN0R+Y1c+qd/C3BboU6j/QOdkcvbpsr6jfMb/wLgBtIHhMrW3wvMGeX1uJ7UAloHvKjC+huAjxSWnZCPU3l+KXBFDa/9GcBdZfO3Amc3+fyWgslgpfmy7e4B3lRYdhY5MOb5AL5YQ5mfBq6u9h4r7K8U8E8nBfPNCu+VAHYr28+9wOSybb5WXlaFMl6U97Fr2bLfAx/Kj/+qWO44XuN5wPrCe/53ld7LlV6LsnPy9rJls/Kyw/L8x4FrCs/bKm9zYJ5fCZxapY7nklr/dX/r7cTJF20bc4ykJyStBn5FCmrvgmfyrFeSAjOkVufWwCWj7O8OYL88idQKLHkb8J14Nvd7GbCrpINqrKvy36iy/s3ATNL1h+OAQ4BvSpoqaSvSP9INY5TxS1K64JOSphXWvRj4cH69npD0BPBNUgt1u7LtFoyouHSSpF9IeiA/7/Ok1lbJF4B/lPQrSf8k6cXVKijpZeV1kHTKGMdUlaQBUjrp3wvH9WnSt5hylY7rDEkLJA3l5727cFy1eAFwa0Q8XrbsRmADKZVRsjgi1pfNLwO2qbbTiLiV1AA4Jdf1INIxld6/PwP+BNwt6RJJp0rarM66A/yBZ9/z+5G+wdXr1rLHy/Lf0rG9GDi8cH7uzetK5+hfgPMkXSvpw5L2LNvfhbled0r6sqTXVOpM0S26tuId4gbSm2EPUk75ryLiwbL15wGvk7QxKfB/PyIeyevuzH+feXNFxNMRcVdE3EVZYJa0JSlPOSf3vllHytXOIH0Q1GIvUkvm4SrrX0QKCmsiYiUpdbE36UPrvaSv97ePUcZiUqrmQOD7haA/iZQS2K9sehHpa/hQ2XZPlu9Q0sHAt0j54dcC+wP/SNmHYUScT0qTfJ2UZ75R0rwqdVxQqMOPxjim0ZT+f84o7HNv4IWFbYvH9TrgX0kB5ej8vH8jpSqapfzDfW2FdWP9/19MDvj57y8i4k8A+QPmAFIq6h5SCvN2STvUWcdn3vN5+lOdz4eyY4vcLOfZY5tEyt/vV5h2J723iYh5pP+PH5BSNrdKektet5D0TWJu3tdFwM+6Nei7l05jnsrBuZqrSEH2DFKwKm+9/JQUfOcCx49RzimkoFhs/RwCfE7SWRHx5MinJbl3zxtIF4g3VNnsfuAkSZtHxMqIWJG7ft4AHAW8Yow6AhARiyQdAVwL/FDSCRGxGlgI7DnG61XJocD9EfGJsuN5XoVy7wPmA/MlfYB0gXJehe1WkS6o16v0zWpy2b7+LGkZKe3xjTr3dxhwU0Q805VVUvFbwdPl5VWxhNQDa7OyVv5LScFpSZ11Kvom8Kn8ofs60rWUZ0TEOtJ5vlbS2aTrFMeRzkOz1fJaVLKQ9KH0p4gofug9IyJ+T0pZnSvpK6SG1AV53eOkb9SXSboQ+B9Srv/OynvrXF35KdUt8lfoC0gXa+8n5QJL654E3kpKC10l6RhJu0raR9J7SN0hS1/B3wpcFhGLyidSa2MD6Z+xZIqk7SRtL+mFkuaQ0k0rSB8u1ZxHavVdKekwSc8HjgW2ILVO3yZJozy//LiXkC4e7gNcIWkGKZf6Bkkfl7S3pD1zquazY+zuTmCWpFMk7SLpHcDJ5RtI+kJ+/XaRtB8pfba4lrrW4UFSL5ijJW0raYu8/Gzg/ZLeLWmPfGx/m7uljuZO4ABJx0raXdJHKOt3ni0F9s77nSlpoxF7SSmWp4Bv5PfO4cC/kz7cx/PB9oz8IfrfpIvxWwDfLa2TdJykMyXtnz+A3wBsRv6QUfodye2SZjVShzJLGfu1qOTLue7flnRQfo8cKWm+pM0kzcipmiOUfmtxEOnDeHE+jvdIOlnSCyTtlo9zJWXdpbtKuy8idOtElQtqFbZ7HimQfrTK+gNIPRCWk76aPkzKj76J9IF8QH7+S6s8/xvAjfnxPJ7ttraelE//FfAhari4RvrqWqrLU6Sc/ImkHhmrgE+O8tzrgS8Vlu1G+rp/Lemi8KuAn+d9rySlV8p7syyl0JU1L/8U6RvOE6Suje8gf3vP679Iap2tztt9C5jV4PmdTeEiLanVd09+ba8vW34yqSW5Or/mv6CsWyiFbpR52VRSt8xHSD1mzgc+Ciwt22aA9E3wccbulnlNPkePUKVbZqH8eVS5wF/Y7i25vMsLyw8Drsvv11XAIsp6r5CuCQUwe5R9V61DcV2l16LSOary+uxOaqE/kut6R37PTM3TN/N7bw3pGsB8YPP83NPzuX2c9J79b6r8L3bDVOodYRMktxh+CewSEfe0uz5m1r8c8CdIvmA5QErpPBYR/6fNVTKzPucc/sQ5mdRtbSbwnjbXxczMLXwzs37hFr6ZWZ/o6H74M2fOjNmzZ7e7GmZmXePmm29+KCIGKq3r6IA/e/ZsFiwY8Yt0MzOrQlLVXys7pWNm1icc8M3M+oQDvplZn3DANzPrEw74ZmZ9oqUBX9KWki7Lo+gtkXRIK8s3M+tnre6W+QXgqog4SdJU0giKZtZD/vwIzP8vWFbtVjtWs4+eAttv3bz9tSzg5/HDDycNm0qkW/U9PdpzzKz7nP0f8Nu7212L3rCm6i1bxqeVKZ2dSWOVf13SbySdJ2mT4kaS5uT7fC4YGhoauRcz61gRcNvSdtfCqmllwJ9CupnHVyJif9JdlD5Y3Cgi5kfEYEQMDgxU/HWwmXWoNWthg8dj7FitzOHfB9wXETfl+cuoEPDNrHutLiRpN54G55zenrr0gu22au7+WhbwI+IBSfdK2iMi7gBeSfPvO2pmbbSqEPA3nQH7F2/Nbm3T6l467wIuyT10/gic1uLyzWwCFVv402u91bi1REsDfkTcQrohtpn1oGILf/rU9tTDKvMvbc2saYot/BkO+B3FAd/MmmZESmdae+phlTngm1nTFFM6buF3Fgd8M2uaES18B/yO4oBvZk3jXjqdzQHfzJpmRErHOfyO4oBvZk3jlE5nc8A3s6bxRdvO5oBvZk3jFn5nc8A3s6ZxwO9sDvhm1jRO6XQ2B3wzaxq38DubA76ZNY1b+J3NAd/Mmmb1muHzbuF3Fgd8M2ua1YWbbjvgdxYHfDNrGg+P3Nkc8M2saXwDlM7mgG9mTeOLtp3NAd/MmmL9Bni6kMOf5tEyO4oDvpk1xZoKwX6SI0xH8ekws6bwBdvO54BvZk2xyn3wO54Dvpk1hS/Ydj4HfDNrimIOf7rvdtVxprSyMElLgceB9cC6iBhsZflmNnFG9MF3D52O09KAn708Ih5qQ7lmNoGKOXyndDqPUzpm1hQeGrnztTrgB/BTSTdLmlNpA0lzJC2QtGBoaKjF1TOz8Rpx0dY5/I7T6oB/WEQcABwLvFPS4cUNImJ+RAxGxODAwECLq2dm47XGLfyO19KAHxH3578PAt8HDmxl+WY2cXzRtvO1LOBL2kTSZqXHwKuARa0q38wmlvvhd75W9tLZFvi+pFK534yIq1pYvplNoBEXbZ3D7zgtC/gR8Udg31aVZ2at5RZ+53O3TDNrCl+07XwO+GbWFL5o2/kc8M2sKUYMj+wcfsdxwDezpvD9bDufA76ZNYUv2nY+B3wzawpftO18Dvhm1hRO6XQ+B3wzawrf07bzOeCbWVO4hd/5HPDNrGHr1qepZJJgajtur2SjcsA3s4ZVuvlJGjbLOokDvpk1zOmc7uCAb2YN8wXb7uCAb2YNcwu/Ozjgm1nD3MLvDg74ZtawShdtrfM44JtZw0aMo+ORMjuSA76ZNWxEC99j4XckB3wza9iqNcPnndLpTA74Ztaw1WuHz/uibWdywDezho1I6TiH35Ec8M2sYb75SXdwwDezhq0u5PCn+aJtR3LAN7OGuYXfHRzwzaxhIy7aOoffkVoe8CVNlvQbSVe2umwzmxj+pW13aEcL/0xgSRvKNbMJUuyH75ROZ2ppwJe0I/Aa4LxWlmtmE6vYwvdF287U6hb+vwLvBzZU20DSHEkLJC0YGhpqWcXMbPw8lk53aFnAl3Qc8GBE3DzadhExPyIGI2JwYGCgRbUzs0b4l7bdoZUt/EOB4yUtBb4FvELSxS0s38wmSLEfvi/adqaW3Vc+IuYCcwEkHQG8LyLe2Krye83t98JPF468WGbWDiufGj7vFn5nalnAt+ZZvgLO+CKsWTv2tmbt4BZ+Z2pLwI+I64Hr21F2L1h4l4O9da6Np8GUye2uhVXiX9p2oadWt7sGZtW99qB218CqcUqnCxV7RBy0Bxy+T3vqYlZux5kwuHu7a2HVOOB3oeKPXPaZDX91aFuqYmZdxCmdLlRs4U/zBTIzq4EDfhfyQFVmNh4NB3xJHjWjxUYEfJ8BM6tBXQFf0j9I+uuy+fOBVZLukLRH02tnFbmFb2bjUW8L/x+AIQBJhwN/A7wBuAX4XFNrZlU54JvZeNTbS2cWcHd+/FrguxHxHUm3AT9vas2squJFWwd8M6tFvS38lcA2+fFRwDX58VpgerMqZaNzDt/MxqPeFv5Pga9JWgjsBvxXXv5Cnm352wRzC9/MxqPeFv47gV8CA8BJEbEiLz8AuLSZFbPqPBStmY1HXS38iFgJvKvC8rObViMb04gWvlM6ZlaDertl7lXe/VLSUZIuljRXksfHaxH30jGz8ag3pXMBsD+ApJ2AHwJbk1I9/9TcqlklESOHRnbAN7Na1Bvw9wQW5scnATdFxKuBNwEnN7NiVtna9bC+7BbwUyZ77HEzq029AX8yUEoovBL4cX78B2DbZlXKqnOXTDMbr3oD/iLgHZJeRgr4V+Xls4CHmlkxq8z5ezMbr3oD/geA00m3J7w0Im7Ly48Hft3EelkVzt+b2XjV2y3zBkkDwOYR8UjZqn8HnqryNGuiVW7hm9k41T08ckSsByZLOkjStLxsaUQ82PTa2QjO4ZvZeNXbD38zSd8FHgRuJOXukfRVSfOaXz0rKgZ83+3KzGpVbwv/M8AOpKEUVpUtvxI4sVmVsuqKOfwZDvhmVqN6B087HjgxIm6RFGXLlwC7NK9aVo1z+GY2XvW28LcCHq6wfDNg/WhPlDRd0q8l/VbS7yR9rM6yjQopHefwzaxG9Qb8/yW18ktKrfy3k3L6o1kDvCIi9gX2A46RdHCd5fe9NW7hm9k41ZvS+RDwE0kvzM99T358IHD4aE+MiACeyLMb5SmqP8MqKaZ0nMM3s1rV1cKPiBuBlwJTScMpvBJYBhwSEQtHey6ApMmSbiH18vlZRNxUYZs5khZIWjA0NFRP9fqCh0Y2s/Gqt4VP/nXtqeMpLPfh30/SlsD3Je0dEYsK28wH5gMMDg76G0CBu2Wa2XjVHfABJO1AurftsG8ItbTy83aPSroOOIY0Po/VyEMrmNl41RXwJe0PXEwaJlmF1UEaTbPacweAtTnYzyDdBP0z9VXXVhVub+gcvpnVqt4W/nzgXtIAasuo76Lr9sBF+c5Yk4DvRMSVdZbf94o5fHfLNLNa1Rvw9wL2j4g76y0oIm4l3y3Lxs/dMs1svOrth38bsN1EVMRqM6KXjgO+mdWo3oD/IeCzko6UtK2krcuniaigDed++GY2XvWmdK7Of3/K8Py9GOOirTVHMaXjHL6Z1aregP/yCamF1cy3ODSz8ao34N8N3JuHSXiGJAE7Na1WVpV/aWtm41VvDv9uYKDC8q3zOptgxRb+jGntqYeZdZ96A34pV1+0KbC68erYaCI8PLKZjV9NKR1J5+aHAXxKUvkNyyeTRsu8pblVs6K162FD2cftlMlpMjOrRa05/H3yXwEvAMrbmU8DC4Fzmlgvq8A3MDezRtQU8CPi5QCSvg6cGRErJ7RWVtGIgO/8vZnVoa5eOhFx2kRVxMbmFr6ZNWLMgC/pR8AbI2JlflxVRBw/2nprjIdVMLNG1NLCfxh4kaRfUfkG5tYixaGR3cI3s3qMGfAj4jRJ64HtSykdSf8JvC0ilk90Be1ZvvmJmTWi1n74xZudvAyY0eS62Bg8rIKZNaLeH16VFD8ArAWcwzezRtQa8IORv7D1DcZbzL+yNbNG1NotU8DFkkqXDacDXyv84ta9dCbYiHF03MI3szrUGvAvKsxf3OyK2NicwzezRtT6S1v/4KoDeGhkM2vEeC/aWhuMyOG7hW9mdXDA7yLO4ZtZIxzwu4hz+GbWCAf8LlLM4btbppnVo2UBX9JOkq6TtFjS7ySd2aqye4Vb+GbWiHpvYt6IdcB7I2KhpM2AmyX9LCIWt7AOXc05fDNrRMta+BGxPCIW5sePA0uAWa0qvxd4aAUza0RbcviSZgP7Aze1o/xutcZDK5hZA1oe8CVtCnwPOKvSrRIlzZG0QNKCoaGhVlevo61yDt/MGtDSgC9pI1KwvyQiLq+0TUTMj4jBiBgcGBhoZfU6nnP4ZtaIVvbSEXA+sCQi/qVV5fYS3wDFzBrRyhb+ocCbgFdIuiVPr25h+V0tYmRKxzl8M6tHy7plRsQv8I1Txu3pdSnol0yZnCYzs1r5l7Zdwvl7M2uUA36XKObvPVKmmdXLAb9LjOiS6fy9mdXJAb9LeBwdM2uUA36XcMA3s0Y54HeJEX3wndIxszo54HcJD6tgZo1q5fDIPWftOrjsF3DXsokva9mK4fMO+GZWLwf8cVq3Hj54Ady4pD3lO+CbWb2c0hmHDRvgny5tX7AH2GLj9pVtZt3JAb9OEfCFH8BPbm5fHWZMhVe9uH3lm1l3ckqnTlf+Gr7z8+HLZs2ENx8JasFIQVMmw767wHZbTXxZZtZbHPDr9N0bhs/P3BzOPQN2eE576mNmViundOoQAfcUbsJ1zukO9mbWHRzw6/D4quE/gJq2ETzft2E3sy7hgF+Hhx4bPj+wRWvy9mZmzeCAX4ehCgHfzKxbOODXwQHfzLqZA34dHPDNrJs54NfBAd/MupkDfh0c8M2smzng12FEwN+yLdUwMxsXB/w6FAP+zM3bUw8zs/FwwK/RuvXwyBPDlzngm1k3ccCv0UMr09AKJVttCht5JCIz6yItC/iSLpD0oKRFrSqzmXzB1sy6XStb+BcCx7SwvKYaenT4vAO+mXWblgX8iLgBWDHmhh3KLXwz63Ydl8OXNEfSAkkLhoaGxn5Ci7hLppl1u44L+BExPyIGI2JwYGCg3dV5hlv4ZtbtOi7gdyoHfDPrdg74NXLAN7Nu18pumZcCvwL2kHSfpLe2quxGRTjgm1n3a9lPhyLi5FaV1WyVbm242Yz21cfMbDyc0qmBb21oZr3AAb8GTueYWS9wwK+BA76Z9QIH/Bo44JtZL3DAr4EDvpn1Agf8Gjjgm1kvcMCvgcfRMbNe4IA/huUr4A/Lhy/bxi18M+tCDvhjuOjqdHvDkp0GYJst21YdM7Nxc8AfxfIVcOVNw5edeqR/dGVm3ckBfxQXXQ3rNzw7P2smHP3i9tXHzKwRfXUb7uUr0lSybj2sXQdPr4MNMXzbNWtHtu5POwqmTJ74epqZTYS+CfhfvgIuvnb8z3fr3sy6XV+kdJavaCzYg1v3Ztb9+iLg/2pJY8/fdXu37s2s+/VFSuemO4bPP3cb2HozmDwJpk6Bjaakx5XssDX8zeFu3ZtZ9+v5gL9uPSy4c/iyj78J9tixPfUxM2uXnk/p3LYUnlrz7PxWm8LuO7StOmZmbdPzAf+m24fPH7QnTOr5ozYzG6nnQ18x4B+8Z3vqYWbWbj0d8B95Au64f/iylzy/PXUxM2u3ng74v74DouwXtHvsmHrnmJn1o57rpbNhAzy+ClY+Bdf9dvg6p3PMrJ/1VMD/+SL4wAXDW/XlDnLAN7M+1tKUjqRjJN0h6S5JH2z2/jeZXj3YbzwN9pnd7BLNzLpHywK+pMnAl4Fjgb2AkyXt1cwyttik+rpjX+Jfy5pZf2tlSudA4K6I+COApG8BfwksblYBW2yc/m4yPT3eYhPYfGPYb9c0PIKZWT9rZcCfBdxbNn8fcFBxI0lzgDkAz33uc+sq4Dmbw8/PcUvezKySjuuWGRHzI2IwIgYHBgbqeq7kYG9mVk0rA/79wE5l8zvmZWZm1gKtDPj/C+wuaWdJU4HXAz9qYflmZn2tZTn8iFgn6e+BnwCTgQsi4netKt/MrN+19IdXEfFj4MetLNPMzJKOu2hrZmYTwwHfzKxPKKqNRdABJA0Bfxrn02cCDzWxOt3Ax9z7+u14wcdcr+dFRMU+7R0d8BshaUFEDLa7Hq3kY+59/Xa84GNuJqd0zMz6hAO+mVmf6OWAP7/dFWgDH3Pv67fjBR9z0/RsDt/MzIbr5Ra+mZmVccA3M+sTPRfwJ/o2ip1A0k6SrpO0WNLvJJ2Zl28t6WeSfp//btXuujabpMmSfiPpyjy/s6Sb8vn+dh6Yr2dI2lLSZZJul7RE0iG9fp4lvTu/rxdJulTS9F47z5IukPSgpEVlyyqeVyXn5mO/VdIB4y23pwJ+K26j2CHWAe+NiL2Ag4F35uP8IHBNROwOXJPne82ZwJKy+c8An4+I3YBHgLe2pVYT5wvAVRGxJ7Av6dh79jxLmgX8AzAYEXuTBlp8Pb13ni8Ejiksq3ZejwV2z9Mc4CvjLbSnAj5lt1GMiKeB0m0Ue0pELI+Ihfnx46QgMIt0rBflzS4CTmhLBSeIpB2B1wDn5XkBrwAuy5v01DFL2gI4HDgfICKejohH6fHzTBrUcYakKcDGwHJ67DxHxA3AisLiauf1L4FvRPI/wJaSth9Pub0W8CvdRnFWm+rSEpJmA/sDNwHbRsTyvOoBYNt21WuC/CvwfmBDnn8O8GhErMvzvXa+dwaGgK/nNNZ5kjahh89zRNwPnAPcQwr0jwE309vnuaTaeW1aXOu1gN9XJG0KfA84KyJWlq+L1N+2Z/rcSjoOeDAibm53XVpoCnAA8JWI2B94kkL6pgfP81akFu3OwA7AJoxMffS8iTqvvRbw++Y2ipI2IgX7SyLi8rz4z6Wvevnvg+2q3wQ4FDhe0lJSqu4VpPz2lvmrP/Te+b4PuC8ibsrzl5E+AHr5PB8J3B0RQxGxFricdO57+TyXVDuvTYtrvRbw++I2ijl3fT6wJCL+pWzVj4BT8+NTgR+2um4TJSLmRsSOETGbdF6vjYhTgOuAk/JmvXbMDwD3StojL3olsJgePs+kVM7BkjbO7/PSMffseS5T7bz+CPjb3FvnYOCxstRPfSKipybg1cCdwB+AD7e7PhN0jIeRvu7dCtySp1eTctrXAL8Hrga2bnddJ+j4jwCuzI93AX4N3AV8F5jW7vo1+Vj3Axbkc/0DYKteP8/Ax4DbgUXAfwDTeu08A5eSrlGsJX2Te2u18wqI1PvwD8BtpB5M4yrXQyuYmfWJXkvpmJlZFQ74ZmZ9wgHfzKxPOOCbmfUJB3wzsz7hgG/WZpKWSnpfu+thvc8B37qCpG0lfT4PHbs6Dy17o6R35SEmOp6keeXD4ZZ5CfBvra6P9Z8pY29i1l55gLhfAiuBj5B+hLQKeCHwNuBh4JttrN/USKOzjktEDDWzPmbVuIVv3eArpBEyByPiWxGxOCLujogrI+IE0q8WkbSFpPm59f+4pP+WNFjaiaQ3S3pC0ivzzTWezDeS2bm8MEmvlXRz/iZxt6RPlt9wI6dg5uWbWDwKXJKXf1rp5jur8jaflTS9VDZwNvBCSZGnN5ft731l+3+upO/nY3hc0uV5aOjS+nm5/q+X9Ie8zQ8kzWzuy269xgHfOpqk5wBHA1+OiCcrbRMRkcdd+U/SsLHHkYaMvgG4tjB2+DRgLvAW4BBgS+CrZeUdTQrgXyJ9g3gLaQyXfy4U+x7Sz/8HgQ/lZU/m7V8A/B1pzJ8P53XfBj4H3AFsn6dvVzjeSaQxVLYFXp6nHYAf5GMsmQ28DjgReFU+3k9Wen3MntHuMSU8eRptAg4ijRt0YmH5fcATefoqafTMJ4AZhe1uAd6fH78572uPsvWnAGvgmWFGbgA+UtjHCXnfpW2WAlfUUPczSDfkKc3PAxZV2G4p8L78+ChgPTC7bP0upG84R5btZzWwRdk2Hy4vy5OnSpNz+NatXka6/d18YDrwYtLdkYaGN4SZDuxaNr8mIu4om18GTCUNSrYi7+dASR8o22YSMAPYjjTgFaQBzYaRdBJwFrAbsGmu3+Q6j+sFwLKIWFpaEBF/lLSMdNvOq/PiP0XEY4Xj2KbOsqzPOOBbp7uL1Crfs3xhRNwNIOmpvGgS8GfSB0FR+c1h1hXWlUYPnFT292OkERmLyi+uDksv5WFrv5Wf+27gUeB40t2bmqV8pMO1FdY5RWujcsC3jhYRD0v6KfD3kr4YEU9U2XQhKe+9ISL+2ECRC4E9I+KuOp93KHB/RHyitEDS8wrbPM3YLf4lwA6SZpda+ZJ2IeXxF9dZJ7Nh3CKwbvB3pPfqzZJOlrSXpOdLOhnYl5TzvprUdfOHko7NN8E5RNLHJFVq9VfzceANkj4uaW9Je0o6SdJnx3jencAsSadI2kXSO4CTC9ssBZ4n6QBJMyVNq7Cfq0ndTi+RNJh7GV1C+iC6to7jMBvBAd86Xm6x7w9cBXwC+A0pAL6H9IOlsyIiSDeBuRb4Gqk3zHeAPUj57VrL+gnwGlLvmF/n6YOkOzGN9rwrgP9HutH6raSLrx8tbPY94Mekm1wMMfIDgXwcf5nXX5enB4AT8jqzcfMNUMzM+oRb+GZmfcIB38ysTzjgm5n1CQd8M7M+4YBvZtYnHPDNzPqEA76ZWZ9wwDcz6xP/Hws68YHv1Q93AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitness value of the best solution = 32.28813571350302\n",
      "Index of the best solution : 8\n"
     ]
    }
   ],
   "source": [
    "# After the generations complete, some plots are showed that summarize how the outputs/fitness values evolve over generations.\n",
    "param_ga_instance.plot_fitness(title=\"PyGAD & Keras - Iteration vs. Fitness\", linewidth=4)\n",
    "\n",
    "# Returning the details of the best solution.\n",
    "solution, solution_fitness, solution_idx = ga_instance.best_solution()\n",
    "print(\"Fitness value of the best solution = {solution_fitness}\".format(solution_fitness=solution_fitness))\n",
    "print(\"Index of the best solution : {solution_idx}\".format(solution_idx=solution_idx))\n",
    "\n",
    "# Fetch the parameters of the best solution.\n",
    "best_solution_weights = pygad.kerasga.model_weights_as_matrix(model=model,\n",
    "                                                              weights_vector=solution)\n",
    "model.set_weights(best_solution_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z5jb0MTrkL60",
    "outputId": "a061f762-b0c5-4d6a-d747-539066511d64"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on testing set:72.45833277702332%\n",
      "Error rate on testing set:27.541667222976685%\n",
      "Accuracy on training set:82.27083086967468%\n",
      "Error rate on training set:17.729169130325317%\n"
     ]
    }
   ],
   "source": [
    "#update acc matrix for testing data\n",
    "acc_meter.update_state(tf.argmax(model(X_test, training=False), axis=1), tf.argmax(y_test, axis=1))\n",
    "test_acc = acc_meter.result().numpy()*100\n",
    "print(\"Accuracy on testing set:{acc}%\".format(acc=test_acc))\n",
    "print(\"Error rate on testing set:{err}%\".format(err=(100-test_acc)))\n",
    "#update acc matrix for training data\n",
    "acc_meter.update_state(tf.argmax(model(X_train, training=False), axis=1), tf.argmax(y_train, axis=1))\n",
    "train_acc = acc_meter.result().numpy()*100\n",
    "print(\"Accuracy on training set:{acc}%\".format(acc=train_acc))\n",
    "print(\"Error rate on training set:{err}%\".format(err=(100-train_acc)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "resnet_kerasGA.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}