{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "VG9PY05cFJ_R",
      "metadata": {
        "id": "VG9PY05cFJ_R"
      },
      "source": [
        "# DSA 2024 S1 Assignment\n",
        "\n",
        "Member number: XXXXX\n",
        "\n",
        "## Purpose:\n",
        "\n",
        "This notebook serves as a record of the data science procedures employed to analyze and understand the characteristics of famous movie quotes. The work was carried out for a movie production company and was done primarily to assist producers with improving both their scripts and the overall production process.\n",
        "\n",
        "Specifically, the project has two key objectives:\n",
        "\n",
        "- **Explore the characteristics of famous movie quotes which the company can use to guide script writing**: The company desires this so that they can boost future box office sales, earn Academy Awards and to improve the memorability of their movies. We employ Natural Language Processing (NLP) techniques and clustering algorithms to assist with this objective (Q2)\n",
        "- **Identify potentially problematic quotes**: many actors have contractual stipulations with regards to the content they are willing to portray (e.g. no swearing and no taboo topics). By flagging potentially problematic quotes, we can better inform our actors and make timely adjustments to our scripts so as to avoid casting complications, reshoots and delays. This culminated in the development of a neural network model that categorises quotes as 'problematic' or 'not problematic'\n",
        "\n",
        "Finally, this notebook is accompanied by an executive summary of the findings and a short-video to explain the model to script writers and improve model adoption."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XtZXmepZWbBz",
      "metadata": {
        "id": "XtZXmepZWbBz"
      },
      "source": [
        "## Packages\n",
        "This section imports the packages that are used in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 324,
      "id": "KQVVeMMzHYK2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KQVVeMMzHYK2",
        "outputId": "0d971830-e7db-4bd2-acb6-1959c1fd2b0b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: fuzzywuzzy in /usr/local/lib/python3.10/dist-packages (0.18.0)\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: symspellpy in /usr/local/lib/python3.10/dist-packages (6.7.7)\n",
            "Requirement already satisfied: editdistpy>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from symspellpy) (0.1.3)\n",
            "Requirement already satisfied: contractions in /usr/local/lib/python3.10/dist-packages (0.1.73)\n",
            "Requirement already satisfied: textsearch>=0.0.21 in /usr/local/lib/python3.10/dist-packages (from contractions) (0.0.24)\n",
            "Requirement already satisfied: anyascii in /usr/local/lib/python3.10/dist-packages (from textsearch>=0.0.21->contractions) (0.3.2)\n",
            "Requirement already satisfied: pyahocorasick in /usr/local/lib/python3.10/dist-packages (from textsearch>=0.0.21->contractions) (2.1.0)\n",
            "Requirement already satisfied: sentence_transformers in /usr/local/lib/python3.10/dist-packages (2.6.1)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.32.0 in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (4.38.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (4.66.2)\n",
            "Requirement already satisfied: torch>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (2.2.1+cu121)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (1.25.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (1.4.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (1.11.4)\n",
            "Requirement already satisfied: huggingface-hub>=0.15.1 in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (0.20.3)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from sentence_transformers) (9.4.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (3.13.4)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (2023.6.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (2.31.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (6.0.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (4.11.0)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.15.1->sentence_transformers) (24.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (3.1.3)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.11.0->sentence_transformers) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.11.0->sentence_transformers) (12.4.127)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence_transformers) (2023.12.25)\n",
            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence_transformers) (0.15.2)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers<5.0.0,>=4.32.0->sentence_transformers) (0.4.2)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence_transformers) (1.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sentence_transformers) (3.4.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.11.0->sentence_transformers) (2.1.5)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence_transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence_transformers) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence_transformers) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.15.1->sentence_transformers) (2024.2.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.11.0->sentence_transformers) (1.3.0)\n",
            "Requirement already satisfied: better_profanity in /usr/local/lib/python3.10/dist-packages (0.7.0)\n",
            "Requirement already satisfied: python-datamuse in /usr/local/lib/python3.10/dist-packages (2.0.1)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.28.1 in /usr/local/lib/python3.10/dist-packages (from python-datamuse) (2.31.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.28.1->python-datamuse) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.28.1->python-datamuse) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.28.1->python-datamuse) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.28.1->python-datamuse) (2024.2.2)\n",
            "Requirement already satisfied: scikeras in /usr/local/lib/python3.10/dist-packages (0.13.0)\n",
            "Requirement already satisfied: keras>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from scikeras) (3.2.1)\n",
            "Requirement already satisfied: scikit-learn>=1.4.2 in /usr/local/lib/python3.10/dist-packages (from scikeras) (1.4.2)\n",
            "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (1.4.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (1.25.2)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (13.7.1)\n",
            "Requirement already satisfied: namex in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (0.0.8)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (3.9.0)\n",
            "Requirement already satisfied: optree in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (0.11.0)\n",
            "Requirement already satisfied: ml-dtypes in /usr/local/lib/python3.10/dist-packages (from keras>=3.2.0->scikeras) (0.2.0)\n",
            "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.4.2->scikeras) (1.11.4)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.4.2->scikeras) (1.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=1.4.2->scikeras) (3.4.0)\n",
            "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from optree->keras>=3.2.0->scikeras) (4.11.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.2.0->scikeras) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.2.0->scikeras) (2.16.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->keras>=3.2.0->scikeras) (0.1.2)\n",
            "Requirement already satisfied: shap in /usr/local/lib/python3.10/dist-packages (0.45.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from shap) (1.25.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from shap) (1.11.4)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from shap) (1.4.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from shap) (2.0.3)\n",
            "Requirement already satisfied: tqdm>=4.27.0 in /usr/local/lib/python3.10/dist-packages (from shap) (4.66.2)\n",
            "Requirement already satisfied: packaging>20.9 in /usr/local/lib/python3.10/dist-packages (from shap) (24.0)\n",
            "Requirement already satisfied: slicer==0.0.7 in /usr/local/lib/python3.10/dist-packages (from shap) (0.0.7)\n",
            "Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from shap) (0.58.1)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from shap) (2.2.1)\n",
            "Requirement already satisfied: llvmlite<0.42,>=0.41.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->shap) (0.41.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->shap) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->shap) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->shap) (2024.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->shap) (1.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->shap) (3.4.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->shap) (1.16.0)\n",
            "\n",
            "GPU device not found\n",
            "Found CPU at /device:CPU:0\n"
          ]
        }
      ],
      "source": [
        "## General packages\n",
        "from google.colab import files # importing data\n",
        "from google.colab import drive\n",
        "import time # measure duration of run time\n",
        "#import re\n",
        "#import string\n",
        "import itertools # for confusion matrix function\n",
        "from collections import Counter # for calculation of token frequencies\n",
        "#import warnings\n",
        "\n",
        "## Packages for data frames and mathematical operations.\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "## Package for fuzzy matching\n",
        "!pip install fuzzywuzzy\n",
        "from fuzzywuzzy import fuzz\n",
        "from fuzzywuzzy import process\n",
        "\n",
        "## Regex\n",
        "import re\n",
        "\n",
        "## Package for plotting and visualisation\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from wordcloud import WordCloud # for visually examining the words\n",
        "from prettytable import PrettyTable # for visually examining the frequent words\n",
        "from mpl_toolkits.mplot3d import Axes3D\n",
        "\n",
        "## Package for generating random numbers\n",
        "import random\n",
        "from random import sample # for generating random numbers to sample randomly from the dataset\n",
        "\n",
        "import warnings\n",
        "\n",
        "## Used to split the data into training and test sets\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "## Scikit-learn libraries to perform clustering and model evaluation\n",
        "from sklearn.cluster import KMeans\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.pipeline import make_pipeline\n",
        "from sklearn import metrics\n",
        "from sklearn.metrics import confusion_matrix, f1_score, classification_report\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "from sklearn.metrics import silhouette_score\n",
        "from sklearn.metrics import davies_bouldin_score\n",
        "from sklearn.metrics import calinski_harabasz_score\n",
        "from sklearn.metrics.cluster import adjusted_rand_score #rand score for external clustering validation\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "\n",
        "## The Natural Language Toolkit (nltk) is a useful package for NLP\n",
        "import nltk\n",
        "nltk.download('stopwords') # For a standard list of stopwords.\n",
        "nltk.download('wordnet') # For lemmatisation.\n",
        "nltk.download('punkt')\n",
        "# nltk.download('averaged_perceptron_tagger')\n",
        "# nltk.download('omw-1.4')\n",
        "\n",
        "# Package for fixing spelling mistakes in tokenised data\n",
        "!pip install symspellpy\n",
        "import symspellpy\n",
        "\n",
        "#from nltk import pos_tag, word_tokenize\n",
        "from nltk.corpus import wordnet\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "\n",
        "# For expanding contractions\n",
        "!pip install contractions\n",
        "import contractions #expand contractions\n",
        "\n",
        "# For dimensionality reduction using PCA and t-SNE\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.manifold import TSNE\n",
        "\n",
        "# For vectorisation using BERT.\n",
        "!pip install sentence_transformers\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "# For identifying swearing:\n",
        "!pip install better_profanity\n",
        "from better_profanity import profanity\n",
        "\n",
        "# For connecting to the datamuse API\n",
        "!pip install python-datamuse\n",
        "from datamuse import Datamuse\n",
        "\n",
        "# for clustering:\n",
        "from scipy.cluster.hierarchy import dendrogram, linkage\n",
        "from sklearn.cluster import AgglomerativeClustering\n",
        "from sklearn.metrics import silhouette_score, calinski_harabasz_score, davies_bouldin_score\n",
        "\n",
        "## Neural network packages and functions\n",
        "  # primarily use tensorflow\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Activation, Dropout\n",
        "%pip install scikeras\n",
        "from scikeras.wrappers import KerasClassifier\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "import tensorflow.keras.backend as K\n",
        "from keras.utils import get_custom_objects\n",
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "from sklearn.dummy import DummyClassifier\n",
        "from keras.regularizers import l1, l2, l1_l2 # add regularisation\n",
        "from tensorflow.keras.metrics import Precision, Recall\n",
        "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report, log_loss, roc_auc_score\n",
        "!pip install shap\n",
        "import shap # visualisation of variable importance\n",
        "\n",
        "# For benchmark model:\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "\n",
        "\n",
        "## We force keras to behave deterministically to improve reproducibility of the results\n",
        "# get device name for doing this later\n",
        "import os # operating system\n",
        "device_name = tf.test.gpu_device_name()\n",
        "print(device_name)\n",
        "if device_name != '/device:GPU:0':\n",
        "    print('GPU device not found')\n",
        "    device_name = \"/device:CPU:0\"\n",
        "    print('Found CPU at {}'.format(device_name))\n",
        "else:\n",
        "    print('Found GPU at {}'.format(device_name))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Z-m4PgeFfTrx",
      "metadata": {
        "id": "Z-m4PgeFfTrx"
      },
      "source": [
        "## Parameters\n",
        "Set global parameters and configurations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 325,
      "id": "DoWN6XFgfc7m",
      "metadata": {
        "id": "DoWN6XFgfc7m"
      },
      "outputs": [],
      "source": [
        "# Set random seed\n",
        "random_seed = 123"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 326,
      "id": "tEXY1Zrcffu7",
      "metadata": {
        "id": "tEXY1Zrcffu7"
      },
      "outputs": [],
      "source": [
        "# Notebook settings\n",
        "pd.set_option('display.max_colwidth', None) # Present full contents of a cell so that features in the datasets are not truncated when printed\n",
        "pd.set_option('display.max_row', None) # Present full rows of a dataset\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xoME0nrg1r9X",
      "metadata": {
        "id": "xoME0nrg1r9X"
      },
      "source": [
        "## Functions\n",
        "This section sets out user defined functions that are used in this notebook.\n",
        "\n",
        "*Note: most functions are defined in ther relevant section*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 327,
      "id": "bdb58870-1f66-4302-b7d5-1576d58e3dbf",
      "metadata": {
        "id": "bdb58870-1f66-4302-b7d5-1576d58e3dbf"
      },
      "outputs": [],
      "source": [
        "def is_duplicate(text1, text2, threshold=80):\n",
        "  \"\"\"\n",
        "  This function checks if two text strings are similar using fuzzy matching.\n",
        "\n",
        "  Args:\n",
        "      text1: The first text string.\n",
        "      text2: The second text string.\n",
        "      threshold: The minimum fuzzy matching score (0-100) considered a duplicate (default: 80).\n",
        "\n",
        "  Returns:\n",
        "      True if the texts are considered duplicates, False otherwise.\n",
        "  \"\"\"\n",
        "  # Apply fuzzy matching algorithm and apply threshold\n",
        "  score = fuzz.ratio(text1.lower(), text2.lower())\n",
        "  return score >= threshold\n",
        "\n",
        "\n",
        "def duplicate_indices(text_data, threshold = 80):\n",
        "  \"\"\"\n",
        "  This function returns the indices in a dataset of duplicates using fuzzy matching.\n",
        "\n",
        "  Args:\n",
        "      text_data: column of text data\n",
        "      threshold: The minimum fuzzy matching score (0-100) considered a duplicate (default: 80).\n",
        "\n",
        "  Returns:\n",
        "      A list of indices\n",
        "  \"\"\"\n",
        "  # Create an empty list to store duplicate indices\n",
        "  duplicates = []\n",
        "  unique_j_values = set()\n",
        "\n",
        "  # Iterate through each text string\n",
        "  for i in range(len(text_data)):\n",
        "    for j in range(i + 1, len(text_data)):\n",
        "      # Check if current and next text are duplicates using the function\n",
        "      if is_duplicate(text_data.iloc[i], text_data.iloc[j], threshold = threshold) and j not in unique_j_values:\n",
        "        duplicates.append((i, j))\n",
        "        unique_j_values.add(j)\n",
        "\n",
        "  return duplicates\n",
        "\n",
        "\n",
        "def extract_first_year(string):\n",
        "  \"\"\"\n",
        "  This function returns the numbers before a hyphen in a string.\n",
        "\n",
        "  Args:\n",
        "      string: as string\n",
        "\n",
        "  Returns:\n",
        "      An integer representing the digits before the hyphen\n",
        "  \"\"\"\n",
        "  if isinstance(string, str):\n",
        "      match = re.match(r'^(\\d+)-?', string)\n",
        "      if match:\n",
        "          return int(match.group(1))  # Extract the number before the '-' and convert to integer\n",
        "      else:\n",
        "          return None  # Return None if the pattern doesn't match\n",
        "  else:\n",
        "      return None  # Return None for non-string inputs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1LOeKQASWwat",
      "metadata": {
        "id": "1LOeKQASWwat"
      },
      "source": [
        "## Import data\n",
        "This section imports the data that is used in this notebook.\n",
        "\n",
        "The data was provided by the Actuaries Institute:\n",
        "\n",
        "*(2023). DSA 2024 S1 Assignment Data [DSA 2024 S1 Assignment Data]. Actuaries Institute.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 328,
      "id": "PIuA9lJII86e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PIuA9lJII86e",
        "outputId": "5fa930f7-1535-4305-a7ca-c310ce55832f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive/\n"
          ]
        }
      ],
      "source": [
        "# Mount google drive to import the required data\n",
        "drive.mount('/content/gdrive/',force_remount=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 329,
      "id": "pJCnCTgqgpuD",
      "metadata": {
        "id": "pJCnCTgqgpuD"
      },
      "outputs": [],
      "source": [
        "# Read in dataset\n",
        "infolder = '/content/gdrive/My Drive/DSA datasets/'\n",
        "file_name = 'DSA 2024 S1 Assignment data.xlsx'\n",
        "\n",
        "data = pd.read_excel(infolder+file_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5jbUFlfRUCmH",
      "metadata": {
        "id": "5jbUFlfRUCmH"
      },
      "source": [
        "## Q1 - Prepare the data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7640511f-7f89-41bf-99b8-fd27f1ff4048",
      "metadata": {
        "id": "7640511f-7f89-41bf-99b8-fd27f1ff4048"
      },
      "source": [
        "### Q1a - Examine the dataset\n",
        "#### Purpose:\n",
        "The purpose of this section it to gain an understanding of the dataset and its features. This will involve summarising the key chararacteristics of the dataset, assessing data and concept drift, and addressing some initial cleaning needs. Additionally, the dataset will be split into training, validation and test sets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ze6Gv4kr2yom",
      "metadata": {
        "id": "ze6Gv4kr2yom"
      },
      "source": [
        "#### Q1a Step 1 - Examine entire dataset\n",
        "**Outline**\n",
        "\n",
        "Firstly, we examine the dataset as a whole to understand the following:\n",
        "- What variables we have and their data type\n",
        "- Summary statistics\n",
        "- Missing values\n",
        "\n",
        "This is to get an initial idea of the data and to highlight any potential issues that may exist."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 330,
      "id": "ff1278c5-cb17-4542-ab1c-631366898f27",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ff1278c5-cb17-4542-ab1c-631366898f27",
        "outputId": "c17cf83c-a6e9-48a5-c194-350acd680a46"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 922 entries, 0 to 921\n",
            "Data columns (total 12 columns):\n",
            " #   Column              Non-Null Count  Dtype  \n",
            "---  ------              --------------  -----  \n",
            " 0   Movie Quote         922 non-null    object \n",
            " 1   Movie Name          922 non-null    object \n",
            " 2   Actor               917 non-null    object \n",
            " 3   Character           917 non-null    object \n",
            " 4   Year                921 non-null    object \n",
            " 5   Genre               921 non-null    object \n",
            " 6   Director            919 non-null    object \n",
            " 7   Writer              919 non-null    object \n",
            " 8   Production Company  919 non-null    object \n",
            " 9   Sentiment_VADER     922 non-null    float64\n",
            " 10  Sentiment_TextBlob  922 non-null    float64\n",
            " 11  label_ChatGPT       921 non-null    object \n",
            "dtypes: float64(2), object(10)\n",
            "memory usage: 86.6+ KB\n"
          ]
        }
      ],
      "source": [
        "# Get initial summary of the dataset\n",
        "  # number of entries\n",
        "  # column names\n",
        "  # data types\n",
        "  # number of missing values\n",
        "data.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-5_Fo3mhp816",
      "metadata": {
        "id": "-5_Fo3mhp816"
      },
      "source": [
        "The above highlights the following about the entire dataset:\n",
        "- We have 922 rows of data. This is a relatively low amount of data given we aim to conduct Natural Language Processing (NLP) and create a neural network (NN). This is a potential concern for the movie production company's objectives:\n",
        "  - **Limited insights**: A small dataset may limit our ability to extract insights about quotes, making script improvement more challenging. The patterns captured may not generalise well or may not extend beyond simple insights.\n",
        "  - **Limited model performance**: Training our NN on limited data may result in low accuracy and generalisability. This may result in unreliable identification of future problematic quotes, which may lead to casting issues, delays and hence additional costs for the movie production company.\n",
        "  \n",
        "  We could acquire more data to remedy this however we choose to use what has been provided.\n",
        "- We have 12 columns in the dataset. `Sentiment_VADER` and `Sentiment_TextBlob` are numeric whereas all other columns have been interpreted as strings. We would have expected `Year` to be numeric so this will need cleaning.\n",
        "- The following columns have missing values:\n",
        "  - 5 missing values: `Actor`, `Character`\n",
        "  - 1 missing value: `Year`, `Genre`, `label_ChatGPT`\n",
        "  - 3 missing values: `Director`, `Writer`, `Production Company`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 331,
      "id": "V_g7LlCUwJ9n",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 397
        },
        "id": "V_g7LlCUwJ9n",
        "outputId": "dfffd373-993a-44b2-fa6c-efff771832f7"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"data\",\n  \"rows\": 922,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 885,\n        \"samples\": [\n          \"A martini...shaken, not stirred.\",\n          \"Fear. Fear attracts the fearful\\u00e2\\u20ac\\u00a6 the strong\\u00e2\\u20ac\\u00a6 the weak\\u00e2\\u20ac\\u00a6 the innocent\\u00e2\\u20ac\\u00a6 the corrupt. Fear. Fear is my ally.\",\n          \"That\\u00e2\\u20ac\\u2122s what you always do, you confuse love for admiration.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Movie Name\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 584,\n        \"samples\": [\n          \"Into the Wild\",\n          \"The Hangover\",\n          \" THE DARK KNIGHT\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Actor\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 400,\n        \"samples\": [\n          \"Keir Dullea\",\n          \"Kevin James\",\n          \"Matthew Broderick\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Character\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 568,\n        \"samples\": [\n          \"Mr. White/Larry\",\n          \"Borat Sagdiyev\",\n          \"Dr. Emmett \\\"Doc\\\" Brown\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Year\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 100,\n        \"samples\": [\n          1977,\n          \"1992\",\n          \"2002\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Genre\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 221,\n        \"samples\": [\n          \"Adventure, Comedy, Crime\",\n          \"Horror, Mystery, Thriller\",\n          \"Crime\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Director\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 360,\n        \"samples\": [\n          \"Darren Aronofsky\",\n          \"John Carpenter\",\n          \"Damien Chazelle\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Writer\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 514,\n        \"samples\": [\n          \"Tina Fey\",\n          \"Yuichiro Kido\",\n          \"Christopher McQuarrie, Jez Butterworth, John-Henry Butterworth\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Production Company\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 323,\n        \"samples\": [\n          \"MGM, Revolution Studios\",\n          \"20th Century Fox Television\",\n          \"Fantastic Mr. Fox With Company\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Sentiment_VADER\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.419473475281569,\n        \"min\": -0.9812,\n        \"max\": 0.9455,\n        \"num_unique_values\": 291,\n        \"samples\": [\n          -0.5478,\n          0.6239,\n          0.128\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Sentiment_TextBlob\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.27458665520729697,\n        \"min\": -1.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 189,\n        \"samples\": [\n          0.2916666666666667,\n          -0.3499999999999999,\n          -0.05\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 24,\n        \"samples\": [\n          \"1. None of the above\",\n          \"3.\\n3. politics\",\n          \"6. none of the above\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "data"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-55a1a276-5db4-4b26-b694-9ff4252a2a9a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>Movie Name</th>\n",
              "      <th>Actor</th>\n",
              "      <th>Character</th>\n",
              "      <th>Year</th>\n",
              "      <th>Genre</th>\n",
              "      <th>Director</th>\n",
              "      <th>Writer</th>\n",
              "      <th>Production Company</th>\n",
              "      <th>Sentiment_VADER</th>\n",
              "      <th>Sentiment_TextBlob</th>\n",
              "      <th>label_ChatGPT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>\"I am Groot.\"</td>\n",
              "      <td>Guardians of the Galaxy</td>\n",
              "      <td>Vin Diesel (voice)</td>\n",
              "      <td>Groot</td>\n",
              "      <td>2014</td>\n",
              "      <td>Action, Adventure, Comedy</td>\n",
              "      <td>James Gunn</td>\n",
              "      <td>James Gunn, Nicole Perlman</td>\n",
              "      <td>Marvel Studios</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6. none of the above</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>\"May the Force be with you.\"</td>\n",
              "      <td>Star Wars: Episode VII - The Force Awakens</td>\n",
              "      <td>Harrison Ford</td>\n",
              "      <td>Han Solo</td>\n",
              "      <td>2015</td>\n",
              "      <td>Action, Adventure, Fantasy</td>\n",
              "      <td>J.J. Abrams</td>\n",
              "      <td>Lawrence Kasdan, J.J. Abrams</td>\n",
              "      <td>Lucasfilm</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6. none of the above</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>\"I can do this all day.\"</td>\n",
              "      <td>Captain America: The First Avenger</td>\n",
              "      <td>Chris Evans</td>\n",
              "      <td>Steve Rogers / Captain America</td>\n",
              "      <td>2011</td>\n",
              "      <td>Action, Adventure, Sci-Fi</td>\n",
              "      <td>Joe Johnston</td>\n",
              "      <td>Christopher Markus, Stephen McFeely</td>\n",
              "      <td>Marvel Studios</td>\n",
              "      <td>0.0000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2. religion</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>\"You have no idea what I'm capable of.\"</td>\n",
              "      <td>The Social Network</td>\n",
              "      <td>Jesse Eisenberg</td>\n",
              "      <td>Mark Zuckerberg</td>\n",
              "      <td>2010</td>\n",
              "      <td>Biography, Drama</td>\n",
              "      <td>David Fincher</td>\n",
              "      <td>Aaron Sorkin</td>\n",
              "      <td>Columbia Pictures</td>\n",
              "      <td>0.1027</td>\n",
              "      <td>0.2</td>\n",
              "      <td>6. none of the above</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>\"You can't stop the signal, Mal.\"</td>\n",
              "      <td>Serenity</td>\n",
              "      <td>Nathan Fillion</td>\n",
              "      <td>Malcolm 'Mal' Reynolds</td>\n",
              "      <td>2005</td>\n",
              "      <td>Action, Adventure, Sci-Fi</td>\n",
              "      <td>Joss Whedon</td>\n",
              "      <td>Joss Whedon</td>\n",
              "      <td>Universal Pictures</td>\n",
              "      <td>0.2235</td>\n",
              "      <td>0.0</td>\n",
              "      <td>6. none of the above</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-55a1a276-5db4-4b26-b694-9ff4252a2a9a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-55a1a276-5db4-4b26-b694-9ff4252a2a9a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-55a1a276-5db4-4b26-b694-9ff4252a2a9a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f068eb88-de6e-417a-81bb-cc2c46534fbd\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f068eb88-de6e-417a-81bb-cc2c46534fbd')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f068eb88-de6e-417a-81bb-cc2c46534fbd button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                               Movie Quote  \\\n",
              "0                            \"I am Groot.\"   \n",
              "1             \"May the Force be with you.\"   \n",
              "2                 \"I can do this all day.\"   \n",
              "3  \"You have no idea what I'm capable of.\"   \n",
              "4        \"You can't stop the signal, Mal.\"   \n",
              "\n",
              "                                   Movie Name               Actor  \\\n",
              "0                     Guardians of the Galaxy  Vin Diesel (voice)   \n",
              "1  Star Wars: Episode VII - The Force Awakens       Harrison Ford   \n",
              "2          Captain America: The First Avenger         Chris Evans   \n",
              "3                          The Social Network     Jesse Eisenberg   \n",
              "4                                    Serenity      Nathan Fillion   \n",
              "\n",
              "                        Character  Year                       Genre  \\\n",
              "0                           Groot  2014   Action, Adventure, Comedy   \n",
              "1                        Han Solo  2015  Action, Adventure, Fantasy   \n",
              "2  Steve Rogers / Captain America  2011   Action, Adventure, Sci-Fi   \n",
              "3                 Mark Zuckerberg  2010            Biography, Drama   \n",
              "4          Malcolm 'Mal' Reynolds  2005   Action, Adventure, Sci-Fi   \n",
              "\n",
              "        Director                               Writer  Production Company  \\\n",
              "0     James Gunn           James Gunn, Nicole Perlman      Marvel Studios   \n",
              "1    J.J. Abrams         Lawrence Kasdan, J.J. Abrams           Lucasfilm   \n",
              "2   Joe Johnston  Christopher Markus, Stephen McFeely      Marvel Studios   \n",
              "3  David Fincher                         Aaron Sorkin   Columbia Pictures   \n",
              "4    Joss Whedon                          Joss Whedon  Universal Pictures   \n",
              "\n",
              "   Sentiment_VADER  Sentiment_TextBlob         label_ChatGPT  \n",
              "0           0.0000                 0.0  6. none of the above  \n",
              "1           0.0000                 0.0  6. none of the above  \n",
              "2           0.0000                 0.0           2. religion  \n",
              "3           0.1027                 0.2  6. none of the above  \n",
              "4           0.2235                 0.0  6. none of the above  "
            ]
          },
          "execution_count": 331,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# View the first 5 observations\n",
        "data.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3foaC90lwhKX",
      "metadata": {
        "id": "3foaC90lwhKX"
      },
      "source": [
        "Some observations can be made with regard to he first 5 rows of the data:\n",
        "- There are quotation marks in `Movie Quote`. These will need to be cleaned later as they are meaningless.\n",
        "- Each value of `Genre` contains multiple genres separated by commas. It appears that a quote can be associated with a maximum of 3 genres. These may need to be split out to conduct proper analysis.\n",
        "- `label_chatGPT` may be unreliable. The third row does not appear to have religious connotations yet chatGPT has labeled it as religious. As it is, if this column is used to create a response label of 'problematic' or not, we may fall victim to the adage 'garbage in, garbage out'. This will greatly hinder our model's ability to actually provide useful/reliable classifications that the movie company can use.\n",
        "- Our two sentiment score columns do not always agree. For example in row 5, `Sentiment_VADER` is positive whereas `Sentiment_TextBlob` is zero.\n",
        "- Many of the features appear to be characteristics of the *movie* rather than the *quote* (e.g. `Director`, `Genre`). This should be taken into account in our modelling stages where we are interested in the *characteristics of quotes*.\n",
        "\n",
        "---\n",
        "Next we calculate summary statistics for our two numeric columns (the sentiment scores)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 332,
      "id": "NFUw31d0xqNi",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "NFUw31d0xqNi",
        "outputId": "d457e431-9893-4c65-d84c-c9ace7af8783"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"data\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": \"Sentiment_VADER\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 325.9476441416064,\n        \"min\": -0.9812,\n        \"max\": 922.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          0.025728633405639918,\n          0.0,\n          922.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Sentiment_TextBlob\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 325.95403982925836,\n        \"min\": -1.0,\n        \"max\": 922.0,\n        \"num_unique_values\": 7,\n        \"samples\": [\n          922.0,\n          0.06425444045402356,\n          0.109375\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-73a9b403-3206-491c-8aa8-c26e771b93eb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sentiment_VADER</th>\n",
              "      <th>Sentiment_TextBlob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>922.000000</td>\n",
              "      <td>922.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.025729</td>\n",
              "      <td>0.064254</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.419473</td>\n",
              "      <td>0.274587</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>-0.981200</td>\n",
              "      <td>-1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>-0.152650</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.318200</td>\n",
              "      <td>0.109375</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.945500</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-73a9b403-3206-491c-8aa8-c26e771b93eb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-73a9b403-3206-491c-8aa8-c26e771b93eb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-73a9b403-3206-491c-8aa8-c26e771b93eb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e0c0a856-9410-4baa-af1d-fc5307c4e9ad\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e0c0a856-9410-4baa-af1d-fc5307c4e9ad')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e0c0a856-9410-4baa-af1d-fc5307c4e9ad button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       Sentiment_VADER  Sentiment_TextBlob\n",
              "count       922.000000          922.000000\n",
              "mean          0.025729            0.064254\n",
              "std           0.419473            0.274587\n",
              "min          -0.981200           -1.000000\n",
              "25%          -0.152650            0.000000\n",
              "50%           0.000000            0.000000\n",
              "75%           0.318200            0.109375\n",
              "max           0.945500            1.000000"
            ]
          },
          "execution_count": 332,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Get summary statistics of the numeric columns\n",
        "data.describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dMQgvtXuyQZ-",
      "metadata": {
        "id": "dMQgvtXuyQZ-"
      },
      "source": [
        "According to the data dictionary, the columns `Sentiment_VADER` and `Sentiment_TextBlob` both represent sentiment scores between -1 and 1, although are created by different libraries. Based on the summary statistics, `Sentiment_VADER` appears to be more volatile (higher std. deviation) although is on average closer to zero than `Sentiment_TextBlob`. Also, `Sentiment_VADER` does not have any observations at the max/min values whereas `Sentiment_TextBlob` does. We hold off on a further analysis of these two variables until after the training-validation-test split, so as to avoid introducing leakage."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "00c8c00a-c189-4ce0-887e-d413ce6894b0",
      "metadata": {
        "id": "00c8c00a-c189-4ce0-887e-d413ce6894b0"
      },
      "source": [
        "**Summary**  \n",
        "- 922 rows of data\n",
        "- Limited amount of data may have implications for the reliability of our model and the insights garnered (see above for details)\n",
        "- Missing values in `Actor, Character, Year, Genre, label_ChatGPT, Director, Writer`, and  `Production Company` columns\n",
        "- `Genre` needs to be split into additional columns\n",
        "- `label_ChatGPT` may be unreliable. Left as it is, we may fall victim to 'garbage in, garbage out'\n",
        "- There are differences between our two sentiment score columns `Sentiment_VADER` and `Sentiment_TextBlob`"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pu3zYh523in1",
      "metadata": {
        "id": "pu3zYh523in1"
      },
      "source": [
        "#### Q1a Step 2 - Duplicates\n",
        "**Outline**\n",
        "\n",
        "In this section we remove duplicates from the **entire** dataset.\n",
        "\n",
        "Duplicates in the data provide a potential source of leakage. For example, if the same row is in both the training and test set. Leakage may result in our model appearing better than it actually would at deployment.\n",
        "\n",
        "Looking at the entire dataset to identify duplicates is itself a potential source of leakage, however we believe that duplicates present a greater risk of leakage. Therefore we remove duplicates from the **entire** dataset.\n",
        "\n",
        "---\n",
        "First we use a naive approach to identify duplicate quotes: look for exact matches."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 333,
      "id": "KGEFavpK3k1X",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KGEFavpK3k1X",
        "outputId": "863b5219-639b-492b-be7a-c8fcd576bcfb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 37 duplicate quotes in the dataset\n"
          ]
        }
      ],
      "source": [
        "# Primarily we are concerned about duplicates in the 'Movie Quote' field\n",
        "# Let's first use a naive approach to identifying duplicates:\n",
        "\n",
        "print('There are' , data['Movie Quote'].duplicated().sum(),'duplicate quotes in the dataset')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TN_hXpeV8G8E",
      "metadata": {
        "id": "TN_hXpeV8G8E"
      },
      "source": [
        "This is an alarming number of duplicates, especially given our small dataset size. However, this naive approach - where we looked for exact matches in the text - may not identify all duplicates as there may be partial matches or differences in formatting.\n",
        "\n",
        "We hence use a fuzzy matching approach to identify more duplicates. Fuzzy matching looks for partial matches between strings and so will identify duplicates that may just have different spelling or punctuation etc. A package is used to assign a similarity score between quotes. We then look at how many duplicates are detected using different thresholds for this score:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 334,
      "id": "4HoAY5GM8Gd5",
      "metadata": {
        "id": "4HoAY5GM8Gd5"
      },
      "outputs": [],
      "source": [
        "# we test the number of duplicate values identified at different thresholds of the fuzzy score\n",
        "  # note this does take a while to run\n",
        "\n",
        "# Note this takes a long time to run and can be commented out if not necessary\n",
        "\n",
        "duplicate_index_80 = duplicate_indices(text_data = data['Movie Quote'], threshold = 80)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 335,
      "id": "n1eqFnwoBdw_",
      "metadata": {
        "id": "n1eqFnwoBdw_"
      },
      "outputs": [],
      "source": [
        "duplicate_index_90 = duplicate_indices(text_data = data['Movie Quote'], threshold = 90)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 336,
      "id": "0crLbh_xFGf5",
      "metadata": {
        "id": "0crLbh_xFGf5"
      },
      "outputs": [],
      "source": [
        "duplicate_index_70 = duplicate_indices(text_data = data['Movie Quote'], threshold = 70)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 337,
      "id": "MCyfO8yc8C-y",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MCyfO8yc8C-y",
        "outputId": "fb7d948e-42c3-4c55-c8b4-f6810ec9ac36"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using a fuzzy thershold of 70, we found 130 duplicate quotes in the dataset\n",
            "Using a fuzzy thershold of 80, we found 123 duplicate quotes in the dataset\n",
            "Using a fuzzy thershold of 90, we found 100 duplicate quotes in the dataset\n"
          ]
        }
      ],
      "source": [
        "print('Using a fuzzy thershold of 70, we found', len(duplicate_index_70), 'duplicate quotes in the dataset')\n",
        "print('Using a fuzzy thershold of 80, we found', len(duplicate_index_80), 'duplicate quotes in the dataset')\n",
        "print('Using a fuzzy thershold of 90, we found', len(duplicate_index_90), 'duplicate quotes in the dataset')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jlpaqJfHF1ua",
      "metadata": {
        "id": "jlpaqJfHF1ua"
      },
      "source": [
        "As we decrease the fuzzy threshold we identify more duplicates, as expected. Under all three levels we tested, a significant number of duplicates were detected (for example 123 duplicates from 922 data points). Between 70 and 90 only 7 additional duplicates were detected; let us look at the additional quotes identified to determine if they are reasonable and help us decide on a threshold:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 338,
      "id": "WtKDHGjEnKGl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WtKDHGjEnKGl",
        "outputId": "ad4d1d4b-7b17-41f4-b42e-905f3d87c940"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Duplicate 1:\n",
            "Yipee kayaye, motherfucker!\n",
            "Yippie-ki-yay, motherfâ€”er!\n",
            "\n",
            "Duplicate 2:\n",
            "We need guns. Lots of guns.\n",
            "Guns, lots of guns.\n",
            "\n",
            "Duplicate 3:\n",
            "I ate his liver with some fava beans and a nice chianti.\n",
            "A census taker once tried to test me. I ate his liver with some fava beans and a nice Chianti.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "additional_quotes = list(set(duplicate_index_70) - set(duplicate_index_80))\n",
        "\n",
        "# look at first 3 examples:\n",
        "for n in range(0,3):\n",
        "  pair = additional_quotes[n]\n",
        "  print(f'Duplicate {n+1}:')\n",
        "  print(data.loc[pair[0],'Movie Quote'])\n",
        "  print(data.loc[pair[1],'Movie Quote'])\n",
        "  print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BOgBsRtapO6F",
      "metadata": {
        "id": "BOgBsRtapO6F"
      },
      "source": [
        "These additional quotes appear to be genuine duplicates that were missed at a threshold of 80. It may not be possible to capture every duplicate and continuing to decrease the threshold may result in non-duplicates being accidentally removed. We therefore proceed with a **threshold of 70**, resulting in us **identifying 130 duplicates**.\n",
        "\n",
        "---\n",
        "As a sense check, let's look at a few of the matches the fuzzy matching came up with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 339,
      "id": "xM3S446jC5L3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xM3S446jC5L3",
        "outputId": "75dcd6d0-6e15-4914-ae35-e8581bba0fa4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " - Index 11: \"I'm always angry.\"\n",
            " - Index 37: \"I'm always angry.\"\n",
            "\n",
            " - Index 116: Yipee kayaye, motherfucker!\n",
            " - Index 303: Yippie-ki-yay, motherfâ€”er!\n",
            "\n",
            " - Index 15: \"There's no place like home.\"\n",
            " - Index 152: There's no place like home.\n",
            "\n",
            " - Index 407: Itâ€™s only after weâ€™ve lost everything that weâ€™re free to do anything.\n",
            " - Index 414: It's only after we've lost everything that we're free to do anything.\n",
            "\n",
            " - Index 17: \"I feel the need - the need for speed.\"\n",
            " - Index 119: I feel the need...the need for speed.\n",
            "\n",
            " - Index 8: \"It's not who I am underneath, but what I do that defines me.\"\n",
            " - Index 543: It's not who I am underneath, but what I do that defines me\n",
            "\n",
            " - Index 188: After all, tomorrow is another day.\n",
            " - Index 197: After all, tomorrow is another day!\n",
            "\n",
            " - Index 153: Yo, Adrian!\n",
            " - Index 249: Yo, Adrian!!.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# As a sense check, let's look at a few of the matches the fuzzy matching came up with:\n",
        "duplicate_index = duplicate_index_70\n",
        "\n",
        "random.seed(random_seed)\n",
        "\n",
        "for i, j in random.sample(duplicate_index, k = 8):\n",
        "  print(f\" - Index {i}: {data['Movie Quote'].iloc[i]}\")\n",
        "  print(f\" - Index {j}: {data['Movie Quote'].iloc[j]}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FvSbvH9XGn9S",
      "metadata": {
        "id": "FvSbvH9XGn9S"
      },
      "source": [
        "We see that it picks up true duplicates as well as differences in punctuation, spelling and additional phrases. As another sense check, let us check that the movie name is the same for each of these duplicates:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 340,
      "id": "mgyXFkbmIDI4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mgyXFkbmIDI4",
        "outputId": "24951451-d582-4d67-8719-77a7bf7c84a8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Of the 130 fuzzy duplicates, 115 have the same movie name.\n",
            "\n",
            "\n",
            "Example of duplicates with different movie name:\n",
            " - Index 1: Movie Quote                  \"May the Force be with you.\"\n",
            "Movie Name     Star Wars: Episode VII - The Force Awakens\n",
            "Name: 1, dtype: object\n",
            " - Index 184: Movie Quote           May the Force be with you.\n",
            "Movie Name      STAR WARS EPISODE IV: A NEW HOPE\n",
            "Name: 184, dtype: object\n",
            "\n",
            " - Index 2: Movie Quote              \"I can do this all day.\"\n",
            "Movie Name     Captain America: The First Avenger\n",
            "Name: 2, dtype: object\n",
            " - Index 857: Movie Quote    I can do this all day\n",
            "Movie Name         Avengers: Endgame\n",
            "Name: 857, dtype: object\n",
            "\n",
            " - Index 1: Movie Quote                  \"May the Force be with you.\"\n",
            "Movie Name     Star Wars: Episode VII - The Force Awakens\n",
            "Name: 1, dtype: object\n",
            " - Index 253: Movie Quote    May the Force be with you.\n",
            "Movie Name                      Star Wars\n",
            "Name: 253, dtype: object\n",
            "\n",
            " - Index 12: Movie Quote    \"The name's Bond, James Bond.\"\n",
            "Movie Name                            Skyfall\n",
            "Name: 12, dtype: object\n",
            " - Index 215: Movie Quote    My name is Bond, James Bond.\n",
            "Movie Name                           Dr. No\n",
            "Name: 215, dtype: object\n",
            "\n",
            " - Index 423: Movie Quote    We need guns. Lots of guns.\n",
            "Movie Name                          Matrix\n",
            "Name: 423, dtype: object\n",
            " - Index 907: Movie Quote        Guns, lots of guns.\n",
            "Movie Name     John Wick 3: Parabellum\n",
            "Name: 907, dtype: object\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# As another sense check, let us check that the movie name is the same for each of these duplicates\n",
        "same_movie = []\n",
        "\n",
        "for i, j in duplicate_index:\n",
        "    same_movie.append(is_duplicate(str(data['Movie Name'].iloc[i]), str(data['Movie Name'].iloc[j])))\n",
        "\n",
        "print(f'Of the {len(duplicate_index)} fuzzy duplicates, {sum(same_movie)} have the same movie name.')\n",
        "print('\\n')\n",
        "\n",
        "random.seed(random_seed)\n",
        "print('Example of duplicates with different movie name:')\n",
        "\n",
        "for i, j in random.sample([x for i, x in enumerate(duplicate_index) if not same_movie[i]], k = 5):\n",
        "  print(f\" - Index {i}: {data[['Movie Quote', 'Movie Name']].iloc[i]}\")\n",
        "  print(f\" - Index {j}: {data[['Movie Quote', 'Movie Name']].iloc[j]}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "w495Rx5EOQkD",
      "metadata": {
        "id": "w495Rx5EOQkD"
      },
      "source": [
        "Based on the above there are 15 fuzzy duplicate quotes that did not have the same movie name. However, upon inspecting further we see that often the quote is from the same series (e.g. Star Wars, Avengers, James Bond). This is not always the case as we see in the final example where the quotes are from 'Matrix' and 'John Wick 3' which are different movie series.\n",
        "\n",
        "We manually review these duplicates with different movie names and exclude any pairs that do not appear to be from the same series (manual review not shown).\n",
        "\n",
        "After this, we remove all the other fuzzy duplicates identified:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 341,
      "id": "X1hchS1Sr23E",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X1hchS1Sr23E",
        "outputId": "aa34fc7e-e082-45db-f594-05c641f6bb9c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Length before removing two pairs: 130. Length after removing two pairs: 128\n"
          ]
        }
      ],
      "source": [
        "# 2 detected duplicates were found to be from different franchises and hence not true duplicates\n",
        "duplicate_index2 = [x for x in duplicate_index if x not in [(423,907), (624, 852)]]\n",
        "\n",
        "# check only two pairs were removed:\n",
        "print(f'Length before removing two pairs: {len(duplicate_index)}. Length after removing two pairs: {len(duplicate_index2)}')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 342,
      "id": "8HjCqNQ7HZwj",
      "metadata": {
        "id": "8HjCqNQ7HZwj"
      },
      "outputs": [],
      "source": [
        "# Now let's remove the fuzzy duplicates\n",
        "data_no_dup = data.drop([j for i,j in duplicate_index2])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 343,
      "id": "q3hiOONXPQs0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q3hiOONXPQs0",
        "outputId": "94fc9993-36ce-48d1-e9e1-6eeb263ef783"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Rows before removing duplicates: 922\n",
            "Number of duplicates: 128\n",
            "Expected number of rows after removing: 794\n",
            "Actual number of rows after removing: 794\n"
          ]
        }
      ],
      "source": [
        "# checks:\n",
        "print(f'Rows before removing duplicates: {data.shape[0]}')\n",
        "print(f'Number of duplicates: {len(duplicate_index2)}')\n",
        "print(f'Expected number of rows after removing: {data.shape[0]-len(duplicate_index2)}')\n",
        "print(f'Actual number of rows after removing: {data_no_dup.shape[0]}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "t541vJdFXswO",
      "metadata": {
        "id": "t541vJdFXswO"
      },
      "source": [
        "We see that the number of rows removed is equal to the expected number hence passing the sense check. Overall, 128 rows were removed from the data, leaving us with 794 data points."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 344,
      "id": "oKU96z4xYONm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oKU96z4xYONm",
        "outputId": "01ab8e8b-b3d8-46ae-8c9f-e5b88b0097b5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 794 entries, 0 to 921\n",
            "Data columns (total 12 columns):\n",
            " #   Column              Non-Null Count  Dtype  \n",
            "---  ------              --------------  -----  \n",
            " 0   Movie Quote         794 non-null    object \n",
            " 1   Movie Name          794 non-null    object \n",
            " 2   Actor               791 non-null    object \n",
            " 3   Character           791 non-null    object \n",
            " 4   Year                793 non-null    object \n",
            " 5   Genre               793 non-null    object \n",
            " 6   Director            793 non-null    object \n",
            " 7   Writer              793 non-null    object \n",
            " 8   Production Company  793 non-null    object \n",
            " 9   Sentiment_VADER     794 non-null    float64\n",
            " 10  Sentiment_TextBlob  794 non-null    float64\n",
            " 11  label_ChatGPT       793 non-null    object \n",
            "dtypes: float64(2), object(10)\n",
            "memory usage: 80.6+ KB\n",
            "Number of NULLs before removing duplicates: 22\n",
            "Number of NULLs after removing duplicates: 12\n"
          ]
        }
      ],
      "source": [
        "# let's get a summary of our data and the number of missing values etc. after removing duplicates\n",
        "data_no_dup.info()\n",
        "print(f'Number of NULLs before removing duplicates: {data.isnull().sum().sum()}')\n",
        "print(f'Number of NULLs after removing duplicates: {data_no_dup.isnull().sum().sum()}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "l2eNk4d3YbiW",
      "metadata": {
        "id": "l2eNk4d3YbiW"
      },
      "source": [
        "The number of missing values in our dataset has reduced substantially from 22 to 12. We do not remove rows with missing values as data quantity is a clear issue. However, missing values will be handled until after our training-validation-test split as missing values may be expected after deployment."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ySLmm_pI3n7W",
      "metadata": {
        "id": "ySLmm_pI3n7W"
      },
      "source": [
        "**Summary**  \n",
        "In this step we removed 128 duplicates from our dataset, leaving us with 794 entries overall. Duplicates were removed to reduce the risk of leakage during our modelling and evaluation stages. Fuzzy matching was used to assess duplicates and several checks were done to ensure we did not throw out data unnecessarily."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wcDq6HtjZR0F",
      "metadata": {
        "id": "wcDq6HtjZR0F"
      },
      "source": [
        "\n",
        "\n",
        "#### Q1a Step 3 - Data and concept drift\n",
        "**Outline**\n",
        "\n",
        "In this section we analyse whether there has been any data drift or concept drift in the **entire** dataset.\n",
        "\n",
        "Although examining our entire dataset for drift may risk introducing leakage, this must be balanced with understanding our data and whether the data is appropriate for our purpose. Changes in societal norms over time, may influence whether movie quotes are 'problematic' or not. Our objective is to assist with the writing of *future* movies and so more recent movies may better mimic this deployment environment. Therefore, we believe it to be justified in conducting this drift analysis on the **entire** dataset to test this hypothesis.\n",
        "\n",
        "- **Data drift** refers to changes in the statistical properties of the variables over time.\n",
        "- **Concept drift** refers to changes in the relationship between features and the response variable over time. We do not yet have a response variable and so our analysis of this kind of drift is limited. Although we could use `label_ChatGPT` as a proxy, this variable is unclean so we do not look at concept drift."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zkGAEFuG2fl7",
      "metadata": {
        "id": "zkGAEFuG2fl7"
      },
      "source": [
        "---\n",
        "First we need to convert `Year` to a numeric variable so we can order the variables chronologically and conduct analysis on data drift over time (previously it was identified that the variable was read as a string)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 345,
      "id": "obnDe1ASe9KS",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "obnDe1ASe9KS",
        "outputId": "5ed45d90-67e0-4a3f-c09d-8106d7153bf8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Non-numeric values in \"Year\" were coerced to NA\n",
            "The non-numeric values were: \n",
            "45              NaN\n",
            "63          Unknown\n",
            "667       2011-2019\n",
            "674       2011-2019\n",
            "678       2011-2019\n",
            "763    2013-present\n",
            "824       2015-2019\n",
            "Name: Year, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# convert Year to numeric and coerce any non-numeric to NA\n",
        "data_no_dup['Year_clean'] = pd.to_numeric(data_no_dup['Year'], errors = 'coerce')\n",
        "\n",
        "print('Non-numeric values in \"Year\" were coerced to NA')\n",
        "print('The non-numeric values were: ')\n",
        "print(data_no_dup.loc[data_no_dup[data_no_dup['Year_clean'].isna()].index, 'Year'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SnXVGjRB-SvS",
      "metadata": {
        "id": "SnXVGjRB-SvS"
      },
      "source": [
        "For a few quotes from TV shows, a year range was provided instead of a single year. One of these ranges goes up to 'present' which is ambiguous.\n",
        "We therefore clean these values by **selecting the year as the first year in the range**. We also take note of the *Unknown* and *Nan* values for future investigation but, leave them for now."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 346,
      "id": "VtB2CNaZ9cfa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VtB2CNaZ9cfa",
        "outputId": "d4e41212-ae6a-4f31-cd60-002d9b7cf252"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "             Year  Year_clean\n",
            "45            nan        <NA>\n",
            "63        Unknown        <NA>\n",
            "667     2011-2019        2011\n",
            "763  2013-present        2013\n",
            "800          2016        2016\n",
            "1            2015        2015\n"
          ]
        }
      ],
      "source": [
        "# We use regular expression to get the first year of the range:\n",
        "data_no_dup['Year'] = data_no_dup['Year'].astype(str)\n",
        "data_no_dup['Year_clean'] = data_no_dup['Year'].apply(lambda x: extract_first_year(x)).astype('Int64')\n",
        "\n",
        "# spot check that conversion was done correctly:\n",
        "print(data_no_dup[['Year', 'Year_clean']].loc[[45,63,667,763,800,1]])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SiGOX8HdFIa0",
      "metadata": {
        "id": "SiGOX8HdFIa0"
      },
      "source": [
        "##### Data Drift Analysis\n",
        "\n",
        "Now we will plot several of our variables against `Year` to examine data drift. We will not plot most variables however, such as `Movie Name`, `Actor`, `Character` , `Director` and `Writer`. This is because they exhibit high cardinality and so it would be meaningless to plot. Also they are *expected* to change over time (e.g. an actor from the 1960s is unlikely to be active today). Furthermore, these variables are not particularly informative for *future* script writing since these past actors and characters are unlikely to recur in future movies. Actors and directors are also unlikely to be known at the time of deployment (i.e. time of finishing a script).\n",
        "\n",
        "Similarly, the `Production Company` is not an important variable as we are working for a particular movie production company. Therefore we also do not analyse this variable in our drift analysis.\n",
        "\n",
        "We examine `Genre`, `Sentiment_VADER`, `Sentiment_TextBlob` and `label_ChatGPT` in our data drift analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EGwcIT0mPjPZ",
      "metadata": {
        "id": "EGwcIT0mPjPZ"
      },
      "source": [
        "---\n",
        "First let us look at `Genre`. Recall that genre can contain up to 3 tags separated by commas, we therefore split `Genre` into tokens and count the number of times each genre appears in a year. We limit our analysis to the top 10 genres in each year due to the high cardinality of this variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 347,
      "id": "x9KLEvSVFF-s",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 489
        },
        "id": "x9KLEvSVFF-s",
        "outputId": "39825347-5143-455f-c7fb-78f082ae23b1"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 56024 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Genre\n",
        "# From our previous analysis we saw that Genre can contain up to 3 tags separated by commas\n",
        "# We therefore split Genre into tokens and count the common tokens by year\n",
        "\n",
        "data_no_dup['Genre_split'] = data_no_dup['Genre'].str.split(', ')\n",
        "\n",
        "data_plot_genre = data_no_dup.explode('Genre_split') # splits genre tokens into rows by duplicating rows. Do this just for plotting\n",
        "data_plot_genre['Genre_split'] = data_plot_genre['Genre_split'].replace('Science Fiction', 'Sci-Fi') # clean up Science Fiction value to be consistent\n",
        "data_plot_genre['Genre_split'] = data_plot_genre['Genre_split'].replace(' Action', 'Action') # clean up Action value to be consistent\n",
        "data_plot_genre['Genre_split'] = data_plot_genre['Genre_split'].replace('Romantic comedy', 'Romantic Comedy') # clean up Science Fiction value to be consistent\n",
        "\n",
        "# Group years into 5-year buckets\n",
        "bucket_size = 5\n",
        "top_x_genres = 10\n",
        "data_plot_genre['Year_bucket'] = (data_plot_genre['Year_clean'] // bucket_size) * bucket_size\n",
        "\n",
        "# Count the occurrences of each genre for each year bucket\n",
        "genre_counts = data_plot_genre.groupby(['Year_bucket', 'Genre_split']).size().unstack(fill_value=0)\n",
        "\n",
        "# Select top 10 genres for each period\n",
        "top_genres = genre_counts.apply(lambda x: x.nlargest(top_x_genres), axis=1)\n",
        "\n",
        "# Plot the count of each token against the 'Year' column\n",
        "top_genres.plot(kind='bar', stacked=True)\n",
        "plt.xlabel('Year Bucket')\n",
        "plt.ylabel('Count')\n",
        "plt.title(f'Top {top_x_genres} genres for each {bucket_size}-year period')\n",
        "plt.legend(title='Genre', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 348,
      "id": "y6x21x10MMbI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 489
        },
        "id": "y6x21x10MMbI",
        "outputId": "00775d65-2cf9-4c46-e2f7-e790b0d2f537"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 55988 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Now we plot bars as percentage of top 10 genres do get better comparison with older years where there is less data.\n",
        "bucket_size = 5\n",
        "top_x_genres = 10\n",
        "\n",
        "genre_percentages = top_genres.div(top_genres.sum(axis=1), axis=0) * 100\n",
        "\n",
        "# Plot the percentage of each genre relative to the top 10 in each year\n",
        "genre_percentages.plot(kind='bar', stacked=True)\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Percentage')\n",
        "plt.title('Percentage of top 10 genres in each 5-year period')\n",
        "plt.legend(title='Genre', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LbXV44o1M9DS",
      "metadata": {
        "id": "LbXV44o1M9DS"
      },
      "source": [
        "From the graphs we see that in the last 50 years or so, the top 10 genres in each period has been somewhat similar with genres like Action, Adventure and Drama dominating.\n",
        "Before 1960, the genre mix is slightly different, however there are a limited number of points in these early years.\n",
        "\n",
        "Despite a different mix of genres, we believe the quotes in these early years to still be relevant for our analysis. The different mix is partially a result of volatility given the low number of movies in these early years. Furthermore, `Genre` has a high cardinality and really our analysis is concerned about characteristics of the quotes themselves not the movies. These older quotes are still timeless and relevant for understanding the characteristics of famous quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oC5yjHOUI3dC",
      "metadata": {
        "id": "oC5yjHOUI3dC"
      },
      "source": [
        "---\n",
        "Next we look at how our sentiment scores (`Sentiment_VADER`, `Sentiment_TextBlob`) have changed over time. We plot the average of each of these scores against year. To reduce the impact of volatility we also plot the average of each score over 5-year buckets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 349,
      "id": "8ru18UYP50KT",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        },
        "id": "8ru18UYP50KT",
        "outputId": "c08f9d3e-d474-4771-ae09-5e12e076afc6"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 94144 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# First let's plot Sentiment_VADER and Sentiment_TextBlob\n",
        "\n",
        "# calculate average value of the columns by Year\n",
        "average_sentiment = data_no_dup.groupby('Year_clean').mean(numeric_only = True)\n",
        "\n",
        "# Plot the average value of each sentiment column against 'Year'\n",
        "average_sentiment.plot(kind='line', marker='o')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Average Sentiment Score')\n",
        "plt.title('Average Sentiment Score Over Time')\n",
        "plt.grid(True)\n",
        "plt.legend(title='Sentiment Columns')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 350,
      "id": "1_P_u8M56_c4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 487
        },
        "id": "1_P_u8M56_c4",
        "outputId": "cdad7b94-7be6-4d56-f9c7-c43a35273841"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 77920 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# do the same but with 5 year bucket groupings:\n",
        "\n",
        "data_plot_sentiment = data_no_dup.copy()\n",
        "\n",
        "bucket_size = 5\n",
        "top_x_genres = 10\n",
        "data_plot_sentiment['Year_bucket'] = (data_plot_sentiment['Year_clean'] // bucket_size) * bucket_size # create year buckets\n",
        "data_plot_sentiment = data_plot_sentiment.drop(columns = ['Year_clean']) # remove so we don't take average over group by\n",
        "\n",
        "# calculate average value of the columns by Year\n",
        "average_sentiment = data_plot_sentiment.groupby('Year_bucket').mean(numeric_only = True)\n",
        "\n",
        "# Plot the average value of each sentiment column against 'Year'\n",
        "average_sentiment.plot(kind='line', marker='o')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Average Sentiment Score')\n",
        "plt.title('Average Sentiment Score Over Time')\n",
        "plt.grid(True)\n",
        "plt.legend(title='Sentiment Columns')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "S8nvNy3L8vYn",
      "metadata": {
        "id": "S8nvNy3L8vYn"
      },
      "source": [
        "Although there is some volatility in the average sentiment scores (particularly for older years where there is less data), overall both sentiment scores have remained reasonably flat over time. This suggests that **there has not been a significant drift in these variables**. This may be because the VADER and TextBlob sentiments are trained/based on current social beliefs.\n",
        "Therefore, with regard to these variables, the entire time period appears to be relevant for our modelling and analysis of potentially problematic quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "agF5Mf-68La_",
      "metadata": {
        "id": "agF5Mf-68La_"
      },
      "source": [
        "---\n",
        "Now we look at `label_ChatGPT`, a proxy for our response currently. This variable has not been cleaned, which limits our analysis, so we create a table of the 3 most common values in each 5-year period"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 351,
      "id": "HFuRLSCHAJEe",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HFuRLSCHAJEe",
        "outputId": "8ec29a47-6439-4ec1-a5e7-e0b23df4ce20"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 5 most common labels in each 5-year bucket:\n",
            "Year_bucket\n",
            "1930                  [6. none of the above, 1. sex, (6. none of the above)]\n",
            "1935                 [6. none of the above, 5. rude, (6. none of the above)]\n",
            "1940             [6. none of the above, 3. politics, (6. none of the above)]\n",
            "1945    [6. none of the above, (6. none of the above), 1. None of the above]\n",
            "1950    [6. none of the above, (6. none of the above), 1. None of the above]\n",
            "1955    [6. none of the above, (6. none of the above), 1. None of the above]\n",
            "1960             [6. none of the above, 3. politics, (6. none of the above)]\n",
            "1965    [6. none of the above, 6. None of the above, (6. none of the above)]\n",
            "1970                        [6. none of the above, 3. politics, 2. religion]\n",
            "1975          [6. none of the above, 5. insensitive, (6. none of the above)]\n",
            "1980               [6. none of the above, 2. religion, 5. none of the above]\n",
            "1985               [6. none of the above, 2. religion, 1. None of the above]\n",
            "1990                            [6. none of the above, 3. politics, 4. rude]\n",
            "1995               [6. none of the above, 1. None of the above, 3. politics]\n",
            "2000                            [6. none of the above, 3. politics, 4. rude]\n",
            "2005                            [6. none of the above, 3. politics, 4. rude]\n",
            "2010                        [6. none of the above, 3. politics, 2. religion]\n",
            "2015               [6. none of the above, 3. politics, 6. None of the above]\n",
            "dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Now we look at label_ChatGPT\n",
        "# This variable has not yet been cleaned (Q1d) so we create a table with the 3 most common values in each 5 year period\n",
        "# Our analysis is limited due to the unclean nature of this variable\n",
        "\n",
        "data_chatGPT_table = data_no_dup.copy()\n",
        "\n",
        "# Group years into 5-year periods\n",
        "data_chatGPT_table['Year_bucket'] = (data_chatGPT_table['Year_clean'] // 5) * 5\n",
        "\n",
        "# Calculate counts of each unique value within each 5-year period\n",
        "data_chatGPT_table = data_chatGPT_table.groupby(['Year_bucket', 'label_ChatGPT']).size().unstack(fill_value=0)\n",
        "\n",
        "# Create a table with the top 5 most common labels in each 5-year bucket\n",
        "top_5_labels = data_chatGPT_table.apply(lambda x: x.nlargest(3).index.tolist(), axis=1)\n",
        "\n",
        "# Print the table\n",
        "print(\"Top 5 most common labels in each 5-year bucket:\")\n",
        "print(top_5_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mE46nMX_A04f",
      "metadata": {
        "id": "mE46nMX_A04f"
      },
      "source": [
        "A rough look at the table suggests that '6. none of the above' is a common label in all periods. It is likely that there is some imbalance in this variable with most quotes falling into this category.\n",
        "Due to the imbalance and the unclean nature of the variable, making definitive conclusion about data drift in this variable is challenging. However, at a preliminary level there does not appear to be a significant shift in the distribution over time."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2gRcx1QXBwYA",
      "metadata": {
        "id": "2gRcx1QXBwYA"
      },
      "source": [
        "**Summary**  \n",
        "- In this step we looked at drift in the following variables `Genre`, `Sentiment_VADER`, `Sentiment_TextBlob` and `label_ChatGPT`.\n",
        "- There did not appear to be any clear evidence of data drift in these variables, with some differences mostly being attributed to volatility.\n",
        "- **This suggests no filters on the time period need to be applied for our training and test sets**.\n",
        "- The other variables were not examined due to high cardinality, a limited relevance to our objective of improving *scripts* (e.g. casting decisions are irrelevant), or they may not be known at deployment."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "i5mxjycWC71h",
      "metadata": {
        "id": "i5mxjycWC71h"
      },
      "source": [
        "\n",
        "#### Q1a Step 4 - Splitting into training, validation and test sets\n",
        "**Outline**\n",
        "\n",
        "In this section we split the dataset into a training, validation and test set. The training set will be used to fit our model, the validation set for tuning our model and the test set will represent unseen data to evaluate how our model will generalise.\n",
        "\n",
        "It is important to do this before conducting EDA so as not to introduce leakage into our analysis. Conducting EDA on the entire dataset may lead us to make modelling decisions that were possibly influenced by our 'unseen data'.\n",
        "\n",
        "Our entire dataset has a limited number of points (794) so we believe it necessary to retain more data for training. This will allow us to better capture a broader range of quote characteristics and potential nuances within the data. We hence opt for an **80-10-10 split**. The limited data in the validation and test set will hinder our ability to evaluate our model somewhat, but we believe this split to be necessary given the complexity of our modelling task (Q3). ***Note:*** As we have limited data, cross-validation is likely a better strategy however, we have been asked to explicitly split into three datasets.\n",
        "\n",
        "**Regarding the test set:** While an argument can be made to prioritize quotes from recent years, potentially reflecting current social norms and script styles, our analysis revealed no clear evidence of data drift across the data timeframe. This suggests that the sentiment towards older quotes and whether they are problematic are still aligned with today. Therefore, we make no specific time-based adjustments when creating our dataset splits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 352,
      "id": "dyWpb-FqJUBn",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dyWpb-FqJUBn",
        "outputId": "fbef84f9-ed95-4348-eb45-403a4c2a7d4c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Actual vs Expected number of rows in each dataset: \n",
            "Training: Actual: 634, Expected: 634.0\n",
            "Validation: Actual: 80, Expected: 80.0\n",
            "Test: Actual: 80, Expected: 80.0\n"
          ]
        }
      ],
      "source": [
        "# create training-validation-test sets:\n",
        "\n",
        "test_size = 0.1\n",
        "val_size = 1/9\n",
        "\n",
        "train_val, test = train_test_split(data_no_dup, test_size = test_size, random_state = random_seed)\n",
        "train, val = train_test_split(train_val,test_size=val_size,random_state = random_seed)\n",
        "\n",
        "# check that splits have been done correctly:\n",
        "print('Actual vs Expected number of rows in each dataset: ')\n",
        "print(f'Training: Actual: {train.shape[0]}, Expected: {np.ceil(data_no_dup.shape[0] - 2 * np.ceil(0.1*data_no_dup.shape[0]))}')\n",
        "print(f'Validation: Actual: {val.shape[0]}, Expected: {np.ceil(0.1*data_no_dup.shape[0])}')\n",
        "print(f'Test: Actual: {test.shape[0]}, Expected: {np.ceil(test_size*data_no_dup.shape[0])}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sHRsvq8SLP0d",
      "metadata": {
        "id": "sHRsvq8SLP0d"
      },
      "source": [
        "**Summary**  \n",
        "- Dataset was split into training, validation and test with an 80-10-10 split"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zTBInmWcLcvN",
      "metadata": {
        "id": "zTBInmWcLcvN"
      },
      "source": [
        "\n",
        "#### Q1a Step 5 - EDA: Movie Name, Actor, Character, Director and Writer\n",
        "**Outline**\n",
        "\n",
        "In this section we conduct exploratory data analysis on `Movie Name`, `Actor`, `Character`, `Director` and `Writer`. These variables all have high cardinality hence why we group our analysis of them together.\n",
        "\n",
        "The high cardinality makes it difficult to create meaningful plots and so we instead explore these variables by looking at the most common values to try and identify patterns. While exploring these variables we also highlight why they may be irrelevant to our business objective and modelling goals.\n",
        "\n",
        "This EDA will be done on the **training** dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "RDusHlx2Nrwf",
      "metadata": {
        "id": "RDusHlx2Nrwf"
      },
      "source": [
        "---\n",
        "**Movie Name**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 353,
      "id": "0jtYqBQBNonv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0jtYqBQBNonv",
        "outputId": "418f2300-7ce1-4a35-bd01-a7506bfb4b52"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 447 unique Movie Names in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 20 Movie Names:  \n",
            "\n",
            "\n",
            "                                                   Count Percentage\n",
            "Movie Name                                                         \n",
            "Mean Girls                                            12      1.89%\n",
            "The Dark Knight                                       11      1.74%\n",
            "Dragon Ball Super: Broly                               8      1.26%\n",
            "Joker                                                  7       1.1%\n",
            "Into the Wild                                          7       1.1%\n",
            "Avengers: Endgame                                      6      0.95%\n",
            "Interstellar                                           6      0.95%\n",
            "Game of Thrones                                        5      0.79%\n",
            "The Lord of the Rings: The Two Towers                  5      0.79%\n",
            "Fantastic Mr. Fox                                      5      0.79%\n",
            "Captain America: The Winter Soldier                    4      0.63%\n",
            "Shawshank Redemption                                   4      0.63%\n",
            "The Lion King                                          4      0.63%\n",
            " THE WIZARD OF OZ                                      4      0.63%\n",
            "Scooby-Doo on Zombie Island                            4      0.63%\n",
            "The Imitation Game                                     4      0.63%\n",
            "Fight Club                                             4      0.63%\n",
            "The Lord of the Rings: The Fellowship of the Ring      3      0.47%\n",
            "Finding Forrester                                      3      0.47%\n",
            "The Hobbit: The Battle of Five Armies                  3      0.47%\n"
          ]
        }
      ],
      "source": [
        "# First let us look at the cardinality of Movie Name i.e the number of unique values\n",
        "\n",
        "# we repeat this several times so create a function to do this:\n",
        "def cardinality_and_count_analysis(data, col_name, top_n):\n",
        "  print(f'There are {len(set(data[col_name]))} unique {col_name}s in the dataset (from', data.shape[0], 'rows)')\n",
        "\n",
        "  # Let's look if there are any values that repeat a lot:\n",
        "  # Creating table of count and percentage of unique values\n",
        "  dat_count=data[col_name].value_counts()\n",
        "  dat_perc=(data[col_name].value_counts(normalize=True) * 100). round(2).astype(str) + '%'\n",
        "\n",
        "  dat_count_perc =  pd.DataFrame({'Count': dat_count,\n",
        "                                  'Percentage': dat_perc\n",
        "  })\n",
        "\n",
        "  #Printing the count and percentage of the top top_n entries\n",
        "\n",
        "  print(f\"Total count and percentage of the top {top_n} {col_name}s:  \")\n",
        "  print('\\n')\n",
        "  print(dat_count_perc.head(top_n))\n",
        "\n",
        "cardinality_and_count_analysis(train, 'Movie Name', 20)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 354,
      "id": "qvRuu4qEQJJA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qvRuu4qEQJJA",
        "outputId": "c02f9bde-0ad3-4393-921b-71a609bf7992"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mean Girls Quotes:\n",
            "507                                                                It's like I have ESPN or something.\n",
            "514                                              That's why her hair is so big.  It's full of secrets.\n",
            "504                                                                                    Boo, you whore!\n",
            "518                                                                                  Is butter a carb?\n",
            "508                                                                We should totally just stab Caesar!\n",
            "509                                                                Whatever. I'm getting cheese fries.\n",
            "506                                      Stop trying to make 'fetch' happen. It's not going to happen.\n",
            "510                                                                                  That is so fetch!\n",
            "526                                                        Four for you, Glen Coco! You go, Glen Coco!\n",
            "528                                                                          She doesn't even go here!\n",
            "527                                                    There's a 30% chance that it's already raining.\n",
            "525    I wish I could bake a cake filled with rainbows and smiles and everyone would eat and be happy.\n",
            "Name: Movie Quote, dtype: object\n",
            "The Dark Knight Quotes:\n",
            "20                                                                                                                     \"Why so serious?\"\n",
            "19                                                                                     \"I'm not a monster. I'm just ahead of the curve.\"\n",
            "620                                                                  You see, in their last moments people show you who they really are.\n",
            "605                                                                                         Some men just want to watch the world burn. \n",
            "634                                          Nobody panics when things go â€œaccording to planâ€_x009d_. Even if the plan is horrifying!\n",
            "624                                                                                                      Let's put a smile on that face!\n",
            "621                                                                                 If youâ€™re good at something, never do it for free.\n",
            "609                                                             It's not about money...itâ€™s about sending a message. Everything burns!\n",
            "603                                                                                  It's not about what I want. It's about what's fair!\n",
            "608    Sometimes the truth isnâ€™t good enough, sometimes people deserve more. Sometimes people deserve to have their faith rewarded    \n",
            "627                                                                Madness, as you know, is like gravity, all it takes is a little push.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Let's just check that we haven't missed any duplicates by looking at the top two movies:\n",
        "print('Mean Girls Quotes:')\n",
        "print(train[train['Movie Name'] == 'Mean Girls']['Movie Quote'])\n",
        "\n",
        "print('The Dark Knight Quotes:')\n",
        "print(train[train['Movie Name'] == 'The Dark Knight']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ykOE0zrrT5mv",
      "metadata": {
        "id": "ykOE0zrrT5mv"
      },
      "source": [
        "We also notice that some quotes have spilled into the `Movie Name` column which is the second column. We detect this by checking for quotation marks in the `Movie Name` column. This may miss some spillage but a manual search through all rows is unfeasible. This spillage will be cleaned in a later step (Q1b)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 355,
      "id": "B2xoLdycXohs",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B2xoLdycXohs",
        "outputId": "ea142438-3080-4ede-8833-8d5e07df5d98"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                       Movie Quote  \\\n",
            "166                        SON OF    \n",
            "65   Surely, you can't be serious.   \n",
            "63                    Who are you?   \n",
            "\n",
            "                                      Movie Name           Actor    Character  \\\n",
            "166              ... That's gonna leave a mark.\"    Jeff Daniels  Harry Dunne   \n",
            "65    \"I am serious, and don't call me Shirley.\"  Leslie Nielsen   Dr. Rumack   \n",
            "63                       \"Your worst nightmare.\"         Unknown      Unknown   \n",
            "\n",
            "        Year    Genre                                  Director  \\\n",
            "166     1994   Comedy                  Peter and Bobby Farrelly   \n",
            "65      1980   Comedy  Jim Abrahams, David Zucker, Jerry Zucker   \n",
            "63   Unknown  Unknown                                   Unknown   \n",
            "\n",
            "                                       Writer  Production Company  \\\n",
            "166                  Peter and Bobby Farrelly     New Line Cinema   \n",
            "65   Jim Abrahams, David Zucker, Jerry Zucker  Paramount Pictures   \n",
            "63                                    Unknown             Unknown   \n",
            "\n",
            "     Sentiment_VADER  Sentiment_TextBlob         label_ChatGPT  Year_clean  \\\n",
            "166           0.0000            0.000000  6. none of the above        1994   \n",
            "65            0.4805            0.083333  6. none of the above        1980   \n",
            "63            0.0000            0.000000  6. none of the above        <NA>   \n",
            "\n",
            "    Genre_split  \n",
            "166    [Comedy]  \n",
            "65     [Comedy]  \n",
            "63    [Unknown]  \n"
          ]
        }
      ],
      "source": [
        "# we also notice that some quotes have spilled into the `Movie Name` column which is the second column\n",
        "# we detect this by checking for quotation marks in our Movie Name column\n",
        "# we may potentially miss some spillage but a manual search through all rows is unfeasible\n",
        "\n",
        "print(train.loc[train['Movie Name'].notna() & train['Movie Name'].str.contains('\"')])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LXV2MOkXRKZc",
      "metadata": {
        "id": "LXV2MOkXRKZc"
      },
      "source": [
        "**Summary for Movie Name**\n",
        "Overall, `Movie Name` has high cardinality with 468 unique values from 639 rows. Therefore using this variable as it is will lead to a large increase in the feature space and we will be struck by the 'curse of dimensionality' particularly given our lack of data. This increase in the feature space will make it difficult for our models to identify the **key** characteristics of famous movie quotes and will also make it harder to *communicate* what these key characteristics are to executives.\n",
        "\n",
        "Furthermore, `Movie Name` is unlikely to be available or be certain at the time of deployment (i.e. while writing a script). New movie names are also unlikely to be the same as the movie names of the past, meaning this variable is useless for our analysis.\n",
        "\n",
        "There are a few movies such as Mean Girls that have several famous quotes, however a given movie's quotes still only makes up a small proportion of the total dataset. This suggests that our dataset is not biased towards a particular movie. This is good as bias in the data could impact how well our results generalise to other movie quotes.\n",
        "\n",
        "We notice some spillage from the first column, `Movie Quote`, into `Movie Name`. This will need to be resolved to correctly analysis our quotes which are the focus of our analysis. The other columns do not appear to be affected by the spillage.\n",
        "\n",
        "In summary, `Movie Name` is useless for our analysis and we will not consider it further."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-cu2A6UoU7WF",
      "metadata": {
        "id": "-cu2A6UoU7WF"
      },
      "source": [
        "---\n",
        "**Actor**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 356,
      "id": "yeLAEqNHU_Lm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yeLAEqNHU_Lm",
        "outputId": "365bef22-8403-4063-ab95-e35855af7ffb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 338 unique Actors in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 20 Actors:  \n",
            "\n",
            "\n",
            "                   Count Percentage\n",
            "Actor                              \n",
            "Heath Ledger          16      2.53%\n",
            "Robert Downey Jr.     13      2.05%\n",
            "Leonardo DiCaprio     10      1.58%\n",
            "Keanu Reeves          10      1.58%\n",
            "Sean Connery           8      1.26%\n",
            "Brad Pitt              8      1.26%\n",
            "Tom Hanks              8      1.26%\n",
            "Will Smith             8      1.26%\n",
            "Daniel Radcliffe       7      1.11%\n",
            "Lindsay Lohan          7      1.11%\n",
            "Emile Hirsch           7      1.11%\n",
            "Chris Evans            7      1.11%\n",
            "Al Pacino              7      1.11%\n",
            "Joaquin Phoenix        7      1.11%\n",
            "George Clooney         6      0.95%\n",
            "Tom Cruise             6      0.95%\n",
            "Jack Nicholson         6      0.95%\n",
            "Mike Myers             6      0.95%\n",
            "Marlon Brando          6      0.95%\n",
            "Robin Williams         6      0.95%\n"
          ]
        }
      ],
      "source": [
        "# Follow same process as with Movie name\n",
        "cardinality_and_count_analysis(train, 'Actor', 20)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bBHzyCMoWxTM",
      "metadata": {
        "id": "bBHzyCMoWxTM"
      },
      "source": [
        "The `Actor` variable has high cardinality with 336 unique values from 639 rows. For similar reasons to `Movie Name`, the high cardinality makes this variable problematic to use in our analysis.\n",
        "\n",
        "Importantly however, we are trying to assist with *scriptwriting* not *casting decisions*. At the time of scriptwriting and hence model deployment, the actor may not yet be known. It is hence not a useful variable for understanding what characteristics of the **script** make a quote famous or problematic; this has nothing to do with the actor. Similarly, new actors may not exist in the training dataset.\n",
        "\n",
        "Again, given our business objective is to help with screenwriting not casting, the `Actor` variable is not useful for our analysis and we do not consider it further.\n",
        "\n",
        "A single actor's quotes only make up a small percentage of our total dataset and the percentages are similar across actors, indicating that the data is not biased towards any particular actor."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZTnLOW_eYkQL",
      "metadata": {
        "id": "ZTnLOW_eYkQL"
      },
      "source": [
        "---\n",
        "**Character**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 357,
      "id": "cC4qRA4VYoGT",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cC4qRA4VYoGT",
        "outputId": "33e92ea5-ea1d-43b7-ccc2-793f434bba95"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 468 unique Characters in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 20 Characters:  \n",
            "\n",
            "\n",
            "                          Count Percentage\n",
            "Character                                 \n",
            "Joker                         9      1.42%\n",
            "Tony Stark/Iron Man           8      1.26%\n",
            "Cady Heron                    7      1.11%\n",
            "Christopher McCandless        7      1.11%\n",
            "Harry Potter                  6      0.95%\n",
            "The Joker                     6      0.95%\n",
            "Arthur Fleck/Joker            5      0.79%\n",
            "Cooper                        5      0.79%\n",
            "Mr. Fox                       5      0.79%\n",
            "Neo                           4      0.63%\n",
            "Alan Turing                   4      0.63%\n",
            "Chris Gardner                 4      0.63%\n",
            "Goku                          4      0.63%\n",
            "Ellis Boyd \"Red\" Redding      4      0.63%\n",
            "Michael Corleone              4      0.63%\n",
            "Aragorn                       4      0.63%\n",
            "Gandalf                       4      0.63%\n",
            "Elliot Alderson               4      0.63%\n",
            "Naruto Uzumaki                3      0.47%\n",
            "Rick Blaine                   3      0.47%\n"
          ]
        }
      ],
      "source": [
        "# Follow same process as with Movie name by looking at cardinality and top characters\n",
        "cardinality_and_count_analysis(train, 'Character', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zQvpyAadZJTI",
      "metadata": {
        "id": "zQvpyAadZJTI"
      },
      "source": [
        "`Character` has high cardinality which makes it difficult to use to identify meaningful patterns.\n",
        "\n",
        "*Furthermore, we are interested in the content of a quote and its cultural impact, not necessarily which character said it. Famous quotes often transcend the individual character and resonate with audiences for broader reasons (which is what we actually want to understand for objective 1).*\n",
        "\n",
        "Also, it should be the content of quotes, not the character, that makes a quote problematic (objective 2).\n",
        "\n",
        "The character name may be known at the time of scriptwriting however, a character's name alone is fairly meaningless and may not be indicative of a good script. For example, if we identify some names/characters as important and the production company decides to use them ad nauseam, it may have the unintended effect of making the movies less memorable. Quote memorability often comes from new and fresh characters that may not be in our training set.\n",
        "\n",
        "We do notice that the 'Joker' or some variation of this character appears commonly in the dataset. There may hence be a slight bias towards this character in the dataset. This may have an impact on our clustering and modelling results and how well they generalise to future quotes. Given our scarcity of data, we ignore this potential issue and leave the dataset as it it.\n",
        "\n",
        "Overall, `Character` is likely unimportant for our analysis and we do not consider it further."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "A076D9dHczqq",
      "metadata": {
        "id": "A076D9dHczqq"
      },
      "source": [
        "---\n",
        "**Director and Writer**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 358,
      "id": "rg7-GNRsc293",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rg7-GNRsc293",
        "outputId": "c4a280c7-8c94-48c3-cdad-85dbb1d82177"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 308 unique Directors in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 20 Directors:  \n",
            "\n",
            "\n",
            "                          Count Percentage\n",
            "Director                                  \n",
            "Christopher Nolan            27      4.26%\n",
            "Peter Jackson                16      2.52%\n",
            "Anthony Russo, Joe Russo     12      1.89%\n",
            "Mark Waters                  12      1.89%\n",
            "Steven Spielberg             10      1.58%\n",
            "Quentin Tarantino             9      1.42%\n",
            "Francis Ford Coppola          9      1.42%\n",
            "Victor Fleming                9      1.42%\n",
            "David Fincher                 9      1.42%\n",
            "Todd Phillips                 8      1.26%\n",
            "Tatsuya Nagamine              8      1.26%\n",
            "Stanley Kubrick               7       1.1%\n",
            "Sean Penn                     7       1.1%\n",
            "Martin Scorsese               7       1.1%\n",
            "Robert Zemeckis               6      0.95%\n",
            "Morten Tyldum                 6      0.95%\n",
            "David Yates                   6      0.95%\n",
            "Frank Darabont                6      0.95%\n",
            "Rob Reiner                    6      0.95%\n",
            "Tim Burton                    5      0.79%\n"
          ]
        }
      ],
      "source": [
        "# First look at director\n",
        "cardinality_and_count_analysis(train, 'Director', 20)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 359,
      "id": "naKrwuaRc_HI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "naKrwuaRc_HI",
        "outputId": "8866ec77-b132-4dd5-8982-f9325cdce191"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 420 unique Writers in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 20 Writers:  \n",
            "\n",
            "\n",
            "                                                   Count Percentage\n",
            "Writer                                                             \n",
            "Jonathan Nolan, Christopher Nolan                     14      2.21%\n",
            "Christopher Markus, Stephen McFeely                   13      2.05%\n",
            "Tina Fey                                              12      1.89%\n",
            "Quentin Tarantino                                      9      1.42%\n",
            "Akira Toriyama                                         8      1.26%\n",
            "Todd Phillips, Scott Silver                            7       1.1%\n",
            "Christopher Nolan, Jonathan Nolan                      7       1.1%\n",
            "Richard Curtis                                         5      0.79%\n",
            "Sylvester Stallone                                     5      0.79%\n",
            "Fran Walsh, Philippa Boyens, Peter Jackson             5      0.79%\n",
            "Irene Mecchi, Jonathan Roberts, Linda Woolverton       5      0.79%\n",
            "Mario Puzo, Francis Ford Coppola                       5      0.79%\n",
            "George Lucas                                           4      0.63%\n",
            "Glenn Leopold                                          4      0.63%\n",
            "Damien Chazelle                                        4      0.63%\n",
            "Steve Conrad                                           4      0.63%\n",
            "Sam Esmail                                             4      0.63%\n",
            "Chuck Palahniuk (novel), Jim Uhls (screenplay)         4      0.63%\n",
            "Noel Langley, Florence Ryerson, Edgar Allan Woolf      4      0.63%\n",
            "Aaron Sorkin                                           4      0.63%\n"
          ]
        }
      ],
      "source": [
        "# Now look at writer\n",
        "cardinality_and_count_analysis(train, 'Writer', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Q0G7RZPSdQIC",
      "metadata": {
        "id": "Q0G7RZPSdQIC"
      },
      "source": [
        "As with previous variables, `Director` and `Writer` have high cardinality making them difficult to use in our analysis due to the 'curse of dimensionality'.\n",
        "\n",
        "At the risk of repeating ourselves, the business objectives are to improve *scripts* and identify if *quotes* are problematic. This is entirely independent of who wrote the script and who will direct the movie. Also, the director may not yet be known at the time of screenwriting and so we cannot use it for our predictions.\n",
        "\n",
        "We therefore will not use these variables in our analysis and do not consider them further."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3p6q5w_9jrZX",
      "metadata": {
        "id": "3p6q5w_9jrZX"
      },
      "source": [
        "**Section Summary**\n",
        "\n",
        "The variables `Movie Name`, `Actor`, `Character`, `Director` and `Writer` all have extremely high cardinality. Using even two of these variables in our analysis would lead to more features than observations. Using just one would lead to high variance error in our modelling techniques due to the curse of dimensionality and limit the usefulness of our insights or classifier.\n",
        "\n",
        "Furthermore, many of these variables are unlikely to be known at the time of model deployment (e.g. the scriptwriting phase) and dimension reduction techniques will not resolve this.\n",
        "\n",
        "While these variables may offer context, our business objective relates to improving *scripts* by understanding the key characteristics of famous quotes and identifying problematic quotes. Therefore we are concerned about characteristics/features that relate to *quotes* not factors that relate to the movie or other external factors.\n",
        "\n",
        "**Therefore, these variables are not useful for our purpose and will not be used.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "GVIFAbEGnTKV",
      "metadata": {
        "id": "GVIFAbEGnTKV"
      },
      "source": [
        "#### Q1a Step 6 - EDA: Year\n",
        "**Outline**\n",
        "\n",
        "In this section we conduct EDA related to `Year`.\n",
        "\n",
        "We cleaned this variable previously and also conducted some analysis relating to data drift.\n",
        "\n",
        "Here we look at the distribution of year, strange values and consider whether it is appropriate for our future analysis.\n",
        "\n",
        "---\n",
        "First let us calculate summary statistics and create a histogram of the non-null values in `Year`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 360,
      "id": "bGYUbxKCq5b3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bGYUbxKCq5b3",
        "outputId": "66ce5772-498a-48a8-fe96-10431f16e00b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "We have 1 NA Years in the training dataset\n"
          ]
        }
      ],
      "source": [
        "# look at missing values:\n",
        "print(f'We have {train[\"Year_clean\"].isna().sum()} NA Years in the training dataset')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 361,
      "id": "SPvj87GQocwO",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SPvj87GQocwO",
        "outputId": "40540d55-a596-4ced-c976-eed7b20a0f3d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "count         633.0\n",
              "mean     1999.78357\n",
              "std       17.357528\n",
              "min          1931.0\n",
              "25%          1994.0\n",
              "50%          2004.0\n",
              "75%          2013.0\n",
              "max          2019.0\n",
              "Name: Year_clean, dtype: Float64"
            ]
          },
          "execution_count": 361,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Get summary statistics of year\n",
        "train['Year_clean'].describe()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 362,
      "id": "yQguQLfno78-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 507
        },
        "id": "yQguQLfno78-",
        "outputId": "f4851074-0ad6-4839-9320-b1fc0c83215f"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 58736 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Now create some plots\n",
        "# do not include NAs in plot\n",
        "\n",
        "plt.figure(figsize=(12, 5))\n",
        "\n",
        "# plot distribution of year with a histogram\n",
        "ax1 = plt.subplot(1,2,1)\n",
        "plt.hist(train.loc[[not NA_test for NA_test in train[\"Year_clean\"].isna()],'Year_clean'], bins=20, edgecolor='black')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Histogram of Years')\n",
        "plt.grid(True)\n",
        "\n",
        "\n",
        "# create violin plot of year - violin plot is similar to a box plot but with more detail\n",
        "ax1 = plt.subplot(1,2,2)\n",
        "sns.violinplot(data=train.loc[[not NA_test for NA_test in train[\"Year_clean\"].isna()],'Year_clean'], inner='quartile')\n",
        "plt.title('Violin Plot of Year')\n",
        "#plt.xlabel('Variables')\n",
        "plt.ylabel('Year')\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 363,
      "id": "mNiVh3c9qIRC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mNiVh3c9qIRC",
        "outputId": "9296e75a-f6c1-4ee8-9f7f-8a5e5f20b131"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Percentage of quotes from post-2000: 60.03%\n"
          ]
        }
      ],
      "source": [
        "perc_gt_2000 = (train['Year_clean'] > 2000).mean() * 100\n",
        "print(f'Percentage of quotes from post-2000: {perc_gt_2000:.2f}%')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8rXYb6u5pkib",
      "metadata": {
        "id": "8rXYb6u5pkib"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**Min Year:** 1931\n",
        "\n",
        "**Max Year:** 2019\n",
        "\n",
        "We see that most quotes come from after the year 2000 (60%), with generally less quotes from a given year the longer ago it was. This gives us some confidence that the quotes in our dataset are relevant to today and future movie scripts (as opposed to if we only had old quotes). This skewed distribution is also expected given that more movies have probably been made each progressive year\n",
        "\n",
        "The maximum year is 2019 whereas we are doing our analysis in 2024. There is hence a time gap in our data. This may mean our dataset is not incorporating the latest trends and scriptwriting styles that could be present today and in the future. Alternatively, quotes in recent years may not have had enough time to reach iconic status, offering an explanation for their absence. This data gap is a potential limitation for our models as they may produce results that are irrelevant to today. However, for 2020 and 2021 there were limited movie releases due to the pandemic; therefore the data gap may not be that substantial.\n",
        "\n",
        "***Should year be used as a feature in our analysis?***\n",
        "\n",
        "We will be deploying our analysis to assist with screenwriting in the *future*. At the time of screenwriting (e.g. deployment), the year a movie will be released is unknown. Therefore, to avoid leakage `Year` will not be used in our analysis. Even though our cluster analysis in Q2 does not make predictions, the year a movie is produced is not a characteristic of a *quote*, which is what we are concerned about. What makes a quote famous should be timeless and related more to how the content resonates with people.\n",
        "\n",
        "**Therefore `Year` will not be used in our consequent analysis in Q2 and Q3**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CIDrpxLSt6PX",
      "metadata": {
        "id": "CIDrpxLSt6PX"
      },
      "source": [
        "#### Q1a Step 7 - EDA: Sentiment_VADER and Sentiment_TextBlob\n",
        "**Outline**\n",
        "\n",
        "In this section we conduct EDA on our two sentiment score columns. These columns are characteristics of the quotes and so may be useful features for our model and cluster analysis in later stages.\n",
        "\n",
        "These variables are related and so we also look at the relationship between these two variables by considering:\n",
        "- Correlation measures\n",
        "- How often do they disagree/agree"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 364,
      "id": "j2R83RdQvK8C",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "j2R83RdQvK8C",
        "outputId": "a50d2358-fbc8-4ea2-c074-a486c8def27e"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"train[['Sentiment_VADER', 'Sentiment_TextBlob']]\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": \"Sentiment_VADER\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 224.12199408787816,\n        \"min\": -0.9812,\n        \"max\": 634.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          0.045459936908517344,\n          0.0,\n          634.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Sentiment_TextBlob\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 224.12719548616357,\n        \"min\": -1.0,\n        \"max\": 634.0,\n        \"num_unique_values\": 7,\n        \"samples\": [\n          634.0,\n          0.07329923370402965,\n          0.16049107142857144\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-d2cbe304-3bde-4b6e-8a12-68757893b3e4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sentiment_VADER</th>\n",
              "      <th>Sentiment_TextBlob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>634.000000</td>\n",
              "      <td>634.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.045460</td>\n",
              "      <td>0.073299</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.441049</td>\n",
              "      <td>0.287228</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>-0.981200</td>\n",
              "      <td>-1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>-0.187450</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.361200</td>\n",
              "      <td>0.160491</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.945500</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d2cbe304-3bde-4b6e-8a12-68757893b3e4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d2cbe304-3bde-4b6e-8a12-68757893b3e4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d2cbe304-3bde-4b6e-8a12-68757893b3e4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9815685f-47b5-412c-a31b-a95387df0174\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9815685f-47b5-412c-a31b-a95387df0174')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9815685f-47b5-412c-a31b-a95387df0174 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       Sentiment_VADER  Sentiment_TextBlob\n",
              "count       634.000000          634.000000\n",
              "mean          0.045460            0.073299\n",
              "std           0.441049            0.287228\n",
              "min          -0.981200           -1.000000\n",
              "25%          -0.187450            0.000000\n",
              "50%           0.000000            0.000000\n",
              "75%           0.361200            0.160491\n",
              "max           0.945500            1.000000"
            ]
          },
          "execution_count": 364,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# look at summary statistics of the two variables in the training set:\n",
        "train[['Sentiment_VADER', 'Sentiment_TextBlob']].describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Iw9sxEwqxGxU",
      "metadata": {
        "id": "Iw9sxEwqxGxU"
      },
      "source": [
        "From the above, we see that both scores have average values close to zero but `Sentiment_VADER` has higher variance.\n",
        "Both scores are between -1 and 1, however only `Sentiment_TextBlob` has values exactly at these edge values (i.e. 1 and -1). This matches the data dictionary which indicated that the sentiment scores should be between -1 and 1.\n",
        "\n",
        "Interestingly, the first quartile of `Sentiment_TextBlob` is the same as its median (value of 0) which suggests that it may be less likely to give negative scores compared to `Sentiment_VADER`. This is particularly relevant given that in Q3 we will try to identify problematic quotes which may be expected to be related to negative sentiment.\n",
        "\n",
        "There appears to be more positive sentiment in our dataset according to both scores. We try to support this by looking at a violin plot of the two sentiment scores:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 365,
      "id": "S9DdBYUhw5bx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "S9DdBYUhw5bx",
        "outputId": "8495e050-7b7f-4d80-a017-a1ee02fec6a7"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 46392 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# create violin plot of the two variables\n",
        "sns.violinplot(data=train[['Sentiment_VADER', 'Sentiment_TextBlob']], inner='quartile')\n",
        "plt.title('Violin Plot of Sentiment scores')\n",
        "plt.xlabel('Variables')\n",
        "plt.ylabel('Value')\n",
        "plt.xticks(ticks=[0, 1], labels=['Sentiment_VADER', 'Sentiment_TextBlob'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_de8GxICyAIu",
      "metadata": {
        "id": "_de8GxICyAIu"
      },
      "source": [
        "The violin plot verifies many of the observations from the summary statistics.\n",
        "- `Sentiment_VADER` has higher volatility and is more likely to give negative scores than `Sentiment_TextBlob`.\n",
        "- `Sentiment_TextBlob` appears to be highly concentrated around zero which suggests it may not be good at distinguishing good/bad sentiments.\n",
        "- Both scores appear to have more quotes with positive sentiment rather than negative sentiment.\n",
        "\n",
        "We now look at the relationship between the two sentiment scores. We would expect that as they are trying to measure a similar thing that they would have some positive relationship:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 366,
      "id": "CzytIy0DzQwA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "CzytIy0DzQwA",
        "outputId": "530efc68-d7b7-4e7b-8e3b-79d314ada333"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 64644 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# create scatter plot of the two sentiment scores\n",
        "\n",
        "plt.scatter(x = train[['Sentiment_VADER']], y = train[['Sentiment_TextBlob']])\n",
        "plt.title('Scatter Plot between Sentiment_VADER and Sentiment_TextBlob')\n",
        "plt.xlabel('Sentiment_VADER')\n",
        "plt.ylabel('Sentiment_TextBlob')\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fOJDMV6M1TGW",
      "metadata": {
        "id": "fOJDMV6M1TGW"
      },
      "source": [
        "From the scatterplot above, we see that the two sentiment scores have an approximate positive linear relationship, which is expected given they are measuring similar concepts.\n",
        "\n",
        "TextBlob tends to predict a score of zero more often than VADER.\n",
        "Furthermore, the two measures do not always agree on the overall sentiment; there are several points where one is positive while the other is negative.\n",
        "\n",
        "We next calculate some correlation measures to quantify the relationship between the two scores:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 367,
      "id": "rg2cJ4ZzzMVt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rg2cJ4ZzzMVt",
        "outputId": "14be8f7c-9016-475d-96f2-828048849d50"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Pearson correlation coefficient: 0.51\n",
            "Spearman rank correlation coefficient: 0.54\n",
            "Kendall's tau: 0.42\n"
          ]
        }
      ],
      "source": [
        "# look at correlation measures between the two sentiments\n",
        "var1 = train['Sentiment_VADER']\n",
        "var2 = train['Sentiment_TextBlob']\n",
        "\n",
        "# Calculate Pearson correlation coefficient - for linear relationships\n",
        "pearson_corr = np.corrcoef(var1, var2)[0, 1]\n",
        "\n",
        "# Calculate Spearman rank correlation coefficient\n",
        "spearman_corr = np.corrcoef(var1.rank(), var2.rank())[0, 1]\n",
        "\n",
        "# Calculate Kendall's tau\n",
        "from scipy.stats import kendalltau\n",
        "kendall_tau, _ = kendalltau(var1, var2)\n",
        "\n",
        "print(f\"Pearson correlation coefficient: {pearson_corr:.2f}\")\n",
        "print(f\"Spearman rank correlation coefficient: {spearman_corr:.2f}\")\n",
        "print(f\"Kendall's tau: {kendall_tau:.2f}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lligC4Zy1r1H",
      "metadata": {
        "id": "lligC4Zy1r1H"
      },
      "source": [
        "The correlation measures roughly agree that there is a positive relationship between the two which again is expected given that they are both measuring sentiment.\n",
        "\n",
        "Next let us look at the amount of times the scores agree/disagree in their sign for a quote:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 368,
      "id": "74BdYFbo2CMd",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "74BdYFbo2CMd",
        "outputId": "8563888f-c6a4-46e9-8ea3-694c033e7ac3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "VADER > 0 and TextBlob < 0: 2.84 %\n",
            "VADER < 0 and TextBlob > 0: 4.42 %\n",
            "VADER == 0 and TextBlob != 0: 6.62 %\n",
            "VADER != 0 and TextBlob == 0: 23.97 %\n",
            "VADER == 0: 34.38 %\n",
            "TextBlob == 0: 51.74 %\n"
          ]
        }
      ],
      "source": [
        "# Finally let us look at the number of times the two scores disagree in sign:\n",
        "\n",
        "print(f'VADER > 0 and TextBlob < 0: {((var1 > 0) & (var2 < 0)).sum()/len(var1)*100:.2f} %')\n",
        "print(f'VADER < 0 and TextBlob > 0: {((var1 < 0) & (var2 > 0)).sum()/len(var1)*100:.2f} %')\n",
        "print(f'VADER == 0 and TextBlob != 0: {((var1 == 0) & (var2 != 0)).sum()/len(var1)*100:.2f} %')\n",
        "print(f'VADER != 0 and TextBlob == 0: {((var1 != 0) & (var2 == 0)).sum()/len(var1)*100:.2f} %')\n",
        "\n",
        "print(f'VADER == 0: {((var1 == 0)).sum()/len(var1)*100:.2f} %')\n",
        "print(f'TextBlob == 0: {((var2 == 0)).sum()/len(var2)*100:.2f} %')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hm297Q5D3pCz",
      "metadata": {
        "id": "hm297Q5D3pCz"
      },
      "source": [
        "We see that the two scores disagree in sign about 37% of the time (2.84 + 4.42 + 6.62 + 23.97).\n",
        "\n",
        "Furthermore, TextBlob gives a value of zero over half the time suggesting it may not be as useful for identifying sentiment; a neutral sentiment for most statements is not particularly useful for distinguishing quotes. Alternatively, perhaps VADER is more sensitive and attributes sentiment where the statement truly is neutral.\n",
        "\n",
        "---\n",
        "\n",
        "Next we look at specific quotes that rank high or low in each of the sentiment scores, and consider the relationship with `label_ChatGPT` which is a proxy for our response for now:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 369,
      "id": "eLZnevAwnAoU",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eLZnevAwnAoU",
        "outputId": "8d796bd3-e9dd-47fa-c7ca-c6d1633741e8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Sentiment_VADER:\n",
            "Top 5 quotes (most positive)\n",
            "                                                                                                                                                                                                                               Movie Quote  \\\n",
            "347                                                                                                                                                           Hope is a good thing, maybe the best of things, and no good thing ever dies.   \n",
            "236  Men, you are about to embark on a great crusade to stamp out runaway decency in the west. Now you men will only be risking your lives, whilst I will be risking an almost certain Academy Award nomination for Best Supporting Actor.   \n",
            "877                                                                                                        My mother always tells me to smile and put on a happy face. She told me I had a purpose to bring laughter and joy to the world.   \n",
            "464                                                                                                                                                         The greatest thing youâ€™ll ever learn is just to love and be loved in return.   \n",
            "601                                                                                                                                                     But when you forgive, you love. And when you love, God's light shines through you.   \n",
            "\n",
            "     Sentiment_VADER         label_ChatGPT  \n",
            "347           0.8934  6. none of the above  \n",
            "236           0.9126           3. politics  \n",
            "877           0.9217  6. none of the above  \n",
            "464           0.9231  6. none of the above  \n",
            "601           0.9455           2. religion  \n",
            "Bottom 5 quotes (most negative)\n",
            "                                                                                                                                                                                     Movie Quote  \\\n",
            "292                                  Come on... Come on! Do it! Do it! Come on. Come on! Kill me! I'm here! Kill me! I'm here! Kill me! Come on! Kill me! I'm here! Come on! Do it now! Kill me!   \n",
            "213                                                                                                                              Life is a banquet, and most poor suckers are starving to death!   \n",
            "577  When you're looking for a cancer kid, he should be hopeless. He should have a wheelchair, he should have trouble talking, he should have a little pet goldfish in a Zip-lock bag, hopeless.   \n",
            "897                                                                          Bruce Lee: My hands are registered as lethal weapons. We get into a fight, I accidentally kill you... I go to jail.   \n",
            "378                                        You little son of a bitch ball! Why you don't you just go HOME? That's your HOME! Are you too good for your HOME? ANSWER ME! SUCK MY WHITE ASS, BALL!   \n",
            "\n",
            "     Sentiment_VADER         label_ChatGPT  \n",
            "292          -0.9812  6. none of the above  \n",
            "213          -0.9321  6. none of the above  \n",
            "577          -0.9201  6. none of the above  \n",
            "897          -0.9118  6. none of the above  \n",
            "378          -0.9007  6. none of the above  \n",
            "\n",
            "---------------\n",
            "Sentiment_TextBlob:\n",
            "Top 5 quotes (most positive)\n",
            "                                                                                                                                   Movie Quote  \\\n",
            "768  A machine doesn't improvise well because you cannot program a fear of death. Our survival instinct is our greatest source of inspiration.   \n",
            "489                                                                         Ye best start believin' in ghost stories, Miss Turner. Yer in one.   \n",
            "878                                                           How truly magnificent! I think this battle is about to come to a rousing finale!   \n",
            "492                                                                                                                    To me, you are perfect.   \n",
            "304                                                                                                                 Welcome to the party, pal!   \n",
            "\n",
            "     Sentiment_TextBlob         label_ChatGPT  \n",
            "768                 1.0  6. none of the above  \n",
            "489                 1.0  6. none of the above  \n",
            "878                 1.0  6. none of the above  \n",
            "492                 1.0  6. none of the above  \n",
            "304                 1.0           2. religion  \n",
            "Bottom 5 quotes (most negative)\n",
            "                                                                                                                                 Movie Quote  \\\n",
            "493                                                                              I will not say: do not weep; for not all tears are an evil.   \n",
            "634                                              Nobody panics when things go â€œaccording to planâ€_x009d_. Even if the plan is horrifying!   \n",
            "856  I remember the story of an evil witch, at the princess she cursed to sleep forever. The story became legend. But this is no fairy tale.   \n",
            "167                                                                                                                                  My bad!   \n",
            "805                                                                                          Marriage is buying a house for someone you hate   \n",
            "\n",
            "     Sentiment_TextBlob         label_ChatGPT  \n",
            "493              -1.000  6. none of the above  \n",
            "634              -1.000  6. none of the above  \n",
            "856              -1.000  6. none of the above  \n",
            "167              -0.875  6. none of the above  \n",
            "805              -0.800  6. none of the above  \n"
          ]
        }
      ],
      "source": [
        "def get_high_low_sentiment_quotes(data, sentiment_col):\n",
        "  # sort data frame based on sentiment score\n",
        "  data_sorted = data.sort_values(by = sentiment_col)\n",
        "\n",
        "  # get top 5 and bottom 5 quotes\n",
        "  bottom_5_quotes = data_sorted.iloc[:5][['Movie Quote', sentiment_col, 'label_ChatGPT']]\n",
        "  top_5_quotes = data_sorted.iloc[-5:][['Movie Quote', sentiment_col, 'label_ChatGPT']]\n",
        "\n",
        "  print(f'{sentiment_col}:')\n",
        "  print('Top 5 quotes (most positive)')\n",
        "  print(top_5_quotes)\n",
        "\n",
        "  print('Bottom 5 quotes (most negative)')\n",
        "  print(bottom_5_quotes)\n",
        "\n",
        "\n",
        "\n",
        "get_high_low_sentiment_quotes(train, 'Sentiment_VADER')\n",
        "print()\n",
        "print('---------------')\n",
        "get_high_low_sentiment_quotes(train, 'Sentiment_TextBlob')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PylPgsJ6o-nZ",
      "metadata": {
        "id": "PylPgsJ6o-nZ"
      },
      "source": [
        "The above output shows the top 5 and bottom 5 ranked quotes according to `Sentiment_VADER` and `Sentiment_TextBlob`. There does not appear to be any cross-over between the top/bottom quotes for each sentiment score.\n",
        "\n",
        "Notably, all of the most negative quotes are labelled as '6. none of the above' by `label_ChatGPT`. We might have expected some relationship between a quote having a low sentiment score and a label representing one of the problematic categories however this does not appear to be the case. Some of the bottom quotes for `Sentiment_VADER` likely should be labelled '4. rude' as they contain swearing, suggesting that `label_ChatGPT` may be unreliable. However, for the most part the quotes with low `Sentiment_VADER` are correctly labelled as '6. none of the above' (based on our judgement). For `Sentiment_TextBlob`, the bottom quotes do not appear particularly problematic and so `label_ChatGPT` appears accurate in these cases.\n",
        "\n",
        "Overall, it is difficult to discern a clear relationship between the sentiment scores and the ChatGPT label. That is not say that there is not a relationship but it is ambiguous from the examples given."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FJFE0kWa5b3_",
      "metadata": {
        "id": "FJFE0kWa5b3_"
      },
      "source": [
        "**Summary**\n",
        "- No missing values or values outside the expected range for our two sentiment scores\n",
        "- `Sentiment_VADER` has greater variance than `Sentiment_TextBlob` in our dataset.\n",
        "- Before scores appear to be negatively skewed in our dataset (more positive samples)\n",
        "- `Sentiment_TextBlob` is more concentrated around zero and has limited negative observations. Over 50% of the time it has a value of zero.\n",
        "- The two measures do not always agree in their sentiment (~ 37% of the time). However they are positively related which is expected given they both measure sentiment.\n",
        "- It is unclear what the relationship between the sentiment scores and `label_ChatGPT` is at this stage. The relationship may be complex and non-linear, indicating that a neural network may be an appropriate modelling technique."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kw67nSmL9cUo",
      "metadata": {
        "id": "kw67nSmL9cUo"
      },
      "source": [
        "#### Q1a Step 8 - EDA: label_ChatGPT\n",
        "**Outline**\n",
        "\n",
        "In this section we conduct EDA on `label_ChatGPT` which will later be used to construct a response variable.\n",
        "\n",
        "The data dictionary indicates that this column should take 1 of 6 unique values. We attempt to verify this here as well as determine the reliability of the labels.\n",
        "\n",
        "---\n",
        "Let us first look at the cardinality of the variable and the distribution of the different categories:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 370,
      "id": "8A5lUOPZ9z2T",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8A5lUOPZ9z2T",
        "outputId": "025091a8-b0f8-418d-829d-4cc75bc7e000"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.series.Series'>\n",
            "Index: 634 entries, 862 to 10\n",
            "Series name: label_ChatGPT\n",
            "Non-Null Count  Dtype \n",
            "--------------  ----- \n",
            "633 non-null    object\n",
            "dtypes: object(1)\n",
            "memory usage: 9.9+ KB\n",
            "\n",
            "There are 21 unique label_ChatGPTs in the dataset (from 634 rows)\n",
            "Total count and percentage of the top 10 label_ChatGPTs:  \n",
            "\n",
            "\n",
            "                              Count Percentage\n",
            "label_ChatGPT                                 \n",
            "6. none of the above            562     88.78%\n",
            "3. politics                      19       3.0%\n",
            "4. rude                           9      1.42%\n",
            "2. religion                       8      1.26%\n",
            "6. None of the above              8      1.26%\n",
            "1. sex                            5      0.79%\n",
            "1. None of the above              4      0.63%\n",
            "5. insensitive                    3      0.47%\n",
            "1. none of the above              2      0.32%\n",
            "Answer: 6. none of the above      2      0.32%\n"
          ]
        }
      ],
      "source": [
        "train['label_ChatGPT'].info()\n",
        "print()\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT', 10)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 371,
      "id": "JPscL2AdPsPK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "JPscL2AdPsPK",
        "outputId": "37261697-c8a5-4080-b8f4-01d2a926968e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7bf474be36d0>"
            ]
          },
          "execution_count": 371,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 53156 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# create a pie chart of label_ChatGPT to understand distribution of labels\n",
        "fig, ax = plt.subplots()\n",
        "\n",
        "totals_chatGPT = train[\"label_ChatGPT\"].value_counts()\n",
        "wedges, texts, autotexts = ax.pie(totals_chatGPT, labels=totals_chatGPT.index, autopct='%1.1f%%')\n",
        "\n",
        "# don't show labels or percentage labels for categories with less than 1%\n",
        "threshold = 0.8\n",
        "for wedge, label, pct_label in zip(wedges, texts, autotexts):\n",
        "    pct_value = pct_label.get_text().rstrip('%')\n",
        "    if float(pct_value) < threshold:\n",
        "        label.set_text('') # remove text label\n",
        "        pct_label.set_text('') # remove % label\n",
        "        wedge.set_label('') # remove from legend\n",
        "\n",
        "ax.legend(bbox_to_anchor=(1.2, 1))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 372,
      "id": "V1KeAmV9yStZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V1KeAmV9yStZ",
        "outputId": "1b097297-fe76-43cd-9081-4b4cf483f4ac"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "We have 1 NAs for the label\n",
            "The quote with NA label:\n",
            "              Movie Quote  \\\n",
            "100  Very nice, how much?   \n",
            "\n",
            "                                                                                  Movie Name  \n",
            "100   BORAT: CULTURAL LEARNINGS OF AMERICA FOR MAKE BENEFIT\\n. GLORIOUS NATION OF KAZAKHSTAN  \n"
          ]
        }
      ],
      "source": [
        "# look at NAs in the label:\n",
        "print()\n",
        "print(f'We have {train[\"label_ChatGPT\"].isna().sum()} NAs for the label')\n",
        "print('The quote with NA label:')\n",
        "print(train.loc[train[\"label_ChatGPT\"].isna(), ['Movie Quote', 'Movie Name']])\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 373,
      "id": "oEVKlowdy2FC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oEVKlowdy2FC",
        "outputId": "1838d74a-fb8b-4af4-a1b1-de67cddb9c69"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The longest chatGPT label is: \n",
            "Longest string: The answer is '6. none of the above'. The quote is not about sex, religion, politics, rudeness, or insensitivity. It is about the inability to let go of something important, and how it can hinder one's ability to bring about change.\n"
          ]
        }
      ],
      "source": [
        "print('The longest chatGPT label is: ')\n",
        "print(f'Longest string: {max(train.loc[[not x for x in train[\"label_ChatGPT\"].isna()], \"label_ChatGPT\"], key=len)}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gqnWfmQH_kzt",
      "metadata": {
        "id": "gqnWfmQH_kzt"
      },
      "source": [
        "Although we expected 6 unique values for this column, there are 20. There are some labels that are close to the correct format such as 'Answer: 6. none of the above'.  These will need to be cleaned at a later stage so that we have 6 categories as expected.\n",
        "\n",
        "It is also clear that the data is extremely imbalanced with 88% of the data being labelled as '6. none of the above'. This percentage does not even include the unclean versions of this category! Similarly the other categories such as 'religion' and 'politics' contain only a few quotes each. Using these values to flag a quote as 'problematic' will lead to a **highly imbalanced response**. This must be addressed later in our modelling. Furthermore, categories like 'sex' contain only 5 observations in our training data; this topic is extremely nuanced and 5 quotes is unlikely to provide enough coverage for a model to understand. This may result in a model that does not generalise well.\n",
        "\n",
        "Additionally, we have identified a missing value for the label too. We can either remove this entry or impute the label value later. The quote is from the movie 'Borat'. Fixing this value will come at a later stage.\n",
        "\n",
        "One label is extremely long, although does appear to specify '6. none of the above'.\n",
        "\n",
        "Next let us look at some examples of quotes in each of the categories, according to the data dictionary (e.g. '2. religion', '3. politics')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 374,
      "id": "SOPwsEPEAsai",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SOPwsEPEAsai",
        "outputId": "5e1a3e85-3b81-4ee0-fbff-dbfac65615dd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "264                                               You must unlearn what you have learned.\n",
            "845                                       The hardest choices require the strongest wills\n",
            "2                                                                \"I can do this all day.\"\n",
            "137                                                    THE POWER OF CHRIST COMPELS YOU!!!\n",
            "298                                                           Death cannot stop true love\n",
            "698                                                We accept the love we think we deserve\n",
            "304                                                            Welcome to the party, pal!\n",
            "601    But when you forgive, you love. And when you love, God's light shines through you.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Let's look at some 'problematic' quotes:\n",
        "\n",
        "# for example let us look at a so-called 'religious' quotes\n",
        "print(train.loc[train['label_ChatGPT'] ==  '2. religion']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ol3sB1IiBVsq",
      "metadata": {
        "id": "ol3sB1IiBVsq"
      },
      "source": [
        "Although some of these quotes are religious, we believe a human would argue that not all of them are (for example \"I can do this all day\"). This highlights a potential unreliability of using chatGPT alone to construct a response variable.\n",
        "\n",
        "If we do just use ChatGPT, we may fall victim to 'garbage in, garbage out' and create a predictive model that is unreliable. In particular, it may lead to more false positives (i.e. flagging a quote as problematic when it isn't). This may lead to delays in the production process and potentially missed opportunities with actors!\n",
        "\n",
        "---\n",
        "\n",
        "Next we look at quotes that have been tagged as 'political':"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 375,
      "id": "BHpP6ntfCQ6-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BHpP6ntfCQ6-",
        "outputId": "84ff93d2-2022-4ae3-e366-5b6b619da158"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "804                                                                                                                                                       Bombs won't stop terrorism, brains will, and we don't have nearly enough of those.\n",
            "562                                                                                                                                                                                               Float like a cadillac sting like a beemer.\n",
            "344                                                                                                                                                                       Lord, it's a miracle! Man up and vanished like a fart in the wind!\n",
            "830                                                                                                                              I have burrowed underneath your brain. I am nested there. I am the scream in your mind. You will cooperate.\n",
            "600                                                                                                           Some people feel like they don't deserve love. They walk away quietly into empty spaces, trying to close the gaps of the past.\n",
            "508                                                                                                                                                                                                      We should totally just stab Caesar!\n",
            "506                                                                                                                                                                            Stop trying to make 'fetch' happen. It's not going to happen.\n",
            "468                                                                       Itâ€™s a dangerous business, going out your doorâ€¦You step into the Road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.\n",
            "750                                                                                                                                                                     The price of freedom is high, and it's the price I am willing to pay\n",
            "236    Men, you are about to embark on a great crusade to stamp out runaway decency in the west. Now you men will only be risking your lives, whilst I will be risking an almost certain Academy Award nomination for Best Supporting Actor.\n",
            "539                                                                                                                                                                                                     You know how I know that you're gay?\n",
            "208                                                                                                                                                                                        When it comes to women, youâ€™re a true democrat.\n",
            "129                                                                                                                                                                                                                         HOLY SCHNIKES!!!\n",
            "826                                                                                                                                                           Every time we get a chance to get ahead they move the finish line, every time.\n",
            "536                                                                                                                                                                                                              I am the Jewish James Bond!\n",
            "667                                                                                                                                                            When you play game of thrones, you win or you die, there is no middle ground!\n",
            "542                                                                                                                                                                                                                         I am the senate!\n",
            "711                                                                                                                                                                             Proximity to power deludes some into thinking they wield it.\n",
            "220                                                                                                                                                                                Gentlemen, you can't fight in here! This is the War Room!\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# now let us look at 'political' quotes\n",
        "print(train.loc[train['label_ChatGPT'] ==  '3. politics']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DhLVbhoHCnMn",
      "metadata": {
        "id": "DhLVbhoHCnMn"
      },
      "source": [
        "Again many of these quotes do appear to be political but some are a stretch: 'Oh, behave!'\n",
        "\n",
        "This again highlights the unreliability of this label which is a concern for future modelling stages.\n",
        "\n",
        "---\n",
        "\n",
        "Next we explore the quotes that have been labelled as relating to 'sex':"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 376,
      "id": "Ol3CzpnqC8QV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ol3CzpnqC8QV",
        "outputId": "c4e82409-d92a-4a67-975f-509671461e40"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "901                                                  I used to use this little gun when I was a prostitude.\n",
            "666    The problem is, I pretty much fancy everyone. It's like I've just got out of prison all of the time.\n",
            "159                                                             Why don't you come up sometime, and see me?\n",
            "7                                                                          \"I wish I knew how to quit you.\"\n",
            "794                                                                            My tastes are very singular.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# now let us look at 'sex' quotes\n",
        "print(train.loc[train['label_ChatGPT'] ==  '1. sex']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DU44ZirRDH4P",
      "metadata": {
        "id": "DU44ZirRDH4P"
      },
      "source": [
        "Slightly more accurate than other labels, but certainly not correct for all of them. For example, \"I wish I knew how to quit you\" may be a reference to love not sex.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "Next we look at quotes that have been labelled as 'none of the above' but contain the key word 'God', which based on our domain knowledge may be associated with 'religion'. We find two quotes in our training data that match this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 377,
      "id": "hEDZFJOpEN80",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hEDZFJOpEN80",
        "outputId": "fe320a05-9e0d-4428-8896-354929324e09"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "853    For I am a Sinner in the hands of an angry God, Bloody Mary, Full of Vodka, Blessed are you among cocktails, pray for me now, at the hour of my death, which I hope is soon. Amen\n",
            "201                                                                                                                                    As God is my witness, I'll never be hungry again.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Let's also look for potential 'problematic' quotes that have been labelled as '6. none of the above'\n",
        "print(train.loc[(train['label_ChatGPT'] == '6. none of the above') & (train['Movie Quote'].str.contains('God'))]['Movie Quote'])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MzalX0ybEMj3",
      "metadata": {
        "id": "MzalX0ybEMj3"
      },
      "source": [
        "We identify two quotes that contain God but are not labelled as religious. I think a reasonable reviewer would label both these quotes as having religious connotations. This again highlights the unreliability of `label_ChatGPT`.\n",
        "\n",
        "If left as they are, observations like this in our data may lead to a higher false negative rate in our predictive model. This could lead to the movie production company breaking its contractual stipulations with actors if quotes are not properly flagged with actors. Consequently, this may lead to expensive reshoots due to recasting, delays in production, legal payouts and poor relationships with actors."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OMCmaQdmDU3q",
      "metadata": {
        "id": "OMCmaQdmDU3q"
      },
      "source": [
        "**Summary**\n",
        "- `label_ChatGPT` is highly imbalanced with over 88% of quotes belonging to the category '6. non of the above'.\n",
        "- `label_ChatGPT` contains more than the intended 6 categories. This will need to be cleaned at a later stage.\n",
        "- The label is often unreliable. Manual adjustments may be needed else we fall victim to 'garbage in, garbage out'. Otherwise, we may see high error rates for our predictive model. For example, high false positives of 'problematic' quotes may slow down production and result in missed opportunities with actors.\n",
        "- Similarly, incorrect labels of quotes as '6. none of the above' may lead to higher false negative rates in our predictive model at deployment. Possible implications of FNs include expensive reshoots due to recasting, delays in production, legal payouts to actors and poor relationships with actors. FNs are very risky/costly in this context!"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BL6yMHrJC8Ap",
      "metadata": {
        "id": "BL6yMHrJC8Ap"
      },
      "source": [
        "#### Q1a Step 9 - EDA: Movie Quotes\n",
        "**Outline**\n",
        "\n",
        "In this section we conduct EDA on the movie quotes. This will help to inform future cleaning steps and possible feature engineering that will assist in the accuracy of our predictive model. The quotes are the main piece of data that we will conduct our NLP analysis on. Our aim for this exploration stage is the following:\n",
        "\n",
        "- Identify possible cleaning needed and potential issues with the quotes.\n",
        "- Understand common words which may later inform stop word selection.\n",
        "- Consider what features engineering could be done with the quotes such as by looking at the use of punctuation (e.g. '!' and '?') or word count."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 378,
      "id": "VWlwHu79IYW1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VWlwHu79IYW1",
        "outputId": "40a8ab03-12bb-48e3-8d69-0c34702374af"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "917                                                                                I didn't know if you were lost. Stick with me. I'll keep you safe.\n",
              "521                                                                                       Your identity is your most valuable possession. Protect it.\n",
              "464                                                                    The greatest thing youâ€™ll ever learn is just to love and be loved in return.\n",
              "855                                                                                     Elsa, the past is not what it seems. You must find the truth.\n",
              "16                                                                                                                          \"You're a wizard, Harry.\"\n",
              "5                                                          \"Life is not the amount of breaths you take, it's the moments that take your breath away.\"\n",
              "527                                                                                                   There's a 30% chance that it's already raining.\n",
              "96                                                                                                                      Ding-dong, the witch is dead!\n",
              "399                                                                        The flower that blooms in adversity is the most rare and beautiful of all.\n",
              "342    Yeah I called her up. She gave me a bunch of crap about me not listening to her, or something. I don't know, I wasn't really paying attention.\n",
              "870                                                                                                      Thereâ€™s nothing you canâ€™t do if you try.\n",
              "713                                                               The only thing we're allowed to believe is that we won't regret the choice we made.\n",
              "309                     You must strive to find your own voice because the longer you wait to begin, the less likely you are going to find it at all.\n",
              "170                                                                                                          Hey! I'm walking here! I'm walking here!\n",
              "657      An idea is like a virus. Resilient. Highly contagious. And even the smallest seed of an idea can grow. It can grow to define or destroy you.\n",
              "Name: Movie Quote, dtype: object"
            ]
          },
          "execution_count": 378,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Let's look at a few random samples from the data:\n",
        "train['Movie Quote'].sample(n=15, random_state=random_seed)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c50DwK0oI84T",
      "metadata": {
        "id": "c50DwK0oI84T"
      },
      "source": [
        "- We see corrupted characters like 'â€™' which appears to mistakenly represent an apostrophe. Other meaningless special character combinations also exist in the dataset. These may be from incorrect character encodings.\n",
        "- Some quotes are surrounding by quotation marks while some are not\n",
        "- Quotes often contain several phrases or sentences that are related, rather than just a single sentence."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fCLQ6QNsX4SC",
      "metadata": {
        "id": "fCLQ6QNsX4SC"
      },
      "source": [
        "Next let us look at the word count in our quotes and their distribution:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 379,
      "id": "v1amyzhdJ-dB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "v1amyzhdJ-dB",
        "outputId": "7bcaf05b-1b90-4c35-b7af-4ee953007fa0"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 22388 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Now let's look at the average length and word count of quotes\n",
        "train['word_count'] = train['Movie Quote'].apply(lambda x: len(x.split()))\n",
        "\n",
        "sns.histplot(train['word_count'])\n",
        "plt.xlabel('Word count')\n",
        "plt.ylabel('Number of quotes')\n",
        "plt.title('Distribution of quote word counts')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 380,
      "id": "vC1Vnkz6KXei",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vC1Vnkz6KXei",
        "outputId": "cff4dd6f-63e8-45d3-c2f1-88b4ebaf501e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Minimum number of words: 1\n",
            "Maximum number of words: 64\n",
            "Average number of words: 12.3\n"
          ]
        }
      ],
      "source": [
        "print(f'Minimum number of words: {min(train[\"word_count\"])}')\n",
        "print(f'Maximum number of words: {max(train[\"word_count\"])}')\n",
        "print(f'Average number of words: {np.mean(train[\"word_count\"]):.1f}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 381,
      "id": "mc2Rq-zELgf2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mc2Rq-zELgf2",
        "outputId": "b32d9437-90ea-41d7-a9e7-dadf4cc52da4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "55     Shall...we...play...a...game?\n",
            "180                         Rosebud.\n",
            "50                         Khaaaaan!\n",
            "154                         Schwing!\n",
            "160              Ssssssssssssmokin'!\n",
            "592                         Bazinga!\n",
            "117                          Hoo-ah!\n",
            "172                     Uhh....yeah.\n",
            "442      Snnnnnooooooooooow!!!!!!!!!\n",
            "104                           Bogus.\n",
            "103                       Excellent!\n",
            "678                            Hodor\n",
            "Name: Movie Quote, dtype: object\n",
            "439    My name is Maximus Decimus Meridius, commander of the Armies of the North, General of the Felix Legions and loyal servant to the true emperor, Marcus Aurelius. Father to a murdered son, husband to a murdered wife. And I will have my vengeance, in this life or the next.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# look at the shortest and longest quote\n",
        "print(train.loc[train['word_count'] ==  1]['Movie Quote'])\n",
        "print(train.loc[train['word_count'] ==  49]['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nNzvUG7RLuvX",
      "metadata": {
        "id": "nNzvUG7RLuvX"
      },
      "source": [
        "The average quote contains about 12 words and we see that the number of quotes with a particular word count tends to decrease with the number of words. This may be expected as shorter quotes are easier to remember and so more likely to become famous.\n",
        "\n",
        "There are several one-word quotes. Many of them represent a catchphrase or a characters name. Notably, some of them include repetitive letters such as \"Sssssmokin'!\". These would need to be removed to be recognised as real words by any vectorisation algorithm.\n",
        "\n",
        "Furthermore, some one-word quotes are not truly one word but rather there are ellipses between words. This will need to be cleaned later too.\n",
        "\n",
        "---\n",
        "Let us next look at the most common words in our quotes. This may help inform possible domain specific stop words later or important words in our quotes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 382,
      "id": "tAH2lS_VNCcr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tAH2lS_VNCcr",
        "outputId": "d4dc98bf-d77c-4dcb-ab2a-bd786a782885"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most common words:\n",
            "the : 237\n",
            "you : 207\n",
            "to : 203\n",
            "a : 189\n",
            "I : 182\n",
            "is : 135\n",
            "of : 109\n",
            "and : 106\n",
            "in : 78\n",
            "it : 66\n",
            "be : 64\n",
            "You : 57\n",
            "that : 57\n",
            "your : 57\n",
            "not : 56\n",
            "have : 53\n",
            "my : 52\n",
            "for : 52\n",
            "are : 48\n",
            "what : 44\n",
            "do : 42\n",
            "can : 40\n",
            "we : 38\n",
            "like : 37\n",
            "The : 37\n",
            "just : 37\n",
            "I'm : 37\n",
            "know : 35\n",
            "as : 34\n",
            "all : 33\n"
          ]
        }
      ],
      "source": [
        "# Next let us look at the most common words -> this may help inform possible stop words or important words\n",
        "\n",
        "# Tokenize the strings into words and count their occurrences\n",
        "word_counts = Counter()\n",
        "for text in train['Movie Quote']:\n",
        "    words = text.split()\n",
        "    word_counts.update(words)\n",
        "\n",
        "# Select the most common words\n",
        "most_common_words = word_counts.most_common(30)  # Select top 20 most common words\n",
        "print(\"Most common words:\")\n",
        "for word, count in most_common_words:\n",
        "    print(word, \":\", count)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pOpdLBBcN5R2",
      "metadata": {
        "id": "pOpdLBBcN5R2"
      },
      "source": [
        "The most common words do not appear to be domain specific and are mostly generic stop words. Suggests we do not need to add any additional words to our stop word corpus later.\n",
        "\n",
        "Next we look at the use of punctuation in quotes. This may be used for feature engineering later since punctuation could relate to the tone of a quote and hence whether it is problematic or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 383,
      "id": "3K3fU6vLOKCq",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3K3fU6vLOKCq",
        "outputId": "7296b395-ed72-4e0e-9c5b-941c64bdae8a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Percentage of quotes containing a question mark: 13.41%\n",
            "Percentage of quotes containing an exclamation mark: 20.98%\n"
          ]
        }
      ],
      "source": [
        "# Finally look at punctuation use in the quotes\n",
        "question_perc = train[\"Movie Quote\"].str.contains(\"\\?\").sum()/train.shape[0]*100\n",
        "exclamation_perc = train[\"Movie Quote\"].str.contains(\"!\").sum()/train.shape[0]*100\n",
        "print(f'Percentage of quotes containing a question mark: {question_perc:.2f}%')\n",
        "print(f'Percentage of quotes containing an exclamation mark: {exclamation_perc:.2f}%')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ECAHYyFyPHt9",
      "metadata": {
        "id": "ECAHYyFyPHt9"
      },
      "source": [
        "A signficant proportion of quotes contain either a '?' or '!'. The presence of punctuation such as an exclamation mark may be indicative of a quote being rude, for example. We may hence consider creating a feature that flags whether an exclaimation mark is present.\n",
        "\n",
        "---\n",
        "Next as a check on our quotes, we look at quotes that do not end with some form of punctuation. We might expect that quotes should end with punctuation if the grammar is correct. A quote that does not end with some form of punctuation may indicate a possible error such as a quote being cutoff:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 384,
      "id": "h1f4J31kPi5z",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h1f4J31kPi5z",
        "outputId": "41e59b47-252a-4047-b372-33c3727dff23"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "You only live twice:Once when you are born And once when you look death in the face\n",
            "The law says that you cannot touch. But I think I see a lot of lawbreakers up in this house tonight\n",
            "Not all treasure is silver and gold mate\n",
            "It's the loneliest feeling not to know who you are\n",
            "A bit of madness is key\n",
            "Manners maketh man\n",
            "ZaldrÄ«zes buzdari iksos daor! (A dragon is not a slave!)\n",
            "A Lannister always pays his debts\n",
            "It is impossible to manufacture or imitate love\n",
            "The people who are crazy enough to think that they can change the world, are the one's who do\n"
          ]
        }
      ],
      "source": [
        "# We also look for quotes that do not end with some form of punctuation : . ! ? \"\n",
        "# This is because it may suggest an error in the quote such as part of it being cutoff\n",
        "\n",
        "# Define regular expression pattern checking if the end of string is not . ! ? \" : ;\n",
        "pattern = r'[^.!?:;\"\\']+$'\n",
        "\n",
        "# Extract quotes that do not end with . ? ! or \"\n",
        "extracted_strings = [string for string in train['Movie Quote'] if re.search(pattern, string)]\n",
        "\n",
        "# Print the first 10 extracted strings\n",
        "for string in extracted_strings[0:10]:\n",
        "  print(string)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fr1f_oRgSU0n",
      "metadata": {
        "id": "fr1f_oRgSU0n"
      },
      "source": [
        "It appears that quotes that do not end with any punctuation are okay and have not been mistakenly cutoff."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "glpihBRRSdUd",
      "metadata": {
        "id": "glpihBRRSdUd"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "- In this section we identified some possible parts of the quotes that need to be cleaned such as special characters, repetitive letters and quotation marks\n",
        "- Identified possible features that could be created to help the business better identify problematic quotes (e.g. use of exclamation marks)\n",
        "- Looked at the length of quotes and the most common words"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jXrVrV_7SppK",
      "metadata": {
        "id": "jXrVrV_7SppK"
      },
      "source": [
        "####**Overall summary for EDA**  \n",
        "\n",
        "In this first section we examined the dataset and its features while considering their relevance to our business objective. Some preliminary cleaning was done and the entire dataset was split into training, validation and test sets.\n",
        "\n",
        "The details are left to the relevant sections but some key points from our analysis include:\n",
        "\n",
        "- No clear evidence of data drift, suggesting that the entire dataset is relevant for informing decisions about screenwriting today and creating a classifier for problematic quotes.\n",
        "- The variables `Movie Name`, `Actor`, `Character`, `Director` and `Writer` were deemed to be irrelevant to our two main problems. This is for a few key reasons:\n",
        "  1. They primarily represent external factors beyond a writer's control, not the inherent qualities of a quote that contribute to its fame or potential issues.\n",
        "  2. They would not be available/known during scriptwriting (deployment time)\n",
        "  3. They had high cardinality so using them would lead to high variance error in our results.\n",
        "- The `label_ChatGPT` column is unreliable and using it, even after some cleaning, is likely to lead to poor modelling results (garbage in, garbage out). This unreliability may lead to high FP or FN rates in our model at deployment, both of which have costs for the movie production company. Additional, analysis and packages may be needed to complement this column when constructing a response variable. This column is also highly imbalanced which will have consequences during our modelling later; some method to address this imbalance should be employed.\n",
        "- The two sentiment score columns are positively related. `Sentiment_VADER` appears more likely to attribute negative sentiment to quotes.\n",
        "- Data quantity is a key issue (our training set only has 639 entries) which may limit the complexity of the insights we can extract with regard to famous quote characteristics.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yNelfxy5UQLm",
      "metadata": {
        "id": "yNelfxy5UQLm"
      },
      "source": [
        "### Q1b - Clean the data\n",
        "#### Purpose:\n",
        "\n",
        "In this section we perform several cleaning steps. Many of these relate to the `Movie Quote` variable which forms the crux of our analysis. They are needed to standardise the text so we can focus on the most informative parts of the text and correctly vectorise it later.\n",
        "The following cleaning steps were taken for the `Movie Quote` variable:\n",
        "\n",
        "- Fixing quote spillage\n",
        "- Lower casing\n",
        "- Substituting and removing corrupted characters\n",
        "- Expanding contractions and shortened words\n",
        "- Removing punctuation\n",
        "- Correcting spelling mistakes\n",
        "- Tokenisation\n",
        "- Remove stop words\n",
        "- Lemmatisation\n",
        "\n",
        "Also `Genre` was split out, cleaned/grouped and converted to new variables.\n",
        "\n",
        "#### Cleaning done in part 1a\n",
        "- Cleaned `Year`: convert to numeric and used regex to extract year from ranges. Spot checks were done.\n",
        "- Removed duplicate quotes using fuzzy matching. Extensive checks were performed.\n",
        "\n",
        "\n",
        "#### References:\n",
        "For verifying spillage:\n",
        "\n",
        "*Airplane! (1980) - Quotes - IMDb. (n.d.). Www.imdb.com. Retrieved March 30, 2024, from https://www.imdb.com/title/tt0080339/quotes/*\n",
        "\n",
        "*Dumb and Dumber (1994) - Quotes - IMDb. (n.d.). Www.imdb.com. https://www.imdb.com/title/tt0109686/quotes/*\n",
        "\n",
        "For encoding mapping:\n",
        "\n",
        "*HTML Windows-1252 Reference. (n.d.). Www.w3schools.com. Retrieved March 30, 2024, from https://www.w3schools.com/charsets/ref_html_ansi.asp*\n",
        "\n",
        "For quote correction:\n",
        "\n",
        "*Dragon Ball Super: Broly (2018) - Quotes - IMDb. (n.d.). Www.imdb.com. Retrieved March 30, 2024, from https://www.imdb.com/title/tt7961060/quotes/?item=qt4471760*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "x4idAyiJWGaT",
      "metadata": {
        "id": "x4idAyiJWGaT"
      },
      "source": [
        "#### Q1b Step 1 - Fixing quotes that have spilled into `Movie Name`\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "During EDA, we identified quotes that accidentally spilled over into the `Movie Name` column. To ensure all quote data is captured, we concatenated the spilled content with the `Movie Quote` column. This step is applied first so that all subsequent cleaning steps are applied consistently to the *entirety* of all quotes.\n",
        "\n",
        "***How:***\n",
        "\n",
        "Since automatic spillage detection for all scenarios is impractical, we implemented a heuristic approach. This method checks for the presence of quotation marks in the `Movie Name` column, assuming they likely indicate spilled quotes. While this might miss some edge cases, it's a reasonable compromise considering most quotes are unaffected by spillage.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "- **Data completeness:** We want to retain all relevant `Movie Quote` information for training our Neural Network (NN) and clustering model, especially considering our limited data.\n",
        "- **Consistency for modelling and future cleaning steps:** Cleaning spillage now ensures the `Movie Quote` column is consistent and avoids issues if the spillage remained unaddressed. Ensures future cleaning steps will be applied to the entirety of all quotes, rather than reintroducing noise if the spillage is addressed later.\n",
        "\n",
        "---\n",
        "First let us print rows that may contain spillage based on our heuristic:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 385,
      "id": "bCGilwLSWGwY",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bCGilwLSWGwY",
        "outputId": "bc5e7202-e483-47d9-b71a-7747b285e59c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                       Movie Quote  \\\n",
            "166                        SON OF    \n",
            "65   Surely, you can't be serious.   \n",
            "63                    Who are you?   \n",
            "\n",
            "                                      Movie Name  \n",
            "166              ... That's gonna leave a mark.\"  \n",
            "65    \"I am serious, and don't call me Shirley.\"  \n",
            "63                       \"Your worst nightmare.\"  \n"
          ]
        }
      ],
      "source": [
        "# First let us print rows will 'spillage' in the training set\n",
        "# We identify spillage by looking for movie names that contain quotation marks\n",
        "print(train.loc[train['Movie Name'].notna() & train['Movie Name'].str.contains('\"'), ['Movie Quote', 'Movie Name']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7zmRrhu9Yn3s",
      "metadata": {
        "id": "7zmRrhu9Yn3s"
      },
      "source": [
        "The training data contains three observations where the movie name contains a quotation mark. They indeed appear to be continuations of quotes rather than movie names i.e. there has been spillage.\n",
        "\n",
        "We cross-reference these quotes via the internet to confirm that they have indeed spilled into the second column. Two of the quotes are from the movies  'Airplane!' and 'Dumb and Dumber'. However, the third quote comes from an 'Unknown' movie; likely because it is commonly said in a range of movies. The movie that a quote came from is unimportant so we correct this quote as well.\n",
        "\n",
        "We correct the quotes column by concatenating the two strings. The movie name is also substituted with the correct movie, although this is not strictly necessary.\n",
        "\n",
        "As a sense check we print the resulting quotes from the three quotes above after the changes, as well as a quote that does not appear to have spillage. This will verify that the change has been made correctly and not affected unintended rows.\n",
        "\n",
        "\n",
        "**References:**\n",
        "\n",
        "Airplane! (1980) - Quotes - IMDb. (n.d.). Www.imdb.com. Retrieved March 30, 2024, from https://www.imdb.com/title/tt0080339/quotes/\n",
        "\n",
        "Dumb and Dumber (1994) - Quotes - IMDb. (n.d.). Www.imdb.com. https://www.imdb.com/title/tt0109686/quotes/"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "N837WQKVOv-G",
      "metadata": {
        "id": "N837WQKVOv-G"
      },
      "source": [
        "**Apply change and do sense check:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 386,
      "id": "Ucm1mI5vak8z",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ucm1mI5vak8z",
        "outputId": "b879c473-dc42-4b19-fc96-6945aab5216c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "After fix:\n",
            "                                                                   Movie Quote  \\\n",
            "65   Surely, you can't be serious.  \"I am serious, and don't call me Shirley.\"   \n",
            "166                                   SON OF   ... That's gonna leave a mark.\"   \n",
            "63                                       Who are you?  \"Your worst nightmare.\"   \n",
            "502                                   Not all treasure is silver and gold mate   \n",
            "\n",
            "                   Movie Name           Actor     Character     Year  \\\n",
            "65                  Airplane!  Leslie Nielsen    Dr. Rumack     1980   \n",
            "166           Dumb and Dumber    Jeff Daniels   Harry Dunne     1994   \n",
            "63                    Unknown         Unknown       Unknown  Unknown   \n",
            "502  Pirates of the Carribean     Johnny Depp  Jack Sparrow     2003   \n",
            "\n",
            "                          Genre                                  Director  \\\n",
            "65                       Comedy  Jim Abrahams, David Zucker, Jerry Zucker   \n",
            "166                      Comedy                  Peter and Bobby Farrelly   \n",
            "63                      Unknown                                   Unknown   \n",
            "502  Action, Adventure, Fantasy                            Gore Verbinski   \n",
            "\n",
            "                                       Writer    Production Company  \\\n",
            "65   Jim Abrahams, David Zucker, Jerry Zucker    Paramount Pictures   \n",
            "166                  Peter and Bobby Farrelly       New Line Cinema   \n",
            "63                                    Unknown               Unknown   \n",
            "502                  Ted Elliot, Terry Rossio  Walt Disney Pictures   \n",
            "\n",
            "     Sentiment_VADER  Sentiment_TextBlob         label_ChatGPT  Year_clean  \\\n",
            "65            0.4805            0.083333  6. none of the above        1980   \n",
            "166           0.0000            0.000000  6. none of the above        1994   \n",
            "63            0.0000            0.000000  6. none of the above        <NA>   \n",
            "502          -0.2235            0.000000  6. none of the above        2003   \n",
            "\n",
            "                      Genre_split  word_count  \n",
            "65                       [Comedy]           5  \n",
            "166                      [Comedy]           2  \n",
            "63                      [Unknown]           3  \n",
            "502  [Action, Adventure, Fantasy]           8  \n"
          ]
        }
      ],
      "source": [
        "# Fix the spillage in the training set\n",
        "condition = train['Movie Name'].notna() & train['Movie Name'].str.contains('\"')\n",
        "train.loc[condition, 'Movie Quote'] = train.loc[condition, 'Movie Quote'] + ' ' + train['Movie Name']\n",
        "\n",
        "train.loc[65, 'Movie Name'] = 'Airplane!'\n",
        "train.loc[166, 'Movie Name'] = 'Dumb and Dumber'\n",
        "train.loc[63, 'Movie Name'] = 'Unknown'\n",
        "\n",
        "print('After fix:')\n",
        "print(train.loc[[65, 166, 63, 502]]) # print the three rows affected + another quote to check"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eZiMJvJRdN_M",
      "metadata": {
        "id": "eZiMJvJRdN_M"
      },
      "source": [
        "The printed output reveals that the `Movie Quote` column has been correctly concatenated for the rows with spillage whilst the quotes are the same for rows without spillage (e.g. the *Pirates of the Carribean* quote above was unaffected by the changes). The `Movie Name` column has also been updated correctly.\n",
        "\n",
        "We now apply this cleaning to our validation and test set. We are unable to impute the correct movie name for these datasets without leakage, so only correct the quote:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 387,
      "id": "6FztzBuldpaa",
      "metadata": {
        "id": "6FztzBuldpaa"
      },
      "outputs": [],
      "source": [
        "# Follow same process as with training set:\n",
        "# validation\n",
        "condition = val['Movie Name'].notna() & val['Movie Name'].str.contains('\"')\n",
        "val.loc[condition, 'Movie Quote'] = val.loc[condition, 'Movie Quote'] + ' ' + val['Movie Name']\n",
        "val.loc[condition, 'Movie Name'] = np.nan\n",
        "\n",
        "\n",
        "# test\n",
        "condition = test['Movie Name'].notna() & test['Movie Name'].str.contains('\"')\n",
        "test.loc[condition, 'Movie Quote'] = test.loc[condition, 'Movie Quote'] + ' ' + test['Movie Name']\n",
        "test.loc[condition, 'Movie Name'] = np.nan\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MRSv8MiLeiYC",
      "metadata": {
        "id": "MRSv8MiLeiYC"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "We attempted to correct for spillage of quotes into another column. We assumed that movie names containing quotation marks were incorrect and instead a result of column spillage.\n",
        "\n",
        "These quotes were concatenated to get the intended quote in the `Movie Quote` column as this column will be the focus of most of our analysis and subsequent cleaning steps."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DYZlTpdBUQLz",
      "metadata": {
        "id": "DYZlTpdBUQLz"
      },
      "source": [
        "#### Q1b Step 2 - Lowercasing\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this step, we convert all text data to lowercase.\n",
        "\n",
        "***How:***\n",
        "\n",
        "Use the string `lower()` function built into Python.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "- **Standardisation**: This step improves data standardization for analysis. Lowercasing ensures consistent treatment of text variations (\"Big\", \"BIG\", \"big\") which reduces vocabulary size and simplifies feature space. This can lead to better performance for common vectorisation techniques (TF-IDF) used in text analysis. We acknowledge the potential loss of information for proper nouns/acronyms, but believe the benefits for model performance outweigh this drawback. Additionally, embedding models like BERT, which might not require lowercasing, but our chosen vectorisation method is not yet finalised.\n",
        "\n",
        "- **Assist later cleaning steps**: Lowercasing will also help us to correct some spelling misakes later such as when certain letters are repeated (e.g. \"Sssssssmokin'\" as identified during EDA)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LWuKdsdNPB2b",
      "metadata": {
        "id": "LWuKdsdNPB2b"
      },
      "source": [
        "**Apply lowercasing and check *before* versus *after*:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 388,
      "id": "yd0tA8O7UQLz",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "yd0tA8O7UQLz",
        "outputId": "7c6d270a-bf35-40e1-e852-1b23b80607d1"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"train[['Quote_clean', 'Movie Quote']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"Quote_clean\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"your identity is your most valuable possession. protect it.\",\n          \"\\\"you're a wizard, harry.\\\"\",\n          \"the greatest thing you\\u00e2\\u20ac\\u2122ll ever learn is just to love and be loved in return.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"Your identity is your most valuable possession. Protect it.\",\n          \"\\\"You're a wizard, Harry.\\\"\",\n          \"The greatest thing you\\u00e2\\u20ac\\u2122ll ever learn is just to love and be loved in return.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-f6c7398b-c160-40ce-9613-58d43a0205ae\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Quote_clean</th>\n",
              "      <th>Movie Quote</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>917</th>\n",
              "      <td>i didn't know if you were lost. stick with me. i'll keep you safe.</td>\n",
              "      <td>I didn't know if you were lost. Stick with me. I'll keep you safe.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>521</th>\n",
              "      <td>your identity is your most valuable possession. protect it.</td>\n",
              "      <td>Your identity is your most valuable possession. Protect it.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>464</th>\n",
              "      <td>the greatest thing youâ€™ll ever learn is just to love and be loved in return.</td>\n",
              "      <td>The greatest thing youâ€™ll ever learn is just to love and be loved in return.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>855</th>\n",
              "      <td>elsa, the past is not what it seems. you must find the truth.</td>\n",
              "      <td>Elsa, the past is not what it seems. You must find the truth.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>\"you're a wizard, harry.\"</td>\n",
              "      <td>\"You're a wizard, Harry.\"</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f6c7398b-c160-40ce-9613-58d43a0205ae')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f6c7398b-c160-40ce-9613-58d43a0205ae button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f6c7398b-c160-40ce-9613-58d43a0205ae');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-18fdafd6-f06f-4ff3-8334-d16976d4951a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-18fdafd6-f06f-4ff3-8334-d16976d4951a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-18fdafd6-f06f-4ff3-8334-d16976d4951a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                        Quote_clean  \\\n",
              "917              i didn't know if you were lost. stick with me. i'll keep you safe.   \n",
              "521                     your identity is your most valuable possession. protect it.   \n",
              "464  the greatest thing youâ€™ll ever learn is just to love and be loved in return.   \n",
              "855                   elsa, the past is not what it seems. you must find the truth.   \n",
              "16                                                        \"you're a wizard, harry.\"   \n",
              "\n",
              "                                                                        Movie Quote  \n",
              "917              I didn't know if you were lost. Stick with me. I'll keep you safe.  \n",
              "521                     Your identity is your most valuable possession. Protect it.  \n",
              "464  The greatest thing youâ€™ll ever learn is just to love and be loved in return.  \n",
              "855                   Elsa, the past is not what it seems. You must find the truth.  \n",
              "16                                                        \"You're a wizard, Harry.\"  "
            ]
          },
          "execution_count": 388,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Convert Movie Quote text to lower case\n",
        "\n",
        "train['Quote_clean'] = train['Movie Quote'].str.lower()\n",
        "val['Quote_clean'] = val['Movie Quote'].str.lower()\n",
        "test['Quote_clean'] = test['Movie Quote'].str.lower()\n",
        "\n",
        "# check that is worked correctly:\n",
        "train[['Quote_clean', 'Movie Quote']].sample(n=5, random_state = random_seed)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OazWMJhHuch0",
      "metadata": {
        "id": "OazWMJhHuch0"
      },
      "source": [
        "We have performed a spot check of three random rows, comparing the quote before and after lowercasing. The output above reveals that all capitals have been converted to lowercase.\n",
        "\n",
        "There is potentially some loss of meaning, such as with the name 'Aurora' being converted to 'aurora' (not shown in output above), which may be interpreted as a spelling mistake in later cleaning steps."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bXWecP1jUQL0",
      "metadata": {
        "id": "bXWecP1jUQL0"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "The quotes were converted to lowercase to increase standardisation and reduce the corpus' vocabulary. It also necessary for some of our future cleaning steps.\n",
        "\n",
        "There is potentially some loss of meaning, such as with the name 'Aurora' being converted to 'aurora' which may be interpreted as a spelling mistake rather than a proper noun."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VieMKzn9UQL0",
      "metadata": {
        "id": "VieMKzn9UQL0"
      },
      "source": [
        "#### Q1b Step 3 - Fixing incorrect character encodings\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "During EDA, we noticed some special characters that were likely incorrect encodings of other characters (for example â€™ appears to be an incorrect encoding of an apostrophe). In this step we attempt to remove and substitute these special characters where necessary.\n",
        "\n",
        "In this step we also remove any obvious non-English text in the training data and correct some other related mistakes in quotes that were observed.\n",
        "\n",
        "Other special characters such as 'true' punctuation were removed at a later step.\n",
        "\n",
        "***How:***\n",
        "\n",
        "Investigated the special characters and used a reference to identify the intended character. Used regular expression to substitute the characters.\n",
        "\n",
        "***Why (Incorrect Encodings):***\n",
        "\n",
        "This is necessary to standardise our text so our modelling can focus on the core meaning of words. For example, \"canâ€™t\" and \"can't\" should be interpreted the same. These incorrect encodings represent noise in the data that may distort our results as words cannot be interpreted as they are intended.\n",
        "\n",
        "For example, let's consider TF-IDF, a common vectorisation technique based on word frequency and uniqueness. If we do not correct these encodings, the vocabulary will be artificially inflated and meaningless words like \"canâ€™t\" would have a high uniqueness score. This may potentially skew our model's results and focus later.\n",
        "\n",
        "***Why (Non-English Text):***\n",
        "\n",
        "- We are an English company and so focus on just this language\n",
        "- Other libraries we use later are language specific and so we only want to include English.\n",
        "- Non-English text adds noise to the vocabulary and might negatively impact our analysis\n",
        "\n",
        "---\n",
        "We now begin cleaning these misencodings. First we look at how many quotes may be affected:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 389,
      "id": "82SLu3ngUQL1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "82SLu3ngUQL1",
        "outputId": "8c59d060-9659-4aa2-8ebc-505b17e94939"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The training data contains 58 rows with â€™ symbols\n",
            "784                                                                            just because someone stumbles and loses their path, doesnâ€™t mean theyâ€™re lost forever.\n",
            "350               these walls are funny. first you hate â€™em. then you get used to â€™em. enough time passes, you get so you depend on them. thatâ€™s institutionalized.\n",
            "464                                                                                        the greatest thing youâ€™ll ever learn is just to love and be loved in return.\n",
            "408                                                                                         donâ€™t let anyone ever make you feel like you donâ€™t deserve what you want.\n",
            "468    itâ€™s a dangerous business, going out your doorâ€¦you step into the road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Previously during EDA, we noticed 'â€™' so will investigate this first\n",
        "\n",
        "# Get rows with 'â€™' in the quote:\n",
        "print('The training data contains', train['Quote_clean'].str.contains('â€™').sum(), 'rows with â€™ symbols')\n",
        "special_char_sample = train['Quote_clean'].loc[train['Quote_clean'].str.contains('â€™')].sample(5, random_state = random_seed).index\n",
        "print(train['Quote_clean'].loc[special_char_sample])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3k35sYKVFSwC",
      "metadata": {
        "id": "3k35sYKVFSwC"
      },
      "source": [
        "We have 58 rows that contain 'â€™' which is quite substantial. Looking at the sample rows above, we can infer that this collection of symbols is likely meant to represent an apostrophe. For example, \"donâ€™t\" appears to be \"don't\". We make this correction now:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VYebLJZ8SkN8",
      "metadata": {
        "id": "VYebLJZ8SkN8"
      },
      "source": [
        "**Substitute 'â€™' and check the results:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 390,
      "id": "pjcLgYajzOwf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pjcLgYajzOwf",
        "outputId": "54baa722-0582-4db8-ed34-02e9dbabfd43"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The training data now contains 0 rows with â€™ symbols after removing them.\n",
            "784                                                                            just because someone stumbles and loses their path, doesn't mean they're lost forever.\n",
            "350                 these walls are funny. first you hate 'em. then you get used to 'em. enough time passes, you get so you depend on them. that's institutionalized.\n",
            "464                                                                                      the greatest thing you'll ever learn is just to love and be loved in return.\n",
            "408                                                                                         don't let anyone ever make you feel like you don't deserve what you want.\n",
            "468    it's a dangerous business, going out your doorâ€¦you step into the road, and if you don't keep your feet, there is no knowing where you might be swept off to.\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Now substitute â€™ with an apostrophe\n",
        "\n",
        "# Define the replacement function\n",
        "def replace_apostrophe(text):\n",
        "  return text.replace('â€™', \"'\")\n",
        "\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(replace_apostrophe)\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(replace_apostrophe)\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(replace_apostrophe)\n",
        "\n",
        "# check our sample:\n",
        "print('The training data now contains', train['Quote_clean'].str.contains('â€™').sum(), 'rows with â€™ symbols after removing them.')\n",
        "print(train['Quote_clean'].loc[special_char_sample])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4O8iEeVO1uJJ",
      "metadata": {
        "id": "4O8iEeVO1uJJ"
      },
      "source": [
        "Our checks verify that there are now no more â€™ symbols in our dataset. Furthermore we see that previously sampled rows now have apostrophes where the â€™ symbols previously were. A quick inspection highlights that these sentences make sense and that apostrophes were appropriate replacements.\n",
        "\n",
        "Next we find other groups of symbols that may represent incorrect encodings:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 391,
      "id": "tWn1CkY22XEk",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tWn1CkY22XEk",
        "outputId": "c04eacbb-0949-4e16-8a43-8ed1359632d3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "664                                                                                                                                                                     zaldrä«zes buzdari iksos daor! (a dragon is not a slave!)\n",
            "261                                         choose the sword, and you will join me. choose the ball and you join your mother, in death. you don't understand my words, but you must choose. soâ€¦ come boy, choose life or death.\n",
            "890                                                                                                                        you've got a lot to learn. here on earth, we like â€¦have to listen to what the others tell you to do.\n",
            "483                                                                                                                                                                                 li'l dice my ass, my name is now li'l zã© now\n",
            "884                                                                                                                        these barbarians pretend to be submissive, but theâ€¦a sleeping beast down, before it can be awakened.\n",
            "477                                                                                                                                                     it is not our abilities that show what we truly areâ€¦ it is our choices.\n",
            "256                                                                                                                                                                                         that's no moonâ€¦it's a space station\n",
            "883                                                                                                                        maybe it's because i'm constantly fighting, destroâ€¦d to be a lower-class warrior... like my kakarot.\n",
            "468                                                                it's a dangerous business, going out your doorâ€¦you step into the road, and if you don't keep your feet, there is no knowing where you might be swept off to.\n",
            "634                                                                                                                                   nobody panics when things go â€œaccording to planâ€_x009d_. even if the plan is horrifying!\n",
            "272    i've seen things you people wouldn't believe. attack ships on fire off the shoulder of orion. i watched c-beams glitter in the dark near the tannhã¤user gate. all those moments will be lost in time, like tears in rain.\n",
            "479                                                                                                                                                                          dadinho ã© o caralho, meu nome ã© zã© pequeno porra!\n",
            "908                                                                                                                        i was so sure my strength was getting close to it'â€¦on this planet... then who would i train against.\n",
            "513                                                                                                                                                       a day may come when the courage of men failsâ€¦ but it is not this day.\n",
            "914                                                                                                                        it looks as though he's managed to transform somehâ€¦ his human form, retaining his speed and agility.\n",
            "522                                                                                                          nunchuck skillsâ€¦ bowhunting skillsâ€¦ computer hacking skillsâ€¦ girls only want boyfriends who have great skills!\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# find other incorrect encodings:\n",
        "\n",
        "def find_incorrect_encodings(text):\n",
        "  # use hexadecimal regex pattern to match any character outside the standard ASCII range such as â, €, ™:\n",
        "  pattern = r\"[^\\x00-\\x7F]\"\n",
        "  return re.findall(pattern, text)\n",
        "\n",
        "incorrect_encoding_rows = train[train['Quote_clean'].apply(lambda x: len(find_incorrect_encodings(x)) > 0)]\n",
        "\n",
        "# Print the rows with potentially incorrect encodings\n",
        "print(incorrect_encoding_rows['Quote_clean'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AsLXQFuz3Riu",
      "metadata": {
        "id": "AsLXQFuz3Riu"
      },
      "source": [
        "***Interpretation:***\n",
        "\n",
        "The above output shows all quotes in the training data that contain characters outside the standard ASCII range such as â, €, © etc. We have 16 such rows. Many of the symbol combinations are the same between quotes, indicating that there is some consistent erroneous encoding (e.g. 'â€¦').\n",
        "\n",
        "After some research, this is confirmed: the special characters appears to be from 'Windows 1252' encoding. We use information about Windows 1252 encoding and general common sense to convert these special characters to their intended ASCII character.\n",
        "\n",
        "**Reference:**\n",
        "\n",
        "HTML Windows-1252 Reference. (n.d.). Www.w3schools.com. Retrieved March 30, 2024, from https://www.w3schools.com/charsets/ref_html_ansi.asp\n",
        "\n",
        "---\n",
        "\n",
        "***Issues***:\n",
        "\n",
        "However, we also notice that some of these quotes appear to be in languages other than English (whether a real language or a fantasy one):\n",
        "\n",
        "- \"dadinho ã© o caralho, meu nome ã© zã© pequeno porra!\" - from Cidade de Deus (Portuguese)\n",
        "- \"zaldrä«zes buzdari iksos daor! (a dragon is not a slave!)\" - from Game of Thrones (High Valerian and English translation)\n",
        "\n",
        "Also, some quotes appear to be genuinely corrupted with missing text rather than just incorrect encodings (these corrupted quotes all come from the movie 'Dragon Ball Super: Broly'). For example:\n",
        "\n",
        "- \"it looks as though he's managed to transform somehâ€¦ his human form, retaining his speed and agility.\"\n",
        "\n",
        "---\n",
        "\n",
        "***Actions:***\n",
        "\n",
        "- **Foreign language quotes:** We remove any quotes identified as entirely in a non-English language. For mixed language quotes like the Game of Thrones ones,  we remove only the non-English parts of the quote. Non-English text is not relevant to our business objective, where our company is presumably English-speaking. Our objective is focused on finding patterns in English and adding foreign languages will just introduce noise. Furthermore, many NLP libraries that we will use are designed specifically for English.\n",
        "\n",
        "- **Missing/corrupted quotes:** To avoid data loss, we impute these quotes using web data. We do not recompute the chatGPT label or sentiment scores for these imputed quotes.\n",
        "\n",
        "Source for quotes:\n",
        "https://www.imdb.com/title/tt7961060/quotes/?item=qt4471760\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 392,
      "id": "vl4fgNDSM_Pa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vl4fgNDSM_Pa",
        "outputId": "9c602772-a898-4a7c-bb96-6bfcef29c5f7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Portuguese quote: \n",
            "Dadinho Ã© o caralho, meu nome Ã© ZÃ© Pequeno porra!\n",
            "\n",
            "High Valerian quote (fantasy language):\n",
            "ZaldrÄ«zes buzdari iksos daor! (A dragon is not a slave!)\n",
            "\n",
            "After fixing:\n",
            "a dragon is not a slave!\n",
            "\n",
            "Corrupted quotes: \n",
            "908    I was so sure my strength was getting close to it'â€¦on this planet... Then who would I train against.\n",
            "883    Maybe it's because I'm constantly fighting, destroâ€¦d to be a lower-class warrior... like my Kakarot.\n",
            "890    You've got a lot to learn. Here on Earth, we like â€¦have to listen to what the others tell you to do.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Corrupted quotes after fixing:\n",
            "908                                                            i was so sure my strength was getting close to it's peak... it'd be a shame for someone so incredible to die on this planet... then, who would i train against.\n",
            "883                                maybe it's because i'm constantly fighting, destroying, that i'd actually like to save something for once. especially someone who is judged to be a lower-class warrior... like my kakarot.\n",
            "890    you've got a lot to learn. here on earth, we like to live together in peace and harmony, although that's not always been the case... you can stop fighting. you don't have to listen to what the others tell you to do.\n",
            "Name: Quote_clean, dtype: object\n",
            "\n",
            "Does the Portuguese entry still exist:\n",
            "False\n"
          ]
        }
      ],
      "source": [
        "# Manually remove/fix non-English text\n",
        "import warnings\n",
        "\n",
        "# appears to be in Portuguese:\n",
        "print('Portuguese quote: ')\n",
        "print(train['Movie Quote'].loc[479]) # we will remove this after manual adjustments\n",
        "\n",
        "print()\n",
        "# contains fantasy language from Game of Thrones (high Valerian)\n",
        "print('High Valerian quote (fantasy language):')\n",
        "print(train['Movie Quote'].loc[664])\n",
        "train['Quote_clean'].loc[664] = 'a dragon is not a slave!' # manually fix\n",
        "print()\n",
        "print('After fixing:')\n",
        "print(train['Quote_clean'].loc[664])\n",
        "\n",
        "\n",
        "print()\n",
        "# manually fix quotes with corrupted data (all from Dragonball Super):\n",
        "print('Corrupted quotes: ')\n",
        "print(train['Movie Quote'].loc[[908, 883, 890]])\n",
        "\n",
        "# source for quotes:\n",
        "# https://www.imdb.com/title/tt7961060/quotes/?item=qt4471760\n",
        "# we apply lower casing to these quotes\n",
        "warnings.filterwarnings(\"ignore\") # To ignore warning messages\n",
        "\n",
        "train['Quote_clean'].loc[908] = \"I was so sure my strength was getting close to it's peak... It'd be a shame for someone so incredible to die on this planet... Then, who would I train against.\".lower()\n",
        "train['Quote_clean'].loc[883] = \"Maybe it's because I'm constantly fighting, destroying, that I'd actually like to save something for once. Especially someone who is judged to be a lower-class warrior... like my Kakarot.\".lower()\n",
        "train['Quote_clean'].loc[890] = \"You've got a lot to learn. Here on Earth, we like to live together in peace and harmony, although that's not always been the case... You can stop fighting. You don't have to listen to what the others tell you to do.\".lower()\n",
        "\n",
        "print()\n",
        "print('Corrupted quotes after fixing:')\n",
        "print(train['Quote_clean'].loc[[908, 883, 890]])\n",
        "\n",
        "# we remove the Portuguese entry:\n",
        "train_copy = train.copy()\n",
        "train = train.drop(479)\n",
        "\n",
        "# Check Portuguese quote is gone:\n",
        "print()\n",
        "print('Does the Portuguese entry still exist:')\n",
        "print(479 in train.index)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HIMKHz2hapNd",
      "metadata": {
        "id": "HIMKHz2hapNd"
      },
      "source": [
        "The **checks** above confirm that the relevant quotes have been fixed or removed:\n",
        "- The Portuguese quote was removed\n",
        "- Only the English remains of the High Valerian quote\n",
        "- The Dragonball quotes have been replaced\n",
        "\n",
        "Next we replace the Windows 1252 encodings with their intended ASCII characters. We base these replacements only on the special characters in the training set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 393,
      "id": "lUZxs5A1I8MS",
      "metadata": {
        "id": "lUZxs5A1I8MS"
      },
      "outputs": [],
      "source": [
        "# finally we replace the incorrect encodings\n",
        "\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('â€¦', ', ')\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('â€œ', '\"')\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('â€_x009d_.', '\"')\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('ã©', 'e') # should be accented e but we just keep as 'e'\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('ã©', 'a') # should be accented a but we just keep as 'a'\n",
        "train['Quote_clean'] = train['Quote_clean'].str.replace('ã¤', 'a') # should be accented a but we just keep as 'a'\n",
        "\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('â€¦', ', ')\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('â€œ', '\"')\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('â€_x009d_.', '\"')\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('ã©', 'e') # should be accented e but we just keep as 'e'\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('ã©', 'a') # should be accented a but we just keep as 'a'\n",
        "val['Quote_clean'] = val['Quote_clean'].str.replace('ã¤', 'a') # should be accented a but we just keep as 'a'\n",
        "\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('â€¦', ', ')\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('â€œ', '\"')\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('â€_x009d_.', '\"')\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('ã©', 'e') # should be accented e but we just keep as 'e'\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('ã©', 'a') # should be accented a but we just keep as 'a'\n",
        "test['Quote_clean'] = test['Quote_clean'].str.replace('ã¤', 'a') # should be accented a but we just keep as 'a'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lPO0ZbS5Lj2N",
      "metadata": {
        "id": "lPO0ZbS5Lj2N"
      },
      "source": [
        "**Checks: spot check affected quotes before vs after and calculate number of rows with non-standard characters**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 394,
      "id": "CuAyqquUmFsl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CuAyqquUmFsl",
        "outputId": "712bf28d-2250-4cd1-d2a9-69b690ca7550"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                                                                            Movie Quote  \\\n",
            "522                                                Nunchuck skillsâ€¦ bowhunting skillsâ€¦ computer hacking skillsâ€¦ Girls only want boyfriends who have great skills!   \n",
            "468  Itâ€™s a dangerous business, going out your doorâ€¦You step into the Road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.   \n",
            "634                                                                         Nobody panics when things go â€œaccording to planâ€_x009d_. Even if the plan is horrifying!   \n",
            "\n",
            "                                                                                                                                                       Quote_clean  \n",
            "522                                              nunchuck skills,  bowhunting skills,  computer hacking skills,  girls only want boyfriends who have great skills!  \n",
            "468  it's a dangerous business, going out your door, you step into the road, and if you don't keep your feet, there is no knowing where you might be swept off to.  \n",
            "634                                                                               nobody panics when things go \"according to plan\" even if the plan is horrifying!  \n"
          ]
        }
      ],
      "source": [
        "# spot check:\n",
        "print(train[['Movie Quote', 'Quote_clean']].loc[[522, 468, 634]])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-gM2LHIcbA7_",
      "metadata": {
        "id": "-gM2LHIcbA7_"
      },
      "source": [
        "In the sample above of affected quotes we no longer have strange symbols and the replacement characters appear to make sense within the quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "imp4V6Y0yL3S",
      "metadata": {
        "id": "imp4V6Y0yL3S"
      },
      "source": [
        "**Check: search if there are anymore non-ASCII characters left**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 395,
      "id": "9NGnp_dCyJXC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9NGnp_dCyJXC",
        "outputId": "dda79266-af14-4088-98a4-10f535cafdf6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of rows with non-standard characters: 0\n"
          ]
        }
      ],
      "source": [
        "# check if there are any strange encodings left:\n",
        "incorrect_encoding_rows = train[train['Quote_clean'].apply(lambda x: len(find_incorrect_encodings(x)) > 0)]\n",
        "\n",
        "# Print the rows with potentially incorrect encodings\n",
        "print(f'Number of rows with non-standard characters: {len(incorrect_encoding_rows)}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wyozJhdrykjz",
      "metadata": {
        "id": "wyozJhdrykjz"
      },
      "source": [
        "No more non-standard characters exist within the training dataset suggesting all encoding issues have been resolved."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "O7rr7JAcUQL1",
      "metadata": {
        "id": "O7rr7JAcUQL1"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we:\n",
        "- Replaced incorrect encodings with the standard symbol so they made sense\n",
        "- Removed non-English text since our focus is on English movie quotes. Also, many NLP libraries we will use are built for English.\n",
        "- Used online resouces to replace seemingly corrupted quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "57_xvkA6boTb",
      "metadata": {
        "id": "57_xvkA6boTb"
      },
      "source": [
        "#### Q1b Step 4 - Expand contractions\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we expand contractions in our quotes (e.g. \"don't\" into \"do not\").\n",
        "\n",
        "***How:***\n",
        "\n",
        "Use function in `contractions` library to expand standard contractions.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "Including abbreviations and contractions will increase the number of unique words in our data, making it more 'sparse'. We standardise them to allow our models to better identify patterns in the data, improving performance. This is particularly important given the lack of observations we have.\n",
        "\n",
        "However, contractions may signal the sentiment of the text which is potentially important for our classifier later (e.g. informal vs formal speech). Ultimately, we believe the sparsity issue to be more material given our lack of data, thus we choose to clean this aspect of the text."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8kJwMOWnTZ51",
      "metadata": {
        "id": "8kJwMOWnTZ51"
      },
      "source": [
        "**Apply contraction standardisation:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 396,
      "id": "qlD2-ydQgY58",
      "metadata": {
        "id": "qlD2-ydQgY58"
      },
      "outputs": [],
      "source": [
        "# Let's create a new column with the contractions cleaned:\n",
        "\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(\"y'know\", 'you know', x)) # manual fix for a contraction observed in EDA that is not in contractions library\n",
        "train['Quote_clean2'] = train['Quote_clean'].apply(lambda x: contractions.fix(x))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CdKpEpcGOrM_",
      "metadata": {
        "id": "CdKpEpcGOrM_"
      },
      "source": [
        "**Check 1: spot check of quotes with and without contractions before and after the fix**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 397,
      "id": "ctbDn1kNOqTD",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ctbDn1kNOqTD",
        "outputId": "347fc245-0f66-4675-dffd-9096a40b9c34"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quote before contractions expands\n",
            "\"you're a wizard, harry.\"\n",
            "----\n",
            "lord, it's a miracle! man up and vanished like a fart in the wind!\n",
            "----\n",
            "that is so fetch!\n",
            "----\n",
            "ssssssssssssmokin'!\n",
            "\n",
            "Quote after contractions expanded\n",
            "\"you are a wizard, harry.\"\n",
            "----\n",
            "lord, it is a miracle! man up and vanished like a fart in the wind!\n",
            "----\n",
            "that is so fetch!\n",
            "----\n",
            "ssssssssssssmokin'!\n"
          ]
        }
      ],
      "source": [
        "# Let us choose some rows to compare before and after cleaning:\n",
        "print('Quote before contractions expands')\n",
        "print(train['Quote_clean'].loc[16])\n",
        "print('----')\n",
        "print(train['Quote_clean'].loc[344])\n",
        "print('----')\n",
        "print(train['Quote_clean'].loc[510]) # no contraction example\n",
        "print('----')\n",
        "print(train['Quote_clean'].loc[160]) # word shortening example\n",
        "print()\n",
        "print('Quote after contractions expanded')\n",
        "print(train['Quote_clean2'].loc[16])\n",
        "print('----')\n",
        "print(train['Quote_clean2'].loc[344])\n",
        "print('----')\n",
        "print(train['Quote_clean2'].loc[510])\n",
        "print('----')\n",
        "print(train['Quote_clean2'].loc[160])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JGrXB6KgkpDJ",
      "metadata": {
        "id": "JGrXB6KgkpDJ"
      },
      "source": [
        "We can see above that contractions have been expanded in our text (\"you're\" became \"you are\"). Note that words ending with an apostrophe like \"sssmokin' \" were not fixed. We will attempt to remedy this at a later stage when we fix spelling mistakes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "unPxxk4OO79A",
      "metadata": {
        "id": "unPxxk4OO79A"
      },
      "source": [
        "**Check 2: look at the most common contactions before and after cleaning**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 398,
      "id": "V7Xn-OMrk4ym",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V7Xn-OMrk4ym",
        "outputId": "91e11056-8240-49db-b472-99fda928c08c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Before contraction cleaning:\n",
            "\n",
            "it's: 61\n",
            "don't: 49\n",
            "i'm: 46\n",
            "you're: 32\n",
            "that's: 22\n",
            "can't: 21\n",
            "i'll: 15\n",
            "there's: 11\n",
            "we're: 11\n",
            "doesn't: 10\n",
            "i've: 9\n",
            "didn't: 9\n",
            "what's: 7\n",
            "he's: 6\n",
            "you've: 5\n",
            "they're: 5\n",
            "we've: 4\n",
            "we'll: 4\n",
            "it'll: 4\n",
            "you'll: 4\n",
            "---\n",
            "After contraction cleaning:\n",
            "\n",
            "life's: 2\n",
            "man's: 2\n",
            "li'l: 2\n",
            "broly's: 1\n",
            "everybody's: 1\n",
            "following's: 1\n",
            "one's: 1\n",
            "grabthar's: 1\n",
            "guy's: 1\n",
            "people's: 1\n",
            "frisco's: 1\n",
            "evil's: 1\n",
            "all's: 1\n",
            "captain's: 1\n",
            "winning's: 1\n",
            "heeeeere's: 1\n",
            "world's: 1\n",
            "god's: 1\n",
            "beggar's: 1\n",
            "woman's: 1\n"
          ]
        }
      ],
      "source": [
        "# look at most common contractions before and after cleaning:\n",
        "\n",
        "# Regular expression to capture contractions\n",
        "contraction_re = r\"\\b\\w+'[\\w\\d]+\\b\"\n",
        "contraction_count_before = Counter()\n",
        "contraction_count_after = Counter()\n",
        "\n",
        "# Display most commonly used contraction before and after cleaning\n",
        "for quote in train['Quote_clean']:\n",
        "    found_contractions_before = re.findall(contraction_re, quote)\n",
        "    contraction_count_before.update(found_contractions_before)\n",
        "for quote in train['Quote_clean2']:\n",
        "    found_contractions_after = re.findall(contraction_re, quote)\n",
        "    contraction_count_after.update(found_contractions_after)\n",
        "\n",
        "print(f\"Before contraction cleaning:\\n\")\n",
        "for contraction, count in contraction_count_before.most_common(20):\n",
        "    print(f\"{contraction}: {count}\")\n",
        "print('---')\n",
        "print(f\"After contraction cleaning:\\n\")\n",
        "for contraction, count in contraction_count_after.most_common(20):\n",
        "    print(f\"{contraction}: {count}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "10QvVCT-mh5d",
      "metadata": {
        "id": "10QvVCT-mh5d"
      },
      "source": [
        "We see that contractions no longer appear in the top words with apostrophes after cleaning. We still maintain possessive words (e.g. \"Luke's\") which is fine and some slang (e.g. \"li'l\").\n",
        "\n",
        "The word \"heeeeere's\" is technically a contraction of \"here is\" but was not corrected by the `contraction ` package due to the word not being recognised as \"here\". This will be fixed later when we correct spelling mistakes.\n",
        "\n",
        "Overall appears the fix has been done correctly. We will now update our `Quote_clean` variable in our three datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 399,
      "id": "ZRT6Ye83nVd5",
      "metadata": {
        "id": "ZRT6Ye83nVd5"
      },
      "outputs": [],
      "source": [
        "# Update Quote_clean for train, val and test sets\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(\"y'know\", 'you know', x)) # manual fix for a contraction observed in EDA\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: contractions.fix(x))\n",
        "\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(lambda x: re.sub(\"y'know\", 'you know', x)) # manual fix for a contraction observed in EDA\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(lambda x: contractions.fix(x))\n",
        "\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(lambda x: re.sub(\"y'know\", 'you know', x)) # manual fix for a contraction observed in EDA\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(lambda x: contractions.fix(x))\n",
        "\n",
        "train = train.drop('Quote_clean2', axis = 1)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rvGw1Uq04IWX",
      "metadata": {
        "id": "rvGw1Uq04IWX"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we:\n",
        "- Expanded contractions in the `Movie quote` column\n",
        "\n",
        "After the change was made we noticed some spelling mistakes that impeded the contraction cleaning. The impact was fairly immaterial but we will fix these at a later stage after correcting spelling errors."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-Qsq-jP_odjG",
      "metadata": {
        "id": "-Qsq-jP_odjG"
      },
      "source": [
        "#### Q1b Step 5 - Remove punctuation and replace numeric words\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "Next we remove all punctuation from the quotes. However, we acknowledge that punctuation may reflect important information about a quote. For example, exclamation marks may suggest that a quote is rude (relevant to our classifier in Q3) or question marks may be an important part of famous quotes that we wish to identify in our cluster analysis. Therefore, to avoid losing some of this information we also create new features representing punctuation flags.\n",
        "\n",
        "We also remove any numeric words and other special characters (e.g. $)\n",
        "\n",
        "***How:***\n",
        "\n",
        "Use regex to lookup and replace punctuation and other special characters.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "**Removing Punctuation:**\n",
        "- Assists with correctly tokenising our quotes later.\n",
        "- This removes variations of the same word like \"hello\" vs. \"hello,\" simplifying the data and reducing vocabulary size, which can improve model performance especially given we have limited data.\n",
        "- Most punctuation does not have much semantic meaning and so will represent noise when we are trying to discern the key characteristics of famous movie quotes\n",
        "\n",
        "**Removing Numbers:**\n",
        "- Numbers can significantly increase the size of our corpus' vocabulary and do not often provide a lot of semantic meaning in movie quotes.\n",
        "- In other domains numbers can be important, but for movie quotes we do not believe this is the case and so remove them to reduce noise."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J-sfIQQSVJyU",
      "metadata": {
        "id": "J-sfIQQSVJyU"
      },
      "source": [
        "---\n",
        "First, we create two features: `exclamation_flag` and `question_flag`. Each represent whether an exclamation mark or question mark are present in the quote. These flags may assist in accurately predicting whether a quote is problematic or not as they can reflect the tone of a quote."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 400,
      "id": "B3_fQOVptVsM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "B3_fQOVptVsM",
        "outputId": "aa607242-cb92-4e94-aee7-fe725a65a152"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"train[['Movie Quote', 'exclamation_flag', 'question_flag']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"You only live twice:Once when you are born And once when you look death in the face\",\n          \"And to think I've been telling my friends it's so cool living with an artist. You never once asked to paint me nude!\",\n          \"I love lamp.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"exclamation_flag\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"question_flag\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-5f45295f-31c2-4d19-9ae1-cd7d5fd3a8de\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>exclamation_flag</th>\n",
              "      <th>question_flag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>862</th>\n",
              "      <td>All I have are negative thoughts.</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>224</th>\n",
              "      <td>You only live twice:Once when you are born And once when you look death in the face</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530</th>\n",
              "      <td>I love lamp.</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>710</th>\n",
              "      <td>Let it go!</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>295</th>\n",
              "      <td>And to think I've been telling my friends it's so cool living with an artist. You never once asked to paint me nude!</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5f45295f-31c2-4d19-9ae1-cd7d5fd3a8de')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5f45295f-31c2-4d19-9ae1-cd7d5fd3a8de button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5f45295f-31c2-4d19-9ae1-cd7d5fd3a8de');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-90bb22f4-a4ae-4cb0-9275-49a9bfe664e1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-90bb22f4-a4ae-4cb0-9275-49a9bfe664e1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-90bb22f4-a4ae-4cb0-9275-49a9bfe664e1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                              Movie Quote  \\\n",
              "862                                                                                     All I have are negative thoughts.   \n",
              "224                                   You only live twice:Once when you are born And once when you look death in the face   \n",
              "530                                                                                                          I love lamp.   \n",
              "710                                                                                                            Let it go!   \n",
              "295  And to think I've been telling my friends it's so cool living with an artist. You never once asked to paint me nude!   \n",
              "\n",
              "     exclamation_flag  question_flag  \n",
              "862                 0              0  \n",
              "224                 0              0  \n",
              "530                 0              0  \n",
              "710                 1              0  \n",
              "295                 1              0  "
            ]
          },
          "execution_count": 400,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        " # first we will create two new features: 'exclamation_flag' and 'question_flag'\n",
        " # These variables will take a 1 or 0 depending of if the quote contains either a ! or ? respectively\n",
        " # we do not count the number of times these punctuation marks are contained within our text as often there are repetitious uses for exaggeration\n",
        "\n",
        "train['exclamation_flag'] = train['Quote_clean'].apply(lambda x: 1 if '!' in x else 0)\n",
        "train['question_flag'] = train['Quote_clean'].apply(lambda x: 1 if '?' in x else 0)\n",
        "\n",
        "val['exclamation_flag'] = val['Quote_clean'].apply(lambda x: 1 if '!' in x else 0)\n",
        "val['question_flag'] = val['Quote_clean'].apply(lambda x: 1 if '?' in x else 0)\n",
        "\n",
        "test['exclamation_flag'] = test['Quote_clean'].apply(lambda x: 1 if '!' in x else 0)\n",
        "test['question_flag'] = test['Quote_clean'].apply(lambda x: 1 if '?' in x else 0)\n",
        "\n",
        "# check that it worked:\n",
        "train[['Movie Quote', 'exclamation_flag', 'question_flag']].head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "S8hnOhXqVm_B",
      "metadata": {
        "id": "S8hnOhXqVm_B"
      },
      "source": [
        "Above are the flag values for the first 5 quotes in our training dataset. An inspection of these quotes and their respective flag values suggests that the features have been created correctly.\n",
        "\n",
        "---\n",
        "\n",
        "Next we remove punctuation from our quotes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 401,
      "id": "Gk3sSJ40urkR",
      "metadata": {
        "id": "Gk3sSJ40urkR"
      },
      "outputs": [],
      "source": [
        "# Next we remove all punctuation and replace it with a space\n",
        "# spaces will be trimmed later when we tokenise\n",
        "\n",
        "# use regex to remove non-alphanumeric characters or spaces:\n",
        "pattern = r\"[^\\w\\s]\"\n",
        "\n",
        "# replace punctuation with a space:\n",
        "train['Quote_clean2'] = train['Quote_clean'].apply(lambda x: re.sub(pattern, ' ', x))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gTQyC6EhWOpR",
      "metadata": {
        "id": "gTQyC6EhWOpR"
      },
      "source": [
        "**Check 1: spot check of a random sample of quotes before and after punctuation was removed**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 402,
      "id": "ukDMOoJEWN8x",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ukDMOoJEWN8x",
        "outputId": "c87e61f5-4e5e-4423-9ed6-09dd68379c01"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quotes before punctuation was removed\n",
            "there is always hope.\n",
            "----\n",
            "i laugh in the face of danger\n",
            "----\n",
            "you are right, aurora. but the drowning man will always try and drag somebody down with him. it are not right, but the man's drowning.\n",
            "----\n",
            "the only thing we are allowed to believe is that we will not regret the choice we made.\n",
            "----\n",
            "hey! i am walking here! i am walking here!\n",
            "----\n",
            "well, la-de-da.\n",
            "----\n",
            "\n",
            "Quotes after punctuation was removed:\n",
            "there is always hope \n",
            "----\n",
            "i laugh in the face of danger\n",
            "----\n",
            "you are right  aurora  but the drowning man will always try and drag somebody down with him  it are not right  but the man s drowning \n",
            "----\n",
            "the only thing we are allowed to believe is that we will not regret the choice we made \n",
            "----\n",
            "hey  i am walking here  i am walking here \n",
            "----\n",
            "well  la de da \n",
            "----\n"
          ]
        }
      ],
      "source": [
        "# spot checks\n",
        "random_sample = train.sample(6, random_state=random_seed).index.tolist()\n",
        "\n",
        "print('Quotes before punctuation was removed')\n",
        "for i in random_sample:\n",
        "  print(train['Quote_clean'].loc[i])\n",
        "  print('----')\n",
        "\n",
        "print()\n",
        "print('Quotes after punctuation was removed:')\n",
        "for i in random_sample:\n",
        "  print(train['Quote_clean2'].loc[i])\n",
        "  print('----')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rnBpF3h6yesl",
      "metadata": {
        "id": "rnBpF3h6yesl"
      },
      "source": [
        "From our spot checks above it appears that punctuation has been correctly removed including full stops, commas, question marks and quotation marks. We do notice that previously possessive words such as \"Luke's\" have now been split into \"Luke s\". We will likely remove these \"s\" characters later as a stop word, as they will appear commonly but do not really provide much semantic meaning that the preceding noun doesn't already communicate.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rednuiAiWrNY",
      "metadata": {
        "id": "rednuiAiWrNY"
      },
      "source": [
        "**Check 2: use a different regex pattern to try to identify common punctuation/special symbols**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 403,
      "id": "zI0FAxvuWrqG",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zI0FAxvuWrqG",
        "outputId": "af2e0fdb-4567-41fb-9d6c-8e32e6f79f73"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Punctuation found in quotes: 0\n"
          ]
        }
      ],
      "source": [
        "pattern2 = r\"[,%$#@!&*(),.?]\"\n",
        "\n",
        "punctuation_counter = Counter()\n",
        "\n",
        "# Display most commonly used contraction before and after cleaning\n",
        "for quote in train['Quote_clean']:\n",
        "    found_punctuation = re.findall(pattern2, quote)\n",
        "    contraction_count_before.update(found_punctuation)\n",
        "\n",
        "print(f'Punctuation found in quotes: {len(punctuation_counter.most_common())}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nU3Ln33PWsIg",
      "metadata": {
        "id": "nU3Ln33PWsIg"
      },
      "source": [
        "Our quick check above reinforces that the regex pattern we used to remove punctuation and special characters was appropriate and has been applied correctly.\n",
        "\n",
        "---\n",
        "Next we remove numeric words from our quotes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 404,
      "id": "Ae6eiy9Myv7K",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ae6eiy9Myv7K",
        "outputId": "667e55dc-6791-4a3e-f1f6-7c9fa28b10e6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Before removing numbers: \n",
            "57                             the numbers all go to 11 \n",
            "434         i will take poor assumptions for  800  alex \n",
            "573        dear 8 pound  6 ounce newborn infant jesus   \n",
            "527    there is a 30  chance that it is already raining \n",
            "858                                     i love you 3000 \n",
            "Name: Quote_clean2, dtype: object\n",
            "\n",
            "After removing numbers:\n",
            "57                             the numbers all go to   \n",
            "434          i will take poor assumptions for     alex \n",
            "573       dear   pound    ounce newborn infant jesus   \n",
            "527    there is a    chance that it is already raining \n",
            "858                                       i love you   \n",
            "Name: Quote_clean2, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# next we remove numeric words\n",
        "\n",
        "# first get rows with words containing numbers in them:\n",
        "num_pattern = r'\\b\\w*\\d\\w*\\b'\n",
        "quotes_with_numeric = train[train['Quote_clean2'].str.contains(num_pattern)]\n",
        "\n",
        "print('Before removing numbers: ')\n",
        "print(quotes_with_numeric['Quote_clean2'])\n",
        "print()\n",
        "\n",
        "# remove numbers:\n",
        "print('After removing numbers:')\n",
        "quotes_with_numeric['Quote_clean2'] = quotes_with_numeric['Quote_clean2'].apply(lambda x: re.sub(num_pattern, ' ', x))\n",
        "\n",
        "print(quotes_with_numeric['Quote_clean2'])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xSPkzvUS3dgh",
      "metadata": {
        "id": "xSPkzvUS3dgh"
      },
      "source": [
        "Looking at the above output which highlights quotes that contained numeric words before and after the cleaning step was applied. We see that there are no numbers within the 'after' quotes. These numbers have been replaced by spaces which has led to excessive whitespace in the quotes. This will be inconsequential when we tokenise our strings later however, we strip our cleaned quotes of excess whitespace for readability.\n",
        "\n",
        "---\n",
        "\n",
        "We now apply these cleaning steps to the training, validation and test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 405,
      "id": "y6Yxd9yu3oou",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y6Yxd9yu3oou",
        "outputId": "f296d333-9d79-4bbd-867f-423a524e54db"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "862                                                                                         all i have are negative thoughts\n",
            "224                                      you only live twice once when you are born and once when you look death in the face\n",
            "530                                                                                                              i love lamp\n",
            "710                                                                                                                let it go\n",
            "295    and to think i have been telling my friends it is so cool living with an artist you never once asked to paint me nude\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "pattern = r\"[^\\w\\s]\"\n",
        "num_pattern = r'\\b\\w*\\d\\w*\\b'\n",
        "\n",
        "# replace punctuation and numbers with a space:\n",
        "# we also strip any excess whitespace\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(pattern, ' ', x))\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(num_pattern, ' ', x))\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(' +', ' ', x))\n",
        "train['Quote_clean'] = train['Quote_clean'].str.strip()\n",
        "\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(lambda x: re.sub(pattern, ' ', x))\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(lambda x: re.sub(num_pattern, ' ', x))\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(lambda x: re.sub(' +', ' ', x))\n",
        "val['Quote_clean'] = val['Quote_clean'].str.strip()\n",
        "\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(lambda x: re.sub(pattern, ' ', x))\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(lambda x: re.sub(num_pattern, ' ', x))\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(lambda x: re.sub(' +', ' ', x))\n",
        "test['Quote_clean'] = test['Quote_clean'].str.strip()\n",
        "\n",
        "print(train['Quote_clean'].head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5FC6A-uOZHZ7",
      "metadata": {
        "id": "5FC6A-uOZHZ7"
      },
      "source": [
        "A final look at some samples from the `Quote_clean` column in our training set, highlights that the quotes do not contain any punctuation. For example \"broly's\" has now become \"broly s\"."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "egMmwQJt4krh",
      "metadata": {
        "id": "egMmwQJt4krh"
      },
      "source": [
        "**Summary**  \n",
        "In this cleaning step we:\n",
        "- Removed punctuation from our quotes\n",
        "- Created two new features `exclamation_flag` and `question_flag`\n",
        "- Removed numeric words from our quotes\n",
        "\n",
        "We will need to correct for lone 's' characters that appear in our text due to possessive words. This will be taken care of during stop word removal."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HSQfXtfFXxAp",
      "metadata": {
        "id": "HSQfXtfFXxAp"
      },
      "source": [
        "#### Q1b Step 6 - Spelling mistakes\n",
        "**Outline**\n",
        "\n",
        "***What and How:***\n",
        "\n",
        "In this step we fix spelling mistakes in our quotes. Movie quotes are generally well-edited and so free from spelling errors compared to a user-generated text field but during EDA we still noticed two types of spelling mistakes in our movie quotes:\n",
        "\n",
        "- **Repeated letters (e.g., \"Ssssssmokin'\"):** These are removed using regular expressions as they represent emphasis during acting, not script elements which is the focus of our analysis.\n",
        "\n",
        "- **Informal spellings and genuine mistakes (e.g., \"smokin'\" vs. \"smoking\"):** We initially used the SymSpellPy package, a dictionary-based approach, to fix these broader errors, however its suggestions proved unreliable. It's changes often introduced more errors than it fixed and so we did not end up using it. This means we must tolerate a level of spelling mistakes in our text. This is fine as the mistakes are generally minor and movie quote data is typically free of spelling mistakes.\n",
        "\n",
        "This step was conducted before tokenisation as SymSpellPy, takes into account surrounding words when suggesting corrections.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "- **Remove acting emphasis markers (repeated letters)**: Our goal is to improve scriptwriting, not acting/directorial choices. These corrections will allow us to capture the intended written word.\n",
        "- **Improve vectorisation of quotes**: later we will use word embedding algorithms to vectorise our quotes. Misspelled words may not have accurate embeddings which will hinder our models' ability to understand the quote. Cleaning will improve our model performance."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_baV72YlgFTk",
      "metadata": {
        "id": "_baV72YlgFTk"
      },
      "source": [
        "\n",
        "---\n",
        "First we set up some functions and the SymSpellPy package for identifying spelling errors. As our text comes from movies and so is already mostly devoid of spelling errors, we use a low maximum edit distance of 2. This means our spellchecker will only correct errors that are up to 2 characters different from the original text. If we set a higher edit distance then we may get more erroneous corrections which is unnecessary given we expect few mistakes (movie scripts are typically well-edited and so devoid of spelling mistakes)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 406,
      "id": "PCyA5Y6AbKH0",
      "metadata": {
        "id": "PCyA5Y6AbKH0"
      },
      "outputs": [],
      "source": [
        "# First let's set up the SymSpell dictionary and create some functions\n",
        "\n",
        "# We load a pre-trained dictionary from SymSpell\n",
        "dictionary = symspellpy.SymSpell(max_dictionary_edit_distance=2) # edit distance of 2 means only up to 2 changes will be made to the word; we do not expect many errors in our text\n",
        "dictionary.load_dictionary(infolder+'en-80k.txt', term_index = 0, count_index = 1)\n",
        "\n",
        "# function to correct spelling mistakes:\n",
        "def correct_spelling(text):\n",
        "    suggestions = dictionary.lookup_compound(text, max_edit_distance=2)\n",
        "    corrected_text = suggestions[0].term if suggestions else text # will use the most likely correction\n",
        "    return corrected_text\n",
        "\n",
        "# function that returns true if there is a spelling mistake:\n",
        "def has_spelling_mistake(text):\n",
        "    suggestions = dictionary.lookup_compound(text, max_edit_distance=2)\n",
        "    return len(suggestions)>0  # Return True there is at least one suggestion"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pLgJuF7R9N6r",
      "metadata": {
        "id": "pLgJuF7R9N6r"
      },
      "source": [
        "Before applying our spellchecker we correct letters that have been repeated for dramatic effect such as in \"ssssssssmokin' \". This is because our spellchecker will not correct these as the edit distance of these words is too high."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 407,
      "id": "dSFwvQsY_2fj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dSFwvQsY_2fj",
        "outputId": "08f75013-e546-4ed3-9eb3-60367abfaf0f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "50                                                                                                                                                                                              khaaaaan\n",
            "160                                                                                                                                                                                    ssssssssssssmokin\n",
            "442                                                                                                                                                                                   snnnnnooooooooooow\n",
            "49                                                                                                                                                                                     heeeeere s johnny\n",
            "556    you know that ringing in your ears that eeeeeeeeee that is the sound of the ear cells dying like their swan song once it is gone you will never hear that frequency again enjoy it while it lasts\n",
            "42                                                                                                                                                                                      they are heeeere\n",
            "Name: Quote_clean, dtype: object\n",
            "\n",
            "Quotes after replacing repeated letters\n",
            "50                                                                                                                                                                                         khan\n",
            "160                                                                                                                                                                                      smokin\n",
            "442                                                                                                                                                                                        snow\n",
            "49                                                                                                                                                                                here s johnny\n",
            "556    you know that ringing in your ears that e that is the sound of the ear cells dying like their swan song once it is gone you will never hear that frequency again enjoy it while it lasts\n",
            "42                                                                                                                                                                                they are here\n",
            "Name: Quote_clean2, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# During EDA we noticed words that contained repeated letters for dramatic effect\n",
        "# we fix these here\n",
        "\n",
        "# Function to check for repeated letters\n",
        "def contains_three_or_more_consecutive_letters(text):\n",
        "    pattern = r'(.)\\1{2,}'  # Matches three or more consecutive occurrences of the same character\n",
        "    return bool(re.search(pattern, text))\n",
        "\n",
        "# Function to replace consecutive occurrences of the same letter with a single occurrence\n",
        "def replace_three_or_more_consecutive_letters(text):\n",
        "    return re.sub(r'(.)\\1{2,}', r'\\1', text)\n",
        "\n",
        "# find quotes with repeated letters based on observations during EDA:\n",
        "repeated_letters = train[train['Quote_clean'].apply(contains_three_or_more_consecutive_letters)]\n",
        "print(repeated_letters['Quote_clean'])\n",
        "\n",
        "# Replace consecutive occurrences of the same letter with a single occurrence in the DataFrame\n",
        "train['Quote_clean2'] = train['Quote_clean'].apply(replace_three_or_more_consecutive_letters)\n",
        "\n",
        "print()\n",
        "print('Quotes after replacing repeated letters')\n",
        "print(train[train['Quote_clean'].apply(contains_three_or_more_consecutive_letters)]['Quote_clean2'])\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Xk5clPkm_XWL",
      "metadata": {
        "id": "Xk5clPkm_XWL"
      },
      "source": [
        "The above output shows quotes with the same letter repeated 3 or more times. In standard English words do not contain the same letter 3 or more times in a row. It is clear that the words with repeated letters are for dramatic effect (e.g. \"heeeere\" instead of \"here\").\n",
        "\n",
        "We replace these repeated letters with single instances of the letter and show the output for the affected quotes. The fix appears to have been applied correctly (e.g. \"heeeere\" is now \"here\").\n",
        "\n",
        "We do not notice that the onomatopoeia of 'eeeeeee' has been replaced with a single 'e'; there is some meaning loss here but it is only a fraction of a much larger quote and again it relates more to an actor's performance rather than the script itself."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YL_b2qd6AgeV",
      "metadata": {
        "id": "YL_b2qd6AgeV"
      },
      "source": [
        "\n",
        "---\n",
        "Next we search for spelling mistakes in our quotes using SymSpellPy and correct them:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 408,
      "id": "ZN_haSTtAlvS",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZN_haSTtAlvS",
        "outputId": "8efbaa3c-92fa-4017-97ce-9bf310ebc16a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "75 quotes out of 633 have been detected as having a spelling mistake (11.85%).\n"
          ]
        }
      ],
      "source": [
        "train['Quote_clean3'] = train['Quote_clean2'].apply(correct_spelling)\n",
        "diff_spelling_indices = train['Quote_clean3'] != train['Quote_clean2']\n",
        "\n",
        "print(f'{diff_spelling_indices.sum()} quotes out of {train.shape[0]} have been detected as having a spelling mistake ({diff_spelling_indices.sum()/train.shape[0]*100 :.2f}%).')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 409,
      "id": "I3GTJzS3DT6X",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I3GTJzS3DT6X",
        "outputId": "923af7c5-24d5-4aca-990a-1a6106495b72"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Examples include:\n",
            "Before:\n",
            "518                                                                                                                        is butter a carb\n",
            "522                              nunchuck skills bowhunting skills computer hacking skills girls only want boyfriends who have great skills\n",
            "547                                                                                      happiness is the most insidious prison of all evey\n",
            "817                                      pwinston tell them tell them all whoever comes whoever it is i will kill them i will kill them all\n",
            "135                                                                                           alright mr demille i am ready for my close up\n",
            "61                                                                                                                           i know kung fu\n",
            "708                                                you have changed bilbo baggins you are not the same hobbit as the one who left the shire\n",
            "332                                                                                you shoot me in a dream you better wake up and apologise\n",
            "535    did you ever hear the tragedy of darth plagueis the wise i thought not it is not a story the jedi would tell you it is a sith legend\n",
            "898                                                                                                    i am as real as a donut motherfucker\n",
            "753                                                                                                                   dress like a kingsman\n",
            "136                                                                                 i ate his liver with some fava beans and a nice chianti\n",
            "755                                                                                                                     wubba lubba dub dub\n",
            "388                                                                               what i need is a real live ghost that is an oxymoron daph\n",
            "417                                                                                                     by grabthar s hammer what a savings\n",
            "Name: Quote_clean2, dtype: object\n",
            "\n",
            "Corrected:\n",
            "518                                                                                                                       is butter a care\n",
            "522                           nun chuck skills bow hunting skills computer hacking skills girls only want boyfriends who have great skills\n",
            "547                                                                                     happiness is the most insidious prison of all even\n",
            "817                                      winston tell them tell them all whoever comes whoever it is i will kill them i will kill them all\n",
            "135                                                                                           alright mr demise i am ready for my close up\n",
            "61                                                                                                                           i know king f\n",
            "708                                               you have changed bilbo bagging you are not the same hobbit as the one who left the shire\n",
            "332                                                                               you shoot me in a dream you better wake up and apologize\n",
            "535    did you ever hear the tragedy of darth plagues the wise i thought not it is not a story the jedi would tell you it is a with legend\n",
            "898                                                                                                   i am as real as a don't motherfucker\n",
            "753                                                                                                                   dress like a kinsman\n",
            "136                                                                                i ate his liver with some java beans and a nice chianti\n",
            "755                                                                                                                    wu ba lu ba dub dub\n",
            "388                                                                              what i need is a real live ghost that is an oxymoron dash\n",
            "417                                                                                                   by grab thar s hammer what a savings\n",
            "Name: Quote_clean3, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('Examples include:')\n",
        "\n",
        "mistake_examples = train[diff_spelling_indices].sample(15, random_state = random_seed)\n",
        "\n",
        "print('Before:')\n",
        "print(mistake_examples['Quote_clean2'])\n",
        "print()\n",
        "print('Corrected:')\n",
        "print(mistake_examples['Quote_clean3'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lEeNMl_0EOjp",
      "metadata": {
        "id": "lEeNMl_0EOjp"
      },
      "source": [
        "After applying the spelling correction algorithm, **we are not satisfied with the results**. Even with a small edit-distance, the corrections appear to introduce mistakes or lead to a loss of meaning in the quote rather than fix genuine spelling mistakes. For example, the correctly spelt word \"carb\" was incorrectly replaced with \"care\" or \"donut\" with \"don't\".\n",
        "\n",
        "This is due to the nature of the algorithm used and the small dictionary size used to find mistakes. Many nouns (e.g. \"Rebecca\") or slang words are not in the dictionary and so are erroneously fixed. This step hence appears to introduce more errors than resolves.\n",
        "\n",
        "Only a small fraction of our quotes are detected to have errors (11%) and most of these detections are false positives (e.g. noun correction) so the impact of true spelling mistakes is likely immaterial; ***we hence decide not to further correct spelling mistakes in our quotes.***\n",
        "\n",
        "---\n",
        "\n",
        "Next we apply the cleaning of repeated letters to our training, validation and test sets:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 410,
      "id": "KI0Z5_v6GKEv",
      "metadata": {
        "id": "KI0Z5_v6GKEv"
      },
      "outputs": [],
      "source": [
        "# We do fix the repeated letters in our training, test and validations datasets\n",
        "\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(replace_three_or_more_consecutive_letters)\n",
        "val['Quote_clean'] = val['Quote_clean'].apply(replace_three_or_more_consecutive_letters)\n",
        "test['Quote_clean'] = test['Quote_clean'].apply(replace_three_or_more_consecutive_letters)\n",
        "\n",
        "# drop unnecessary columns\n",
        "train = train.drop('Quote_clean2', axis = 1)\n",
        "train = train.drop('Quote_clean3', axis = 1)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "j8w4qYM39jH4",
      "metadata": {
        "id": "j8w4qYM39jH4"
      },
      "source": [
        "We also manually fix some spelling errors identified in the training set. The output below shows the affected quotes before and after the fix, making it clear that the change was applied correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 411,
      "id": "NkGS_IhuImAn",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NkGS_IhuImAn",
        "outputId": "0654c361-c838-4d6d-9fd1-6eff5b4b56ba"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Manual correction 1:\n",
            "Before\n",
            "160    smokin\n",
            "Name: Quote_clean, dtype: object\n",
            "After:\n",
            "smoking\n",
            "\n",
            "Manual correction 2:\n",
            "Before\n",
            "49    here s johnny\n",
            "Name: Quote_clean, dtype: object\n",
            "After:\n",
            "here is johnny\n"
          ]
        }
      ],
      "source": [
        "# make manual adjustments to training dataset:\n",
        "# smokin can be corrected, as well as 'here s johnny'. They were not fixed at previous stages due to the repeated letters\n",
        "print('Manual correction 1:')\n",
        "print('Before')\n",
        "print(train.loc[train['Quote_clean'].str.contains(r\"smokin\"), 'Quote_clean']) # check only one quote affected\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(r\"^smokin$\", 'smoking', x))\n",
        "print('After:')\n",
        "print(train.loc[160, 'Quote_clean'])\n",
        "\n",
        "print()\n",
        "\n",
        "print('Manual correction 2:')\n",
        "print('Before')\n",
        "print(train.loc[train['Quote_clean'].str.contains(\"here s johnny\"), 'Quote_clean']) # check only one quote affected\n",
        "train['Quote_clean'] = train['Quote_clean'].apply(lambda x: re.sub(r\"here s johnny\", 'here is johnny', x))\n",
        "print('After:')\n",
        "print(train.loc[49, 'Quote_clean'])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KCZAovHV977N",
      "metadata": {
        "id": "KCZAovHV977N"
      },
      "source": [
        "**Check: look at random sample of quotes to ensure there were not incidental changes**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 412,
      "id": "UugUETPdLwfW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UugUETPdLwfW",
        "outputId": "aec667cc-f87c-49dd-9666-d0e5e0fa6ba9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "480                                                                                                                 there is always hope\n",
            "859                                                                                                        i laugh in the face of danger\n",
            "810    you are right aurora but the drowning man will always try and drag somebody down with him it are not right but the man s drowning\n",
            "713                                               the only thing we are allowed to believe is that we will not regret the choice we made\n",
            "170                                                                                              hey i am walking here i am walking here\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "# final check to see how our quote clean is looking and nothing has been accidentally altered:\n",
        "\n",
        "print(train['Quote_clean'].sample(5, random_state = random_seed))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0zxw9LmJ-Ga2",
      "metadata": {
        "id": "0zxw9LmJ-Ga2"
      },
      "source": [
        "The random sample of quotes appear as expected based on previous cleaning steps. The nouns in the quotes have not been altered given we did not apply the SymSpellPy cleaning step."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jBSFlOxzNNtz",
      "metadata": {
        "id": "jBSFlOxzNNtz"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we:\n",
        "- Corrected repeated letters in quotes: e.g. 'ssssssmokin' -> 'smokin'\n",
        "- Chose **not** to correct other spelling errors as their impact was immaterial and the automated process introduced more errors than resolved.\n",
        "- Some manual corrections were made where appropriate\n",
        "\n",
        "Overall, the movie quotes are mostly free of material spelling errors after this step.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Xyo-Dx9qg_dF",
      "metadata": {
        "id": "Xyo-Dx9qg_dF"
      },
      "source": [
        "#### Q1b Step 7 - Tokenisation\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we tokenise our quotes (i.e. split into individual words).\n",
        "\n",
        "***How:***\n",
        "\n",
        "We do this by using the NLTK library which has a tokeniser function. This function does not just split on whitespace but can also automatically split compound words (e.g. \"cannot\" to \"can not\").\n",
        "\n",
        "***Why:***\n",
        "\n",
        "- **Essential for Vectorisation (if applicable)**: Tokenisation is a crucial step for many vectorisation techniques, including TF-IDF. It breaks down text into the fundamental units (words) that these methods analyse for frequency and relationships. This allows models to understand the meaning conveyed by individual words and their combinations.\n",
        "\n",
        "- **Enables Further Cleaning**: Tokens are the building blocks for further text cleaning steps like stop word removal and lemmatisation.\n",
        "\n",
        "- **Flexibility for Future Analysis**: While embedding models like BERT do not require tokenisation as it gathers meaning from context, we have not finalised our vectorisation method yet. Tokenisation ensures the data is prepared for various analysis approaches (e.g TF-IDF). If we choose to use BERT then this cleaning step is not relevant.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eKknvMRp-u8N",
      "metadata": {
        "id": "eKknvMRp-u8N"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "First we apply tokenisation using the NLTK library:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 413,
      "id": "UXcAkk3ajPxO",
      "metadata": {
        "id": "UXcAkk3ajPxO"
      },
      "outputs": [],
      "source": [
        "# create new column with tokens:\n",
        "train['Quote_tokens'] = train['Quote_clean'].apply(lambda x: list(nltk.word_tokenize(x)))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9WeJOHFzlT44",
      "metadata": {
        "id": "9WeJOHFzlT44"
      },
      "source": [
        "**Check 1: spot check**\n",
        "\n",
        "We print the first 5 rows to get a rough look at if the tokenisation was performed appropriately:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 414,
      "id": "O9gD8X93leOI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O9gD8X93leOI",
        "outputId": "0c21ef36-3ec5-4543-aa00-6fdf5f8e65e7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                               Quote_clean  \\\n",
            "862                                                                                       all i have are negative thoughts   \n",
            "224                                    you only live twice once when you are born and once when you look death in the face   \n",
            "530                                                                                                            i love lamp   \n",
            "710                                                                                                              let it go   \n",
            "295  and to think i have been telling my friends it is so cool living with an artist you never once asked to paint me nude   \n",
            "\n",
            "                                                                                                                                        Quote_tokens  \n",
            "862                                                                                                          [all, i, have, are, negative, thoughts]  \n",
            "224                                           [you, only, live, twice, once, when, you, are, born, and, once, when, you, look, death, in, the, face]  \n",
            "530                                                                                                                                  [i, love, lamp]  \n",
            "710                                                                                                                                    [let, it, go]  \n",
            "295  [and, to, think, i, have, been, telling, my, friends, it, is, so, cool, living, with, an, artist, you, never, once, asked, to, paint, me, nude]  \n"
          ]
        }
      ],
      "source": [
        "print(train[['Quote_clean', 'Quote_tokens']].head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-Sg0WgkTkU46",
      "metadata": {
        "id": "-Sg0WgkTkU46"
      },
      "source": [
        "An initial look at the first 5 rows suggests that the tokenisation has been done correctly since `Quote_tokens` contains a list of the words in `Quote_clean`. Since NLTK can tokenise compound words we check for this by looking at the number of unique words before and after tokenisation:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TlkNswQtnGC6",
      "metadata": {
        "id": "TlkNswQtnGC6"
      },
      "source": [
        "**Check 2**\n",
        "\n",
        "Since NLTK can split compound words we check that this has been done appropriately by looking at the number of unique words before and after tokenisation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 415,
      "id": "P_qIUwFRkTs4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P_qIUwFRkTs4",
        "outputId": "35bef3bd-a77f-497d-f3da-de58697d7548"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 23 quotes that have a different number of words after tokenisation.\n"
          ]
        }
      ],
      "source": [
        "columns = ['Quote_clean', 'Quote_tokens']\n",
        "tokenisation_check = train[columns]\n",
        "\n",
        "# Checks the number of words match - use split() to do simple tokenisation\n",
        "tokenisation_check['Check'] = tokenisation_check.apply(lambda row: 1 if len(row['Quote_clean'].split()) == len(row['Quote_tokens']) else 0, axis=1)\n",
        "\n",
        "print('There are '+ str(len(tokenisation_check[tokenisation_check['Check']==0])) +' quotes that have a different number of words after tokenisation.')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fDuTCtudnt7T",
      "metadata": {
        "id": "fDuTCtudnt7T"
      },
      "source": [
        "Let's look at the mismatches to check for reasonableness and errors:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 416,
      "id": "wFMBsPnGn7nt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "wFMBsPnGn7nt",
        "outputId": "b5be056b-e3f4-403f-c069-9d78b1e62567"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"tokenisation_check[tokenisation_check['Check']==0]\",\n  \"rows\": 23,\n  \"fields\": [\n    {\n      \"column\": \"Quote_clean\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 23,\n        \"samples\": [\n          \"some people cannot believe in themselves until someone else believes in them first\",\n          \"at some point you got to decide for yourself who you going to be cannot let nobody make that decision for you\",\n          \"the law says that you cannot touch but i think i see a lot of lawbreakers up in this house tonight\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Quote_tokens\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Check\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Unique_tokens\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"not , can , cannot\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-6d10f6e4-fadd-4ede-8787-cb9e97275c10\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Quote_clean</th>\n",
              "      <th>Quote_tokens</th>\n",
              "      <th>Check</th>\n",
              "      <th>Unique_tokens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>697</th>\n",
              "      <td>the law says that you cannot touch but i think i see a lot of lawbreakers up in this house tonight</td>\n",
              "      <td>[the, law, says, that, you, can, not, touch, but, i, think, i, see, a, lot, of, lawbreakers, up, in, this, house, tonight]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>724</th>\n",
              "      <td>people who cannot throw something important away can never hope to change anything</td>\n",
              "      <td>[people, who, can, not, throw, something, important, away, can, never, hope, to, change, anything]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>555</th>\n",
              "      <td>i cannot really remember when i last had any hope and i certainly cannot remember when anyone else did either because really since women stopped being able to have babies what is left to hope for</td>\n",
              "      <td>[i, can, not, really, remember, when, i, last, had, any, hope, and, i, certainly, can, not, remember, when, anyone, else, did, either, because, really, since, women, stopped, being, able, to, have, babies, what, is, left, to, hope, for]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>you cannot handle the truth</td>\n",
              "      <td>[you, can, not, handle, the, truth]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>680</th>\n",
              "      <td>because if we cannot protect the earth you can be damn sure we will avenge it</td>\n",
              "      <td>[because, if, we, can, not, protect, the, earth, you, can, be, damn, sure, we, will, avenge, it]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>391</th>\n",
              "      <td>no matter how the wind howls the mountain cannot bow to it</td>\n",
              "      <td>[no, matter, how, the, wind, howls, the, mountain, can, not, bow, to, it]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>568</th>\n",
              "      <td>do not ever let somebody tell you you cannot do something not even me alright</td>\n",
              "      <td>[do, not, ever, let, somebody, tell, you, you, can, not, do, something, not, even, me, alright]</td>\n",
              "      <td>0</td>\n",
              "      <td>can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>870</th>\n",
              "      <td>there is nothing you cannot do if you try</td>\n",
              "      <td>[there, is, nothing, you, can, not, do, if, you, try]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>283</th>\n",
              "      <td>i cannot believe i just gave my panties to a geek</td>\n",
              "      <td>[i, can, not, believe, i, just, gave, my, panties, to, a, geek]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>806</th>\n",
              "      <td>at some point you got to decide for yourself who you going to be cannot let nobody make that decision for you</td>\n",
              "      <td>[at, some, point, you, got, to, decide, for, yourself, who, you, going, to, be, can, not, let, nobody, make, that, decision, for, you]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>583</th>\n",
              "      <td>spider pig spider pig does whatever a spider pig does can he swing from a web no he cannot he is a pig look out he is a spider pig</td>\n",
              "      <td>[spider, pig, spider, pig, does, whatever, a, spider, pig, does, can, he, swing, from, a, web, no, he, can, not, he, is, a, pig, look, out, he, is, a, spider, pig]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>720</th>\n",
              "      <td>some people say it is more hygienic than a handshake but who is to say you cannot get shit on your fist</td>\n",
              "      <td>[some, people, say, it, is, more, hygienic, than, a, handshake, but, who, is, to, say, you, can, not, get, shit, on, your, fist]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>429</th>\n",
              "      <td>there are no accidents no coincidences no escapes you cannot cheat death</td>\n",
              "      <td>[there, are, no, accidents, no, coincidences, no, escapes, you, can, not, cheat, death]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>138</th>\n",
              "      <td>i am going to make him an offer he cannot refuse</td>\n",
              "      <td>[i, am, going, to, make, him, an, offer, he, can, not, refuse]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>179</th>\n",
              "      <td>i am sorry dave i am afraid i cannot do that</td>\n",
              "      <td>[i, am, sorry, dave, i, am, afraid, i, can, not, do, that]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>some people cannot believe in themselves until someone else believes in them first</td>\n",
              "      <td>[some, people, can, not, believe, in, themselves, until, someone, else, believes, in, them, first]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>836</th>\n",
              "      <td>just because something works does not mean it cannot be improved</td>\n",
              "      <td>[just, because, something, works, does, not, mean, it, can, not, be, improved]</td>\n",
              "      <td>0</td>\n",
              "      <td>can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>298</th>\n",
              "      <td>death cannot stop true love</td>\n",
              "      <td>[death, can, not, stop, true, love]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>559</th>\n",
              "      <td>people cannot do something themselves they want to tell you you cannot do it if you want something go get it period</td>\n",
              "      <td>[people, can, not, do, something, themselves, they, want, to, tell, you, you, can, not, do, it, if, you, want, something, go, get, it, period]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>570</th>\n",
              "      <td>the past cannot hurt you anymore unless you let it</td>\n",
              "      <td>[the, past, can, not, hurt, you, anymore, unless, you, let, it]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>220</th>\n",
              "      <td>gentlemen you cannot fight in here this is the war room</td>\n",
              "      <td>[gentlemen, you, can, not, fight, in, here, this, is, the, war, room]</td>\n",
              "      <td>0</td>\n",
              "      <td>not , can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>768</th>\n",
              "      <td>a machine does not improvise well because you cannot program a fear of death our survival instinct is our greatest source of inspiration</td>\n",
              "      <td>[a, machine, does, not, improvise, well, because, you, can, not, program, a, fear, of, death, our, survival, instinct, is, our, greatest, source, of, inspiration]</td>\n",
              "      <td>0</td>\n",
              "      <td>can , cannot</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>surely you cannot be serious i am serious and do not call me shirley</td>\n",
              "      <td>[surely, you, can, not, be, serious, i, am, serious, and, do, not, call, me, shirley]</td>\n",
              "      <td>0</td>\n",
              "      <td>can , cannot</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6d10f6e4-fadd-4ede-8787-cb9e97275c10')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6d10f6e4-fadd-4ede-8787-cb9e97275c10 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6d10f6e4-fadd-4ede-8787-cb9e97275c10');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a7d1cac2-f5bf-4719-9ce0-1bea04d9b605\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a7d1cac2-f5bf-4719-9ce0-1bea04d9b605')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a7d1cac2-f5bf-4719-9ce0-1bea04d9b605 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                                                                                             Quote_clean  \\\n",
              "697                                                                                                   the law says that you cannot touch but i think i see a lot of lawbreakers up in this house tonight   \n",
              "724                                                                                                                   people who cannot throw something important away can never hope to change anything   \n",
              "555  i cannot really remember when i last had any hope and i certainly cannot remember when anyone else did either because really since women stopped being able to have babies what is left to hope for   \n",
              "23                                                                                                                                                                           you cannot handle the truth   \n",
              "680                                                                                                                        because if we cannot protect the earth you can be damn sure we will avenge it   \n",
              "391                                                                                                                                           no matter how the wind howls the mountain cannot bow to it   \n",
              "568                                                                                                                        do not ever let somebody tell you you cannot do something not even me alright   \n",
              "870                                                                                                                                                            there is nothing you cannot do if you try   \n",
              "283                                                                                                                                                    i cannot believe i just gave my panties to a geek   \n",
              "806                                                                                        at some point you got to decide for yourself who you going to be cannot let nobody make that decision for you   \n",
              "583                                                                   spider pig spider pig does whatever a spider pig does can he swing from a web no he cannot he is a pig look out he is a spider pig   \n",
              "720                                                                                              some people say it is more hygienic than a handshake but who is to say you cannot get shit on your fist   \n",
              "429                                                                                                                             there are no accidents no coincidences no escapes you cannot cheat death   \n",
              "138                                                                                                                                                     i am going to make him an offer he cannot refuse   \n",
              "179                                                                                                                                                         i am sorry dave i am afraid i cannot do that   \n",
              "9                                                                                                                     some people cannot believe in themselves until someone else believes in them first   \n",
              "836                                                                                                                                     just because something works does not mean it cannot be improved   \n",
              "298                                                                                                                                                                          death cannot stop true love   \n",
              "559                                                                                  people cannot do something themselves they want to tell you you cannot do it if you want something go get it period   \n",
              "570                                                                                                                                                   the past cannot hurt you anymore unless you let it   \n",
              "220                                                                                                                                              gentlemen you cannot fight in here this is the war room   \n",
              "768                                                             a machine does not improvise well because you cannot program a fear of death our survival instinct is our greatest source of inspiration   \n",
              "65                                                                                                                                  surely you cannot be serious i am serious and do not call me shirley   \n",
              "\n",
              "                                                                                                                                                                                                                                     Quote_tokens  \\\n",
              "697                                                                                                                    [the, law, says, that, you, can, not, touch, but, i, think, i, see, a, lot, of, lawbreakers, up, in, this, house, tonight]   \n",
              "724                                                                                                                                            [people, who, can, not, throw, something, important, away, can, never, hope, to, change, anything]   \n",
              "555  [i, can, not, really, remember, when, i, last, had, any, hope, and, i, certainly, can, not, remember, when, anyone, else, did, either, because, really, since, women, stopped, being, able, to, have, babies, what, is, left, to, hope, for]   \n",
              "23                                                                                                                                                                                                            [you, can, not, handle, the, truth]   \n",
              "680                                                                                                                                              [because, if, we, can, not, protect, the, earth, you, can, be, damn, sure, we, will, avenge, it]   \n",
              "391                                                                                                                                                                     [no, matter, how, the, wind, howls, the, mountain, can, not, bow, to, it]   \n",
              "568                                                                                                                                               [do, not, ever, let, somebody, tell, you, you, can, not, do, something, not, even, me, alright]   \n",
              "870                                                                                                                                                                                         [there, is, nothing, you, can, not, do, if, you, try]   \n",
              "283                                                                                                                                                                               [i, can, not, believe, i, just, gave, my, panties, to, a, geek]   \n",
              "806                                                                                                        [at, some, point, you, got, to, decide, for, yourself, who, you, going, to, be, can, not, let, nobody, make, that, decision, for, you]   \n",
              "583                                                                           [spider, pig, spider, pig, does, whatever, a, spider, pig, does, can, he, swing, from, a, web, no, he, can, not, he, is, a, pig, look, out, he, is, a, spider, pig]   \n",
              "720                                                                                                              [some, people, say, it, is, more, hygienic, than, a, handshake, but, who, is, to, say, you, can, not, get, shit, on, your, fist]   \n",
              "429                                                                                                                                                       [there, are, no, accidents, no, coincidences, no, escapes, you, can, not, cheat, death]   \n",
              "138                                                                                                                                                                                [i, am, going, to, make, him, an, offer, he, can, not, refuse]   \n",
              "179                                                                                                                                                                                    [i, am, sorry, dave, i, am, afraid, i, can, not, do, that]   \n",
              "9                                                                                                                                              [some, people, can, not, believe, in, themselves, until, someone, else, believes, in, them, first]   \n",
              "836                                                                                                                                                                [just, because, something, works, does, not, mean, it, can, not, be, improved]   \n",
              "298                                                                                                                                                                                                           [death, can, not, stop, true, love]   \n",
              "559                                                                                                [people, can, not, do, something, themselves, they, want, to, tell, you, you, can, not, do, it, if, you, want, something, go, get, it, period]   \n",
              "570                                                                                                                                                                               [the, past, can, not, hurt, you, anymore, unless, you, let, it]   \n",
              "220                                                                                                                                                                         [gentlemen, you, can, not, fight, in, here, this, is, the, war, room]   \n",
              "768                                                                            [a, machine, does, not, improvise, well, because, you, can, not, program, a, fear, of, death, our, survival, instinct, is, our, greatest, source, of, inspiration]   \n",
              "65                                                                                                                                                          [surely, you, can, not, be, serious, i, am, serious, and, do, not, call, me, shirley]   \n",
              "\n",
              "     Check       Unique_tokens  \n",
              "697      0  not , can , cannot  \n",
              "724      0        not , cannot  \n",
              "555      0  not , can , cannot  \n",
              "23       0  not , can , cannot  \n",
              "680      0        not , cannot  \n",
              "391      0  not , can , cannot  \n",
              "568      0        can , cannot  \n",
              "870      0  not , can , cannot  \n",
              "283      0  not , can , cannot  \n",
              "806      0  not , can , cannot  \n",
              "583      0        not , cannot  \n",
              "720      0  not , can , cannot  \n",
              "429      0  not , can , cannot  \n",
              "138      0  not , can , cannot  \n",
              "179      0  not , can , cannot  \n",
              "9        0  not , can , cannot  \n",
              "836      0        can , cannot  \n",
              "298      0  not , can , cannot  \n",
              "559      0  not , can , cannot  \n",
              "570      0  not , can , cannot  \n",
              "220      0  not , can , cannot  \n",
              "768      0        can , cannot  \n",
              "65       0        can , cannot  "
            ]
          },
          "execution_count": 416,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Examine quotes with unmatched number of words after tokenisation\n",
        "tokenisation_check['Unique_tokens'] = ''\n",
        "\n",
        "# Iterate through each row in the DataFrame\n",
        "for index, row in tokenisation_check.iterrows():\n",
        "    # Convert the two columns into sets\n",
        "    cleaned_set = set(row['Quote_clean'].split())\n",
        "    tokens_set = set(row['Quote_tokens'])\n",
        "\n",
        "    # Find the unique tokens in each set\n",
        "    cleaned_unique = cleaned_set - tokens_set\n",
        "    tokens_unique = tokens_set - cleaned_set\n",
        "\n",
        "    # Combine the unique tokens into a single set\n",
        "    unique_set = cleaned_unique.union(tokens_unique)\n",
        "\n",
        "    # Store the unique tokens in the 'Unique_tokens' column\n",
        "    tokenisation_check.at[index, 'Unique_tokens'] = ' , '.join(unique_set)\n",
        "\n",
        "# Display results\n",
        "tokenisation_check[tokenisation_check['Check']==0]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oG5KaeG3oSJ5",
      "metadata": {
        "id": "oG5KaeG3oSJ5"
      },
      "source": [
        "As pointed out earlier, it appears all the differences are due to \"cannot\" being tokenised as \"can not\". This is reasonable since some of these \"cannot\"s were likely introduced due to expansion of contractions. This will also reduce the size of our vocablary without loss of meaning.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fLRGJeh__Qs4",
      "metadata": {
        "id": "fLRGJeh__Qs4"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "Finally, we apply tokenisation to our validation and test sets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 417,
      "id": "6o_96NZwvR1r",
      "metadata": {
        "id": "6o_96NZwvR1r"
      },
      "outputs": [],
      "source": [
        "# now we apply tokenisation to our test and validation sets:\n",
        "\n",
        "val['Quote_tokens'] = val['Quote_clean'].apply(lambda x: list(nltk.word_tokenize(x)))\n",
        "test['Quote_tokens'] = test['Quote_clean'].apply(lambda x: list(nltk.word_tokenize(x)))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "K45fqDvHoxGj",
      "metadata": {
        "id": "K45fqDvHoxGj"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we tokenised our Movie quotes. While this step may not be necessary if BERT is used for vectorisation, it gives us the flexibility to use other vectorisation techniques that rely on words being split into tokens (e.g TF-IDF) if we believe these methods are more appropriate. It also faciliates future cleaning steps (stop word removal, lemmatisation)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Dwz068UMpUj8",
      "metadata": {
        "id": "Dwz068UMpUj8"
      },
      "source": [
        "#### Q1b Step 8 - Remove stop words\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we remove stop words from our movie quotes. Stop words are those words that appear frequently but add little meaning (e.g. 'a', 'the'). They can add noise to our data. We remove these words to help our algorithms focus on the meaning-rich parts of the text and thus extract more valuable insights.\n",
        "\n",
        "***How:***\n",
        "\n",
        "We use the NLTK corpus of stopwords to identify and remove stopwords. This is suitable for our standard English movie quotes.\n",
        "We also check for any domain-specific stop words specific to movie quotes but find none.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "Stop word removal is primarily beneficial for traditional frequency-based vectorisation techniques (e.g., TF-IDF) by reducing noise and allowing our models to focus on meaning-rich words.\n",
        "\n",
        "However, for contextual models like BERT, stop words can be informative and contribute to its understanding of a quote. For example, the stop word 'not' provides important context as BERT parses a sentence. Therefore, stop word removal is not necessary if we choose to use BERT for vectorisation."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_5JzDsKWAEEU",
      "metadata": {
        "id": "_5JzDsKWAEEU"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "First we intialise the stopword corpus from the NLTK library and add some additional words to the corpus based on prior observations.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 418,
      "id": "o8QRa6MWt203",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o8QRa6MWt203",
        "outputId": "377a01e1-ab54-428d-b6fa-55dbb380aefc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 181 stopwords in our current stop word list\n"
          ]
        }
      ],
      "source": [
        "# Define stopwords\n",
        "stopword = nltk.corpus.stopwords.words('english')\n",
        "\n",
        "# We add 'a' and 's' to our stop word list\n",
        "# 'a' is a common word that adds little meaning\n",
        "# 's' appeared as a result of prior cleaning steps on possessive terms\n",
        "stop_list = ['a', 's']\n",
        "\n",
        "# Add custom list to stopwords\n",
        "stopword.extend(stop_list)\n",
        "print('There are', len(stopword), 'stopwords in our current stop word list')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 419,
      "id": "TLSF5KTYuZ7e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TLSF5KTYuZ7e",
        "outputId": "ca6a2203-e179-446d-a606-c629bd6d9db6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself']\n"
          ]
        }
      ],
      "source": [
        "# let us look at some of the stop words in our list:\n",
        "\n",
        "print(stopword[1:21])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AqcqiqpZAU2C",
      "metadata": {
        "id": "AqcqiqpZAU2C"
      },
      "source": [
        "There are 181 stopwords in our stopword list. We now remove these stopwords from our lists of tokens:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 420,
      "id": "Ipo_vd8JvZtX",
      "metadata": {
        "id": "Ipo_vd8JvZtX"
      },
      "outputs": [],
      "source": [
        "# Remove stop words\n",
        "train['Quote_tokens_noSW'] = train['Quote_tokens'].apply(\n",
        "    lambda x: [word for word in x if word not in stopword])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TAymrBjtvkke",
      "metadata": {
        "id": "TAymrBjtvkke"
      },
      "source": [
        "**Check 1: spot check 5 random rows**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 421,
      "id": "Qqoe9KwKvrbB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qqoe9KwKvrbB",
        "outputId": "46d2d7c3-237d-4f52-c374-50a37b561ad9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                                                                                                                                                                                                                                                              Quote_tokens  \\\n",
            "386  [people, live, their, lives, bound, by, what, they, accept, as, correct, and, true, that, is, how, they, define, reality, but, what, does, it, mean, to, be, correct, or, true, merely, vague, concepts, their, reality, may, all, be, a, mirage, can, we, consider, them, to, simply, be, living, in, their, own, world, shaped, by, their, beliefs]   \n",
            "588                                                                                                                                                                                                                                                                       [as, long, as, the, concept, of, winners, exists, there, must, also, be, losers]   \n",
            "580                                                                                                                                                                                                                                        [if, we, admit, that, human, life, can, be, ruled, by, reason, then, all, possibility, of, life, is, destroyed]   \n",
            "714                                                                                                                                                                                                                             [you, have, got, to, be, willing, to, crash, and, burn, if, you, are, afraid, of, failing, you, will, not, get, very, far]   \n",
            "889                                                                                                                                                                                                                                    [i, do, what, i, do, because, of, my, dad, he, was, a, hero, he, gave, his, life, for, the, pursuit, of, knowledge]   \n",
            "\n",
            "                                                                                                                                                                              Quote_tokens_noSW  \n",
            "386  [people, live, lives, bound, accept, correct, true, define, reality, mean, correct, true, merely, vague, concepts, reality, may, mirage, consider, simply, living, world, shaped, beliefs]  \n",
            "588                                                                                                                                        [long, concept, winners, exists, must, also, losers]  \n",
            "580                                                                                                                           [admit, human, life, ruled, reason, possibility, life, destroyed]  \n",
            "714                                                                                                                                      [got, willing, crash, burn, afraid, failing, get, far]  \n",
            "889                                                                                                                                                 [dad, hero, gave, life, pursuit, knowledge]  \n"
          ]
        }
      ],
      "source": [
        "sampled_df = train.sample(n=5, random_state = 100) # random sample\n",
        "\n",
        "print(sampled_df[['Quote_tokens', 'Quote_tokens_noSW']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kU2jTkFgwbRA",
      "metadata": {
        "id": "kU2jTkFgwbRA"
      },
      "source": [
        "In the above spot checks we identify that stop words like \"the\", \"is\" \"i\" etc. have been removed from the lists of tokens."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_kIIL7Xdwk8k",
      "metadata": {
        "id": "_kIIL7Xdwk8k"
      },
      "source": [
        "**Check 2: most common words before and after stop words removal**\n",
        "\n",
        "We conduct this check to confirm that common words have been removed. We also conduct this check to identify any additional stop words that may need to be removed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 422,
      "id": "-ukQTYd1wzYu",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "-ukQTYd1wzYu",
        "outputId": "ce60a813-9fcb-4e0f-9a7b-d37f9d20ef65"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most Common Tokens before removing stop words:\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_most_common\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"Token\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"you\",\n          \"can\",\n          \"we\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Frequency\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 88,\n        \"min\": 65,\n        \"max\": 358,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          358,\n          207,\n          108\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_most_common"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-f4ccf385-6bf4-4e21-8d2a-621d15617297\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Token</th>\n",
              "      <th>Frequency</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>you</td>\n",
              "      <td>358</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>the</td>\n",
              "      <td>280</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>is</td>\n",
              "      <td>269</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>i</td>\n",
              "      <td>267</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>to</td>\n",
              "      <td>230</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>a</td>\n",
              "      <td>207</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>not</td>\n",
              "      <td>180</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>it</td>\n",
              "      <td>174</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>and</td>\n",
              "      <td>128</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>of</td>\n",
              "      <td>115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>are</td>\n",
              "      <td>113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>do</td>\n",
              "      <td>108</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>that</td>\n",
              "      <td>90</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>in</td>\n",
              "      <td>84</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>have</td>\n",
              "      <td>76</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>we</td>\n",
              "      <td>74</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>be</td>\n",
              "      <td>70</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>can</td>\n",
              "      <td>69</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>am</td>\n",
              "      <td>69</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>me</td>\n",
              "      <td>65</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f4ccf385-6bf4-4e21-8d2a-621d15617297')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f4ccf385-6bf4-4e21-8d2a-621d15617297 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f4ccf385-6bf4-4e21-8d2a-621d15617297');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-41a70f14-f9e0-43fc-bb7a-f97ee659e00f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-41a70f14-f9e0-43fc-bb7a-f97ee659e00f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-41a70f14-f9e0-43fc-bb7a-f97ee659e00f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_7cc23403-8cbe-49b2-9a08-f8183c6402c8\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_most_common')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_7cc23403-8cbe-49b2-9a08-f8183c6402c8 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_most_common');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   Token  Frequency\n",
              "0    you        358\n",
              "1    the        280\n",
              "2     is        269\n",
              "3      i        267\n",
              "4     to        230\n",
              "5      a        207\n",
              "6    not        180\n",
              "7     it        174\n",
              "8    and        128\n",
              "9     of        115\n",
              "10   are        113\n",
              "11    do        108\n",
              "12  that         90\n",
              "13    in         84\n",
              "14  have         76\n",
              "15    we         74\n",
              "16    be         70\n",
              "17   can         69\n",
              "18    am         69\n",
              "19    me         65"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most Common Tokens after removing stop words:\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_most_common_noSW\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"Token_noSW\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"like\",\n          \"ever\",\n          \"look\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Frequency_noSW\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 7,\n        \"min\": 18,\n        \"max\": 44,\n        \"num_unique_values\": 14,\n        \"samples\": [\n          22,\n          20,\n          44\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_most_common_noSW"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-6e23b65d-f0a1-4ff8-8a0f-30b0ff4abe98\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Token_noSW</th>\n",
              "      <th>Frequency_noSW</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>like</td>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>know</td>\n",
              "      <td>43</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>life</td>\n",
              "      <td>32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>never</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>one</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>people</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>get</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>love</td>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>want</td>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>go</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>time</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>always</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>man</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>something</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>come</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>look</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>let</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>ever</td>\n",
              "      <td>19</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>would</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>thing</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6e23b65d-f0a1-4ff8-8a0f-30b0ff4abe98')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6e23b65d-f0a1-4ff8-8a0f-30b0ff4abe98 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6e23b65d-f0a1-4ff8-8a0f-30b0ff4abe98');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-92b7a4c9-3397-489e-be96-46e4c704dd52\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-92b7a4c9-3397-489e-be96-46e4c704dd52')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-92b7a4c9-3397-489e-be96-46e4c704dd52 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_495fce70-ef9b-4cec-8f6b-0ec9b66593a0\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_most_common_noSW')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_495fce70-ef9b-4cec-8f6b-0ec9b66593a0 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_most_common_noSW');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   Token_noSW  Frequency_noSW\n",
              "0        like              44\n",
              "1        know              43\n",
              "2        life              32\n",
              "3       never              31\n",
              "4         one              31\n",
              "5      people              30\n",
              "6         get              29\n",
              "7        love              28\n",
              "8        want              25\n",
              "9          go              23\n",
              "10       time              23\n",
              "11     always              23\n",
              "12        man              22\n",
              "13  something              21\n",
              "14       come              20\n",
              "15       look              19\n",
              "16        let              19\n",
              "17       ever              19\n",
              "18      would              18\n",
              "19      thing              18"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# frequency of words - most common word before and after removing stop words\n",
        "n_most_common_words = 20\n",
        "\n",
        "all_tokens = [token for sublist in train['Quote_tokens'].dropna() for token in sublist]\n",
        "token_freq = Counter(all_tokens)\n",
        "df_most_common = pd.DataFrame(token_freq.most_common(n_most_common_words), columns=['Token', 'Frequency'])\n",
        "\n",
        "all_tokens_noSW = [token for sublist in train['Quote_tokens_noSW'].dropna() for token in sublist]\n",
        "token_freq_noSW = Counter(all_tokens_noSW)\n",
        "df_most_common_noSW = pd.DataFrame(token_freq_noSW.most_common(n_most_common_words), columns=['Token_noSW', 'Frequency_noSW'])\n",
        "\n",
        "print(\"Most Common Tokens before removing stop words:\")\n",
        "display(df_most_common)\n",
        "print(\"Most Common Tokens after removing stop words:\")\n",
        "display(df_most_common_noSW)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JR0OJt4_xW9N",
      "metadata": {
        "id": "JR0OJt4_xW9N"
      },
      "source": [
        "We can see that all of the top 20 most common words have now been removed. The current top words all appear to communicate some meaning and each appear less than 40 times within our training set. For example, the word 'know' and 'life' have high-semantic meaning and so are likely not stop words.\n",
        "\n",
        "None of the current top words appear to be domain-specific and none of them have a frequency significantly greater than others. This suggests that we do not need to remove any additional stop words.\n",
        "\n",
        "---\n",
        "\n",
        "We now apply stop word removal to the validation and test sets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 423,
      "id": "ai-C0K2-8ztG",
      "metadata": {
        "id": "ai-C0K2-8ztG"
      },
      "outputs": [],
      "source": [
        "# apply stop word removal to validation and test sets:\n",
        "\n",
        "val['Quote_tokens_noSW'] = val['Quote_tokens'].apply(\n",
        "    lambda x: [word for word in x if word not in stopword])\n",
        "\n",
        "test['Quote_tokens_noSW'] = test['Quote_tokens'].apply(\n",
        "    lambda x: [word for word in x if word not in stopword])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "t_wuzq5-yqtC",
      "metadata": {
        "id": "t_wuzq5-yqtC"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we removed stop words from our quotes.\n",
        "\n",
        "It was determined that there were no additional domain-specific stop words that needed to be removed."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oGTcKbNizi_-",
      "metadata": {
        "id": "oGTcKbNizi_-"
      },
      "source": [
        "#### Q1b Step 9 - Lemmatisation\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we conduct lemmatisation on our movie quote tokens. Lemmatisation is the process of reducing words to their root form whilst ensuring that these root forms are dictionary words. The related technique, stemming, only removes suffixes/prefixes and hence is not as accurate as lemmatisation. This is why we choose lemmatisation over stemming.\n",
        "\n",
        "***How:***\n",
        "\n",
        "We conduct lemmatisation using the WordNet lemmatiser that was developed on a large set of English text.\n",
        "\n",
        "***Why:***\n",
        "Lemmatisation is done in order to:\n",
        "- Normalise words that are semantically equivalent even if their form is different (e.g. 'woman' and 'women'). This allows our algorithm to focus on meaning which will be useful for our 'problematic quote' classifier later.\n",
        "- Dimension reduction by decreasing our corpus' vocabulary size. This can improve model performance.\n",
        "\n",
        "However, the BERT embedder, which is used later, can understand words based on context and does not need tokens to be lemmatised. This cleaning step is therefore only relevant if we use a frequency-based vectorisation method. We have not yet finalised the method we will use so apply the step here for flexibility.\n",
        "\n",
        "---\n",
        "Apply lemmatisation to the training set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 424,
      "id": "oeilG1ME27DC",
      "metadata": {
        "id": "oeilG1ME27DC"
      },
      "outputs": [],
      "source": [
        "# Initialise lemmatiser\n",
        "lemmatiser = WordNetLemmatizer()\n",
        "\n",
        "# Perform lemmatisation on training set\n",
        "train['Quote_tokens_noSW_lemm'] = [[lemmatiser.lemmatize(word) for word in quote] for quote in train['Quote_tokens_noSW']]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aWSiCH9H36Y4",
      "metadata": {
        "id": "aWSiCH9H36Y4"
      },
      "source": [
        "**Check 1: spot check on random samples**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 425,
      "id": "IVQEcbm84FTL",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IVQEcbm84FTL",
        "outputId": "7bd62c48-e6a9-4329-e2fd-21c82b45b7b4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Before lemmatisation:\n",
            "                                                                 Quote_tokens_noSW\n",
            "397  [single, grain, rice, tip, scale, one, man, may, difference, victory, defeat]\n",
            "360                                                      [life, always, hard, kid]\n",
            "96                                                       [ding, dong, witch, dead]\n",
            "869            [pearl, harbor, greatest, intelligence, failure, american, history]\n",
            "133                                                          [alive, alive, alive]\n",
            "140                                      [know, fredo, broke, heart, broke, heart]\n",
            "694                                 [genius, billionaire, playboy, philanthropist]\n",
            "91                                                             [run, forrest, run]\n",
            "495                            [thing, romance, people, get, together, right, end]\n",
            "144                                                     [goonies, never, say, die]\n",
            "\n",
            "After lemmatisation:\n",
            "                                                            Quote_tokens_noSW_lemm\n",
            "397  [single, grain, rice, tip, scale, one, man, may, difference, victory, defeat]\n",
            "360                                                      [life, always, hard, kid]\n",
            "96                                                       [ding, dong, witch, dead]\n",
            "869            [pearl, harbor, greatest, intelligence, failure, american, history]\n",
            "133                                                          [alive, alive, alive]\n",
            "140                                      [know, fredo, broke, heart, broke, heart]\n",
            "694                                 [genius, billionaire, playboy, philanthropist]\n",
            "91                                                             [run, forrest, run]\n",
            "495                            [thing, romance, people, get, together, right, end]\n",
            "144                                                       [goony, never, say, die]\n"
          ]
        }
      ],
      "source": [
        "sampled_df = train.sample(n=10, random_state = 223)\n",
        "print('Before lemmatisation:')\n",
        "print(sampled_df[['Quote_tokens_noSW']])\n",
        "\n",
        "print()\n",
        "print('After lemmatisation:')\n",
        "print(sampled_df[['Quote_tokens_noSW_lemm']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18HxzTlv6SX1",
      "metadata": {
        "id": "18HxzTlv6SX1"
      },
      "source": [
        "We do see that some words have been lemmatised. For example, 'children' has become 'child', and 'demons' has become 'demon'. These do correctly appear to be root forms of the respective words.\n",
        "\n",
        "From this random sample, it appears that most words are unchanged."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Qmie-bAi60Lm",
      "metadata": {
        "id": "Qmie-bAi60Lm"
      },
      "source": [
        "**Check 2: check if number of unique tokens has reduced**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9RN6Kr5M7Jvu",
      "metadata": {
        "id": "9RN6Kr5M7Jvu"
      },
      "source": [
        "We see below that the number of unique tokens has reduced after conducting lemmatisation, indicating that it has been done successfully. There is not a substantial decrease in the number of unique words (< 100 word decrease in number of unique tokens)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 426,
      "id": "DspgHISh65qI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DspgHISh65qI",
        "outputId": "fe262617-b818-440a-cc3d-cd373db44657"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of unique tokens before lemmatisation:  1700\n",
            "Number of unique tokens after lemmatisation:  1609\n"
          ]
        }
      ],
      "source": [
        "# Create a list of all tokens in the dataset\n",
        "tokens_lemmatised = list(itertools.chain.from_iterable(train['Quote_tokens_noSW_lemm']))\n",
        "tokens_noSW = itertools.chain.from_iterable(train['Quote_tokens_noSW'])\n",
        "\n",
        "# Create a set of unique tokens\n",
        "unique_tokens_lemmatised = set(tokens_lemmatised)\n",
        "unique_tokens_noSW = set(tokens_noSW)\n",
        "\n",
        "# Print the number of unique tokens\n",
        "print(\"Number of unique tokens before lemmatisation: \", len(unique_tokens_noSW))\n",
        "print(\"Number of unique tokens after lemmatisation: \", len(unique_tokens_lemmatised))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "houMNJ0q7uql",
      "metadata": {
        "id": "houMNJ0q7uql"
      },
      "source": [
        "**Check 3: look at if frequency of most common words has changed**\n",
        "\n",
        "We expect that the frequency of some of the most common words changes after lemmatisation as some related words are reduced to the more common form.\n",
        "\n",
        "We see that 'life' has more occurences after lemmatisation likely due to related words such as 'live' and 'lives' being reduced to 'life'.\n",
        "\n",
        "This indicates lemmatisation has been done correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 427,
      "id": "ljTn3sE87dp2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 743
        },
        "id": "ljTn3sE87dp2",
        "outputId": "67bd564a-4d08-4035-b2c9-db89ad038557"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most Common Tokens before lemmatisation:\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_most_common\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"Token\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"want\",\n          \"know\",\n          \"people\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Frequency\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6,\n        \"min\": 23,\n        \"max\": 44,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          25,\n          43,\n          29\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_most_common"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-6ed9a2c5-f718-4e63-94a3-a84fa46bd12d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Token</th>\n",
              "      <th>Frequency</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>like</td>\n",
              "      <td>44</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>know</td>\n",
              "      <td>43</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>life</td>\n",
              "      <td>32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>never</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>one</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>people</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>get</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>love</td>\n",
              "      <td>28</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>want</td>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>go</td>\n",
              "      <td>23</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6ed9a2c5-f718-4e63-94a3-a84fa46bd12d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6ed9a2c5-f718-4e63-94a3-a84fa46bd12d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6ed9a2c5-f718-4e63-94a3-a84fa46bd12d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5b17ca75-7d3c-483f-a4fb-aedd9b737803\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5b17ca75-7d3c-483f-a4fb-aedd9b737803')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5b17ca75-7d3c-483f-a4fb-aedd9b737803 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_8128dde1-518a-4dbd-8a1b-473f1a5c8fad\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_most_common')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_8128dde1-518a-4dbd-8a1b-473f1a5c8fad button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_most_common');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "    Token  Frequency\n",
              "0    like         44\n",
              "1    know         43\n",
              "2    life         32\n",
              "3   never         31\n",
              "4     one         31\n",
              "5  people         30\n",
              "6     get         29\n",
              "7    love         28\n",
              "8    want         25\n",
              "9      go         23"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most Common Tokens after lemmatisation:\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_most_common_lemm\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"Token_lemm\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"love\",\n          \"know\",\n          \"never\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Frequency_lemm\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 7,\n        \"min\": 25,\n        \"max\": 46,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          40,\n          30,\n          46\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_most_common_lemm"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-c7169c3e-0c21-4110-9921-be7aa32b6ffd\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Token_lemm</th>\n",
              "      <th>Frequency_lemm</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>like</td>\n",
              "      <td>46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>know</td>\n",
              "      <td>46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>life</td>\n",
              "      <td>40</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>one</td>\n",
              "      <td>34</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>get</td>\n",
              "      <td>32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>never</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>people</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>thing</td>\n",
              "      <td>30</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>love</td>\n",
              "      <td>29</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>want</td>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c7169c3e-0c21-4110-9921-be7aa32b6ffd')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c7169c3e-0c21-4110-9921-be7aa32b6ffd button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c7169c3e-0c21-4110-9921-be7aa32b6ffd');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7c9ed5db-d595-45aa-894b-0fae8bd823da\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7c9ed5db-d595-45aa-894b-0fae8bd823da')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7c9ed5db-d595-45aa-894b-0fae8bd823da button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_18a1ae7c-e241-410a-8b4f-1ec2b8c84660\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_most_common_lemm')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_18a1ae7c-e241-410a-8b4f-1ec2b8c84660 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_most_common_lemm');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "  Token_lemm  Frequency_lemm\n",
              "0       like              46\n",
              "1       know              46\n",
              "2       life              40\n",
              "3        one              34\n",
              "4        get              32\n",
              "5      never              31\n",
              "6     people              30\n",
              "7      thing              30\n",
              "8       love              29\n",
              "9       want              25"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# frequency of words - most common word before and after lemmatisation\n",
        "all_tokens = [token for sublist in train['Quote_tokens_noSW'].dropna() for token in sublist]\n",
        "token_freq = Counter(all_tokens)\n",
        "df_most_common = pd.DataFrame(token_freq.most_common(10), columns=['Token', 'Frequency'])\n",
        "\n",
        "all_tokens_lemm = [token for sublist in train['Quote_tokens_noSW_lemm'].dropna() for token in sublist]\n",
        "token_freq_lemm = Counter(all_tokens_lemm)\n",
        "df_most_common_lemm = pd.DataFrame(token_freq_lemm.most_common(10), columns=['Token_lemm', 'Frequency_lemm'])\n",
        "\n",
        "print(\"Most Common Tokens before lemmatisation:\")\n",
        "display(df_most_common)\n",
        "print(\"Most Common Tokens after lemmatisation:\")\n",
        "display(df_most_common_lemm)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 428,
      "id": "AKtfbhFU8iRy",
      "metadata": {
        "id": "AKtfbhFU8iRy"
      },
      "outputs": [],
      "source": [
        "# We now apply lemmatisation to the validation and test set:\n",
        "\n",
        "val['Quote_tokens_noSW_lemm'] = [[lemmatiser.lemmatize(word) for word in quote] for quote in val['Quote_tokens_noSW']]\n",
        "test['Quote_tokens_noSW_lemm'] = [[lemmatiser.lemmatize(word) for word in quote] for quote in test['Quote_tokens_noSW']]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X7qoDIZg8hZb",
      "metadata": {
        "id": "X7qoDIZg8hZb"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we applied lemmatisation to our movie quote tokens.\n",
        "\n",
        "This was the final step of cleaning our quotes in preparation for vectorisation and modelling.  We next conduct cleaning on the `Genre` column."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1rjVhfLt9PAN",
      "metadata": {
        "id": "1rjVhfLt9PAN"
      },
      "source": [
        "#### Q1b Step 10 - Cleaning the `Genre` column\n",
        "**Outline**\n",
        "\n",
        "***What and How:***\n",
        "\n",
        "The `Genre` column contains up to 3 genres separated by commas. In this format it is not useable since our models cannot identify the individual genres within the string. We hence split up this string into its constituent genres and create dummy variables for each genre.\n",
        "\n",
        "The individual genres within the column are also not clean and have high cardinality. We hence attempt to clean the individual names and group some genres together to reduce our feature space.\n",
        "\n",
        "***Why:***\n",
        "- Get `Genre` in a form that can be analysed and used in our models\n",
        "- Grouping of genre reduces the dimension of the feature space which will likely improve our model performance by reducing sparse data and noise.\n",
        "\n",
        "It should be noted that genre is really a characteristic of a movie rather than a quote. The objective of Q2 is to understand *quotes* not *movies* and so **genre columns will not be used in the Q2 analysis**.\n",
        "\n",
        "***Note:***\n",
        "Will genre be known at the time of deployment?\n",
        "\n",
        "Typically when writing a script, a screenwriter will have a genre in mind such as whether it is a comedy or horror etc. Therefore we believe that the genre will be known at the time the script is finished and hence known at deployment. This means that **the genre columns could be used in our neural network for Q3.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HUcg_sPqCG7k",
      "metadata": {
        "id": "HUcg_sPqCG7k"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "First, we split the genre list into its constitutent tokens and print out the counts of each unique genre."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 429,
      "id": "NN3dFBha0n9q",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NN3dFBha0n9q",
        "outputId": "07a600aa-f04d-4433-b12a-02b03f6efb80"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Unique values and their counts:\n",
            "Drama              278\n",
            "Adventure          201\n",
            "Comedy             183\n",
            "Action             177\n",
            "Fantasy             99\n",
            "Crime               97\n",
            "Thriller            86\n",
            "Sci-Fi              86\n",
            "Romance             54\n",
            "Animation           51\n",
            "Biography           38\n",
            "Family              33\n",
            "Horror              20\n",
            "Mystery             17\n",
            "History             13\n",
            "War                 12\n",
            "Sport               10\n",
            "Music                9\n",
            "Romantic comedy      6\n",
            "Anime                5\n",
            "Science Fiction      4\n",
            "Romantic Comedy      4\n",
            "Animated             4\n",
            "Western              3\n",
            "Film Noir            3\n",
            "Musical              2\n",
            "Teen                 1\n",
            " Action              1\n",
            "Dark Fantasy         1\n",
            "Sports drama         1\n",
            "Mockumentary         1\n",
            "Biographical         1\n",
            "Unknown              1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Total number of unique values: 33\n"
          ]
        }
      ],
      "source": [
        "# First we split the genre string into its constituent tokens/genres\n",
        "train['Genre_split'] = train['Genre'].str.split(', ')\n",
        "\n",
        "# how many unique genres do we have and what are they:\n",
        "\n",
        "# Flatten the lists in the column\n",
        "flattened_list = [genre for sublist in train['Genre_split'] for genre in sublist]\n",
        "\n",
        "# Get unique values and their counts\n",
        "unique_values = pd.Series(flattened_list).value_counts()\n",
        "\n",
        "print(\"Unique values and their counts:\")\n",
        "print(unique_values)\n",
        "print(\"\\nTotal number of unique values:\", len(unique_values))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_XABpjMS3G_u",
      "metadata": {
        "id": "_XABpjMS3G_u"
      },
      "source": [
        "We have 33 unique genres which is a signficant amount given our lack of data.  Many of these genres only contain a handful of observations, meaning that the individual Genres feature is quite sparse.\n",
        "\n",
        "We will next clean some of these genres up such as ' Action' becoming 'Action' to reduce some of this sparsity. Then we will use domain-knowledge to group up similar categories to further reduce the feature space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 430,
      "id": "pc7C0GKW3pmE",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pc7C0GKW3pmE",
        "outputId": "52d86c25-fc76-41ea-a3b3-84604773aff7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "New genres after cleaning:\n",
            "Unique values and their counts:\n",
            "Drama           279\n",
            "Adventure       201\n",
            "Comedy          193\n",
            "Action          178\n",
            "Crime           100\n",
            "Fantasy         100\n",
            "Thriller         95\n",
            "Sci-Fi           90\n",
            "Romance          64\n",
            "Animation        60\n",
            "Biography        38\n",
            "Family           33\n",
            "Horror           20\n",
            "History          13\n",
            "War              12\n",
            "Music            11\n",
            "Sport            11\n",
            "Western           3\n",
            "Teen              1\n",
            "Mockumentary      1\n",
            "Biographical      1\n",
            "Unknown           1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Total number of unique values: 22\n"
          ]
        }
      ],
      "source": [
        "# Clean up genres\n",
        "\n",
        "# Create function that will replace a genre 'prev' with a new genre 'new'\n",
        "def replace_genre(genre_list, prev, new):\n",
        "    # check value is a list\n",
        "    if isinstance(genre_list, list):\n",
        "        return [new if genre == prev else genre for genre in genre_list]\n",
        "    else:\n",
        "        return new if genre_list == prev else genre_list\n",
        "\n",
        "# Create a function that will replace a genre with two genres\n",
        "# For example, splitting 'Romantic Comedy' into 'Romance' and 'Comedy'\n",
        "def replace_double_genre(genre_list, prev, new1, new2):\n",
        "    new_list = []\n",
        "    if isinstance(genre_list, list):\n",
        "        for genre in genre_list:\n",
        "            if genre == prev: # replace with two genres:\n",
        "                new_list.append(new1)\n",
        "                new_list.append(new2)\n",
        "            else:\n",
        "                new_list.append(genre)\n",
        "        return list(set(new_list)) # return unique values\n",
        "    else:\n",
        "        return [new1, new2] if genre_list == prev else genre_list\n",
        "\n",
        "# Clean up Genres that can easily be grouped (use domain knowledge and common sense)\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Science Fiction', 'Sci-Fi')) # replace Science Fiction with 'Sci-Fi'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, ' Action', 'Action')) # replace ' Action' with 'Action'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Anime', 'Animation')) # replace 'Anime' with 'Animation'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Animated', 'Animation')) # replace 'Animated' with 'Animation'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Film Noir', 'Crime')) # replace 'Film Noir' with 'Crime'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Musical', 'Music')) # replace 'Musical' with 'Music'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Biograpical', 'Biography')) # replace 'Biographical' with 'Biography'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Mystery', 'Thriller')) # replace 'Mystery' with 'Thriller'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Dark Fantasy', 'Fantasy')) # replace 'Mystery' with 'Thriller'\n",
        "\n",
        "# Clean up double genres such as 'Romantic Comedy' and 'Science fiction/Action'\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_double_genre(x, 'Science fiction/Action', 'Sci-Fi', 'Action'))\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_double_genre(x, 'Romantic Comedy', 'Romance', 'Comedy'))\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_double_genre(x, 'Romantic comedy', 'Romance', 'Comedy'))\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_double_genre(x, 'Sports drama', 'Sport', 'Drama'))\n",
        "\n",
        "## Let us look at the unique values now:\n",
        "# Flatten the lists in the column\n",
        "flattened_list = [genre for sublist in train['Genre_split'] for genre in sublist]\n",
        "\n",
        "# Get unique values and their counts\n",
        "unique_values = pd.Series(flattened_list).value_counts()\n",
        "\n",
        "print('New genres after cleaning:')\n",
        "print(\"Unique values and their counts:\")\n",
        "print(unique_values)\n",
        "print(\"\\nTotal number of unique values:\", len(unique_values))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "UPvV-KyO9ORa",
      "metadata": {
        "id": "UPvV-KyO9ORa"
      },
      "source": [
        "After some intial cleaning we are down to 22 unique categories in the training set. This is still likely too many, especially given some contain only a few quotes/movies.\n",
        "We attempt to group similar genres together and where a genre has less than 40 observations we remove it. Again this is to keep our feature space lean which is essential given our lack of data if we want to produce results that are useful and generalise well (more features increased variance error)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 431,
      "id": "eoOsCk779w3U",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eoOsCk779w3U",
        "outputId": "821ee823-ecad-4b71-e84f-76eccb306632"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Final genres after grouping and deletion:\n",
            "Unique values and their counts:\n",
            "ActionAdventure    379\n",
            "Drama              279\n",
            "Comedy             193\n",
            "NA                 125\n",
            "ThrillerHorror     115\n",
            "Crime              100\n",
            "Fantasy            100\n",
            "Sci-Fi              90\n",
            "Romance             64\n",
            "Animation           60\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Total number of unique values: 10\n"
          ]
        }
      ],
      "source": [
        "# Create function that will remove all genres not in 'keep_list'\n",
        "def remove_genre(genre_list, keep_list):\n",
        "    if isinstance(genre_list, list):\n",
        "        return [genre if genre in keep_list else 'NA' for genre in genre_list]\n",
        "    else:\n",
        "        return genre_list if genre_list in keep_list else 'NA'\n",
        "\n",
        "# Group together Thriller and Horror\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Thriller', 'ThrillerHorror'))\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Horror', 'ThrillerHorror'))\n",
        "\n",
        "# Group together Action and Adventure due to their similarities\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Action', 'ActionAdventure'))\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: replace_genre(x, 'Adventure', 'ActionAdventure'))\n",
        "\n",
        "genre_keep_list = ['Drama', 'Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy', 'Sci-Fi', 'Animation', 'Romance']\n",
        "\n",
        "train['Genre_split'] = train['Genre_split'].apply(lambda x: remove_genre(x, genre_keep_list))\n",
        "\n",
        "\n",
        "# Flatten the lists in the column\n",
        "flattened_list = [genre for sublist in train['Genre_split'] for genre in sublist]\n",
        "\n",
        "# Get unique values and their counts\n",
        "unique_values = pd.Series(flattened_list).value_counts()\n",
        "\n",
        "print('Final genres after grouping and deletion:')\n",
        "print(\"Unique values and their counts:\")\n",
        "print(unique_values)\n",
        "print(\"\\nTotal number of unique values:\", len(unique_values))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MNbGIqDp_VNd",
      "metadata": {
        "id": "MNbGIqDp_VNd"
      },
      "source": [
        "We now have 9 unique genre categories which is substantially less than the original 33. This reduces a lot of the noise in the genre categories. Note the 'NA' category will not be kept.\n",
        "\n",
        "We now convert the `Genre_split` column into binary variables for each genre representing whether the quote is from the particular genre or not."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 432,
      "id": "ZQwoz5fMBghp",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZQwoz5fMBghp",
        "outputId": "3544963e-a096-4593-ae34-749cc0648259"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                              Movie Quote  \\\n",
            "862                                                                                     All I have are negative thoughts.   \n",
            "224                                   You only live twice:Once when you are born And once when you look death in the face   \n",
            "530                                                                                                          I love lamp.   \n",
            "710                                                                                                            Let it go!   \n",
            "295  And to think I've been telling my friends it's so cool living with an artist. You never once asked to paint me nude!   \n",
            "\n",
            "                                Movie Name            Actor  \\\n",
            "862                                  Joker  Joaquin Phoenix   \n",
            "224                    You only live twice     Sean Connery   \n",
            "530  Anchorman: The Legend of Ron Burgundy     Will Ferrell   \n",
            "710                                 Frozen     Idina Menzel   \n",
            "295                *batteries not included      Hume Cronyn   \n",
            "\n",
            "              Character  Year                         Genre  \\\n",
            "862  Arthur Fleck/Joker  2019        Crime, Drama, Thriller   \n",
            "224          James Bond  1967   Action, Adventure, Thriller   \n",
            "530        Ron Burgundy  2004                        Comedy   \n",
            "710                Elsa  2013  Animation, Adventure, Comedy   \n",
            "295         Frank Riley  1987        Comedy, Drama, Fantasy   \n",
            "\n",
            "                     Director                                      Writer  \\\n",
            "862             Todd Phillips                 Todd Phillips, Scott Silver   \n",
            "224             Lewis Gilbert  Roald Dahl, Harold Jack Bloom, Ian Fleming   \n",
            "530                Adam McKay                    Will Ferrell, Adam McKay   \n",
            "710  Chris Buck, Jennifer Lee      Jennifer Lee, Chris Buck, Shane Morris   \n",
            "295           Matthew Robbins                                 Mick Garris   \n",
            "\n",
            "                      Production Company  Sentiment_VADER  ...  \\\n",
            "862  DC Films, Village Roadshow Pictures          -0.5719  ...   \n",
            "224                      Eon Productions          -0.5994  ...   \n",
            "530                  DreamWorks Pictures           0.6369  ...   \n",
            "710        Walt Disney Animation Studios           0.0000  ...   \n",
            "295                   Universal Pictures           0.7171  ...   \n",
            "\n",
            "                                                        Quote_tokens_noSW_lemm  \\\n",
            "862                                                        [negative, thought]   \n",
            "224                                     [live, twice, born, look, death, face]   \n",
            "530                                                               [love, lamp]   \n",
            "710                                                                  [let, go]   \n",
            "295  [think, telling, friend, cool, living, artist, never, asked, paint, nude]   \n",
            "\n",
            "    Drama  Comedy ActionAdventure  Crime ThrillerHorror  Fantasy  Sci-Fi  \\\n",
            "862     1       0               0      1              1        0       0   \n",
            "224     0       0               1      0              1        0       0   \n",
            "530     0       1               0      0              0        0       0   \n",
            "710     0       1               1      0              0        0       0   \n",
            "295     1       1               0      0              0        1       0   \n",
            "\n",
            "    Animation Romance  \n",
            "862         0       0  \n",
            "224         0       0  \n",
            "530         0       0  \n",
            "710         1       0  \n",
            "295         0       0  \n",
            "\n",
            "[5 rows x 30 columns]\n"
          ]
        }
      ],
      "source": [
        "# create binary columns for each of the genres we are keeping\n",
        "for genre in genre_keep_list:\n",
        "    train[genre] = train['Genre_split'].apply(lambda x: 1 if genre in x else 0)\n",
        "\n",
        "# print first 5 rows to check it has been done correctly:\n",
        "print(train.head(5))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "NjzNaA_xB4Gf",
      "metadata": {
        "id": "NjzNaA_xB4Gf"
      },
      "source": [
        "We see that 9 binary columns have been added to our data frame, each of which represent whether `Genre` contains the respective genre.\n",
        "For example, the first row's `Genre` value is 'Sci-Fi, Drama, Romance' and so the flags for 'Sci-Fi', 'Drama' and 'Romance' have a value for 1, while all other genre flags are set to 0.\n",
        "\n",
        "We now apply these changes to the validation and test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 433,
      "id": "gWsgwwXwCcOC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gWsgwwXwCcOC",
        "outputId": "974a00f5-2bf0-4bf2-efd5-1fd33d4b71fd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Validation dataset:\n",
            "Before: data has 80 rows and 19 columns\n",
            "After: data has 80 rows and 28 columns\n",
            "Test dataset:\n",
            "Before: data has 80 rows and 19 columns\n",
            "After: data has 80 rows and 28 columns\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Define a function to clean the genre column and create the binary variables:\n",
        "def add_genre_columns(data, genre_keep_list):\n",
        "\n",
        "  print(f'Before: data has {data.shape[0]} rows and {data.shape[1]} columns')\n",
        "  data['Genre_split'] = data['Genre'].str.split(', ')\n",
        "\n",
        "  # Clean up Genres that can easily be grouped (use domain knowledge and common sense)\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Science Fiction', 'Sci-Fi')) # replace Science Fiction with 'Sci-Fi'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, ' Action', 'Action')) # replace ' Action' with 'Action'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Anime', 'Animation')) # replace 'Anime' with 'Animation'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Animated', 'Animation')) # replace 'Animated' with 'Animation'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Film Noir', 'Crime')) # replace 'Film Noir' with 'Crime'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Musical', 'Music')) # replace 'Musical' with 'Music'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Biograpical', 'Biography')) # replace 'Biographical' with 'Biography'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Mystery', 'Thriller')) # replace 'Mystery' with 'Thriller'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Dark Fantasy', 'Fantasy')) # replace 'Mystery' with 'Thriller'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Thriller', 'ThrillerHorror'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Horror', 'ThrillerHorror'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Action', 'ActionAdventure'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_genre(x, 'Adventure', 'ActionAdventure'))\n",
        "\n",
        "  # Clean up double genres such as 'Romantic Comedy' and 'Science fiction/Action'\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_double_genre(x, 'Science fiction/Action', 'Sci-Fi', 'Action'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_double_genre(x, 'Romantic Comedy', 'Romance', 'Comedy'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_double_genre(x, 'Romantic comedy', 'Romance', 'Comedy'))\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: replace_double_genre(x, 'Sports drama', 'Sport', 'Drama'))\n",
        "\n",
        "  data['Genre_split'] = data['Genre_split'].apply(lambda x: remove_genre(x, genre_keep_list))\n",
        "\n",
        "  # create binary columns for each of the genres we are keeping\n",
        "  for genre in genre_keep_list:\n",
        "      data[genre] = data['Genre_split'].apply(lambda x: 1 if genre in x else 0)\n",
        "\n",
        "  # drop working column:\n",
        "  data = data.drop(columns = ['Genre_split'])\n",
        "\n",
        "  print(f'After: data has {data.shape[0]} rows and {data.shape[1]} columns')\n",
        "  return data\n",
        "\n",
        "# We expect 9 columns to be added to each data frame\n",
        "val = val.drop(columns = ['Genre_split'])\n",
        "test = test.drop(columns = ['Genre_split'])\n",
        "\n",
        "print('Validation dataset:')\n",
        "val = add_genre_columns(data = val, genre_keep_list = genre_keep_list)\n",
        "\n",
        "print('Test dataset:')\n",
        "test = add_genre_columns(data = test, genre_keep_list = genre_keep_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bCkYs18HHR4O",
      "metadata": {
        "id": "bCkYs18HHR4O"
      },
      "source": [
        "We see that 9 columns have been added to each of the validation and test set (one for each of the genres in our keep list). This suggests that the cleaning and feature engineering was completely successfully for these datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "K1hZ7qvzHdKF",
      "metadata": {
        "id": "K1hZ7qvzHdKF"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this cleaning step we cleaned the `Genre` column:\n",
        "- Split list of genres into individual words\n",
        "- Cleaned and grouped genres to get 9 unique genre categories\n",
        "- Create 9 binary features representing whether a quote comes from the particular genre\n",
        "\n",
        "We may use the constructed genre columns in Q3 (neural network) but not Q2 (cluster analysis). This is because the cluster analysis is focused on *characteristics of quotes* whereas genre is a characteristic of the *movie*."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iFh43M0fUQL2",
      "metadata": {
        "id": "iFh43M0fUQL2"
      },
      "source": [
        "**Overall summary**  \n",
        "*Provide a **clear** and **concise** overall summary here of your conclusions from this section of the notebook.*\n",
        "\n",
        "***Cleaning steps performed:***\n",
        "- Year column converted to numeric and cleaned\n",
        "- Duplicates removed\n",
        "- Fixed spillage of quote variable into second column\n",
        "- Standardised quotes with lowercasing\n",
        "- Removed special characters and fixed incorrect encodings\n",
        "- Removed non-English text\n",
        "- Correct some quotes with missing text\n",
        "- Expanded contractions in our quotes\n",
        "- Removed punctuation and numeric words\n",
        "- Corrected some spelling errors\n",
        "- Tokenised the quotes (only required for frequency-based vectorisation methods)\n",
        "- Removed stop words. No additional domain-specific stop words were removed.\n",
        "- Lemmatisation of quotes\n",
        "- Cleaning and grouping of the `Genre` column\n",
        "\n",
        "***Features added:***\n",
        "- `Quote_tokens_noSW_lemm` contains our cleaned, lemmatised tokens from the `Movie Quote` column. Vectorisation may be applied to this column if a frequency-based method is used.\n",
        "- `Quote_tokens` contains cleaned tokenised quotes. These tokens will be concatenated and used for vectorisation if we choose to use BERT.\n",
        "- Flags for the presence of exclamation or question marks in a quote: `exclamation_flag`, `question_flag`\n",
        "- Binary variables for the following genres: `Drama`,  `Comedy`, `ActionAdventure`,  `Crime`, `ThrillerHorror`,  `Fantasy`,  `Sci-Fi`, `Romance`, `Animation`\n",
        "\n",
        "***Checks performed:***\n",
        "\n",
        "At each step of cleaning several checks were employed, including:\n",
        "- Spot checks of random rows\n",
        "- Checks of the specific rows affected by a change\n",
        "- Comparing before and after to ensure changes are as expected\n",
        "\n",
        "Not all cleaning steps passed our reasonableness checks. For example, we found that the automated spell-checker introduced more errors than it fixed and so this step was rejected.\n",
        "\n",
        "After cleaning our quotes, we are ready to vectorise them so our models/machine can understand the text."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "r2D2yMinUS_R",
      "metadata": {
        "id": "r2D2yMinUS_R"
      },
      "source": [
        "### Q1c - Vectorise Movie Quotes\n",
        "#### Purpose:\n",
        "\n",
        "Our clustering and neural network models require matrices of numbers as inputs and cannot directly understand or use strings/words. Therefore, we must convert our quotes to numerical vectors; this is known as *vectorisation*.\n",
        "\n",
        "In this section, we choose an appropriate vectorisation method for our problem and apply it."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_O-pIPbrUS_m",
      "metadata": {
        "id": "_O-pIPbrUS_m"
      },
      "source": [
        "#### Q1c Step 1 - Selection and Justification of Vectorisation Method\n",
        "**Outline**\n",
        "\n",
        "Vectorisation is the process of converting our array of tokens into a vector of numbers that a machine can understand for our modelling. In this section we discuss our business objectives and which vectorisation method will represent the data in a way that allows us to best fulfil these objectives.\n",
        "\n",
        "We decide to select **BERT-based word embeddings** as our vectorisation method. The key reasons for this is that it captures semantic meaning and context which is crucial for our two business objectives. Also, we are able to leverage learnings from a massive text corpus which is useful given our limited data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZyAYMNMHFWLh",
      "metadata": {
        "id": "ZyAYMNMHFWLh"
      },
      "source": [
        "---\n",
        "***Justification of vectorisation method:***\n",
        "\n",
        "We have two business objectives and need to choose a vectorisation technique aligned with both of them.\n",
        "\n",
        "\n",
        "**Objective 1: Understanding quote characteristics**\n",
        "\n",
        "For the first objective we want to understand thematic and stylistic characteristics of quotes to guide scriptwriters. Themes depend on word meaning, and stylistic elements may depend on word relationships in a quote.\n",
        "\n",
        "This suggests that an **embedding vectorisation method** is well-suited for this objective, as this method captures semantic meaning by placing similar quotes close together in the vector space. This will help to identify quotes with similar themes. Furthermore, BERT, a popular LLM that can be used for embedding, considers word order and relationships between words which may help in understanding quote style.\n",
        "\n",
        "\n",
        "**Objective 2: Classifying problematic quotes**\n",
        "\n",
        "Whether a quote is problematic is a vague concept as it can relate to tone, the presence of swearing or taboo references. While specific words like swear words may indicate rudeness, the context of words often signals their intended meaning. So identifying problematic quotes may rely on the meaning of words *in context*.\n",
        "\n",
        "A simple vectorisation method like bag-of-words or TF-IDF may not be sufficient:\n",
        "- They may miss context-dependent meanings of words\n",
        "- They may not capture the semantic nuances of taboo themes which is not just dependent on the presence of particular words\n",
        "\n",
        "Again, an **embedding vectorisation method** is likely better suited for this objective. In particular, BERT considers the context of a word when creating its embeddings. Additionally, the semantic meaning captured by word embedding may be more appropriate for capturing taboo themes in quotes.\n",
        "\n",
        "Therefore, word embedding via BERT is likely an appropriate vectorisation method for our objectives. Using a pre-trained model like BERT will allow us to leverage learnings from a much larger corpus of data and capture the semantic meaning that is crucial for both objectives.\n",
        "\n",
        "This method does have some drawbacks in our context which we discuss below:\n",
        "\n",
        "- **Issue 1: Computational cost and time**\n",
        "\n",
        "  Embedding with BERT can be time-consuming. Since our classifier does not need real-time response (likely running after scripts are finished), this drawback is manageable. Investment in more computing power could address this further if needed, although this does come with a cost.\n",
        "\n",
        "- **Issue 2: High dimensionality of embeddings**\n",
        "\n",
        "  Word embedding techniques represent each word as a high-dimensional vector. Given our dataset has a limited number of observations, this will lead to a very sparse feature space and hence poor modelling outcomes. This can be resolved using dimension reduction techniques such as PCA or t-SNE to substantially decrease the feature space at the cost of some information loss."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1zFIED3AUS_n",
      "metadata": {
        "id": "1zFIED3AUS_n"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In conclusion, we choose to vectorise our quotes using embedding with BERT. This method captures semantic meaning and context which is crucial for both of our key objectives. The drawbacks of the method can be mitigated with dimensionality reduction techniques and by considering how our model will be used at deployment."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TkFVnz9PUS_n",
      "metadata": {
        "id": "TkFVnz9PUS_n"
      },
      "source": [
        "#### Q1c Step 2 - Applying the BERT vectorisation method\n",
        "**Outline**\n",
        "\n",
        "\n",
        "In this step we apply BERT to vectorise our cleaned movie quotes in the training set. BERT is applied to full strings rather than tokens as it considers word order and context to vectorise.\n",
        "\n",
        "Stop word removal and lemmatisation are not necessary for BERT and so we will apply the model to our cleaned quotes without these steps applied.\n",
        "- Stop words such as 'not' can often provide important context in a sentence for BERT.\n",
        "- Trained on a massive dataset, BERT can comprehend various word forms (e.g., \"play,\" \"played,\" \"playing\"). Lemmatisation, which converts words to their base form, is unnecessary.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 434,
      "id": "TcOAqESAUS_n",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TcOAqESAUS_n",
        "outputId": "23a1ac9d-3e47-4340-acd4-fa728d97bf0a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Example of 3 inputs into BERT (stop words removed):\n",
            "['all i have are negative thoughts', 'you only live twice once when you are born and once when you look death in the face', 'i love lamp']\n"
          ]
        }
      ],
      "source": [
        "## Prepare data for BERT\n",
        "# concatenate tokens:\n",
        "train['text_bert_noSW'] = train['Quote_tokens_noSW'].apply(lambda x: ' '.join(x)) # we do this for common word analysis later (don't want stop words)\n",
        "train['text_bert_withSW'] = train['Quote_tokens'].apply(lambda x: ' '.join(x))\n",
        "\n",
        "bert_input_withSW = train['text_bert_withSW'].tolist()\n",
        "\n",
        "print('Example of 3 inputs into BERT (stop words removed):')\n",
        "print(bert_input_withSW[0:3])\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 435,
      "id": "N6JE8jG_noHi",
      "metadata": {
        "id": "N6JE8jG_noHi"
      },
      "outputs": [],
      "source": [
        "# Create model to perform embeddings using BERT:\n",
        "BERT_sent_transformer = SentenceTransformer('distilbert-base-nli-mean-tokens')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 436,
      "id": "FXD7IFUKm31I",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84,
          "referenced_widgets": [
            "c80ec3a9b8a24707951a77ba917fed01",
            "5cd4d7e96b5c46e7b5ff4c65e1ef5b6c",
            "e8b415f8e1ed4d34852942164fe77e4c",
            "13e0bd3c56ee4909a2a827f965f3d1a8",
            "54b7dcc8fe5a4ad2b122ff88b2b08e38",
            "a4400eb9ff1e4366a9b99f6d60149bbe",
            "9b68ac5354ab4087adba39f584d85a50",
            "90397a505a1d4853b96742e74e60c6bc",
            "499f1ae29f25402fb098629dde767fd7",
            "094c2072533c40fbb34946335b6ce067",
            "2ce2a19413674a5dbb6ac4b9f49306b2"
          ]
        },
        "id": "FXD7IFUKm31I",
        "outputId": "3638ebd5-8c23-431a-93aa-0ea2cbda1a53"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Calculating BERT vector embeddings for Quotes ...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c80ec3a9b8a24707951a77ba917fed01",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Batches:   0%|          | 0/20 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Runtime: 22.080670833587646 seconds\n"
          ]
        }
      ],
      "source": [
        "print('Calculating BERT vector embeddings for Quotes ...')\n",
        "start_time = time.time()\n",
        "\n",
        "# Obtain embeddings\n",
        "embeddings_withSW = BERT_sent_transformer.encode(bert_input_withSW, show_progress_bar=True)\n",
        "embedding_BERT_withSW = np.array(embeddings_withSW)\n",
        "\n",
        "end_time = time.time()\n",
        "runtime = end_time - start_time\n",
        "print(\"Runtime:\", runtime, \"seconds\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 437,
      "id": "vHw2mMx7oNlh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vHw2mMx7oNlh",
        "outputId": "1988603e-9989-4fe7-dfdd-5f9d716bc9c1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Example of a BERT embedding of a quote (first 10 values):\n",
            "[-1.1044853  -0.13048436  0.16367202 -0.24875136 -0.9008343   0.03358212\n",
            "  0.39454222 -0.32769725 -0.80392337  1.1224158 ]\n"
          ]
        }
      ],
      "source": [
        "# Let's print out an example of an embedding from BERT:\n",
        "\n",
        "print('Example of a BERT embedding of a quote (first 10 values):')\n",
        "print(embeddings_withSW[0, 0:10])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kyX5orqls79T",
      "metadata": {
        "id": "kyX5orqls79T"
      },
      "source": [
        "**Reasonableness Check 1: do we have a vector for each row of our training data**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3K8GQue-to17",
      "metadata": {
        "id": "3K8GQue-to17"
      },
      "source": [
        "The BERT embedding outputted a matrix with 633 rows and 768 columns. We have a row/vector for each of our training data quotes (633) which is expected. The columns represent dimensions in our vector-space. Therefore, the embedding appears to have been applied to each quote."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 438,
      "id": "aYhv3_y4tGk_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aYhv3_y4tGk_",
        "outputId": "5545ad2b-3366-4bba-d9a5-26ac61c3110d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Our training data set has 633 rows of data\n",
            "\n",
            "Embedding on quotes with stop words\n",
            "Shape of BERT weights for training data : (633, 768)\n"
          ]
        }
      ],
      "source": [
        "print(f'Our training data set has {train.shape[0]} rows of data' )\n",
        "\n",
        "print()\n",
        "print('Embedding on quotes with stop words')\n",
        "print('Shape of BERT weights for training data :', embeddings_withSW.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6h4LyE7XuL12",
      "metadata": {
        "id": "6h4LyE7XuL12"
      },
      "source": [
        "**Reasonable Check 2: Cosine similarity between quotes**\n",
        "\n",
        "We expect that two quotes with similar themes/meanings will be close together in the vector-space generated by the embeddings. Conversely, two quotes that relate to different themes should be further apart in the vector-space.\n",
        "\n",
        "We can assess the proximity of two quotes in the vector space using cosine similarity. This score is in the interval [-1, 1], with a higher score signifying that the two vectors are similar.\n",
        "\n",
        "Therefore, as a reasonableness check on the embedding, I have selected two quotes related to 'love' and one which is not, and then calculated the respective cosine similarities. We would expect the two quotes about 'love' to have a higher cosine similarity with each other than with the other quote."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 439,
      "id": "eHsYnNxbwWem",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eHsYnNxbwWem",
        "outputId": "57a864fa-0bdc-4de2-d36e-fae514ec37b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quote 1 about love:\n",
            "i love you i really love you ditto\n",
            "\n",
            "Quote 2 about love:\n",
            "i love you\n",
            "\n",
            "Miscellaneous quote:\n",
            "may the force be with you\n"
          ]
        }
      ],
      "source": [
        "# The following quotes have been chosen\n",
        "\n",
        "print('Quote 1 about love:')\n",
        "print(train['Quote_clean'].loc[318])\n",
        "row_num1 = train.index.get_loc(318) # get row number in matrix\n",
        "\n",
        "print()\n",
        "print('Quote 2 about love:')\n",
        "print(train['Quote_clean'].loc[858])\n",
        "row_num2 = train.index.get_loc(858)\n",
        "\n",
        "print()\n",
        "print('Miscellaneous quote:')\n",
        "print(train['Quote_clean'].loc[1])\n",
        "row_num3 = train.index.get_loc(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 440,
      "id": "Kv9wIoHRwQ-t",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kv9wIoHRwQ-t",
        "outputId": "b15817f9-5094-432d-acd6-0046cd0efbd2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Comparing sentences:\n",
            "  Sentence at index 318: i love you i really love you ditto\n",
            "  Sentence at index 858: i love you\n",
            "Cosine similarity: 0.9156298637390137\n",
            "---\n",
            "Comparing sentences:\n",
            "  Sentence at index 318: i love you i really love you ditto\n",
            "  Sentence at index 1: may the force be with you\n",
            "Cosine similarity: 0.4414983093738556\n",
            "---\n",
            "Comparing sentences:\n",
            "  Sentence at index 858: i love you\n",
            "  Sentence at index 1: may the force be with you\n",
            "Cosine similarity: 0.584220826625824\n",
            "---\n"
          ]
        }
      ],
      "source": [
        "# Calculate cosine similarity between sample texts\n",
        "sample_text = [row_num1, row_num2, row_num3]\n",
        "index_value = [318, 858, 1]\n",
        "\n",
        "for i in range(len(sample_text)):\n",
        "    for j in range(i+1, len(sample_text)):\n",
        "\n",
        "        vec1 = embedding_BERT_withSW[sample_text[i]]\n",
        "        vec2 = embedding_BERT_withSW[sample_text[j]]\n",
        "        similarity_score = cosine_similarity([vec1], [vec2])[0][0]\n",
        "\n",
        "        # Retrieve the actual sentences being compared\n",
        "        sentence1 = train.loc[index_value[i], 'text_bert_withSW']\n",
        "        sentence2 = train.loc[index_value[j], 'text_bert_withSW']\n",
        "\n",
        "        print(f\"Comparing sentences:\")\n",
        "        print(f\"  Sentence at index {index_value[i]}: {sentence1}\")\n",
        "        print(f\"  Sentence at index {index_value[j]}: {sentence2}\")\n",
        "        print(f\"Cosine similarity: {similarity_score}\")\n",
        "        print(\"---\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WQSzpgLU0fge",
      "metadata": {
        "id": "WQSzpgLU0fge"
      },
      "source": [
        "We see that the cosine similarity between our two quotes about love is very high (~0.92) which is expected and suggests that quotes with similar meaning are close in the vector-space.\n",
        "\n",
        "Additionally, the cosine similarity between the love quote and \"may the force be with you\" is considerably lower which we should expect. This suggests quotes that are less similar also have embeddings that are less similar.\n",
        "\n",
        "Therefore, the BERT embeddding passes this reasonableness check as similar quotes have similar vector representations and vice versa."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "r60IarqfnHhR",
      "metadata": {
        "id": "r60IarqfnHhR"
      },
      "source": [
        "**Interpretation of Results**\n",
        "\n",
        "Our dataset is relatively small so the BERT embedding process was reasonably quick, taking about 15 seconds.\n",
        "\n",
        "The embedding outputted a matrix with 633 rows (one for each training quote) and 768 columns (each representing a dimension in the vector space).  If we were to use these vectors as they are we would have more features than observations (i.e. overparamaterised as 633 < 768) and we will be unable to train a model effectively, without significant variance error. We need the number of features to be significantly lower than the number of observations to reduce the chance of overfitting.\n",
        "\n",
        "Therefore, in the next step we conduct dimensionality reduction on the embedding vector.\n",
        "\n",
        "---\n",
        "\n",
        "First, we apply the BERT embedding to the validation and test sets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 441,
      "id": "QaGbvMjfS2jR",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84,
          "referenced_widgets": [
            "c5dacd0307b04c77908d0171fbae6413",
            "ae21a1422fe447f18fececf73a745673",
            "2e12e9e8989245d49fdfd876ab26902d",
            "9fe2b848f3ad47368718b5accc11e128",
            "cc838166f5af415fa3bbcee7591f7a41",
            "a177f37cef344140be45f6d0657e8fe4",
            "cd3a0ef9887a4aaf8a23e38b7e9eb058",
            "afa5e07d340e481a862dd688661eb528",
            "29d708f2da85420b9324b4fb84a2216a",
            "e82a4acb617f4b6ea90cb82c4fa3a0e9",
            "aeee4a5796154124a3b517dc7a99017f"
          ]
        },
        "id": "QaGbvMjfS2jR",
        "outputId": "b36479e6-570e-47bc-f796-d3b97bf85287"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Calculating BERT vector embeddings for Quotes WITH stop words...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c5dacd0307b04c77908d0171fbae6413",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Batches:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Runtime: 3.1715192794799805 seconds\n"
          ]
        }
      ],
      "source": [
        "## VALIDATION DATASET\n",
        "\n",
        "## Prepare validation data for BERT\n",
        "# concatenate tokens:\n",
        "val['text_bert_noSW'] = val['Quote_tokens_noSW'].apply(lambda x: ' '.join(x))\n",
        "val['text_bert_withSW'] = val['Quote_tokens'].apply(lambda x: ' '.join(x))\n",
        "\n",
        "bert_input_withSW_val = val['text_bert_withSW'].tolist()\n",
        "\n",
        "print('Calculating BERT vector embeddings for Quotes WITH stop words...')\n",
        "start_time = time.time()\n",
        "\n",
        "# Obtain embeddings\n",
        "embeddings_withSW_val = BERT_sent_transformer.encode(bert_input_withSW_val, show_progress_bar=True)\n",
        "embedding_BERT_withSW_val = np.array(embeddings_withSW_val)\n",
        "\n",
        "end_time = time.time()\n",
        "runtime = end_time - start_time\n",
        "print(\"Runtime:\", runtime, \"seconds\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 442,
      "id": "UcKH9aZPTWbW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UcKH9aZPTWbW",
        "outputId": "7ae5080e-26bb-4818-9e56-424b7cf14e40"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Our validation data set has 80 rows of data\n",
            "\n",
            "Embedding on quotes with stop words\n",
            "Shape of BERT weights for validation data : (80, 768)\n"
          ]
        }
      ],
      "source": [
        "# CHECK VALIDATION EMBEDDING\n",
        "\n",
        "print(f'Our validation data set has {val.shape[0]} rows of data' )\n",
        "\n",
        "print()\n",
        "print('Embedding on quotes with stop words')\n",
        "print('Shape of BERT weights for validation data :', embeddings_withSW_val.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 443,
      "id": "p7tjw2x4UQW5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84,
          "referenced_widgets": [
            "69a687b6617447df9ba34917429ac22d",
            "e0073a2535c041568c4b312bba3cbb5e",
            "f8f5ba9019ed41b09c30ab64fbece01f",
            "e3b0d0bb057549719d12c1cf6c5c8264",
            "9d09cb0d16084877b5ebf4f585a9e3cd",
            "766cdbc2a9bc4143a8f210e939c4fb7f",
            "dff23367b0e44e09b9555ed731cb5c5b",
            "536283bf66984e71a756ce6085f9f416",
            "88a991949e6f4f58b00dca1afc9c2452",
            "0aa437253e9d4b72b329daa1025ac32c",
            "218cfa52b5e247f38a1a4512f70b6d23"
          ]
        },
        "id": "p7tjw2x4UQW5",
        "outputId": "a834f733-6a4e-446a-f08e-6358ec7ae755"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Calculating BERT vector embeddings for Quotes WITH stop words...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "69a687b6617447df9ba34917429ac22d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Batches:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Runtime: 3.53800892829895 seconds\n"
          ]
        }
      ],
      "source": [
        "## TEST DATASET\n",
        "\n",
        "## Prepare test data for BERT\n",
        "# concatenate tokens:\n",
        "test['text_bert_noSW'] = test['Quote_tokens_noSW'].apply(lambda x: ' '.join(x))\n",
        "test['text_bert_withSW'] = test['Quote_tokens'].apply(lambda x: ' '.join(x))\n",
        "\n",
        "bert_input_withSW_test = test['text_bert_withSW'].tolist()\n",
        "\n",
        "print('Calculating BERT vector embeddings for Quotes WITH stop words...')\n",
        "start_time = time.time()\n",
        "\n",
        "# Obtain embeddings\n",
        "embeddings_withSW_test = BERT_sent_transformer.encode(bert_input_withSW_test, show_progress_bar=True)\n",
        "embedding_BERT_withSW_test = np.array(embeddings_withSW_test)\n",
        "\n",
        "end_time = time.time()\n",
        "runtime = end_time - start_time\n",
        "print(\"Runtime:\", runtime, \"seconds\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 444,
      "id": "SminZMqrUU0Y",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SminZMqrUU0Y",
        "outputId": "cdd6e8c8-4bbd-49fc-b33c-19cc34ebc7d9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Our test data set has 80 rows of data\n",
            "\n",
            "Embedding on quotes with stop words\n",
            "Shape of BERT weights for test data : (80, 768)\n"
          ]
        }
      ],
      "source": [
        "# CHECK TEST EMBEDDING\n",
        "\n",
        "print(f'Our test data set has {test.shape[0]} rows of data' )\n",
        "\n",
        "print()\n",
        "print('Embedding on quotes with stop words')\n",
        "print('Shape of BERT weights for test data :', embeddings_withSW_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "u1RkeVqVUxXD",
      "metadata": {
        "id": "u1RkeVqVUxXD"
      },
      "source": [
        "As a quick check we compare the number of rows in the validation and test embeddings with the respective datasets. All row counts are equal indicating that the embedding has been applied correctly."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "D5Y4y2mRUS_n",
      "metadata": {
        "id": "D5Y4y2mRUS_n"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "- BERT embedding was applied to the cleaned quotes (prior to stop word removal) for the training, validation and test datasets\n",
        "- The vectorisation appeared to be reasonable as quotes with similar themes were close in the vector-space and vice versa (using cosine similarity)\n",
        "- Checks on the dimensions of the BERT embeddings indicate that it was applied correctly to all quotes\n",
        "- Dimensionality reduction will need to be applied to the BERT embeddings as it contains more columns than observations we have in our training data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ehdH2fhSDj7H",
      "metadata": {
        "id": "ehdH2fhSDj7H"
      },
      "source": [
        "#### Q1c Step 3 - Dimensionality Reduction\n",
        "**Outline**\n",
        "\n",
        "As noted previously, our vectorisation led to more features than observations. Therefore, we must conduct dimensionality reduction on our word embedding vector to reduce the feature space.\n",
        "\n",
        "In this step we use Principal Component Analysis (PCA) to do this. PCA preserves as much variance/information as possible when reducing the dimensions. This is crucial as we want to retain as much explanatory power in the data when trying to understand the key characteristics of the quotes.\n",
        "\n",
        "However, the components that best explain the variance in the embeddings are not necessarily the components that will best explain whether a quote is problematic or not in Q3. This is an unfortunate but necessary drawback of the PCA method."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gh7yAeog_HZ4",
      "metadata": {
        "id": "gh7yAeog_HZ4"
      },
      "source": [
        "**Principal Component Analysis**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 445,
      "id": "C9Wa4UXIA7q2",
      "metadata": {
        "id": "C9Wa4UXIA7q2"
      },
      "outputs": [],
      "source": [
        "# Define a function to obtain the explained variances from the PCA features.\n",
        "def plot_explained_variances(pca):\n",
        "\n",
        "    exp_var_pca = pca.explained_variance_ratio_\n",
        "\n",
        "    # Calculate the cumulative sum of variances explained.\n",
        "    # This will be used to create a scree plot for visualising the variance\n",
        "    # explained by each principal component.\n",
        "\n",
        "    cum_sum_var = np.cumsum(exp_var_pca)\n",
        "\n",
        "    # Create the plot\n",
        "    plt.plot(range(1,len(exp_var_pca)+1), exp_var_pca, alpha=0.5,  linewidth=2,\n",
        "             label='Individual explained variance',color='dodgerblue')\n",
        "    plt.ylabel('Explained variance ratio')\n",
        "    plt.xlabel('Principal component')\n",
        "    plt.legend(loc='best')\n",
        "    plt.tight_layout()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 446,
      "id": "anE9eLK3GABD",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 539
        },
        "id": "anE9eLK3GABD",
        "outputId": "eb84867b-00cb-443f-9068-168231730f52"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 33312 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The first 50 PCs explain 81.36% of the variance\n",
            "The first 4 PCs explain 30.73% of the variance\n",
            "The first 7 PCs explain 41.05% of the variance\n"
          ]
        }
      ],
      "source": [
        "## Applying PCA to our embedding on quotes with stop words\n",
        "\n",
        "# set up scaler as PCA is sensitive to scale\n",
        "scaler_withSW = StandardScaler()\n",
        "n_PCs = 50\n",
        "pca_withSW = PCA(n_components=n_PCs) # initialise PCA object\n",
        "\n",
        "# scale the data:\n",
        "scaled_embedding_withSW = pd.DataFrame(scaler_withSW.fit_transform(embeddings_withSW))\n",
        "\n",
        "#fit PCA model to data and get returned PCs\n",
        "pca_values_withSW = pca_withSW.fit_transform(scaled_embedding_withSW)\n",
        "\n",
        "plot_explained_variances(pca_withSW)\n",
        "\n",
        "print(f'The first {n_PCs} PCs explain {pca_withSW.explained_variance_ratio_.sum()*100:.2f}% of the variance')\n",
        "print(f'The first 4 PCs explain {pca_withSW.explained_variance_ratio_[0:4].sum()*100:.2f}% of the variance')\n",
        "print(f'The first 7 PCs explain {pca_withSW.explained_variance_ratio_[0:7].sum()*100:.2f}% of the variance')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "D1p9fXjpGZz6",
      "metadata": {
        "id": "D1p9fXjpGZz6"
      },
      "source": [
        "The Scree plot above shows the explained variance by each principal component (PC) for our word embeddings. The first PC captures around 12% of the variance, the first 4 PCs cumulatively explain 30%, and the first 50 explain 81%.\n",
        "\n",
        "There is a trade-off in determing an ideal number of PCs for our future analysis. Including more PCs will capture more information and possibly lead to more informative results. However, we need to reduce the feature space to reduce model complexity, especially given our limited data.\n",
        "\n",
        "To balance the trade-off we look at the 'elbow' in the plot. The 'elbow' is ambiguous but appears to be around **5 to 7 PCs**. We will aim to use this many components for our clustering and neural network training. While the first 7 PCs only explain about 40% of the variance, indicating significant information loss, adding more components would contradict our goal of reducing the feature space.\n",
        "\n",
        "Having greater than 3 PCs makes visualising our vectors more difficult. This is relevant for Q2 where we conduct clustering. To mitigate this issue we will employ other visualisation techniques to understand the meaning of clusters (e.g. wordclouds, bar charts).\n",
        "\n",
        "---\n",
        "\n",
        "Next, we show a plot of the first two PCs:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 447,
      "id": "DPk6Ik9SHIel",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 452
        },
        "id": "DPk6Ik9SHIel",
        "outputId": "35f30134-594f-4262-ecb4-2232c9a16a64"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 56972 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# first let us refit to only get the first 7 PCs:\n",
        "# initialise PCA object\n",
        "pca_withSW = PCA(n_components=7)\n",
        "\n",
        "#fit PCA model to data and get returned PCs\n",
        "pca_values_withSW = pca_withSW.fit_transform(scaled_embedding_withSW)\n",
        "\n",
        "plt.scatter(pca_values_withSW[:, 0], pca_values_withSW[:, 1])\n",
        "plt.title(\"Visualisation of first 2 Principal Components\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OCGhYwr3HUMC",
      "metadata": {
        "id": "OCGhYwr3HUMC"
      },
      "source": [
        "The points appear to be quite spread out along the first two PCs which is expected given they explain the most variance. However, they are still relatively grouped together since the first two PCs only explain ~20% of the variance. Additional PCs may be able to further separate out the points but this is difficult to visualise."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hJVOlduER06_",
      "metadata": {
        "id": "hJVOlduER06_"
      },
      "source": [
        "**Check: Principal component analysis should result in matrix with dimensions 638 x 7**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 448,
      "id": "5ncZ31fpSEES",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ncZ31fpSEES",
        "outputId": "83b3b943-3b60-4a63-a064-01ccfdba2977"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "With stop words:\n",
            "Shape of Principal Components for training data : (633, 7)\n"
          ]
        }
      ],
      "source": [
        "print('With stop words:')\n",
        "print('Shape of Principal Components for training data :', pca_values_withSW.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "M4NOUk5IValB",
      "metadata": {
        "id": "M4NOUk5IValB"
      },
      "source": [
        "Next, we apply the principal component analysis to the test and validation datasets. To avoid leakage, we do **not** re-train the scaler or the principal components themselves; we just apply what has already been trained:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 449,
      "id": "gvO09czGVttj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gvO09czGVttj",
        "outputId": "413c6519-9fea-4d09-dea0-81e525cd8c4e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "VALIDATION SET CHECKS:\n",
            "\n",
            "Shape of Principal Components for validation data : (80, 7)\n",
            "\n",
            "TEST SET CHECKS:\n",
            "\n",
            "Shape of Principal Components for test data : (80, 7)\n"
          ]
        }
      ],
      "source": [
        "# scale the validation and test data:\n",
        "# NOTE ONLY APPLY 'TRANSFORM' NOT 'FIT'\n",
        "scaled_embedding_withSW_val = pd.DataFrame(scaler_withSW.transform(embeddings_withSW_val))\n",
        "scaled_embedding_withSW_test = pd.DataFrame(scaler_withSW.transform(embeddings_withSW_test))\n",
        "\n",
        "#Transform scaled vectors into principal components:\n",
        "pca_values_withSW_val = pca_withSW.fit_transform(scaled_embedding_withSW_val)\n",
        "pca_values_withSW_test = pca_withSW.fit_transform(scaled_embedding_withSW_test)\n",
        "\n",
        "# CHECK DIMENSIONS OF PCs:\n",
        "\n",
        "print('VALIDATION SET CHECKS:')\n",
        "print()\n",
        "print('Shape of Principal Components for validation data :', pca_values_withSW_val.shape)\n",
        "print()\n",
        "print('TEST SET CHECKS:')\n",
        "print()\n",
        "print('Shape of Principal Components for test data :', pca_values_withSW_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_ulcOniLW3cf",
      "metadata": {
        "id": "_ulcOniLW3cf"
      },
      "source": [
        "As expected, we see 7 principal components for each of the 80 points in the test and validation datasets, so this step passes the reasonableness check."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YxlAttQgDuPJ",
      "metadata": {
        "id": "YxlAttQgDuPJ"
      },
      "source": [
        "**Summary**  \n",
        "In this section, we applied principal component analysis to the BERT word embeddings.\n",
        "- **Five to seven** principal components were determined to provide an optimal balance between explainability and dimension reduction, as per the elbow method. We will target this many PCs in our consequent analysis.\n",
        "- In the training data, the first 7 PCs explained about 41% of the variance"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QOXoCFrwUS_o",
      "metadata": {
        "id": "QOXoCFrwUS_o"
      },
      "source": [
        "**Overall summary**  \n",
        "\n",
        "In this section we:\n",
        "- Selected BERT embedding as our vectorisation method as it captures semantic meaning which is aligned with our objectives\n",
        "- Applied BERT embeddings on the quotes including stop words as these stop words can add additional context.\n",
        "- Conducted dimensionality reduction using Principal Component Analysis\n",
        "- Ultimately, around 7 principal components was deemed 'optimal' for both embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nWIFmH_uVVm7",
      "metadata": {
        "id": "nWIFmH_uVVm7"
      },
      "source": [
        "### Q1d - Construct response variable\n",
        "#### Purpose:\n",
        "\n",
        "In this section, we create a binary response variable indicating whether a quote is 'problematic' or 'not problematic'. This will be used as the response for our neural network classifier.\n",
        "Unfortunately, the column `label_chatGPT`, which will be used to derive this binary response, is unclean and unreliable. Therefore, we first attempt to clean this variable before constructing our response.\n",
        "\n",
        "#### References:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uE2rbV2aVVnL",
      "metadata": {
        "id": "uE2rbV2aVVnL"
      },
      "source": [
        "#### Q1d Step 1 - Cleaning `label_ChatGPT`\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "Our previous EDA revealed that the `label_ChatGPT` column was unclean as it contained missing values and labels that were not one of the intended six categories. We therefore attempt to clean it before using it to construct a response. The following steps are taken:\n",
        "\n",
        "- Clean the variable so it only takes one of six values\n",
        "- Impute missing values\n",
        "\n",
        "***Why:***\n",
        "\n",
        "`label_ChatGPT` will be used to construct our binary response. We will map the values ['1. sex', '2. religion', '3. politics', '4. rude', '5. insensitive'] to 'problematic' and the value '6. none of the above' to 'not problematic'. However, currently this mapping is not possible as the label contains values outside these six values. We hence allocate each label to the likely intended category using regex and inspection of the quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nsEObFAMdJmF",
      "metadata": {
        "id": "nsEObFAMdJmF"
      },
      "source": [
        "As indicated by the data dictionary, the label is only meant to take one of six values:\n",
        "\n",
        "1. sex\n",
        "2. religion\n",
        "3. politics\n",
        "4. rude\n",
        "5. insensitive\n",
        "6. none of the above\n",
        "\n",
        "During our EDA, we observed that in the training data `label_ChatGPT` contained 21 unique values. Clearly ChatGPT did not always follow the prompt provided when labelling the quote. Additionally, in our EDA we found a missing value in the label that will need to be cleaned.\n",
        "\n",
        "We now attempt to allocate each label to one of the six categories using regex and fuzzy matching. First let us have a look at the unique labels:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 450,
      "id": "0OWCOFKeVVnL",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0OWCOFKeVVnL",
        "outputId": "14fed3e5-820c-4918-f902-8c446d842ab0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "label_ChatGPT contains 1 NA values\n",
            "There are 21 unique label_ChatGPTs in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPTs:  \n",
            "\n",
            "\n",
            "                                                                                                                                                                                                                                          Count  \\\n",
            "label_ChatGPT                                                                                                                                                                                                                                     \n",
            "6. none of the above                                                                                                                                                                                                                        561   \n",
            "3. politics                                                                                                                                                                                                                                  19   \n",
            "4. rude                                                                                                                                                                                                                                       9   \n",
            "2. religion                                                                                                                                                                                                                                   8   \n",
            "6. None of the above                                                                                                                                                                                                                          8   \n",
            "1. sex                                                                                                                                                                                                                                        5   \n",
            "1. None of the above                                                                                                                                                                                                                          4   \n",
            "5. insensitive                                                                                                                                                                                                                                3   \n",
            "1. none of the above                                                                                                                                                                                                                          2   \n",
            "Answer: 6. none of the above                                                                                                                                                                                                                  2   \n",
            "5. rude                                                                                                                                                                                                                                       2   \n",
            "6. none of the above. The quote is a general statement about the difficulty of lying to someone who already expects to be lied to, and does not specifically mention any of the listed topics.                                                1   \n",
            "3.\\n3. politics                                                                                                                                                                                                                               1   \n",
            "1. politics                                                                                                                                                                                                                                   1   \n",
            "Unfortunately, the quote from 'About Time' does not fit into any of the given categories and can be seen as a general reflection on life. Therefore, the answer is '6. none of the above'.                                                    1   \n",
            "To be able to use this...\\n\\n6. none of the above                                                                                                                                                                                             1   \n",
            "s\\n\\n6. none of the above                                                                                                                                                                                                                     1   \n",
            "The answer is '6. none of the above'. The quote is not about sex, religion, politics, rudeness, or insensitivity. It is about the inability to let go of something important, and how it can hinder one's ability to bring about change.      1   \n",
            "6. none of the above (The quote is from a crime/drama movie, not about sex, religion, or politics specifically, and may not be considered rude or insensitive by some)                                                                        1   \n",
            "(6. none of the above)                                                                                                                                                                                                                        1   \n",
            "\n",
            "                                                                                                                                                                                                                                         Percentage  \n",
            "label_ChatGPT                                                                                                                                                                                                                                        \n",
            "6. none of the above                                                                                                                                                                                                                         88.77%  \n",
            "3. politics                                                                                                                                                                                                                                   3.01%  \n",
            "4. rude                                                                                                                                                                                                                                       1.42%  \n",
            "2. religion                                                                                                                                                                                                                                   1.27%  \n",
            "6. None of the above                                                                                                                                                                                                                          1.27%  \n",
            "1. sex                                                                                                                                                                                                                                        0.79%  \n",
            "1. None of the above                                                                                                                                                                                                                          0.63%  \n",
            "5. insensitive                                                                                                                                                                                                                                0.47%  \n",
            "1. none of the above                                                                                                                                                                                                                          0.32%  \n",
            "Answer: 6. none of the above                                                                                                                                                                                                                  0.32%  \n",
            "5. rude                                                                                                                                                                                                                                       0.32%  \n",
            "6. none of the above. The quote is a general statement about the difficulty of lying to someone who already expects to be lied to, and does not specifically mention any of the listed topics.                                                0.16%  \n",
            "3.\\n3. politics                                                                                                                                                                                                                               0.16%  \n",
            "1. politics                                                                                                                                                                                                                                   0.16%  \n",
            "Unfortunately, the quote from 'About Time' does not fit into any of the given categories and can be seen as a general reflection on life. Therefore, the answer is '6. none of the above'.                                                    0.16%  \n",
            "To be able to use this...\\n\\n6. none of the above                                                                                                                                                                                             0.16%  \n",
            "s\\n\\n6. none of the above                                                                                                                                                                                                                     0.16%  \n",
            "The answer is '6. none of the above'. The quote is not about sex, religion, politics, rudeness, or insensitivity. It is about the inability to let go of something important, and how it can hinder one's ability to bring about change.      0.16%  \n",
            "6. none of the above (The quote is from a crime/drama movie, not about sex, religion, or politics specifically, and may not be considered rude or insensitive by some)                                                                        0.16%  \n",
            "(6. none of the above)                                                                                                                                                                                                                        0.16%  \n"
          ]
        }
      ],
      "source": [
        "# First let us look at the unique values of label_chatGPT in our training data:\n",
        "print(f'label_ChatGPT contains {train[\"label_ChatGPT\"].isna().sum()} NA values')\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "v9JUBuC11dCL",
      "metadata": {
        "id": "v9JUBuC11dCL"
      },
      "source": [
        "Before conducting any cleaning of the labels, let us first deal with the missing value.\n",
        "\n",
        "From our EDA we identified the related quote to be from Borat, an irreverent movie. Based on our domain knowledge of the film, the quote is directed towards a woman and is highly disrespectful (see quote below). To avoid losing additional data, we decide to impute the missing value as '5. insensitive' based on our knowledge of the film. Other categories such as '1.sex' may be justified as well but ultimately they will all be grouped together when we make a binary response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 451,
      "id": "AR_uVzqJ1bp3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AR_uVzqJ1bp3",
        "outputId": "4cf56bad-4f31-408e-b5b9-56557069d81c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The quote relating to the missing value is:\n",
            "              Movie Quote\n",
            "100  Very nice, how much?\n",
            "\n",
            "We impute the label for this quote to be \"5. insensitive\"\n",
            "Number of missing values after change: 0\n"
          ]
        }
      ],
      "source": [
        "print('The quote relating to the missing value is:')\n",
        "print(train.loc[train[\"label_ChatGPT\"].isna(), ['Movie Quote']])\n",
        "\n",
        "print()\n",
        "print('We impute the label for this quote to be \"5. insensitive\"')\n",
        "train.loc[100, 'label_ChatGPT'] = '5. insensitive'\n",
        "\n",
        "print(f'Number of missing values after change: {train[\"label_ChatGPT\"].isna().sum()}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9CAy5y_yxGzl",
      "metadata": {
        "id": "9CAy5y_yxGzl"
      },
      "source": [
        "Next, we attempt to clean labels that are likely intending to be in the category '6. none of the above'. We lowercase all labels and search for '6. none of the above' using regex to initially clean this category:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 452,
      "id": "aIBTCYRcfp9f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aIBTCYRcfp9f",
        "outputId": "c76c9cdf-b526-40e9-9a99-2bb0ce177884"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 10 unique label_ChatGPT_cleans in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPT_cleans:  \n",
            "\n",
            "\n",
            "                      Count Percentage\n",
            "label_ChatGPT_clean                   \n",
            "6. none of the above    578     91.31%\n",
            "3. politics              19       3.0%\n",
            "4. rude                   9      1.42%\n",
            "2. religion               8      1.26%\n",
            "1. none of the above      6      0.95%\n",
            "1. sex                    5      0.79%\n",
            "5. insensitive            4      0.63%\n",
            "5. rude                   2      0.32%\n",
            "1. politics               1      0.16%\n",
            "3.\\n3. politics           1      0.16%\n"
          ]
        }
      ],
      "source": [
        "# we see a lot of labels that contain '6. none of the above' but are in the incorrect format\n",
        "# we also see some quotes containing a different number and 'none of the above'. We will investigate these separately\n",
        "\n",
        "def replace_label(label, pattern, replace):\n",
        "  if pattern in label:\n",
        "    return replace\n",
        "  return label\n",
        "\n",
        "\n",
        "# First let us allocate any labels containing '6' and 'none of the above' to the proper label \"6. none of the above\"\n",
        "\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT'].apply(str.lower) # first lowercase all labels to help with matching\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '6. none of the above', '6. none of the above'))\n",
        "\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT_clean', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZZb4n932iPdB",
      "metadata": {
        "id": "ZZb4n932iPdB"
      },
      "source": [
        "This first step of allocating any labels containing '6. none of the above' to the corresponding label, reduced the unique label values to 10.\n",
        "\n",
        "We now investigate those points that contain 'none of the above' but with a number other than '6' next to them. This is conflicting information and we want to better understand which label is appropriate for these quotes (not sure if ChatGPT intended them to be in '1. sex' or '6. none of the above', for example):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 453,
      "id": "dQEGJbVtigKB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dQEGJbVtigKB",
        "outputId": "cb6d3d4b-22b2-4c78-a801-5cf2cc4a1e67"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The number 1 should indicate the topic \"sex\", while \"none of the above\" indicates category 6\n",
            "\n",
            "Quotes with the label \"1. none of the above\"\n",
            "171                                                            Sounds like SOMEBODY'S got a case of the Mondays!\n",
            "864                                                           Is it just me, or is it getting crazier out there?\n",
            "487    It takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.\n",
            "107                                                     YES, they deserved to die, and I hope they burn in hell!\n",
            "398                                                      You're taking all the caviar? That caviar is a garnish!\n",
            "695                                                    I love you, but you don't know what you're talking about.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# First look at labels with '1' and 'none of the above'\n",
        "  # normally '1' would indicate the topic 'sex'\n",
        "\n",
        "print('The number 1 should indicate the topic \"sex\", while \"none of the above\" indicates category 6')\n",
        "print()\n",
        "\n",
        "print('Quotes with the label \"1. none of the above\"')\n",
        "print(train.loc[train['label_ChatGPT_clean'] == '1. none of the above', 'Movie Quote'])\n",
        "print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XwjHy7rGj1Rp",
      "metadata": {
        "id": "XwjHy7rGj1Rp"
      },
      "source": [
        "The six quotes labelled as '1. none of the above' do not appear to be related to sex. However, it cannot be said that all quotes should be classified as '6. none of the above'. The quote - \"YES, they deserved to die, and I hope they burn in hell!\" - would likely be seen as religous or insensitive. This is demonstrative of the unreliability of ChatGPT's labels.\n",
        "\n",
        "We will label these six quotes as '6. none of the above' as is likely intended. As for the last quote, we will attempt to fix ChatGPT's unreliability in a later step.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 454,
      "id": "268VEdNgmkl4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "268VEdNgmkl4",
        "outputId": "85d49ac9-ad28-4404-99be-ee9ef4171ece"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 9 unique label_ChatGPT_cleans in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPT_cleans:  \n",
            "\n",
            "\n",
            "                      Count Percentage\n",
            "label_ChatGPT_clean                   \n",
            "6. none of the above    584     92.26%\n",
            "3. politics              19       3.0%\n",
            "4. rude                   9      1.42%\n",
            "2. religion               8      1.26%\n",
            "1. sex                    5      0.79%\n",
            "5. insensitive            4      0.63%\n",
            "5. rude                   2      0.32%\n",
            "1. politics               1      0.16%\n",
            "3.\\n3. politics           1      0.16%\n"
          ]
        }
      ],
      "source": [
        "# convert '1. none of the above' -> '6. none of the above'\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '1. none of the above', '6. none of the above'))\n",
        "\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT_clean', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4Zylg1x3m4Vp",
      "metadata": {
        "id": "4Zylg1x3m4Vp"
      },
      "source": [
        "We now investigate the remaining labels that do not fit into one of the intended six categories:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 455,
      "id": "I8YYXpSLnGxZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I8YYXpSLnGxZ",
        "outputId": "3badec69-ab30-4206-b5da-0a8ab78b2779"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quotes with the label \"5. rude\" (should be \"4. rude\")\n",
            "181    Pay no attention to that man behind the curtain!\n",
            "104                                              Bogus.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes with the label \"3.\n",
            "3. politics\"\n",
            "496    Why are you trying so hard to fit in when you were born to stand out?\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes with the label \"1. politics\" (should be \"3. politics\")\n",
            "849    Sometimes, when you're feeling helpless, the secret is to help someone else. Get out of your own head. Trust me. The next time someone asks for help, say yes.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('Quotes with the label \"5. rude\" (should be \"4. rude\")')\n",
        "print(train.loc[train['label_ChatGPT_clean'] == '5. rude', 'Movie Quote'])\n",
        "print()\n",
        "\n",
        "print('Quotes with the label \"3.\\n3. politics\"')\n",
        "print(train.loc[train['label_ChatGPT_clean'] == '3.\\n3. politics', 'Movie Quote'])\n",
        "\n",
        "print()\n",
        "print('Quotes with the label \"1. politics\" (should be \"3. politics\")')\n",
        "print(train.loc[train['label_ChatGPT_clean'] == '1. politics', 'Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Bg9wh6JE56j0",
      "metadata": {
        "id": "Bg9wh6JE56j0"
      },
      "source": [
        "The quotes labelled as '5. rude' do appear to be rude and so we change their label to the correct '4. rude'. This suggests that the text in the quote is more accurate than the number.\n",
        "\n",
        "The other two quotes do not appear to be inherently political. We hence choose to disagree with ChatGPT and label them as '6. none of the above'. While this may be introducing our own bias, this is reasonable given ChatGPT's unreliability and own bias."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 456,
      "id": "U7DoCRKn6qiv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U7DoCRKn6qiv",
        "outputId": "4597cf3a-28fd-4ce1-cad8-106b31b0e5c2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 6 unique label_ChatGPT_cleans in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPT_cleans:  \n",
            "\n",
            "\n",
            "                      Count Percentage\n",
            "label_ChatGPT_clean                   \n",
            "6. none of the above    586     92.58%\n",
            "3. politics              19       3.0%\n",
            "4. rude                  11      1.74%\n",
            "2. religion               8      1.26%\n",
            "1. sex                    5      0.79%\n",
            "5. insensitive            4      0.63%\n"
          ]
        }
      ],
      "source": [
        "# Fix labels above:\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '5. rude', '4. rude'))\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '1. politics', '6. none of the above'))\n",
        "train['label_ChatGPT_clean'] = train['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '3.\\n3. politics', '6. none of the above'))\n",
        "\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT_clean', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ugGbcYg27FIC",
      "metadata": {
        "id": "ugGbcYg27FIC"
      },
      "source": [
        "Our training label now contains just 6 unique labels, corresponding to those specified in the data dictionary. We can observe that over 90% of the quotes have been labelled as '6. none of the above', indicating that we have highly imbalanced data.\n",
        "\n",
        "---\n",
        "\n",
        "Next we create a function that will clean the validation and test values for `label_ChatGPT`. We will not be able to see the values contained to avoid leakage. Therefore, we will use fuzzy matching to automatically allocate the labels into one of the six categories. We also impute any missing values with the default value '6. none of the above'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 457,
      "id": "dO_XAFPu82OK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dO_XAFPu82OK",
        "outputId": "6593e406-ab32-44c7-a2d6-f67cbbf22ee1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All values in the label_ChatGPT_clean column are one of the six categories\n",
            "All values in the label_ChatGPT_clean column are one of the six categories\n"
          ]
        }
      ],
      "source": [
        "# categories of the label we permit\n",
        "categories = ['1. sex', '2. religion', '3. politics', '4. rude', '5. insensitive', '6. none of the above']\n",
        "\n",
        "# Function to find the closest matching category for a given value\n",
        "def find_closest_category(value):\n",
        "\n",
        "    # no change if value already one of the categories\n",
        "    if value in categories:\n",
        "      return value\n",
        "    # else find the closest match:\n",
        "    closest_match, score = process.extractOne(value, categories)\n",
        "    return closest_match\n",
        "\n",
        "\n",
        "def clean_chatGPT(data):\n",
        "\n",
        "  # fill NA values with '6. none of the above' by default\n",
        "  data['label_ChatGPT_clean'] = data['label_ChatGPT'].fillna('6. none of the above')\n",
        "\n",
        "  # convert all to lowercase\n",
        "  data['label_ChatGPT_clean'] = data['label_ChatGPT_clean'].apply(str.lower)\n",
        "\n",
        "  # replace any labels that contain '6. none of the above' with '6. none of the above'\n",
        "  data['label_ChatGPT_clean'] = data['label_ChatGPT_clean'].apply(lambda x: replace_label(x, '6. none of the above', '6. none of the above'))\n",
        "\n",
        "  # replace any labels that contain 'none of the above' with '6. none of the above'\n",
        "  data['label_ChatGPT_clean'] = data['label_ChatGPT_clean'].apply(lambda x: replace_label(x, 'none of the above', '6. none of the above'))\n",
        "\n",
        "  # Now use fuzzy matching to correct any other labels that are not one of the six categories\n",
        "  data['label_ChatGPT_clean'] = data['label_ChatGPT_clean'].apply(find_closest_category)\n",
        "\n",
        "  if set(data[\"label_ChatGPT_clean\"]).issubset(categories):\n",
        "    print(f'All values in the label_ChatGPT_clean column are one of the six categories')\n",
        "  else:\n",
        "    print(f'ERROR: some values in the label_ChatGPT_clean column are not one of the six categories')\n",
        "\n",
        "  return(data['label_ChatGPT_clean'])\n",
        "\n",
        "\n",
        "val['label_ChatGPT_clean'] = clean_chatGPT(val)\n",
        "test['label_ChatGPT_clean'] = clean_chatGPT(test)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 458,
      "id": "6VGpObpm0uE3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 116
        },
        "id": "6VGpObpm0uE3",
        "outputId": "be3f53f7-7ec2-4c45-b5c4-d25d482b5cd6"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1f437438-c1e3-4652-8a4c-4b03835adf4d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>Movie Name</th>\n",
              "      <th>Actor</th>\n",
              "      <th>Character</th>\n",
              "      <th>Year</th>\n",
              "      <th>Genre</th>\n",
              "      <th>Director</th>\n",
              "      <th>Writer</th>\n",
              "      <th>Production Company</th>\n",
              "      <th>Sentiment_VADER</th>\n",
              "      <th>...</th>\n",
              "      <th>ActionAdventure</th>\n",
              "      <th>Crime</th>\n",
              "      <th>ThrillerHorror</th>\n",
              "      <th>Fantasy</th>\n",
              "      <th>Sci-Fi</th>\n",
              "      <th>Animation</th>\n",
              "      <th>Romance</th>\n",
              "      <th>text_bert_noSW</th>\n",
              "      <th>text_bert_withSW</th>\n",
              "      <th>label_ChatGPT_clean</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>0 rows × 33 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1f437438-c1e3-4652-8a4c-4b03835adf4d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1f437438-c1e3-4652-8a4c-4b03835adf4d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1f437438-c1e3-4652-8a4c-4b03835adf4d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [Movie Quote, Movie Name, Actor, Character, Year, Genre, Director, Writer, Production Company, Sentiment_VADER, Sentiment_TextBlob, label_ChatGPT, Year_clean, Genre_split, word_count, Quote_clean, exclamation_flag, question_flag, Quote_tokens, Quote_tokens_noSW, Quote_tokens_noSW_lemm, Drama, Comedy, ActionAdventure, Crime, ThrillerHorror, Fantasy, Sci-Fi, Animation, Romance, text_bert_noSW, text_bert_withSW, label_ChatGPT_clean]\n",
              "Index: []\n",
              "\n",
              "[0 rows x 33 columns]"
            ]
          },
          "execution_count": 458,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Check no missing values in the training data for the label:\n",
        "train[train['label_ChatGPT_clean'].isna()]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f6RV99BrEGbh",
      "metadata": {
        "id": "f6RV99BrEGbh"
      },
      "source": [
        "The output above indicates that the validation and test set values for the chatGPT label have been cleaned with all values being one of our six specified categories. Similarly there are no missing labels in our training data.\n",
        "\n",
        "Our three datasets now all have a `label_ChatGPT_clean` column which only contains values belonging to our six categories."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5y4THKt0VVnM",
      "metadata": {
        "id": "5y4THKt0VVnM"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "- Initially our training data had 21 unique values for `label_ChatGPT`\n",
        "- After cleaning, the label now only contains values belonging to one of the six categories specified in the data dictionary\n",
        "- This cleaning was done through regex after investigating what label was likely intended\n",
        "- We corroborated the idea that the chatGPT labels are still unreliable. For example it missed clear references to religion. We try to improve its reliability in the next step."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OZn08x1DVVnM",
      "metadata": {
        "id": "OZn08x1DVVnM"
      },
      "source": [
        "#### Q1d Step 2 - Improving the reliability of the ChatGPT label in the training set\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "Our previous EDA revealed that the `label_ChatGPT` column was highly unreliable as it often missing the presence of swearing and included a significant number of incorrect labels (e.g. quotes labelled as religious that were clearly not). In this section, we attempt to improve the columns's reliability by correcting erroneous labels. This will be done by:\n",
        "\n",
        "  - flagging swearing in quotes using the `better_profanity` library\n",
        "  - manually reviewing our training data\n",
        "  - searching for key words in our quotes that relate to the labels\n",
        "\n",
        "For now, we do this only for the training data.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "If we used the label in its unreliable state our model would fall victim to 'garbage in, garbage out' and our classifier would likely be useless for the movie production studio. It is hence paramount we fix the label so it is objective and gives our classifier a higher chance to make accurate predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Y68r4oEQJyOl",
      "metadata": {
        "id": "Y68r4oEQJyOl"
      },
      "source": [
        "#####Flag swearing in quotes\n",
        "\n",
        "One of the categories in `label_ChatGPT` relates to swearing ('4. rude'). In our previous EDA, we noticed that there were quotes that contained swearing yet had been erroneously labelled '6. none of the above'. We use the `better_profanity` library to identify swearwords in our quotes. If a quote is detected to contain a swear word it will be given the label '4. rude'. It does not matter if a quote's previous label is overwritten, since categories 1-5 will be grouped together when we create the binary response.\n",
        "\n",
        "---\n",
        "We now test the profanity library on our quotes to verify that it correctly identifies swearwords. This library essentially does a lookup against a list of swearwords to identify if profanity is present."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 459,
      "id": "IAU6QzHTVVnM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IAU6QzHTVVnM",
        "outputId": "a52ea53a-7256-478a-edf5-83558faa073d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Yipee kayaye, motherfucker!\n",
            "Does the above quote contain swearing: True\n",
            "Current ChatGPT label: 6. none of the above\n",
            "\n",
            "\"Life is not the amount of breaths you take, it's the moments that take your breath away.\"\n",
            "Does the above quote contain swearing: False\n",
            "Current ChatGPT label: 6. none of the above\n"
          ]
        }
      ],
      "source": [
        "# Identifying swearing:\n",
        "# load the list of swearwords:\n",
        "profanity.load_censor_words()\n",
        "\n",
        "# Let us do a test first:\n",
        "print(train.loc[116, 'Movie Quote'])\n",
        "print(f'Does the above quote contain swearing: {profanity.contains_profanity(train.loc[116, \"Movie Quote\"])}')\n",
        "print(f'Current ChatGPT label: {train.loc[116, \"label_ChatGPT_clean\"]}')\n",
        "print()\n",
        "print(train.loc[5, 'Movie Quote'])\n",
        "print(f'Does the above quote contain swearing: {profanity.contains_profanity(train.loc[5, \"Movie Quote\"])}')\n",
        "print(f'Current ChatGPT label: {train.loc[5, \"label_ChatGPT_clean\"]}')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "odAA0ZrrL40d",
      "metadata": {
        "id": "odAA0ZrrL40d"
      },
      "source": [
        "The test above reveals that the `better_profanity` library correctly identifies that there is swearing in the first quote but not in the second. Notably, chatGPT did **not** detect swearing in the first quote, reinforcing how the label is currently unreliable.\n",
        "\n",
        "We now proceed to apply this package to update our label:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 460,
      "id": "BHSthiw2Mm_q",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BHSthiw2Mm_q",
        "outputId": "4f2be0fe-8630-4276-9600-58d6548e200a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                    Movie Quote  \\\n",
            "5    \"Life is not the amount of breaths you take, it's the moments that take your breath away.\"   \n",
            "116                                                                 Yipee kayaye, motherfucker!   \n",
            "631                                                                              Fuck them kids   \n",
            "181                                            Pay no attention to that man behind the curtain!   \n",
            "\n",
            "      label_ChatGPT_clean label_ChatGPT_reliable  \n",
            "5    6. none of the above   6. none of the above  \n",
            "116  6. none of the above                4. rude  \n",
            "631  6. none of the above                4. rude  \n",
            "181               4. rude                4. rude  \n"
          ]
        }
      ],
      "source": [
        "# search for profanity in our quotes and label them as '4. rude' if they do\n",
        "train['label_ChatGPT_reliable'] = train.apply(lambda row: '4. rude' if profanity.contains_profanity(row['Movie Quote']) else row['label_ChatGPT_clean'], axis = 1)\n",
        "\n",
        "# spot check of quotes with and without swearing:\n",
        "print(train.loc[[5, 116, 631, 181] ,['Movie Quote', \"label_ChatGPT_clean\", \"label_ChatGPT_reliable\"]])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BocOW6Rh947h",
      "metadata": {
        "id": "BocOW6Rh947h"
      },
      "source": [
        "The above output shows the quote, its previous label and its new label after applying the check for profanity. We see that the second and third quotes (which contain profanity) are labelled as rude where they previously were not. The two other quotes that did not contain profanity had no change in their label.\n",
        "\n",
        "This fix should improve the reliability of our training response label and allow us to construct a more useful classifier for the production company. If we had not made this change, our classifier may have been poor at flagging quotes containing swearing which may have led to more contract stipulations being broken with actors."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LFlaDf3xd7fA",
      "metadata": {
        "id": "LFlaDf3xd7fA"
      },
      "source": [
        "**Reasonableness Check: we expect that the number of quotes labelled as rude has increased**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 461,
      "id": "zo_epDrneDSM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zo_epDrneDSM",
        "outputId": "2e554204-9fdc-411b-8a8e-7cac3f9b2378"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Label before we made adjustment for profanity:\n",
            "There are 6 unique label_ChatGPT_cleans in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPT_cleans:  \n",
            "\n",
            "\n",
            "                      Count Percentage\n",
            "label_ChatGPT_clean                   \n",
            "6. none of the above    586     92.58%\n",
            "3. politics              19       3.0%\n",
            "4. rude                  11      1.74%\n",
            "2. religion               8      1.26%\n",
            "1. sex                    5      0.79%\n",
            "5. insensitive            4      0.63%\n",
            "\n",
            "Label after we made adjustment for profanity:\n",
            "There are 6 unique label_ChatGPT_reliables in the dataset (from 633 rows)\n",
            "Total count and percentage of the top 20 label_ChatGPT_reliables:  \n",
            "\n",
            "\n",
            "                        Count Percentage\n",
            "label_ChatGPT_reliable                  \n",
            "6. none of the above      542     85.62%\n",
            "4. rude                    57       9.0%\n",
            "3. politics                17      2.69%\n",
            "2. religion                 8      1.26%\n",
            "1. sex                      5      0.79%\n",
            "5. insensitive              4      0.63%\n"
          ]
        }
      ],
      "source": [
        "print('Label before we made adjustment for profanity:')\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT_clean', 20)\n",
        "\n",
        "print()\n",
        "print('Label after we made adjustment for profanity:')\n",
        "cardinality_and_count_analysis(train, 'label_ChatGPT_reliable', 20)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a_9NbQjsiOy7",
      "metadata": {
        "id": "a_9NbQjsiOy7"
      },
      "source": [
        "We see that the amount of quotes in the '4. rude' category has increased from 11 to 57 as expected. Most of the quotes that changed label came from '6. none of the above' which is also expected given this is the largest category. Two quotes also moved from the '3. politics' label; this is fine as when we construct our binary response these categories will be combined. Therefore, the changes appear to pass this basic 'smell' test."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6pQGOLneCOmJ",
      "metadata": {
        "id": "6pQGOLneCOmJ"
      },
      "source": [
        "##### Manual Review and related word search\n",
        "\n",
        "Our previous EDA on the `label_ChatGPT` column revealed that the label often erroneously tagged quotes as category 1-5 and tagged quotes as '6. none of the above' when they had clear connotations relating to categories 1-5. This issue with the label's reliability is harder to fix than a simple search for profanity. We attempt to tackle this problem in two ways:\n",
        "\n",
        "- *Manual Review:* we have a relatively small training set and only a fraction of quotes are not in '6. none of the above'. We will manually review quotes currently in categories 1-5. This may introduce some of our own bias. However, unlike ChatGPT, which is also biased, we can explain our reasoning for any labels. We therefore believe this to be appropriate.\n",
        "- *Related word search*: We will attempt to find mislabelling of quotes as '6' by searching for key words in the training set. For example, by searching for words related to 'religion'; this will be done using the DataMuse API. This method will likely still miss some of the nuance related to the topics.\n",
        "\n",
        "The DataMuse API is a query engine that allows us to search for words most related to a topic. We will get the top 15 words related to each category 1-5 and search for them in our training data. Searching for a greater number of related words tends to lead to more tangential words and poor results.\n",
        "\n",
        "During our EDA, we looked at the relationship between `label_ChatGPT` and our sentiment scores, `Sentiment_VADER` and `Sentiment_TextBlob`. There was an ambiguous relationship between the sentiment scores and the ChatGPT label, and so we do not use the sentiment scores to try and identify mislabellings in the training data."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JlsHgIktChlj",
      "metadata": {
        "id": "JlsHgIktChlj"
      },
      "source": [
        "\n",
        "---\n",
        "**Category 1: sex**\n",
        "\n",
        "Let us first review any quotes currently flagged as '1. sex'. We will manually fix any quotes that we believe should not be part of the category."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 462,
      "id": "vTjM8gApC-eb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vTjM8gApC-eb",
        "outputId": "60260348-80dd-4512-e4c2-9012118d990f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The following quotes are labelled as \"1. sex\":\n",
            "901                                                  I used to use this little gun when I was a prostitude.\n",
            "666    The problem is, I pretty much fancy everyone. It's like I've just got out of prison all of the time.\n",
            "159                                                             Why don't you come up sometime, and see me?\n",
            "7                                                                          \"I wish I knew how to quit you.\"\n",
            "794                                                                            My tastes are very singular.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('The following quotes are labelled as \"1. sex\":')\n",
        "print(train.loc[train['label_ChatGPT_reliable'] ==  '1. sex']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "szS926VbG60B",
      "metadata": {
        "id": "szS926VbG60B"
      },
      "source": [
        "Overall, the above quotes do appear to be related to the category, apart from the first quote - \"I wish I knew how to quit you\" - which is really more an innocuous reference to love not sex. Overall, the labelling of '1. sex' appears to be reliable.\n",
        "\n",
        "We now modify the label of \"I wish I knew how to quit you\":\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 463,
      "id": "Tq51GXcYLwSP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tq51GXcYLwSP",
        "outputId": "dfe6dd71-f004-4b02-eae0-6ca82e82bc4b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "After correction the label is:\n",
            "label_ChatGPT_reliable                6. none of the above\n",
            "Movie Quote               \"I wish I knew how to quit you.\"\n",
            "Name: 7, dtype: object\n"
          ]
        }
      ],
      "source": [
        "train.loc[7, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "\n",
        "print('After correction the label is:')\n",
        "print(train.loc[7, ['label_ChatGPT_reliable', 'Movie Quote']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nj6ur-s1Mf-g",
      "metadata": {
        "id": "nj6ur-s1Mf-g"
      },
      "source": [
        "Next we attempt to find quotes labelled as '6. none of the above' but actually related to '1. sex'. We do this by using the DataMuse API and searching for words related to 'sex'. This database contains information about words semantically related to other words. We therefore connect to the API and query about what the top words related to 'sex' are. We then search for these words in our quotes and print them out:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 464,
      "id": "ROAqDHJvIHsa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ROAqDHJvIHsa",
        "outputId": "dac3a7a2-3fe5-4e1e-886c-10b3ea58914a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['scenes', 'porn', 'pornography', 'toys', 'violence']\n"
          ]
        }
      ],
      "source": [
        "# Connect to the Datemuse API:\n",
        "api = Datamuse()\n",
        "\n",
        "# Query the database using the API for words related to 'sex'\n",
        "api_lookup_sex = api.words(rel_trg = 'sex', max = 15, topics = ['movie'])\n",
        "related_words_sex = [item['word'] for item in api_lookup_sex]\n",
        "related_words_sex.append('sex')\n",
        "\n",
        "# we now print the first 5 words that the query returned:\n",
        " # (I inspected the full list so I could be confident in the results)\n",
        "print(related_words_sex[0:5])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1by_GlUsNYo1",
      "metadata": {
        "id": "1by_GlUsNYo1"
      },
      "source": [
        "The words returned are related to sex in some capacity however, this topic is nuanced and a key word search is likely to miss quotes related to this theme. Regardless, we use these words to do a naive search of our quotes for references to sex:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 465,
      "id": "UFYb0A5lKxDJ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UFYb0A5lKxDJ",
        "outputId": "830aa8cb-4d7a-4a5f-91f3-aaf1f065b30f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                                                                                                                                                                                                                                                           Quote_clean  \\\n",
            "384                                                                                                                                                                                                                                                                                                              draw me like one of your french girls   \n",
            "323  you are a you are a complex freudian hallucination having something to do with my mother and i do not know why you have wings but you have very lovely legs and you are a very nice tiny person and what am i saying i do not know who my mother was i am an orphan and i have never taken drugs because i missed the sixties i was an accountant   \n",
            "639                                                                                                                                                                                                                                                              i love rumors facts can be so misleading but rumors true or false are often revealing   \n",
            "114                                                                                                                                                                                                                                                                                                                      i sell drugs to the community   \n",
            "522                                                                                                                                                                                                                                         nunchuck skills bowhunting skills computer hacking skills girls only want boyfriends who have great skills   \n",
            "655                                                                                                                                                                                                                                                                                       you cocky cock you will pay for your crimes against humanity   \n",
            "\n",
            "      label_ChatGPT_clean  \n",
            "384  6. none of the above  \n",
            "323  6. none of the above  \n",
            "639  6. none of the above  \n",
            "114  6. none of the above  \n",
            "522  6. none of the above  \n",
            "655  6. none of the above  \n"
          ]
        }
      ],
      "source": [
        "\n",
        "# search for words containing at least one of the related words:\n",
        "quote_contains_word = train['Quote_clean'].str.contains('|'.join(related_words_sex))\n",
        "\n",
        "print(train.loc[quote_contains_word, ['Quote_clean', 'label_ChatGPT_clean']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JdXlSYOQO6fw",
      "metadata": {
        "id": "JdXlSYOQO6fw"
      },
      "source": [
        "Our search for key words related to 'sex' finds only six quotes with a match. Based on our judgement and knowledge of the films, none of these quotes appear to be related to 'sex'. Futhermore, the search did not even return the quotes that are currently tagged as '1. sex', highlighting how this approach may not be effective as it misses a lot of nuance.\n",
        "\n",
        "We choose not to change the label of the quotes identified as they do not appear to be related to 'sex'.\n",
        "\n",
        "Although this approach may not be effective for identifying quotes belonging to this category, it may also suggest that there are no more mislabelled quotes in the training data that are related to 'sex'. Overall, this category appears to be reliable.\n",
        "\n",
        "Now we consider the next category and follow a similar approach."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8IqhqWYzPqze",
      "metadata": {
        "id": "8IqhqWYzPqze"
      },
      "source": [
        "**Category 2: religion**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "harK-yph6VDt",
      "metadata": {
        "id": "harK-yph6VDt"
      },
      "source": [
        "First we look at the quotes in our training data that have been tagged as '2. religion' to verify their reliability:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 466,
      "id": "tYq5s03ZP0JZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tYq5s03ZP0JZ",
        "outputId": "431c7426-d41f-4136-8364-e661ebb96091"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The following quotes are labelled as \"2. religion\":\n",
            "264                                               You must unlearn what you have learned.\n",
            "845                                       The hardest choices require the strongest wills\n",
            "2                                                                \"I can do this all day.\"\n",
            "137                                                    THE POWER OF CHRIST COMPELS YOU!!!\n",
            "298                                                           Death cannot stop true love\n",
            "698                                                We accept the love we think we deserve\n",
            "304                                                            Welcome to the party, pal!\n",
            "601    But when you forgive, you love. And when you love, God's light shines through you.\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('The following quotes are labelled as \"2. religion\":')\n",
        "print(train.loc[train['label_ChatGPT_reliable'] ==  '2. religion']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ijuC-VTBQC-s",
      "metadata": {
        "id": "ijuC-VTBQC-s"
      },
      "source": [
        "Many of these quotes do **not** appear to be related to religion and so we change their label as required. For example, \"I can do this all day\" does not have any obvious religious connotations. We change six quotes in the training data to '6. none of the above' from '2. religion'. This leaves us with only two quotes in the training data labelled as 'religious' which exacerbates the problem of the categories being highly imbalanced."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 467,
      "id": "WRoZ-HnGQTwT",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRoZ-HnGQTwT",
        "outputId": "ed7aedd9-56bd-4d0a-862c-ba0860435961"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "After correction the label for the following quotes are:\n",
            "    label_ChatGPT_reliable                                      Movie Quote\n",
            "298   6. none of the above                      Death cannot stop true love\n",
            "698   6. none of the above           We accept the love we think we deserve\n",
            "2     6. none of the above                         \"I can do this all day.\"\n",
            "264   6. none of the above          You must unlearn what you have learned.\n",
            "845   6. none of the above  The hardest choices require the strongest wills\n",
            "304   6. none of the above                       Welcome to the party, pal!\n"
          ]
        }
      ],
      "source": [
        "# Correct mislabelling of quotes as religious:\n",
        "train.loc[298, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[698, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[2, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[264, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[845, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[304, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "\n",
        "print('After correction the label for the following quotes are:')\n",
        "print(train.loc[[298,698,2,264,845, 304], ['label_ChatGPT_reliable', 'Movie Quote']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9GGMwSzd62vq",
      "metadata": {
        "id": "9GGMwSzd62vq"
      },
      "source": [
        "Next we use DataMuse to search for words related to religion. We also search for some additional words that we believe are related to religion based on our domain knowledge:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 468,
      "id": "yZZES5EyRCMa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yZZES5EyRCMa",
        "outputId": "5c2631a3-5a4f-41c3-9766-2630c024ceff"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['cult', 'mythology', 'morality', 'politics', 'culture']\n"
          ]
        }
      ],
      "source": [
        "# Next we query the database for word related to religion\n",
        "\n",
        "# Query the database using the API for words related to 'sex'\n",
        "api_lookup_religion = api.words(rel_trg = 'religion', max = 15, topics = ['movie'])\n",
        "related_words_religion = [item['word'] for item in api_lookup_religion] # convert to list\n",
        "related_words_religion.append('god')\n",
        "related_words_religion.append('christ')\n",
        "related_words_religion.append('pray')\n",
        "related_words_religion.append('jesus')\n",
        "\n",
        "# we now print the first 5 words that the query returned:\n",
        " # (I inspected the full list so I could be confident in the results)\n",
        "print(related_words_religion[0:5])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 469,
      "id": "EA1scIkUSMGT",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EA1scIkUSMGT",
        "outputId": "9a6a4dd8-fcc5-4a2a-da68-898ced912f4e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                                                                                                                                                                                                       Quote_clean  \\\n",
            "145                                                                                                                                                                                                                                   they may take our lives but they will never take our freedom   \n",
            "386  people live their lives bound by what they accept as correct and true that is how they define reality but what does it mean to be correct or true merely vague concepts their reality may all be a mirage can we consider them to simply be living in their own world shaped by their beliefs   \n",
            "853                                                                                                                     for i am a sinner in the hands of an angry god bloody mary full of vodka blessed are you among cocktails pray for me now at the hour of my death which i hope is soon amen   \n",
            "750                                                                                                                                                                                                                           the price of freedom is high and it is the price i am willing to pay   \n",
            "137                                                                                                                                                                                                                                                                the power of christ compels you   \n",
            "573                                                                                                                                                                                                                                                          dear pound ounce newborn infant jesus   \n",
            "601                                                                                                                                                                                                                 but when you forgive you love and when you love god s light shines through you   \n",
            "201                                                                                                                                                                                                                                              as god is my witness i will never be hungry again   \n",
            "325                                                                                                                                                             hook you let those kids out of that net in less than one minute or you better get an attorney and hope to god he is better than me   \n",
            "\n",
            "      label_ChatGPT_clean  \n",
            "145  6. none of the above  \n",
            "386  6. none of the above  \n",
            "853  6. none of the above  \n",
            "750           3. politics  \n",
            "137           2. religion  \n",
            "573  6. none of the above  \n",
            "601           2. religion  \n",
            "201  6. none of the above  \n",
            "325               4. rude  \n"
          ]
        }
      ],
      "source": [
        "# search for words containing at least one of the related words:\n",
        "quote_contains_word = train['Quote_clean'].str.contains('|'.join(related_words_religion))\n",
        "\n",
        "print(train.loc[quote_contains_word, ['Quote_clean', 'label_ChatGPT_clean']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hQ6m5IdXS7Z_",
      "metadata": {
        "id": "hQ6m5IdXS7Z_"
      },
      "source": [
        "Using the 15 words most-related to religion from WordMuse along with some additional words from our domain knowledge, the word search does appear to identify some quotes related to religion. Not all quotes identified by the keyword-search are religious.\n",
        "\n",
        "We re-label the training quotes outputted that do appear to be about religion but have not been classified as such.  Even though we are adding labels to this category, the imbalanced label issue still persists."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 470,
      "id": "r6Mm1Tr1TnrQ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r6Mm1Tr1TnrQ",
        "outputId": "6218b0ae-d401-4b27-f111-697be5ffb866"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "After correction the label for the following quotes are:\n",
            "    label_ChatGPT_reliable  \\\n",
            "853            2. religion   \n",
            "201            2. religion   \n",
            "573            2. religion   \n",
            "\n",
            "                                                                                                                                                                           Movie Quote  \n",
            "853  For I am a Sinner in the hands of an angry God, Bloody Mary, Full of Vodka, Blessed are you among cocktails, pray for me now, at the hour of my death, which I hope is soon. Amen  \n",
            "201                                                                                                                                  As God is my witness, I'll never be hungry again.  \n",
            "573                                                                                                                                      Dear 8-pound, 6-ounce newborn infant Jesus...  \n"
          ]
        }
      ],
      "source": [
        "train.loc[853, 'label_ChatGPT_reliable'] = categories[1] # change to '1. religion'\n",
        "train.loc[201, 'label_ChatGPT_reliable'] = categories[1]\n",
        "train.loc[573, 'label_ChatGPT_reliable'] = categories[1]\n",
        "\n",
        "print('After correction the label for the following quotes are:')\n",
        "print(train.loc[[853,201,573], ['label_ChatGPT_reliable', 'Movie Quote']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AxFkCLFnT5MG",
      "metadata": {
        "id": "AxFkCLFnT5MG"
      },
      "source": [
        "**Category 3: politics**\n",
        "\n",
        "We follow a similar approach to the prior categories: look at quotes that are currently labelled 'politics' and analyse quotes with words related to politics using WordMuse."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 471,
      "id": "S4xYOuavT7ZM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S4xYOuavT7ZM",
        "outputId": "32fa91e3-0a7b-4aae-875f-8ca66a0ecfbc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The following quotes are labelled as \"3. politics\":\n",
            "804                                                                                                                                                       Bombs won't stop terrorism, brains will, and we don't have nearly enough of those.\n",
            "562                                                                                                                                                                                               Float like a cadillac sting like a beemer.\n",
            "830                                                                                                                              I have burrowed underneath your brain. I am nested there. I am the scream in your mind. You will cooperate.\n",
            "600                                                                                                           Some people feel like they don't deserve love. They walk away quietly into empty spaces, trying to close the gaps of the past.\n",
            "508                                                                                                                                                                                                      We should totally just stab Caesar!\n",
            "506                                                                                                                                                                            Stop trying to make 'fetch' happen. It's not going to happen.\n",
            "468                                                                       Itâ€™s a dangerous business, going out your doorâ€¦You step into the Road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.\n",
            "750                                                                                                                                                                     The price of freedom is high, and it's the price I am willing to pay\n",
            "236    Men, you are about to embark on a great crusade to stamp out runaway decency in the west. Now you men will only be risking your lives, whilst I will be risking an almost certain Academy Award nomination for Best Supporting Actor.\n",
            "208                                                                                                                                                                                        When it comes to women, youâ€™re a true democrat.\n",
            "129                                                                                                                                                                                                                         HOLY SCHNIKES!!!\n",
            "826                                                                                                                                                           Every time we get a chance to get ahead they move the finish line, every time.\n",
            "536                                                                                                                                                                                                              I am the Jewish James Bond!\n",
            "667                                                                                                                                                            When you play game of thrones, you win or you die, there is no middle ground!\n",
            "542                                                                                                                                                                                                                         I am the senate!\n",
            "711                                                                                                                                                                             Proximity to power deludes some into thinking they wield it.\n",
            "220                                                                                                                                                                                Gentlemen, you can't fight in here! This is the War Room!\n",
            "Name: Movie Quote, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('The following quotes are labelled as \"3. politics\":')\n",
        "print(train.loc[train['label_ChatGPT_reliable'] ==  '3. politics']['Movie Quote'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "t2AD8FoiUrql",
      "metadata": {
        "id": "t2AD8FoiUrql"
      },
      "source": [
        "We use our domain knowledge of movies and politics to remove the 'politics' label for quotes that are clearly unrelated to the topic. For example, the quote \"Float like a cadillac sting like a beemer\" from the *Pixar* movie *Cars*. In total we remove five 'politics' labels, leaving 12 quotes in the training set with this label:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 472,
      "id": "a72i8pCLagZ3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a72i8pCLagZ3",
        "outputId": "56f3e12c-c123-4103-b0e4-600a8b7de33b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "After correction the label for the following quotes are:\n",
            "    label_ChatGPT_reliable  \\\n",
            "468   6. none of the above   \n",
            "600   6. none of the above   \n",
            "562   6. none of the above   \n",
            "129   6. none of the above   \n",
            "506   6. none of the above   \n",
            "\n",
            "                                                                                                                                                            Movie Quote  \n",
            "468  Itâ€™s a dangerous business, going out your doorâ€¦You step into the Road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.  \n",
            "600                                      Some people feel like they don't deserve love. They walk away quietly into empty spaces, trying to close the gaps of the past.  \n",
            "562                                                                                                                          Float like a cadillac sting like a beemer.  \n",
            "129                                                                                                                                                    HOLY SCHNIKES!!!  \n",
            "506                                                                                                       Stop trying to make 'fetch' happen. It's not going to happen.  \n"
          ]
        }
      ],
      "source": [
        "# Correct mislabelling of quotes as political:\n",
        "train.loc[468, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[600, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[562, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[129, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "train.loc[506, 'label_ChatGPT_reliable'] = categories[5] # change to none of the above\n",
        "\n",
        "print('After correction the label for the following quotes are:')\n",
        "print(train.loc[[468,600,562,129,506], ['label_ChatGPT_reliable', 'Movie Quote']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "P5I72M_F73JJ",
      "metadata": {
        "id": "P5I72M_F73JJ"
      },
      "source": [
        "Next we query the DataMuse API for words related to politics:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 473,
      "id": "JHcPn-gnbNQ_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JHcPn-gnbNQ_",
        "outputId": "679c22e6-978e-41bf-b1a5-f2b29705a873"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['religion', 'election', 'party', 'mayor', 'culture']\n"
          ]
        }
      ],
      "source": [
        "# Next we query the database for word related to politics\n",
        "\n",
        "# Query the database using the API for words related to 'politics'\n",
        "api_lookup_politics = api.words(rel_trg = 'politics', max = 15, topics = ['movie'])\n",
        "related_words_politics = [item['word'] for item in api_lookup_politics] # convert to list\n",
        "\n",
        "# we now print the first 5 words that the query returned:\n",
        " # (I inspected the full list so I could be confident in the results)\n",
        "print(related_words_politics[0:5])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 474,
      "id": "VOrrw6nXcMPj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VOrrw6nXcMPj",
        "outputId": "37b96092-9010-4398-c4dc-055c9d67a379"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                       Quote_clean   label_ChatGPT_clean  \\\n",
            "331  party on wayne party on garth  6. none of the above   \n",
            "304       welcome to the party pal           2. religion   \n",
            "\n",
            "    label_ChatGPT_reliable  \n",
            "331   6. none of the above  \n",
            "304   6. none of the above  \n"
          ]
        }
      ],
      "source": [
        "# search for words containing at least one of the related words:\n",
        "quote_contains_word = train['Quote_clean'].str.contains('|'.join(related_words_politics))\n",
        "\n",
        "print(train.loc[quote_contains_word, ['Quote_clean', 'label_ChatGPT_clean', 'label_ChatGPT_reliable']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QbSBTDmhclT2",
      "metadata": {
        "id": "QbSBTDmhclT2"
      },
      "source": [
        "The related word search does not reveal many quotes that could be related to politics. This key word search fails to match the 12 other quotes that are already labelled 'poitics', again highlighting how this key word search method may not very effective for nuanced topics.\n",
        "\n",
        "The word search reveals a quote containing the word 'party' which does have associations with politics but in the case of this quote is referring to a different meaning. An issue with this naive word-search method is that it fails to account for different meanings of words, as illustrated by this example.\n",
        "\n",
        "It is difficult to find quotes related to politics due to the vague nature of the topic. However, the key word search suggests that there may not be any more 'obvious' quotes in the training data that should be re-labelled to 'politics'. Overall, this label in the training data is likely reasonably reliable after our manual adjustments."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BAp9lhHcdJvj",
      "metadata": {
        "id": "BAp9lhHcdJvj"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "**Category 5. Insensitive**\n",
        "\n",
        "Again we look at the quotes currently labelled as insensitive and make any necessary corrections. It is difficult to identify 'insensitive' quotes based on a word search as the theme is nuanced and context dependent. Nevertheless, we attempt a related word search to see what we find. First, we look at the quotes currently labelled as 'insensitive':"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 475,
      "id": "9rLpTCc4d5SU",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9rLpTCc4d5SU",
        "outputId": "2486323e-eaca-471f-9d0c-fe570ba87d44"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The following quotes are labelled as \"5. insensitive\":\n",
            "88                                                                                                                                                                               get off my plane\n",
            "100                                                                                                                                                                            very nice how much\n",
            "883    maybe it is because i am constantly fighting destroying that i would actually like to save something for once especially someone who is judged to be a lower class warrior like my kakarot\n",
            "257                  all right but apart from the sanitation medicine education wine public order irrigation roads the fresh water system and public health what have the romans ever done for us\n",
            "Name: Quote_clean, dtype: object\n"
          ]
        }
      ],
      "source": [
        "print('The following quotes are labelled as \"5. insensitive\":')\n",
        "print(train.loc[train['label_ChatGPT_reliable'] ==  '5. insensitive']['Quote_clean'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WC_s7NPCekkJ",
      "metadata": {
        "id": "WC_s7NPCekkJ"
      },
      "source": [
        "We accept that the above four quotes can be interpreted as insensitive. We do not want to introduce our own biases about what is/is not insensitive so leave these quote labels as they are.\n",
        "\n",
        "The data dictionary for `label_ChatGPT` gave an example of a quote that should be labelled insensitive. We check that this quote has indeed been labelled as such and find that it has been labelled as '6. none of the above. We therefore fix the label of this quote as this is clearly an error according to the data dictionary:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 476,
      "id": "nEh8HoA_eW0y",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nEh8HoA_eW0y",
        "outputId": "757cfa9b-1ff2-4f4f-c97a-64657ede8a0e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Label before Fix:\n",
            "Movie Quote               When you're looking for a cancer kid, he should be hopeless. He should have a wheelchair, he should have trouble talking, he should have a little pet goldfish in a Zip-lock bag, hopeless.\n",
            "label_ChatGPT_reliable                                                                                                                                                                           6. none of the above\n",
            "Name: 577, dtype: object\n",
            "\n",
            "Label after Fix:\n",
            "Movie Quote               When you're looking for a cancer kid, he should be hopeless. He should have a wheelchair, he should have trouble talking, he should have a little pet goldfish in a Zip-lock bag, hopeless.\n",
            "label_ChatGPT_reliable                                                                                                                                                                                 5. insensitive\n",
            "Name: 577, dtype: object\n"
          ]
        }
      ],
      "source": [
        "\n",
        "print('Label before Fix:')\n",
        "print(train.loc[577, ['Movie Quote', 'label_ChatGPT_reliable']])\n",
        "\n",
        "# now fix it:\n",
        "train.loc[577, 'label_ChatGPT_reliable'] = categories[4] # change to none of the above\n",
        "\n",
        "print()\n",
        "print('Label after Fix:')\n",
        "print(train.loc[577, ['Movie Quote', 'label_ChatGPT_reliable']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HIWrPbxlf0Xl",
      "metadata": {
        "id": "HIWrPbxlf0Xl"
      },
      "source": [
        "Next we search for words related to insensitive, although we are doubtful of the efficacy of this approach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 477,
      "id": "RaHW0RGef5MC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RaHW0RGef5MC",
        "outputId": "bb70e243-2bcc-4aa0-ca60-29af8f780398"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['remark', 'emotions', 'victims', 'remarks', 'victim']\n"
          ]
        }
      ],
      "source": [
        "# Next we query the database for word related to insensitive\n",
        "\n",
        "# Query the database using the API for words related to 'insensitive'\n",
        "api_lookup_insensitive = api.words(rel_trg = 'insensitive', max = 15, topics = ['movie'])\n",
        "related_words_insensitive = [item['word'] for item in api_lookup_insensitive] # convert to list\n",
        "\n",
        "# we now print the first 5 words that the query returned:\n",
        " # (I inspected the full list so I could be confident in the results)\n",
        "print(related_words_insensitive[0:5])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 478,
      "id": "x4bxWqRXgNF0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x4bxWqRXgNF0",
        "outputId": "6bfb4ed4-faf4-4e29-df97-f65e106fdc3c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                                                                                                                                                                                                                                                                                       Quote_clean  \\\n",
            "876                                                                                                                                                                                                           even if we do ignore the previous data broly s latent abilities are quite remarkable   \n",
            "386  people live their lives bound by what they accept as correct and true that is how they define reality but what does it mean to be correct or true merely vague concepts their reality may all be a mirage can we consider them to simply be living in their own world shaped by their beliefs   \n",
            "726                                                                                                                                                                   we are all traveling through time together everyday of our lives all we can do is do our best to relish this remarkable life   \n",
            "\n",
            "      label_ChatGPT_clean  \n",
            "876  6. none of the above  \n",
            "386  6. none of the above  \n",
            "726  6. none of the above  \n"
          ]
        }
      ],
      "source": [
        "# search for words containing at least one of the related words:\n",
        "quote_contains_word = train['Quote_clean'].str.contains('|'.join(related_words_insensitive))\n",
        "\n",
        "print(train.loc[quote_contains_word, ['Quote_clean', 'label_ChatGPT_clean']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qc7OgK5xgqe1",
      "metadata": {
        "id": "qc7OgK5xgqe1"
      },
      "source": [
        "The quotes identified do not appear to be insensitive so we reject the use of key word search for identifying insensitive quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kaQYUZuHVVnM",
      "metadata": {
        "id": "kaQYUZuHVVnM"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this step we:\n",
        "\n",
        "- **Reclassified profane quotes**: Quotes containing profanity were assigned the label \"4. rude\".\n",
        "- **Reviewed labels with domain knowledge**: We reviewed categories 1-5, leveraging our movie knowledge to correct inconsistencies and improve label accuracy.\n",
        "- **Searching for keywords had limited effectiveness**: While keyword searches aided in identifying mislabeled religious quotes (category 2), this approach proved less effective for other nuanced categories.\n",
        "\n",
        "Overall, these adjustments and the manual review significantly improved the quality of the 'label_ChatGPT' column. This enhanced label reliability will directly translate to a more robust response variable, ultimately strengthening our model's effectiveness and usefulness for the movie production studio. However, data quality may still be a major issue as we are unable to review every single quote. It would also be useful to have some clear criteria about what makes a quote problematic based on the stipulations of our actors.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "76U988TLtS2H",
      "metadata": {
        "id": "76U988TLtS2H"
      },
      "source": [
        "#### Q1d Step 3 - Improving the reliability of the ChatGPT label in the test and validation set\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In the last step, we attempted to improve the unreliability of `label_ChatGPT` in our training dataset. We now do the same for our validation and test set. These datasets are to remain unseen and so we cannot manually review the labels like in the training set.\n",
        "\n",
        "In this step, we only apply a search for profanity when updating the label. A discussion of our approach and the reasons behind it are provided.\n",
        "\n",
        "***Why:***\n",
        "\n",
        "The test set is meant to represent unseen data to mimic the time of deployment; we will use it evaluate the usefulness of our chosen model. If the `label_ChatGPT` column, and hence our constructed response, is unreliable then it will be a poor representation of deployment and which quotes are actually problematic. Therefore, if we do not improve the reliability, our model evaluation will not be a good assessment of the model in practice."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ob_Hoe8aZHVb",
      "metadata": {
        "id": "ob_Hoe8aZHVb"
      },
      "source": [
        "\n",
        "**Discussion:**\n",
        "\n",
        "During EDA, we identified inconsistencies in labeling quotes containing profanity (often not marked as \"4. rude\"). To ensure a consistent evaluation across datasets, we will search for swear words and adjust labels in all sets. This is necessary because:\n",
        "\n",
        "- **Unrepresentative Test Set:** An unaddressed test set with mislabeled swearing would poorly reflect real-world deployment.\n",
        "- **Flawed Model Evaluation:** The model's ability to identify swearing (a key actor stipulation) would not be accurately assessed. This could lead to an overestimation of the model's performance, leading to risky deployment for the movie company and potential costs due to legal breaches.\n",
        "\n",
        "When improving `label_ChatGPT` in the training data we manually reviewed quotes in categories 1-5. This is not possible for the validation and test sets as these datasets are to remain unseen.\n",
        "\n",
        "Furthermore, the previous step revealed that using key-words to relabel categories was ineffective and unreliable. We therefore will not apply decision rules based on key-word searches to relabel quotes in the validation and test set. We could re-label quotes based on common words for each label in the training set but this would likely introduce leakage into our validation and test sets so is also avoided.\n",
        "\n",
        "Therefore, profanity detection is the only step we take to improve the reliability of the ChatGPT label for the validation and test set.\n",
        "\n",
        "We acknowledge that there might still be a level of unreliability for  `label_ChatGPT` in the validation and test sets, due to the inability to apply manual review like with the training set. This is unavoidable and should be considered at the evaluation stage. In particular, the test set label may be unreliable for categories relating to nuanced taboo topics (e.g. sex, religion) as we have not applied direct fixes for these categories.\n",
        "\n",
        "---\n",
        "\n",
        "We now apply the re-labelling to the validation and test set (profanity detection)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 479,
      "id": "hZ6LJd5DjQ93",
      "metadata": {
        "id": "hZ6LJd5DjQ93"
      },
      "outputs": [],
      "source": [
        "# define a function to apply the rules-based fix to the chatGPT label:\n",
        "\n",
        "def rules_based_fix_to_ChatGPT_label(data):\n",
        "  # copy column:\n",
        "  data['label_ChatGPT_reliable'] = data['label_ChatGPT_clean']\n",
        "\n",
        "  # Identify quotes with profanity and label them '4. rude'\n",
        "  data['label_ChatGPT_reliable'] = data.apply(lambda row: '4. rude' if profanity.contains_profanity(row['Movie Quote']) else row['label_ChatGPT_reliable'], axis = 1)\n",
        "\n",
        "  # Check that all values in the column are one of the 6 categories\n",
        "  if set(data[\"label_ChatGPT_reliable\"]).issubset(categories):\n",
        "    print(f'All values in the label_ChatGPT_reliable column are one of the six categories')\n",
        "  else:\n",
        "    print(f'ERROR: some values in the label_ChatGPT_reliable column are not one of the six categories')\n",
        "\n",
        "\n",
        "  return data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 480,
      "id": "V18KRRgrk2iz",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V18KRRgrk2iz",
        "outputId": "b8717bd7-b2ec-484f-8196-fbd579cf08fa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All values in the label_ChatGPT_reliable column are one of the six categories\n"
          ]
        }
      ],
      "source": [
        "# validation test set:\n",
        "val = rules_based_fix_to_ChatGPT_label(val)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 481,
      "id": "ZRcftn54lSvh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZRcftn54lSvh",
        "outputId": "0e3300e4-1ffb-40f6-a443-2989a77e10ee"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All values in the label_ChatGPT_reliable column are one of the six categories\n"
          ]
        }
      ],
      "source": [
        "# validation test set:\n",
        "test = rules_based_fix_to_ChatGPT_label(test)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "w2XClxr_jHag",
      "metadata": {
        "id": "w2XClxr_jHag"
      },
      "source": [
        "We have applied the profanity detection to the validation and test sets. The output reveals that `label_ChatGPT_reliable` only contains values equal to one of the six categories in both the validation and test sets (sense check).\n",
        "\n",
        "---\n",
        "\n",
        "Next we use `label_ChatGPT_reliable` to construct out binary response representing whether a column is problematic or not."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91Mp8gaErDwg",
      "metadata": {
        "id": "91Mp8gaErDwg"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "- relabelled quotes in validation and test set based on profanity detection\n",
        "- did not apply decision-rules for relabelling based on key words as this was found to be unreliable and so require manual validation that is not possible."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZWOuBc0jser-",
      "metadata": {
        "id": "ZWOuBc0jser-"
      },
      "source": [
        "#### Q1d Step 4 - Justification and creation of binary response\n",
        "\n",
        "***Outline Justification of response variable constructed***\n",
        "\n",
        "Our business objective requires us to build a model that classifies quotes as 'problematic' or 'not problematic'. We therefore need to construct a binary response variable representing this. According to the information provided, actors may have stipulations relating to:\n",
        "- swearing\n",
        "- taboo topics like sex, religion and politics\n",
        "- offensive or insensitive quotes\n",
        "\n",
        "The `label_ChatGPT` attempts to flag quotes that may breach these stipulations. Therefore, we label a quote as 'problematic' if the cleaned `label_ChatGPT` takes a value in categories 1-5 (e.g. '1. sex', '2. religion', '3. politics', '4. rude' and '5. insensitive').\n",
        "\n",
        "Obviously, this does not cover all possible taboo topics but they line up with the potential stipulations provided in the brief. Note, we are not required to identify what about a quote makes it problematic hence why a binary variable suffices for the business objective. Any quotes where `label_ChatGPT` takes the value '6. none of the above' will be marked as 'not problematic'.\n",
        "\n",
        "As mentioned previously, the sentiment score columns (e.g. `Sentiment_VADER`) were found to have an ambiguous relationship with `label_ChatGPT` in our EDA. For example, a low sentiment score often corresponded to non-problematic quotes that would likely not lead to an actor contract breach. Other times, low-sentiment quotes were labelled as '6. none of the above' but likely would constitute a contract breach. However, we believe most of the latter issue was corrected with our prior cleaning of `label_ChatGPT`.\n",
        "\n",
        "We therefore do not believe it is justified to use the sentiment scores in directly constructing the binary response as it will lead to an inaccurate response. A neural network may pick up on any non-linear relationship between the sentiment scores and our response, so we will test them as features.\n",
        "\n",
        "Overall, the cleaned version of `label_ChatGPT` will be used to flag quotes that will likely lead to a breach of any actor stipulation.\n",
        "\n",
        "---\n",
        "\n",
        "We will now construct the binary response variable. The new column will be labelled `problematic` and will take the value `1` if the quote is problematic and `0` if the quote is not problematic. We make the response numeric as this will be easier for when we create our neural network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 482,
      "id": "P41BsUMrvjYM",
      "metadata": {
        "id": "P41BsUMrvjYM"
      },
      "outputs": [],
      "source": [
        "# Create function for constructing the binary response:\n",
        "\n",
        "def construct_problematic_binary_response(data):\n",
        "\n",
        "  # map categories 1-5 to '1' and category '6. none of the above' to '0'\n",
        "  data['problematic'] = data.apply(lambda row: 1 if row['label_ChatGPT_reliable'] in categories[0:5] else 0, axis = 1)\n",
        "\n",
        "  return data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 483,
      "id": "qh6A0MHsxUUJ",
      "metadata": {
        "id": "qh6A0MHsxUUJ"
      },
      "outputs": [],
      "source": [
        "# apply function to our training set:\n",
        "train = construct_problematic_binary_response(train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 484,
      "id": "VNQtfOCvCvAA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 380
        },
        "id": "VNQtfOCvCvAA",
        "outputId": "d20dc26b-b3fd-4e26-db5b-a246389db45a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Have a look at first ten rows and their response flag:\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"train[['text_bert_withSW', 'label_ChatGPT_reliable', 'problematic']]\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"text_bert_withSW\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"let us learn to show our friendship for a man when he is alive and not after he is dead\",\n          \"you only live twice once when you are born and once when you look death in the face\",\n          \"there is no place like home\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT_reliable\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"4. rude\",\n          \"6. none of the above\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"problematic\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-9145d8a1-fa9a-4569-91e3-b1e1347ceeaa\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text_bert_withSW</th>\n",
              "      <th>label_ChatGPT_reliable</th>\n",
              "      <th>problematic</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>862</th>\n",
              "      <td>all i have are negative thoughts</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>224</th>\n",
              "      <td>you only live twice once when you are born and once when you look death in the face</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>530</th>\n",
              "      <td>i love lamp</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>710</th>\n",
              "      <td>let it go</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>295</th>\n",
              "      <td>and to think i have been telling my friends it is so cool living with an artist you never once asked to paint me nude</td>\n",
              "      <td>4. rude</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>there is no place like home</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>449</th>\n",
              "      <td>what you need to do is get me my chips with dip cuz you are delinquent motherfucker</td>\n",
              "      <td>4. rude</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>697</th>\n",
              "      <td>the law says that you can not touch but i think i see a lot of lawbreakers up in this house tonight</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>703</th>\n",
              "      <td>let us learn to show our friendship for a man when he is alive and not after he is dead</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>749</th>\n",
              "      <td>we used to look up at the sky and wonder at our place in the stars now we just look down and worry about our place in the dirt</td>\n",
              "      <td>6. none of the above</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9145d8a1-fa9a-4569-91e3-b1e1347ceeaa')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9145d8a1-fa9a-4569-91e3-b1e1347ceeaa button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9145d8a1-fa9a-4569-91e3-b1e1347ceeaa');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6f2def7c-85aa-4101-b5ab-c53adf7a0535\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6f2def7c-85aa-4101-b5ab-c53adf7a0535')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6f2def7c-85aa-4101-b5ab-c53adf7a0535 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                   text_bert_withSW  \\\n",
              "862                                                                                                all i have are negative thoughts   \n",
              "224                                             you only live twice once when you are born and once when you look death in the face   \n",
              "530                                                                                                                     i love lamp   \n",
              "710                                                                                                                       let it go   \n",
              "295           and to think i have been telling my friends it is so cool living with an artist you never once asked to paint me nude   \n",
              "15                                                                                                      there is no place like home   \n",
              "449                                             what you need to do is get me my chips with dip cuz you are delinquent motherfucker   \n",
              "697                             the law says that you can not touch but i think i see a lot of lawbreakers up in this house tonight   \n",
              "703                                         let us learn to show our friendship for a man when he is alive and not after he is dead   \n",
              "749  we used to look up at the sky and wonder at our place in the stars now we just look down and worry about our place in the dirt   \n",
              "\n",
              "    label_ChatGPT_reliable  problematic  \n",
              "862   6. none of the above            0  \n",
              "224   6. none of the above            0  \n",
              "530   6. none of the above            0  \n",
              "710   6. none of the above            0  \n",
              "295                4. rude            1  \n",
              "15    6. none of the above            0  \n",
              "449                4. rude            1  \n",
              "697   6. none of the above            0  \n",
              "703   6. none of the above            0  \n",
              "749   6. none of the above            0  "
            ]
          },
          "execution_count": 484,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "print('Have a look at first ten rows and their response flag:')\n",
        "train[['text_bert_withSW', 'label_ChatGPT_reliable', 'problematic']].head(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6DIq-Q6lvm9S",
      "metadata": {
        "id": "6DIq-Q6lvm9S"
      },
      "source": [
        "**Check on response: no missing values and all values are 0 or 1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 485,
      "id": "LMTmyWEZ0JTl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LMTmyWEZ0JTl",
        "outputId": "6e5a4022-ae9f-4d40-bf40-3a4cdec75860"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 2 unique values for \"problematic\" in the training data (from 633 rows)\n",
            "Total count and percentage of the the values in 'problematic':  \n",
            "             Count  Percentage\n",
            "problematic                   \n",
            "0              552   87.203791\n",
            "1               81   12.796209\n",
            "\n",
            "There are 0 NULL values in the \"problematic\" column\n"
          ]
        }
      ],
      "source": [
        "print(f'There are {len(set(train[\"problematic\"]))} unique values for \"problematic\" in the training data (from', train.shape[0], 'rows)')\n",
        "\n",
        "dat_count=train['problematic'].value_counts()\n",
        "dat_perc=train['problematic'].value_counts()*100/train['problematic'].count()\n",
        "\n",
        "dat_count_perc =  pd.DataFrame({'Count': dat_count,\n",
        "                                'Percentage': dat_perc\n",
        "})\n",
        "\n",
        "#Printing the count and percentage of the top top_n entries\n",
        "\n",
        "print(f\"Total count and percentage of the the values in 'problematic':  \")\n",
        "print(dat_count_perc)\n",
        "\n",
        "print()\n",
        "print(f'There are {train[\"problematic\"].isna().sum()} NULL values in the \"problematic\" column')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QgcHthaA6JM8",
      "metadata": {
        "id": "QgcHthaA6JM8"
      },
      "source": [
        "The above output highlights that 'problematic' only takes two values (0 and 1) which is expected. There are no NULL or NA values in the column indicating that the transformation worked for all rows. About 13% of the data takes the values 1 i.e. 'problematic' indicating that the variable is quite unbalanced.\n",
        "\n",
        "Overall, the output is as expected and passes this reasonableness check."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bdKxY1N9vyOv",
      "metadata": {
        "id": "bdKxY1N9vyOv"
      },
      "source": [
        "**Check on response: breakdown the binary response by ChatGPT label**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 486,
      "id": "XXR51hhS1J4A",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XXR51hhS1J4A",
        "outputId": "6697477e-fd97-48cf-9ff1-8c1a746cc4bd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Label: 6. none of the above\n",
            "\tProblematic: 0, Count: 552\n",
            "Label: 4. rude\n",
            "\tProblematic: 1, Count: 55\n",
            "Label: 3. politics\n",
            "\tProblematic: 1, Count: 12\n",
            "Label: 5. insensitive\n",
            "\tProblematic: 1, Count: 5\n",
            "Label: 1. sex\n",
            "\tProblematic: 1, Count: 4\n",
            "Label: 2. religion\n",
            "\tProblematic: 1, Count: 5\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# get unique values in label_CHatGPT_reliable -> should be same as 'categories' but we check just in case\n",
        "unique_labels = train['label_ChatGPT_reliable'].unique()\n",
        "\n",
        "# Create an empty dictionary to store counts\n",
        "label_counts = {}\n",
        "for label in unique_labels:\n",
        " # Count occurrences of each label_ChatGPT value within the 'problematic' column\n",
        " label_counts[label] = train[train['label_ChatGPT_reliable'] == label]['problematic'].value_counts().to_dict()\n",
        "\n",
        "# Print the results\n",
        "for label, counts in label_counts.items():\n",
        " print(f\"Label: {label}\")\n",
        " for problematic, count in counts.items():\n",
        "   print(f\"\\tProblematic: {problematic}, Count: {count}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "u2tU_vQi6s7P",
      "metadata": {
        "id": "u2tU_vQi6s7P"
      },
      "source": [
        "In this step, we get the count of each unique value of the response for each ChatGPT label. Based on how we constructed the response we would expect that '6. none of the above' only maps to the value 0 and categories 1-5 only map to the value 1 for the response. This appears to be the case in the output above.\n",
        "\n",
        "Therefore, the response maps to `label_ChatGPT` how we would expect and passes this reasonableness check.\n",
        "\n",
        "---\n",
        "\n",
        "Now that the construction of the response passed our reasonableness checks on the training data, we now construct the response for our validation and test sets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 487,
      "id": "e8XiMENi8eht",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e8XiMENi8eht",
        "outputId": "55939c85-3f5d-458c-eead-d9160d289919"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 2 unique values for \"problematic\" in the validation data (from 80 rows)\n"
          ]
        }
      ],
      "source": [
        "## VALIDATION\n",
        "\n",
        "# apply function to our validation set:\n",
        "val = construct_problematic_binary_response(val)\n",
        "print(f'There are {len(set(val[\"problematic\"]))} unique values for \"problematic\" in the validation data (from', val.shape[0], 'rows)')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 488,
      "id": "_VU2Y4UZ9fPC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_VU2Y4UZ9fPC",
        "outputId": "ffe91aa5-864b-452b-8322-29b13486db14"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 2 unique values for \"problematic\" in the test data (from 80 rows)\n"
          ]
        }
      ],
      "source": [
        "## TEST\n",
        "# apply function to our test set:\n",
        "test = construct_problematic_binary_response(test)\n",
        "print(f'There are {len(set(test[\"problematic\"]))} unique values for \"problematic\" in the test data (from', test.shape[0], 'rows)')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WivA5GQa93Bm",
      "metadata": {
        "id": "WivA5GQa93Bm"
      },
      "source": [
        "The response has now been constructed in the test and validation data sets. Based on the output, the response construction appears to have worked."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EAbluR7-VVnM",
      "metadata": {
        "id": "EAbluR7-VVnM"
      },
      "source": [
        "**Summary**  \n",
        "- Justified the method used to construct the binary response variable\n",
        "- Constructed a binary reponse 'problematic', representing whether a quote is problematic or not\n",
        "- Did reasonableness checks on the response"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oab23DF7-J0O",
      "metadata": {
        "id": "oab23DF7-J0O"
      },
      "source": [
        "**Overall summary**  \n",
        "- Cleaned `label_ChatGPT` so it only took one of six values\n",
        "- Improved the reliability of `label_ChatGPT` in the training set by:\n",
        "  - detecting profanity in quotes\n",
        "  - manual review of training labels\n",
        "- Improved the reliability of `label_ChatGPT` in the validation and test sets by detecting profanity in the quotes\n",
        "- Constructed a binary response variable, `problematic`, that takes the value 1 if a quote is problematic and 0 otherwise. A quote was considered problematic if `label_ChatGPT` belonged to categories 1-5.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tXIkXc2hVZg1",
      "metadata": {
        "id": "tXIkXc2hVZg1"
      },
      "source": [
        "## Q2 - Explore characteristics of famous movie quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EKWXjyBPWBew",
      "metadata": {
        "id": "EKWXjyBPWBew"
      },
      "source": [
        "### Q2a - Clustering\n",
        "#### Purpose:\n",
        "\n",
        "To explore the characteristics of famous movie quotes which the company can use to guide script writing. The company desires this so that they can boost future box office sales, earn Academy Awards and to improve the memorability of their movies. We employ clustering algorithms to assist with this objective.\n",
        "\n",
        "\n",
        "#### References:\n",
        "\n",
        "*O’Neill, P., Guo, E., Temple, J., & Basu, T. (2012). The secret science of memorable quotes. Retrieved from https://www.technologyreview.com/2012/04/03/186934/the-secret-science-of-memorable-quotes/*\n",
        "\n",
        "*Marija, M. (2024). The impact of memes and humor in gen Z and gen y marketing. Retrieved from https://phuconcepts.com/the-impact-of-memes-and-humor-in-gen-z-and-gen-y-marketing/*\n",
        "\n",
        "*Mishra, A., Getaneh, M., & Martin, K. (2023). How can you use humor to build brand loyalty with Gen z Consumers? Retrieved from https://www.linkedin.com/advice/0/how-can-you-use-humor-build-brand-loyalty-gen-z-consumers-eefbc*\n",
        "\n",
        "*Haus, T. (2020). The power of monologues in the theatre. Retrieved from https://www.theatrehaus.com/2020/04/the-power-of-monologues-in-the-theatre/*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jSTLTiopzrh6",
      "metadata": {
        "id": "jSTLTiopzrh6"
      },
      "source": [
        "#### Q2a Step 1 - Justification of clustering algorithm\n",
        "**Outline**\n",
        "\n",
        "A range of clustering algorithms exist however, we need to choose one that is suited for our business objective. In this section, we discuss different clustering techniques and their relevance to our objective of understanding the key characteristics of famous movie quotes. Ultimately, we decide that **Agglomerative Hierarchical clustering** is the most appropriate technique due to its added flexibility and ability to provide more nuanced insights.\n",
        "\n",
        "We also decide to use **Euclidean distance** as a distance measure. A justification is provided below.\n",
        "\n",
        "We will decide on a linkage method after applying a range of them and better understanding the consequent results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VH1tiAonDliL",
      "metadata": {
        "id": "VH1tiAonDliL"
      },
      "source": [
        "**Justification of clustering algorithm**\n",
        "\n",
        "We will be conducting clustering to help us identify different types of famous movie quotes and to present the findings to the board. We must choose a clustering algorithm that is suitable for our objective and data.\n",
        "\n",
        "We will consider two clustering algorithms: k-means and agglomerative hierarchical clustering. Next we discuss the strengths and weaknesses of each algorithm in context of our goals and data:\n",
        "\n",
        "- **Communicate to executives:** our results need to be communicable to non-technical movie executives. Both methods are appropriate for this goal as long as a reasonable number of clusters (not too many, not too little) are used in the executive report.\n",
        "- **Usefulness of insights:** a key advantage of hierarchical clustering is that it creates a nested hierarchy of clusters. This may allow us to provide more useful and nuanced insights to the executives such as by identifying sub-categories. K-means clusters do not have this nested hierarchy and so some key insights may be missed. Since executives are often interested in the bigger picture, we need to tailor the level of detail. By using hierarchical clustering, we can easily zoom in on specific sub-categories if needed, while still having the broader cluster view available. Ultimately, we believe this added flexibility will be more useful for our goal.\n",
        "- **Shape of clusters:** K-means tends to work better when the clusters are hyper-spherical (higher-order sphere) whereas hierarchical clustering is more flexible. K-means also tends to create similarly sized clusters which may not be appropriate for out data. We do not have a strong prior assumption about our cluster shapes. Therefore, a more flexible method like hierarchcial clustering may be more appropriate.\n",
        "- **Small dataset**: Agglomerative hierarchical clustering can become messy to understand and computationally intensive with a large dataset. Conversely, K-means is relatively fast and easier to understand, even with large data. In our case, we do not have a particularly large dataset and so hierarchical clustering's speed will likely be acceptable.\n",
        "- **Do not intend to use clusters for inference on unseen observations**: K-means is preferable for allocating labels to unseen data since a point is just added to the cluster it is closest to. However, for agglomerative this is not possible and the algorithm would have to be run again. For our objective, we are only interested in understanding patterns in the training data so this feature of the algorithm is unimportant.\n",
        "\n",
        "Overall, we decide to use **agglomerative hierarchical clustering** to do our clustering analysis. This is because it is more flexible and suitable for our data. We will be able to provide key quote categories and potential sub-categories that can be explored further in the future. K-means is likely a valid choice but we believe the nested structure from hierarchical clustering will allow us to provide more useful insights to the executives.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Bsp-jBdYD_MJ",
      "metadata": {
        "id": "Bsp-jBdYD_MJ"
      },
      "source": [
        "**Justification of distance measure used**\n",
        "\n",
        "We will use Euclidean distance for our distance measure in the clustering.\n",
        "Firstly, all of our features used for clustering (see next step) are continuous variables so Euclidean distance is appropriate. If we had mixed data types then a measure like Gower's may be preferable.\n",
        "\n",
        "Secondly, the scale of our variables is important and we want similarity in the magnitude to be reflectd in the clusters. For example, we want to group quotes that have embeddings with similar magnitude not just if the embedding values tend to move in the same direction (as with a correlation-based measure).\n",
        "\n",
        "Note that Euclidean distance requires us to scale our variables."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vPKyRGij5A5a",
      "metadata": {
        "id": "vPKyRGij5A5a"
      },
      "source": [
        "**Summary**  \n",
        "- Decided to use Agglomerative Hierarchical clustering\n",
        "- Decided to use Euclidean distance as the distance measure in our clustering"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "L9VcdJJs8a5N",
      "metadata": {
        "id": "L9VcdJJs8a5N"
      },
      "source": [
        "#### Q2a Step 2 - Applying the clustering algorithm\n",
        "\n",
        "**Outline**\n",
        "\n",
        "In this step, we apply agglomerative hierarchical clustering to create nested clusters of our quotes.\n",
        "\n",
        "We first consider which features should be used in our clustering. Next we attempt to choose an appropriate linkage method. Using the selected method we then apply agglomerative clustering and choose a rough cutoff point for the clusters that will be used to communicate the findings to the executives. Finally, we internally validate the clustering results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cg10BaFKsFeB",
      "metadata": {
        "id": "cg10BaFKsFeB"
      },
      "source": [
        "**Selection of features for clustering**\n",
        "\n",
        "From our clustering analysis we want to better understand different types of famous movie **quotes**. Therefore, the features we use in the clustering should represent *characteristics of the quotes themselves* and **not** the type of movie they are from or who said them. We believe the following features represent characteristics of the quotes and will provide useful insight to the board:\n",
        "\n",
        "- First 7 principal components of our BERT embedding of the quotes\n",
        "- The sentiment scores: `Sentiment_VADER` and `Sentiment_TextBlob`\n",
        "- The `word_count` column\n",
        "\n",
        "We do not use the genre columns in our clustering as this would lead us to grouping quotes together based on the type of movie they are from rather than due to intrinsic similarities between quotes. Furthermore, other features like `Movie Name` and `Actor` are high cardinality and are unlikely to provide useful insights to the executives."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ddq83psU3pgI",
      "metadata": {
        "id": "Ddq83psU3pgI"
      },
      "source": [
        "\n",
        "---\n",
        "\n",
        "##### Data and function setup\n",
        "\n",
        "We now apply the agglomerative hierachical clustering. We will apply it using three linkage methods and compare the results to determine the most appropriate (single linkage, complete linkage and average linkage). Scaling is required when using Euclidean distance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 489,
      "id": "OXkXP2fJ7SP5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OXkXP2fJ7SP5",
        "outputId": "49123fc7-ff76-4551-e0eb-c49b247b92ce"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The training data had 633 rows. The clustering data has 633 rows.\n",
            "We expect to use 10 features for clustering. The clustering data has 10 columns.\n"
          ]
        }
      ],
      "source": [
        "# Prepare data\n",
        "\n",
        "#First let us get our data and features we intend to use for clustering:\n",
        "clustering_data = pd.concat([pd.DataFrame(pca_values_withSW), train[['Sentiment_VADER', 'Sentiment_TextBlob', 'word_count']].reset_index(drop=True)], axis = 1, ignore_index=True)\n",
        "\n",
        "# check that concatentation and column selection is correct:\n",
        "print(f'The training data had {train.shape[0]} rows. The clustering data has {clustering_data.shape[0]} rows.')\n",
        "print(f'We expect to use {7+3} features for clustering. The clustering data has {clustering_data.shape[1]} columns.')\n",
        "\n",
        "clustering_data.columns = ['PC1','PC2','PC3','PC4','PC5','PC6', 'PC7','Sentiment_VADER', 'Sentiment_TextBlob', 'word_count']\n",
        "\n",
        "# Now apply scaling:\n",
        "cluster_scaler = StandardScaler()\n",
        "clustering_scaled_data = cluster_scaler.fit_transform(clustering_data)\n",
        "clustering_data = pd.DataFrame(clustering_scaled_data, columns = clustering_data.columns) # add column names\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rS02l7W8OccU",
      "metadata": {
        "id": "rS02l7W8OccU"
      },
      "source": [
        "Now we create some functions for fitting the hierarchical clustering algorithm and plotting dendrograms."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 490,
      "id": "7ZZzW4iG8MfD",
      "metadata": {
        "id": "7ZZzW4iG8MfD"
      },
      "outputs": [],
      "source": [
        "# Create function for doing clustering:\n",
        "def perform_hierarchical_clustering(data, linkage_method='single', n_clusters = 20):\n",
        "\n",
        "  # initialise model\n",
        "  cluster_model = AgglomerativeClustering(n_clusters=n_clusters, linkage=linkage_method, metric = 'euclidean', compute_distances=True)\n",
        "  # fit model to the data\n",
        "  clustering_output = cluster_model.fit_predict(data)\n",
        "\n",
        "  # output model fit and the output (labels)\n",
        "  return cluster_model, clustering_output\n",
        "\n",
        "# Function for creating dendrograms given we use scikit learn for clustering\n",
        "# REFEERENCE: scikit learn\n",
        "# https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html#sphx-glr-auto-examples-cluster-plot-agglomerative-dendrogram-py\n",
        "def plot_dendrogram(model, **kwargs):\n",
        "    # Create linkage matrix and then plot the dendrogram\n",
        "\n",
        "    # create the counts of samples under each node\n",
        "    counts = np.zeros(model.children_.shape[0])\n",
        "    n_samples = len(model.labels_)\n",
        "    for i, merge in enumerate(model.children_):\n",
        "        current_count = 0\n",
        "        for child_idx in merge:\n",
        "            if child_idx < n_samples:\n",
        "                current_count += 1  # leaf node\n",
        "            else:\n",
        "                current_count += counts[child_idx - n_samples]\n",
        "        counts[i] = current_count\n",
        "\n",
        "    linkage_matrix = np.column_stack(\n",
        "        [model.children_, model.distances_, counts]\n",
        "    ).astype(float)\n",
        "\n",
        "    # Plot the corresponding dendrogram\n",
        "    dendrogram(linkage_matrix, **kwargs)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "D0gMv7fpOwhE",
      "metadata": {
        "id": "D0gMv7fpOwhE"
      },
      "source": [
        "##### **Selecting linkage method**\n",
        "\n",
        "Next we attempt to do some 'tuning' to decide which linkage method to use in our hierarchical clustering. The linkage method is used to determine the distance between clusters when agglomerating the closest clusters. Different linkage methods will result in different shapes in the clustering outcomes. We will consider the following linkage methods:\n",
        "- single: measures distance between clusters as the smallest pairwise distance between observations in the clusters. This can create elongated, chain-like clusters which may be useful\n",
        "- complete: uses the largest pairwise distance to determine the distance between clusters. Tends to create hyper-spherical clusters.\n",
        "- average: uses the average pairwise distance between observations to determine the distance between clusters. Tends to create balanced, hyper-spherical clusters.\n",
        "\n",
        "We do not yet understand what 'shape' is appropriate for our clusters. We therefore do some data visualisation of the clusters created under each method to help us select an appropriate linkage method. We will fit each linkage method then plot *dendrograms* and a *2D plot*. The dendrogram will give us an idea of the size of clusters and how similar they are at each merging step. The 2D plot will help give us a rough idea of the balance of clusters at a high level."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "D3ugQURGGodJ",
      "metadata": {
        "id": "D3ugQURGGodJ"
      },
      "source": [
        "**Single linkage clustering:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 491,
      "id": "gZalJFmLIGSe",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "gZalJFmLIGSe",
        "outputId": "493dcfc0-19a6-441c-b089-20e991b6a08d"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 132020 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# SINGLE LINKAGE CLUSTERING\n",
        "# fit clustering algorithm:\n",
        "clustering_single_linkage, labels_single_linkage = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'single', n_clusters = 5)\n",
        "\n",
        "# for all our 2D plots we apply a cutoff of 5 clusters to get a rough idea of the clusters created and their balance\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "plt.title(\"Hierarchical Clustering - Single linkage\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_single_linkage)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 492,
      "id": "IMXXo6QoLwnl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        },
        "id": "IMXXo6QoLwnl",
        "outputId": "fac7bf3f-c3ac-44cb-9ac4-0926816364b2"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 26404 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.title(\"Dendrogram - single linkage\")\n",
        "# plot the top three levels of the dendrogram\n",
        "plot_dendrogram(clustering_single_linkage, truncate_mode=\"level\", p=4)\n",
        "plt.xlabel(\"Number of points in node (or index of point if no parenthesis)\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3fGgxVDYpV1B",
      "metadata": {
        "id": "3fGgxVDYpV1B"
      },
      "source": [
        "Above we applied agglomerative hierarchical clustering using *single linkage clustering*. A dendrogram and 2D plot of the top 5 clusters is shown. **Single linkage is clearly not appropriate for our dataset**. The plots reveal that the top 5 clusters comprises of a single large cluster and 4 small clusters containing single points. This is a product of how the method chains together close-together points. The clustering outcomes will not provide useful insights to communicate to the board and so we do not use this linkage method.\n",
        "\n",
        "---\n",
        "**Complete linkage clustering:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 493,
      "id": "_gAGqaUt5ixF",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "_gAGqaUt5ixF",
        "outputId": "667cbac8-4624-4200-a68e-15c2f1ac1c53"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 141768 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# COMPLETE LINKAGE CLUSTERING\n",
        "clustering_complete_linkage, labels_complete_linkage = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'complete', n_clusters = 5)\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "plt.title(\"Hierarchical Clustering - Single linkage\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_complete_linkage)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 494,
      "id": "EZGNf2keB5n0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EZGNf2keB5n0",
        "outputId": "4204425d-07d2-4143-eec0-c020295ee39a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "633"
            ]
          },
          "execution_count": 494,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "train.shape[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 495,
      "id": "zbAyYuMu5ij-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 486
        },
        "id": "zbAyYuMu5ij-",
        "outputId": "f136eb60-102a-4c4d-ede2-de5a08b03f2c"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 34368 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.title(\"Dendrogram - complete linkage\")\n",
        "# plot the top three levels of the dendrogram\n",
        "plot_dendrogram(clustering_complete_linkage, truncate_mode=\"level\", p=4)\n",
        "plt.xlabel(\"Number of points in node (or index of point if no parenthesis)\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FgCjK0m8qjJ8",
      "metadata": {
        "id": "FgCjK0m8qjJ8"
      },
      "source": [
        "The plots reveal a clear hierarchical clustering structure in the dendrogram and relatively balanced clusters in the dendrogram. This is a huge improvement on the single linkage clustering. The clustering from this method will likely provide more useful insights for the executives. Some of the top clusters are smaller than others, in terms of number of quotes, but there are enough points in them for useful insights to still be made.\n",
        "\n",
        "---\n",
        "\n",
        "**Average linkage clustering:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 496,
      "id": "6xzkkB-V59B5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "6xzkkB-V59B5",
        "outputId": "6c3b40fc-b4d0-4f61-fff2-0aa48b8fc1dc"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 134648 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# AVERAGE LINKAGE CLUSTERING\n",
        "clustering_avg_linkage, labels_avg_linkage = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'average', n_clusters = 5)\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "plt.title(\"Hierarchical Clustering - average linkage\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_avg_linkage)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 497,
      "id": "JrlrzP4X59Yh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 476
        },
        "id": "JrlrzP4X59Yh",
        "outputId": "c928e0aa-01df-430a-de3f-5337ec87c860"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 29496 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plt.title(\"Dendrogram - average linkage\")\n",
        "# plot the top three levels of the dendrogram\n",
        "plot_dendrogram(clustering_avg_linkage, truncate_mode=\"level\", p=4)\n",
        "plt.xlabel(\"Number of points in node (or index of point if no parenthesis)\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CNKh9_Lwrf3u",
      "metadata": {
        "id": "CNKh9_Lwrf3u"
      },
      "source": [
        "The results for average linkage indicate that the clusters it generates are also not very balanced (similar to single linkage). In the 2D plot we see a few clusters contain most of the points. Also, looking at the dendrogram, there appears to be several individual data points that are merged in the final few steps; if several of the top clusters just contain a couple quotes then this will not provide useful insight to the executives."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SmbyeKqZDtC-",
      "metadata": {
        "id": "SmbyeKqZDtC-"
      },
      "source": [
        "Therefore overall we decide to proceed with **complete linkage clustering**:\n",
        "- at the top level, this method created balanced clusters with sizeable numbers of quotes which allows more reliable/useful insights to be made about the characteristics of quotes in each cluster. This was not the case for the other methods.\n",
        "- promotes tight clusters which is likely to result in more distinct characteristics within each cluster"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ye7fzYOa3A0v",
      "metadata": {
        "id": "Ye7fzYOa3A0v"
      },
      "source": [
        "From an initial look at the dendrogram for complete linkage and the height at which merges occur, a good number of clusters to analyse may lie somewhere between 3 to 8. When choosing the number of cluster we must also take into account our context and who we will be communicating results to - executives of a movie production company. We therefore want to choose enough clusters to be able to communicate useful insights without going too granular so as to provide needless detail to the executives. Further sub-clusters can always be analysed later after getting more direction from the board.\n",
        "\n",
        "We will next use several validation measures to better justify the number of clusters and evaluate the appropriateness of the clustering overall."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CaURxppl1Oxo",
      "metadata": {
        "id": "CaURxppl1Oxo"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "In this section we:\n",
        "- Justified which features to use in our clustering. We decided on only using features that related to *characteristics of the quotes* themselves\n",
        "- Applied agglomerative hierarchical clustering using three linkage methods: single, complete and average\n",
        "- Decided that *complete* linkage was the most appropriate as it produced the most balanced clusters which will be more useful for our analysis\n",
        "- Based on rough, initial impressions of the clustering outcomes, using a number of clusters between 3-8 will likely be suitable. This will be tested in the next step and must be balanced with the expectations of our stakeholders.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6_0LrJGbBW0I",
      "metadata": {
        "id": "6_0LrJGbBW0I"
      },
      "source": [
        "#### Q2a Step 3 - Evaluation of clustering outputs\n",
        "\n",
        "**Outline**\n",
        "\n",
        "In this step we conduct internal validation to evaluate the clustering outcomes. This will be done by calculating several validation measures for different numbers of clusters. The measures used are:\n",
        "- Within-cluster sum of squares, WCSS: measures similarity of points within clusters; will typically decrease with the number of clusters\n",
        "- Davies-Bouldin (DB) index: measures both inter-cluster and intra-cluster variation\n",
        "- Silhouette score: calculate a coefficient representing how well the points are assigned to their clusters; measures consistency within clusters\n",
        "- Calinski-Harabasz (CH) Index: metric that evaluates both inter-cluster and intra-cluster variation\n",
        "\n",
        "We use a range of measure to get a more holistic assessment of the clustering outcomes. It will also help us to choose an appropriate number of clusters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 498,
      "id": "b9jGh8Xc_RL3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b9jGh8Xc_RL3",
        "outputId": "92fea6b3-97ae-4619-87c8-30e398fdaa54"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to score model: 10 seconds\n"
          ]
        }
      ],
      "source": [
        "# Calculate the WCSS, Silhouette score, Davies-Bouldin index and Calinski-Harabasz index for each cluster K\n",
        "time_score_start = time.time()\n",
        "\n",
        "# create arrays to hold our metrics for each clustering cutoff\n",
        "WCSS = []\n",
        "SS = []\n",
        "DB = []\n",
        "CH = []\n",
        "\n",
        "K = range(2,11)\n",
        "for j in K:\n",
        "\n",
        "    clustering_model, labels = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'complete', n_clusters = j)\n",
        "\n",
        "    # WCSS is a bit more involved for agg hierarchical:\n",
        "    intra_disp = 0.0 # initialise\n",
        "    for k in range(len(labels)):\n",
        "        cluster_k = clustering_data[labels == k] # get points in a cluster\n",
        "        mean_k = np.mean(cluster_k, axis=0) # get cluster centroid\n",
        "        intra_disp += np.sum(np.sum((cluster_k - mean_k) ** 2)) # calculate WCSS for this cluster\n",
        "\n",
        "    WCSS.append(intra_disp)\n",
        "    SS.append(silhouette_score(clustering_data, labels)) #calculates silhouette score metric\n",
        "    DB.append(davies_bouldin_score(clustering_data, labels)) #calculates DB Index score\n",
        "    CH.append(calinski_harabasz_score(clustering_data, labels)) #calculates calinski harabasz\n",
        "\n",
        "time_score_end = time.time()\n",
        "print('Time taken to score model: {:.0f} seconds'.format(time_score_end - time_score_start))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 499,
      "id": "boja0i8TFiP1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "boja0i8TFiP1",
        "outputId": "dd2c7ecf-5448-4811-bc3e-bebf4ddd3d6d"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"summary_table\",\n  \"rows\": 9,\n  \"fields\": [\n    {\n      \"column\": \"n_clusters\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2,\n        \"min\": 2,\n        \"max\": 10,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          9,\n          3,\n          7\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"WCSS\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 507.17238859842416,\n        \"min\": 4243.495470107803,\n        \"max\": 5741.146083444336,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          4330.5592433106185,\n          5391.811800576485,\n          4643.341409261468\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Silhouette Score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.04569450688048309,\n        \"min\": 0.04273634907701704,\n        \"max\": 0.16370777627509067,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          0.04793471063916719,\n          0.1425065843483748,\n          0.04594224981537615\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Davies-Bouldin Index\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.27742080715245915,\n        \"min\": 1.940989649669482,\n        \"max\": 2.871816910318169,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          2.431147822823006,\n          1.940989649669482,\n          2.4082971925883068\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Calinski-Harabasz Index\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 10.204610112565762,\n        \"min\": 34.03620464621393,\n        \"max\": 64.71997331997981,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          36.01298822148117,\n          54.81075633737989,\n          37.89829294396914\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "summary_table"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-a50e40cb-e73b-42ff-971b-58610c4e3c33\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>n_clusters</th>\n",
              "      <th>WCSS</th>\n",
              "      <th>Silhouette Score</th>\n",
              "      <th>Davies-Bouldin Index</th>\n",
              "      <th>Calinski-Harabasz Index</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>5741.146083</td>\n",
              "      <td>0.163708</td>\n",
              "      <td>2.517430</td>\n",
              "      <td>64.719973</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3</td>\n",
              "      <td>5391.811801</td>\n",
              "      <td>0.142507</td>\n",
              "      <td>1.940990</td>\n",
              "      <td>54.810756</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4</td>\n",
              "      <td>5144.575069</td>\n",
              "      <td>0.081243</td>\n",
              "      <td>2.153666</td>\n",
              "      <td>48.311880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5</td>\n",
              "      <td>4945.829038</td>\n",
              "      <td>0.055514</td>\n",
              "      <td>2.871817</td>\n",
              "      <td>43.939012</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>6</td>\n",
              "      <td>4871.753327</td>\n",
              "      <td>0.048261</td>\n",
              "      <td>2.616558</td>\n",
              "      <td>37.535589</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>7</td>\n",
              "      <td>4643.341409</td>\n",
              "      <td>0.045942</td>\n",
              "      <td>2.408297</td>\n",
              "      <td>37.898293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>8</td>\n",
              "      <td>4409.700578</td>\n",
              "      <td>0.052004</td>\n",
              "      <td>2.289105</td>\n",
              "      <td>38.881394</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>9</td>\n",
              "      <td>4330.559243</td>\n",
              "      <td>0.047935</td>\n",
              "      <td>2.431148</td>\n",
              "      <td>36.012988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>10</td>\n",
              "      <td>4243.495470</td>\n",
              "      <td>0.042736</td>\n",
              "      <td>2.635515</td>\n",
              "      <td>34.036205</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a50e40cb-e73b-42ff-971b-58610c4e3c33')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a50e40cb-e73b-42ff-971b-58610c4e3c33 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a50e40cb-e73b-42ff-971b-58610c4e3c33');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-258a68fc-8e92-43ea-bb55-942b4cfe85c4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-258a68fc-8e92-43ea-bb55-942b4cfe85c4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-258a68fc-8e92-43ea-bb55-942b4cfe85c4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_08e202d4-27be-4a24-9a9b-a7b363340ede\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('summary_table')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_08e202d4-27be-4a24-9a9b-a7b363340ede button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('summary_table');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   n_clusters         WCSS  Silhouette Score  Davies-Bouldin Index  \\\n",
              "0           2  5741.146083          0.163708              2.517430   \n",
              "1           3  5391.811801          0.142507              1.940990   \n",
              "2           4  5144.575069          0.081243              2.153666   \n",
              "3           5  4945.829038          0.055514              2.871817   \n",
              "4           6  4871.753327          0.048261              2.616558   \n",
              "5           7  4643.341409          0.045942              2.408297   \n",
              "6           8  4409.700578          0.052004              2.289105   \n",
              "7           9  4330.559243          0.047935              2.431148   \n",
              "8          10  4243.495470          0.042736              2.635515   \n",
              "\n",
              "   Calinski-Harabasz Index  \n",
              "0                64.719973  \n",
              "1                54.810756  \n",
              "2                48.311880  \n",
              "3                43.939012  \n",
              "4                37.535589  \n",
              "5                37.898293  \n",
              "6                38.881394  \n",
              "7                36.012988  \n",
              "8                34.036205  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Summary table of the scores of each metric for each cluster K\n",
        "summary_table = pd.DataFrame({\n",
        "    'n_clusters': K,\n",
        "    'WCSS': WCSS,\n",
        "    'Silhouette Score': SS,\n",
        "    'Davies-Bouldin Index': DB,\n",
        "    'Calinski-Harabasz Index': CH\n",
        "})\n",
        "\n",
        "display(summary_table)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 500,
      "id": "Z984PhRHOCg7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "Z984PhRHOCg7",
        "outputId": "9ee4c9ea-3816-4ccc-cd84-41793bb8650f"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 132420 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 4 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Plot each metric side-by-side\n",
        "\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Plot WCSS vs number of clusters\n",
        "ax1 = plt.subplot(2,2,1)\n",
        "plt.plot(K, WCSS, marker='o', label='WCSS')\n",
        "plt.xlabel('Number of Clusters')\n",
        "plt.ylabel('WCSS')\n",
        "plt.title('WCSS vs. Number of Clusters')\n",
        "plt.grid(True)\n",
        "\n",
        "# Plot silhoutte scores\n",
        "ax2 = plt.subplot(2,2,2)\n",
        "plt.plot(K, SS, marker='s', label='Silhouette Score')\n",
        "plt.xlabel('Number of Clusters')\n",
        "plt.ylabel('Silhouette Score')\n",
        "plt.title('Silhouette Score vs. Number of Clusters')\n",
        "plt.grid(True)\n",
        "\n",
        "# Plot Davies-Bouldin index\n",
        "ax3 = plt.subplot(2,2,3)\n",
        "plt.plot(K, DB, marker='s', label='Silhouette Score')\n",
        "plt.xlabel('Number of Clusters')\n",
        "plt.ylabel('Davies-Bouldin')\n",
        "plt.title('Davies-Bouldin Index vs. Number of Clusters')\n",
        "plt.grid(True)\n",
        "\n",
        "# Plot Calinski-Harabasz Index\n",
        "ax4 = plt.subplot(2,2,4)\n",
        "plt.plot(K, CH, marker='s', label='Silhouette Score')\n",
        "plt.xlabel('Number of Clusters')\n",
        "plt.ylabel('Calinski-Harabasz Index')\n",
        "plt.title('Calinski-Harabasz Index vs. Number of Clusters')\n",
        "plt.grid(True)\n",
        "\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pYILtm9TPFOG",
      "metadata": {
        "id": "pYILtm9TPFOG"
      },
      "source": [
        "**Interpretation of metrics and output**\n",
        "\n",
        "Above we have calculated our four metrics at different cluster cutoff levels. We do not look at the number of clusters beyond 10 because the executives are likely only interested in a few high-level clusters.\n",
        "\n",
        "- **WCSS:** Ideally, WCSS should decrease as the number of clusters increase. While we do see this decrease, there is not a clear \"kink\" to easily inform the selection of cluster numbers. Arguably, 8 clusters could be a cutoff due to minimal improvement in WCSS afterwards.\n",
        "- **Silhouette score**: ideally we want to maximise this score. While the score is highest for 2 and 3 clusters before dropping off significantly, this may be too high-level to provide useful insights. The values after 3 clusters are less than 0.1, which is low, indicating that our clusters may not be well separated or that points within clusters are not very similar. We also see that 8 clusters is a local maximum, corroborating WCSS somewhat. However, based on the silhouette score alone, using 5 or fewer clusters appears to be better than 8 clusters.\n",
        "- **Davies-Bouldin Index:** For this metric, we wish to minimise the score. Looking at the chart above, 3 clusters appears optimal, but it may lack enough granularity for providing useful information to the executives. A significant dip at 8 clusters, suggests this may also be a good choice and that there is value in splitting at a more granular level. There is also a low score at 4 clusters, offering another possibility.\n",
        "- **Calinski-Harabasz Index:** Similar to the silhouette score, we wish to maximise this score. The plot has a similar shape to the silhouette score, with 2 clusters being the best and 8 clusters being a local maximum. However, as with silhoutte score, using 5 or fewer clusters appears to be preferable to 8 clusters based on the score alone.\n",
        "\n",
        "**Key takeaways and decision**\n",
        "\n",
        "Our internal validation metrics offer somewhat conflicting suggestions for an appropriate number of clusters. There are some key takeaways:\n",
        "- While 2 or 3 clusters may be optimal based on the silhouette, DB and CH scores, this may be too high-level to provide any valuable insights\n",
        "- 8 clusters appears to also be a reasonable choice, for more granular analysis, despite lower scores compared to fewer clusters. However, this analysis may be too granular for executives.\n",
        "- The Davies-Bouldin index offers a few choices: 3, 4 and 8 clusters\n",
        "\n",
        "Due to the hierarchical structure of the clusters, we can easily break down the top 4 clusters into 8 granular sub-categories.\n",
        "\n",
        "**Considering these findings and our intended stakeholders, we believe it may be beneficial to analyse multiple cluster configurations: 3, 4 and 8. We can provide high-level findings by looking at the 3 and 4 cluster configurations. However, the board will likely require slightly more granular information to direct its strategy and so we use 8 clusters to provide this. This will allow us to balance statistical optimality with the need for actionable insights.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CQVXkTpO_aJ-",
      "metadata": {
        "id": "CQVXkTpO_aJ-"
      },
      "source": [
        "**Overall summary**  \n",
        "\n",
        "- Justified that agglomerative hierarchical clustering would suitable for our clustering task\n",
        "- Justified Euclidean distance as an appropriate distance measure\n",
        "- Selected the features to use in the clustering based on whether they measured a characteristic of the *quotes themselves* rather than traits about the film and who made it.\n",
        "- Applied hierarchical clustering using single linkage, complete linkage and average linkage. Ultimately, we found that complete linkage led to more balanced clusters and so selected it as our linkage method.\n",
        "- Evaluated the clustering outputs using a range of internal metrics.\n",
        "- Will further analyse the 3 and 4 cluster configurations. However, this may be too high level to provide insight beyond generic comments. Therefore we will also look at 8 clusters to complement our analysis and provide more actionable detail to the executives.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nhSpAKNJWacv",
      "metadata": {
        "id": "nhSpAKNJWacv"
      },
      "source": [
        "### Q2b - Summarise key characteristics\n",
        "#### Purpose:\n",
        "In this section we intend to extract meaningful insights from the clustering done previously. We want to understand the key characteristics of the movie quotes in each cluster and determine if they make intuitive sense (manual validation).\n",
        "\n",
        "The findings will be presented in an executive summary for the executives of the movie production company (see *Q2b - Executive Summary*).\n",
        "\n",
        "#### References:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "i3Q84uz5Wac4",
      "metadata": {
        "id": "i3Q84uz5Wac4"
      },
      "source": [
        "#### Q2b Step 1 - Manual Validation: word clouds\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we will evaluate the clustering output using manual validation. This involves manually reviewing the clustering outcomes and checking that they have an intuitive understanding.\n",
        "\n",
        "This will first be done through word clouds of each cluster. We will look at word clouds for when the number of clusters is 3, 4 and 8, based on prior discussion. The hierarchical nature of clusters will allow us to channel down into sub-clusters if more detail is needed.\n",
        "\n",
        "***Why:***\n",
        "- A word cloud highlights the most common words of quotes in a cluster. This will help us to get an understanding of the themes or semantics that a particular cluster may relate to. This is a key characteristic of quotes and so may be insightful for the executives.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MN5El5C5VBwd",
      "metadata": {
        "id": "MN5El5C5VBwd"
      },
      "source": [
        "\n",
        "##### Three Clusters\n",
        "\n",
        "First let us analyse the results if we have 3 clusters:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 501,
      "id": "3fKumOWsWac4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3fKumOWsWac4",
        "outputId": "c588cb7a-0d32-4836-aa35-a3691b1f77c4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0    510\n",
              "2     88\n",
              "1     35\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 501,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "### 3 CLUSTERS ###\n",
        "K=3\n",
        "# fit clustering:\n",
        "clustering_model, labels_3 = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'complete', n_clusters = 3)\n",
        "\n",
        "# Add labels to our training data:\n",
        "train['cluster_3'] = labels_3\n",
        "\n",
        "# Let us look at the number of quotes in each cluster:\n",
        "pd.DataFrame(labels_3).value_counts()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zN-sDfAuP063",
      "metadata": {
        "id": "zN-sDfAuP063"
      },
      "source": [
        "We see that the three clusters are not very balanced in their size with 500 quotes in one cluster and less than 100 in the other two. The lack of quotes in the latter clusters may make it hard to make meaningful insights related to them. Alternatively, it may be difficult to get much insight from the large cluster as it may contain a mix of different quotes. At this level of clustering we are looking for broad themes which we can dissect further as we break up the clusters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 502,
      "id": "H0hyVMERNu4R",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "H0hyVMERNu4R",
        "outputId": "bdbaa1d0-28d1-4f46-a039-5f9d3bb22603"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 182568 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Next we create some plots\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "ax1 = plt.subplot(1,2,1)\n",
        "plt.title(\"Hierarchical Clustering - 3 clusters\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_3)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "\n",
        "# create dendogram\n",
        "ax2= plt.subplot(1,2,2)\n",
        "plt.title(\"Dendrogram - 3 clusters\")\n",
        "plot_dendrogram(clustering_model, no_labels = True, truncate_mode=\"level\", p=3, color_threshold = clustering_model.distances_[-K]+0.03)\n",
        "plt.xlabel(\"Number of points in node\")\n",
        "plt.axhline(y = clustering_model.distances_[-K]+0.1, c='k') # create horizontal line for cutoff\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cPRRhxeXRDB_",
      "metadata": {
        "id": "cPRRhxeXRDB_"
      },
      "source": [
        "The 2D plot does not show our 3 clusters clearly separated which is expected given that the clustering was done over a 10-dimensional space. The dendrogram shows our 3 clusters (each coloured differently). We can also see the two sub-clusters that would be achieved if we increased the number of clusters to 4 (see the orange clusters in the dendrogram)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 503,
      "id": "prBpUeF9O002",
      "metadata": {
        "id": "prBpUeF9O002"
      },
      "outputs": [],
      "source": [
        "# Next we write a function to create a wordcloud\n",
        "\n",
        "# we use the text with stop words removed for creating our word clouds:\n",
        "def get_wordcloud(data, cluster_num, cluster_label_col,excluded_terms=[]):\n",
        "\n",
        "    print(f'Getting wordcloud for cluster {cluster_num + 1} ...')\n",
        "\n",
        "    # get list of quotes in cluster (use text without stop words)\n",
        "    body_list = list(data['text_bert_noSW'][data[cluster_label_col] == cluster_num])\n",
        "\n",
        "    # remove excluded words\n",
        "    for word in excluded_terms:\n",
        "        body_list = [w.replace(word, '') for w in body_list]\n",
        "\n",
        "    # join all words together in one string\n",
        "    tokens = ' '.join(body_list)\n",
        "    wordcloud = WordCloud(width=800, height=560,\n",
        "                          background_color='white', collocations=False,\n",
        "                          min_font_size=10).generate(tokens)\n",
        "\n",
        "    # plot the WordCloud image\n",
        "    plt.figure(figsize=(8, 5.6), facecolor=None)\n",
        "    plt.imshow(wordcloud)\n",
        "    plt.axis('off')\n",
        "    plt.title(f\"Cluster {cluster_num+1}: {len(body_list)} quotes\") # show number of quotes in cluster\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "# Define a function to get most frequent words for each cluster\n",
        "def get_frequent_words(data, cluster_num, cluster_label_col, hidden_words = None):\n",
        "    cluster_data = data[data[cluster_label_col] == cluster_num]\n",
        "    words = ' '.join(cluster_data['text_bert_noSW']).split()\n",
        "    words = [word for word in words if word.lower() not in hidden_words]\n",
        "    counter = Counter(words)\n",
        "    most_frequent_words = counter.most_common(15)  # Get top 15 frequent words\n",
        "    return [word for word, _ in most_frequent_words]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 504,
      "id": "sd-YdeY0aW5K",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "sd-YdeY0aW5K",
        "outputId": "3a22c7a0-6b25-4ee4-ae6b-78105c4b74b9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 1 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 344036 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 2 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 337100 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 3 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 350104 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "### Now create the word clouds for our 3 clusters\n",
        "\n",
        "# we hide common words that were found during our stop words analysis:\n",
        "hidden_words = ['life', 'know', 'like', 'get', 'one', 'never', 'people', 'get', 'want', 'go']\n",
        "#hidden_words = []\n",
        "\n",
        "for k in range(0,K):\n",
        "  get_wordcloud(train, k, 'cluster_3', hidden_words)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 505,
      "id": "RIThjPrHepF0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RIThjPrHepF0",
        "outputId": "343ccb94-2a78-4354-821e-564f2ab1cd02"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+---------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|                                                    Most Common Words for 3 Clusters                                                   |\n",
            "+-----------+--------------------------------------------------------------------------------------------------------+------------------+\n",
            "|  Cluster  |                                           Most Common Words                                            | Number of quotes |\n",
            "+-----------+--------------------------------------------------------------------------------------------------------+------------------+\n",
            "| Cluster 1 |      got, always, time, something, let, man, going, look, even, ever, home, day, say, back, tell       |       510        |\n",
            "| Cluster 2 | come, kill, walk, forward, would, keep, hard, hope, choose, death, let, little, tortilla, dark, really |        35        |\n",
            "| Cluster 3 |  love, good, great, thing, best, man, happy, always, would, world, better, think, bright, oh, really   |        88        |\n",
            "+-----------+--------------------------------------------------------------------------------------------------------+------------------+\n"
          ]
        }
      ],
      "source": [
        "# Table of frequent words for each wordcloud cluster\n",
        "table = PrettyTable()\n",
        "table.title = 'Most Common Words for 3 Clusters'\n",
        "table.field_names = ['Cluster', 'Most Common Words', 'Number of quotes']\n",
        "\n",
        "# Get frequent words for each cluster and add them to the table\n",
        "for cluster in range(0, K):\n",
        "    frequent_words = get_frequent_words(train, cluster, 'cluster_3', hidden_words)\n",
        "    frequent_words_str = ', '.join(frequent_words)\n",
        "    table.add_row([f\"Cluster {cluster + 1}\", frequent_words_str, len(train[train['cluster_3']==cluster])])\n",
        "\n",
        "print(table)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 506,
      "id": "WLZjwhc0byv-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WLZjwhc0byv-",
        "outputId": "8a0448ac-6532-4a37-8ae8-3f829d36c833"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quotes from cluster 1:\n",
            "229                                                                     Public opinion has a way of changing.\n",
            "884    These barbarians pretend to be submissive, but theâ€¦a sleeping beast down, before it can be awakened.\n",
            "638                                                         I love you too, but I shouldn't have married you.\n",
            "621                                                      If youâ€™re good at something, never do it for free.\n",
            "497                                                                                    We're goin' streaking!\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 2:\n",
            "261                                                    Choose the sword, and you will join me. Choose the ball and you join your mother, in death. You donâ€™t understand my words, but you must choose. Soâ€¦ come boy, choose life or death.\n",
            "265                                                                                        Little pigs, little pigs, let me come in! Not by the hair on your chinny chin chin? Then I'll huff... And I'll puff... And I'll blow your house in!\n",
            "565                    Nobody is gonna hit as hard as life, but it ainâ€™t how hard you can hit. Itâ€™s how hard you can get hit and keep moving forward. Itâ€™s how much you can take, and keep moving forward. Thatâ€™s how winning is done.\n",
            "668    A tortilla is either corn or wheat. But a corn tortilla folded and filled is a taco, whereas a filled wheat tortilla is a burrito. Deep fry a burrito, it's a chimichanga.Toast a tortilla, it's a tostada. Roll it, it's an enchilada.\n",
            "257                                                     All right, but apart from the sanitation, medicine, education, wine, public order, irrigation, roads, the fresh water system and public health, what have the Romans ever done for us?\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 3:\n",
            "344                                                                 Lord, it's a miracle! Man up and vanished like a fart in the wind!\n",
            "877    My mother always tells me to smile and put on a happy face. She told me I had a purpose to bring laughter and joy to the world.\n",
            "660                   It is a curious thing, Harry, but perhaps those who are best suited to power are those who have never sought it.\n",
            "548                                                                                             All's fair in Love and Basketball baby\n",
            "451                                 Ask any racer. Any real racer. It don't matter if you win by an inch or a mile. Winning's winning.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Look at a couple example quotes from each cluster:\n",
        "K=3\n",
        "for cluster in range(0,K):\n",
        "  print(f'Quotes from cluster {cluster + 1}:')\n",
        "  print(train.loc[train['cluster_3']==cluster, 'Movie Quote'].sample(n=5, random_state = random_seed))\n",
        "  print()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VnVd1ffsXEhL",
      "metadata": {
        "id": "VnVd1ffsXEhL"
      },
      "source": [
        "**Summary and Interpretation**\n",
        "\n",
        "Based on the word analysis in the three clusters, the following observations can be made:\n",
        "- It is difficult to discern a clear theme from the quotes in Cluster 1, our largest cluster. As there are so many quotes in this cluster, the top words just reflect the top words in our corpus of quotes which is not particularly insightful. This may be expected given the cluster represents over 80% of our data. We will break this cluster up in consequent steps.\n",
        "- The quotes in Cluster 2 appear to relate to **death** and other morbid themes with common words like 'kill', 'hard', 'death'. However, looking at the quotes themselves reveals that the quotes in this cluster are quite long.\n",
        "- Conversely the quotes in Cluster 3 appear to relate to more **positive and happy themes** with common words such as 'love', 'best', 'great', and 'bright'.\n",
        "\n",
        "Overall, using three clusters does not appear to reveal much about the different characteristics of quotes given that one cluster makes up 80% of the data. More granular cluster analysis is hence needed."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PWiI9Zb0Yta7",
      "metadata": {
        "id": "PWiI9Zb0Yta7"
      },
      "source": [
        "##### Four Clusters\n",
        "\n",
        "Next we look at four clusters. Our previous dendrogram revealed that this will result in us breaking up our largest cluster. We hence only need to look at the two sub-clusters produced as the other two clusters will be the same as under 3 clusters (i.e. cluster 2 and 3 above)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 507,
      "id": "oFbvhc8gY_U-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oFbvhc8gY_U-",
        "outputId": "f847fd0e-e0fa-4aa2-a945-6a48800604dd"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0    455\n",
              "2     88\n",
              "3     55\n",
              "1     35\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 507,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "### 4 CLUSTERS ###\n",
        "K=4\n",
        "# fit clustering:\n",
        "clustering_model, labels_4 = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'complete', n_clusters = K)\n",
        "\n",
        "# Add labels to our training data:\n",
        "train['cluster_4'] = labels_4\n",
        "\n",
        "# Let us look at the number of quotes in each cluster:\n",
        "pd.DataFrame(labels_4).value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "N5tBKll_ZMrI",
      "metadata": {
        "id": "N5tBKll_ZMrI"
      },
      "source": [
        "A smaller cluster has been fragmented off the largest cluster with 55 quotes. We still have one large cluster which may result in us encountering a similar problem to before where the cluster was too large to provide any insight beyond generic remarks about the corpus of quotes. As expected two of our clusters are unchanged since they were not split; we will not repeat our analysis for them here."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 508,
      "id": "BN3XToK2ZgWV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "BN3XToK2ZgWV",
        "outputId": "ce18e3ff-12de-4454-e0ec-593f20d90858"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 187012 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Next we create some plots\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "ax1 = plt.subplot(1,2,1)\n",
        "plt.title(\"Hierarchical Clustering - 4 clusters\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_4)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "\n",
        "# create dendogram\n",
        "ax2= plt.subplot(1,2,2)\n",
        "plt.title(\"Dendrogram - 4 clusters\")\n",
        "plot_dendrogram(clustering_model, no_labels = True, truncate_mode=\"level\", p=3, color_threshold = clustering_model.distances_[-K]+0.03)\n",
        "plt.xlabel(\"Number of points in node\")\n",
        "plt.axhline(y = clustering_model.distances_[-K]+0.1, c='k') # create horizontal line for cutoff\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 509,
      "id": "sg8uUJGLZlcR",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 998
        },
        "id": "sg8uUJGLZlcR",
        "outputId": "99f54cf7-81e4-4c70-cbe0-8dd1ab0c28db"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 1 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 340720 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 4 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 338456 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "### Now create the word clouds for our 4 clusters\n",
        "\n",
        "# we hide common words that were found during our stop words analysis:\n",
        "hidden_words = ['life', 'know', 'like', 'get', 'one', 'never', 'people', 'get', 'want', 'go']\n",
        "#hidden_words = []\n",
        "\n",
        "for k in [0,3]:\n",
        "  get_wordcloud(train, k, 'cluster_4', hidden_words)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 510,
      "id": "r4HwCE2zZuSP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r4HwCE2zZuSP",
        "outputId": "7b98bfbc-43cc-4985-a925-0b27723d0732"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|                                                      Most Common Words for 4 Clusters                                                      |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n",
            "|  Cluster  |                                              Most Common Words                                              | Number of quotes |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n",
            "| Cluster 1 |      something, got, time, let, always, man, ever, home, day, tell, look, even, everything, going, die      |       455        |\n",
            "| Cluster 2 |    come, kill, walk, forward, would, keep, hard, hope, choose, death, let, little, tortilla, dark, really   |        35        |\n",
            "| Cluster 3 |     love, good, great, thing, best, man, happy, always, would, world, better, think, bright, oh, really     |        88        |\n",
            "| Cluster 4 | hate, evil, always, going, bad, madness, means, even, must, mean, road, worst, dangerous, weapons, remember |        55        |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n"
          ]
        }
      ],
      "source": [
        "# Table of frequent words for each wordcloud cluster\n",
        "table = PrettyTable()\n",
        "table.title = 'Most Common Words for 4 Clusters'\n",
        "table.field_names = ['Cluster', 'Most Common Words', 'Number of quotes']\n",
        "\n",
        "# Get frequent words for each cluster and add them to the table\n",
        "  # we only need to look at the clusters that changed\n",
        "for cluster in range(0,K):\n",
        "    frequent_words = get_frequent_words(train, cluster, 'cluster_4', hidden_words)\n",
        "    frequent_words_str = ', '.join(frequent_words)\n",
        "    table.add_row([f\"Cluster {cluster + 1}\", frequent_words_str, len(train[train['cluster_4']==cluster])])\n",
        "\n",
        "print(table)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 511,
      "id": "_WTJG2GWdz_W",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_WTJG2GWdz_W",
        "outputId": "242dd405-0270-4291-f7b7-7c2f7b78e944"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quotes from cluster 1:\n",
            "770                                                                                        Mankind was born on Earth. It was never meant to die here.\n",
            "342    Yeah I called her up. She gave me a bunch of crap about me not listening to her, or something. I don't know, I wasn't really paying attention.\n",
            "182                                                                                                                  Forget it, Jake. It's Chinatown.\n",
            "84                                                                                                                                        HULK SMASH!\n",
            "624                                                                                                                   Let's put a smile on that face!\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 4:\n",
            "868                                                                                      Things are very uncomfortable between us, but weâ€™re stuck in a house together.\n",
            "404                                                                                                                                            Time is always against us.\n",
            "121                                                                                                                    You're only supposed to blow the bloody doors off!\n",
            "634                                                                           Nobody panics when things go â€œaccording to planâ€_x009d_. Even if the plan is horrifying!\n",
            "468    Itâ€™s a dangerous business, going out your doorâ€¦You step into the Road, and if you donâ€™t keep your feet, there is no knowing where you might be swept off to.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Look at a couple example quotes from each cluster:\n",
        "K=4\n",
        "for cluster in [0,3]:\n",
        "  print(f'Quotes from cluster {cluster + 1}:')\n",
        "  print(train.loc[train['cluster_4']==cluster, 'Movie Quote'].sample(n=5, random_state = random_seed))\n",
        "  print()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VJ43fMq1aiuX",
      "metadata": {
        "id": "VJ43fMq1aiuX"
      },
      "source": [
        "**Summary and Interpretation**\n",
        "\n",
        "Based on the word analysis in the four clusters, the following observations can be made:\n",
        "- The largest cluster contains 70% of our data and so the word cloud and common words are still too general to provide any useful insight. We will have to break the analysis down to 8 clusters to extract more valuable information.\n",
        "- In cluster 4, the new cluster, the most common words relate to evil and nastiness with common words like 'evil, 'hate' and 'madness'. This appears to be distinct from Cluster 2 which relates more specifically to death. Looking at a sample of quotes from the cluster, the quotes appear to relate to a particular stressful situation such as having a plan fall apart.\n",
        "\n",
        "We now look at eight clusters which may give us substantially deeper insights."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uvD9NVHretWm",
      "metadata": {
        "id": "uvD9NVHretWm"
      },
      "source": [
        "##### Eight Clusters\n",
        "\n",
        "Next we look at eight clusters. This will give much more granular insight for our analysis, although some clusters may be small and be distorted by outliers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 512,
      "id": "KnTfFzHWetWx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KnTfFzHWetWx",
        "outputId": "7740fc11-725f-465b-f76d-9bb9607ab063"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2    197\n",
              "3    120\n",
              "0     88\n",
              "6     88\n",
              "1     55\n",
              "7     50\n",
              "4     26\n",
              "5      9\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 512,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "### 8 CLUSTERS ###\n",
        "K=8\n",
        "# fit clustering:\n",
        "clustering_model, labels_8 = perform_hierarchical_clustering(data = clustering_data, linkage_method = 'complete', n_clusters = K)\n",
        "\n",
        "# Add labels to our training data:\n",
        "train['cluster_8'] = labels_8\n",
        "\n",
        "# Let us look at the number of quotes in each cluster:\n",
        "pd.DataFrame(labels_8).value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "K3_nq7GpetWx",
      "metadata": {
        "id": "K3_nq7GpetWx"
      },
      "source": [
        "We see most clusters have approximately a similar number of quotes in each of them (50-150). However, we have a couple very small clusters, with one containing just 9 quotes. These may be outlier quotes and the clusters may be too immaterial to bother reporting to the executives. These two small clusters appear to be a fragmentation of our prior 'cluster 2' which appeared to relate to death.\n",
        "\n",
        "Also, there is still a larger cluster with almost 200 quotes in it (30% of our data). We may be able to extract a broad theme from this larger cluster and it is likely important to report to board given a large proportion of quotes fall in it.\n",
        "\n",
        "Our prior 'cluster 3' and 'cluster 4' still exist at this level although they now have different numbers associated with them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 513,
      "id": "Cu4xGGYHetWx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "Cu4xGGYHetWx",
        "outputId": "92a9adc4-3993-470f-fd67-a0a5e9168afd"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 192556 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Next we create some plots\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "#create 2D plot:\n",
        "# the clustering is done over 10 dimensions but we can only plot 2 so we choose the first two PCs of the BERT embedding\n",
        "ax1 = plt.subplot(1,2,1)\n",
        "plt.title(\"Hierarchical Clustering - 8 clusters\")\n",
        "plt.scatter(clustering_data.iloc[:,0], clustering_data.iloc[:,1], c=labels_8)\n",
        "plt.xlabel(\"PC1 of BERT embedding\")\n",
        "plt.xlabel(\"PC2 of BERT embedding\")\n",
        "\n",
        "# create dendogram\n",
        "ax2= plt.subplot(1,2,2)\n",
        "plt.title(\"Dendrogram - 8 clusters\")\n",
        "plot_dendrogram(clustering_model, no_labels = True, truncate_mode=\"level\", p=4, color_threshold = clustering_model.distances_[-K]+0.03)\n",
        "plt.xlabel(\"Number of points in node\")\n",
        "plt.axhline(y = clustering_model.distances_[-K]+0.1, c='k') # create horizontal line for cutoff\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 514,
      "id": "-W-CspChetWy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "-W-CspChetWy",
        "outputId": "f2d7aa6e-f948-4ef4-9dbc-c67d4f0342fa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 1 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 352284 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 2 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 368432 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 3 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 348468 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 4 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 325460 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 5 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 353656 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 6 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 291060 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 7 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 370692 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Getting wordcloud for cluster 8 ...\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 361612 characters]",
            "text/plain": [
              "<Figure size 800x560 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "### Now create the word clouds for our 8 clusters\n",
        "\n",
        "# we hide common words that were found during our stop words analysis:\n",
        "hidden_words = ['life', 'know', 'like', 'get', 'one', 'never', 'people', 'get', 'want', 'go']\n",
        "#hidden_words = []\n",
        "\n",
        "for k in range(0,K):\n",
        "  get_wordcloud(train, k, 'cluster_8', hidden_words)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 515,
      "id": "DG0Jp8Z0etWy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DG0Jp8Z0etWy",
        "outputId": "441d3b75-63b1-4d2f-9bf5-8312562d54d5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|                                                      Most Common Words for 8 Clusters                                                      |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n",
            "|  Cluster  |                                              Most Common Words                                              | Number of quotes |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n",
            "| Cluster 1 |     love, good, great, thing, best, man, happy, always, would, world, better, think, bright, oh, really     |        88        |\n",
            "| Cluster 2 | hate, evil, always, going, bad, madness, means, even, must, mean, road, worst, dangerous, weapons, remember |        55        |\n",
            "| Cluster 3 |     let, everything, something, money, always, yeah, hey, man, take, ever, time, toga, make, dude, half     |       197        |\n",
            "| Cluster 4 |      day, back, let, die, time, got, home, ever, keep, come, tell, something, change, everything, love      |       120        |\n",
            "| Cluster 5 |    forward, would, walk, keep, hard, hope, tortilla, dark, hit, love, little, lives, things, part, better   |        26        |\n",
            "| Cluster 6 |   come, kill, choose, death, join, must, let, hopeless, murdered, write, first, head, think, dead, strong   |        9         |\n",
            "| Cluster 7 | something, got, say, hard, stand, problem, change, someone, sometimes, may, mean, really, look, would, feel |        88        |\n",
            "| Cluster 8 |          pig, look, even, man, spider, need, see, good, rage, right, home, dead, says, used, place          |        50        |\n",
            "+-----------+-------------------------------------------------------------------------------------------------------------+------------------+\n"
          ]
        }
      ],
      "source": [
        "# Table of frequent words for each wordcloud cluster\n",
        "table = PrettyTable()\n",
        "table.title = 'Most Common Words for 8 Clusters'\n",
        "table.field_names = ['Cluster', 'Most Common Words', 'Number of quotes']\n",
        "\n",
        "# Get frequent words for each cluster and add them to the table\n",
        "  # we only need to look at the clusters that changed\n",
        "for cluster in range(0,K):\n",
        "    frequent_words = get_frequent_words(train, cluster, 'cluster_8', hidden_words)\n",
        "    frequent_words_str = ', '.join(frequent_words)\n",
        "    table.add_row([f\"Cluster {cluster + 1}\", frequent_words_str, len(train[train['cluster_8']==cluster])])\n",
        "\n",
        "print(table)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 516,
      "id": "pu4OsIRUetWy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pu4OsIRUetWy",
        "outputId": "5b130ce5-b9af-4f73-b21c-5fd1850e2a8d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Quotes from cluster 1:\n",
            "858                                                                                                                             I love you 3000.\n",
            "189                                                                        In case I don't see ya...good afternoon, good evening, and goodnight.\n",
            "768    A machine doesn't improvise well because you cannot program a fear of death. Our survival instinct is our greatest source of inspiration.\n",
            "818                                         Do you know what a Cinnabon is? It's bread! And frosting! And cinnamon! They heat it up! Really hot!\n",
            "207                                                                                             This is the beginning of a beautiful friendship.\n",
            "771                                                                     Hardest time to lie to somebody is when they're expecting to be lied to.\n",
            "332                                                                                   You shoot me in a dream, you better wake up and apologise.\n",
            "415                                                                     I was the warm little center that the life of this world crowded around.\n",
            "99                                                                                                    Greed, for lack of a better word, is good.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 2:\n",
            "474    I don't like sand. It's all coarse, and rough, and irritating. And it gets everywhere.\n",
            "527                                           There's a 30% chance that it's already raining.\n",
            "389                              You can feel the chill in the air, cut, who opened a window?\n",
            "627                     Madness, as you know, is like gravity, all it takes is a little push.\n",
            "865                                               If you want to get crazy. We can get crazy.\n",
            "658                         There's a benefit to losing: You get to learn from your mistakes.\n",
            "789                    Examined from another angle, your faults and weaknesses can be weapons\n",
            "800                                                                   A bit of madness is key\n",
            "105                                                                The horror.....the horror.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 3:\n",
            "589    Weâ€™ve all got both light and darkness inside us. What matters is the part we choose to act on. Thatâ€™s who we really are.\n",
            "855                                                                   Elsa, the past is not what it seems. You must find the truth.\n",
            "91                                                                                                               Run, Forrest, run!\n",
            "256                                                                                       Thatâ€™s no moonâ€¦itâ€™s a space station\n",
            "763                                                         Everyoneâ€™s a whore, Grace. We just sell different parts of ourselves.\n",
            "54                                                                                                 A martini...shaken, not stirred.\n",
            "840                                                                                                              Is this your king?\n",
            "53                                                                                                               Bond...James Bond.\n",
            "363                                                                                                                Hack the planet.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 4:\n",
            "767                                   This worldâ€™s a treasure, but itâ€™s been telling us to leave for a while now.\n",
            "705                     The people who are crazy enough to think that they can change the world, are the one's who do\n",
            "462    Once you meet someone, you never really forget them. It just takes a while for your memory to come back to you\n",
            "490                        This is the day you will always remember as the day you almost caught Captain Jack Sparrow\n",
            "224                               You only live twice:Once when you are born And once when you look death in the face\n",
            "429                                       There are No Accidents. No Coincidences. No Escapes. YOU CAN'T CHEAT DEATH.\n",
            "770                                                        Mankind was born on Earth. It was never meant to die here.\n",
            "703                          Let us learn to show our friendship for a man when he is alive and not after he is dead.\n",
            "765                                                                                    Come find me when you wake up!\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 5:\n",
            "853                                                                                                                                For I am a Sinner in the hands of an angry God, Bloody Mary, Full of Vodka, Blessed are you among cocktails, pray for me now, at the hour of my death, which I hope is soon. Amen\n",
            "756                                                                                                                 Having a family doesnâ€™t mean that you stop being an individual. You know the best thing you can do for the people that depend on you? Be honest with them, even if it means setting them free.\n",
            "889                                                                                                                                                                                                                  I do what I do because of my dad. He was a hero. He gave his life for the pursuit of knowledge.\n",
            "272                                                                                       I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion. I watched C-beams glitter in the dark near the TannhÃ¤user Gate. All those moments will be lost in time, like tears in rain.\n",
            "438                                                                                                                                                                                   If you utter so much as one syllable, I'LL HUNT YOU DOWN AND GUT YOU LIKE A FISH! If you'd like to fax me, press the star key.\n",
            "386    People live their lives bound by what they accept as correct and true. That's how they define \"reality\". But what does it mean to be \"correct\" or \"true\"? Merely vague concepts... Their \"reality\" may all be a mirage. Can we consider them to simply be living in their own world, shaped by their beliefs?\n",
            "604                                                          About three things I was absolutely certain. First, Edward was a vampire. Second, there was part of him-and I didn't know how potent that part might be- that thirsted for my blood. and third, I was unconditionally and irrevocably in love with him.\n",
            "642                                                                                                     You've ever been to Del Frisco's? They cater, so for lunch I would love a twenty ounce porterhouse steak, medium maybe a little charred, with all the trimmings pommes frites, asparagus, and butter squash.\n",
            "565                                                                                          Nobody is gonna hit as hard as life, but it ainâ€™t how hard you can hit. Itâ€™s how hard you can get hit and keep moving forward. Itâ€™s how much you can take, and keep moving forward. Thatâ€™s how winning is done.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 6:\n",
            "585    ...And I also know how important it is in life not necessarily to be strong but to feel strong. To measure yourself at least once. To find yourself at least once in the most ancient of human conditions. Facing the blind death stone alone, with nothing to help you but your hands and your own head\n",
            "610                                                                                                                       If you let my daughter go now, that'll be the end of it. I will not look for you, I will not pursue you. But if you don't, I will look for you, I will find you, and I will kill you.\n",
            "577                                                                                                                 When you're looking for a cancer kid, he should be hopeless. He should have a wheelchair, he should have trouble talking, he should have a little pet goldfish in a Zip-lock bag, hopeless.\n",
            "292                                                                                                                                                 Come on... Come on! Do it! Do it! Come on. Come on! Kill me! I'm here! Kill me! I'm here! Kill me! Come on! Kill me! I'm here! Come on! Do it now! Kill me!\n",
            "440                                                                                                                                        No thinking -- that comes later. You must write your first draft with your heart. You rewrite with your head. The first key to writing is... to write, not to think!\n",
            "366                                                                                                                                             I used to think that if none of your family or friends knew you were dead, it was like not really being dead. People can invent the best and the worst for you.\n",
            "650                                    Well, how carefully should I tread? Because apparently I just killed two people, and you were about to let me walk right out that door! How MISGUIDED are you? I feed you a couple of bullshit legal precedents, and there you go - you jump on it like a bitch in heat.\n",
            "439                               My name is Maximus Decimus Meridius, commander of the Armies of the North, General of the Felix Legions and loyal servant to the true emperor, Marcus Aurelius. Father to a murdered son, husband to a murdered wife. And I will have my vengeance, in this life or the next.\n",
            "261                                                                                                                     Choose the sword, and you will join me. Choose the ball and you join your mother, in death. You donâ€™t understand my words, but you must choose. Soâ€¦ come boy, choose life or death.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 7:\n",
            "471                                                                                                     Do you like dags?\n",
            "729                             The truth is a matter of circumstances, itâ€™s not all things to all people all the time.\n",
            "844                                                                            I just wanted to take another look at you.\n",
            "592                                                                                                              Bazinga!\n",
            "848    Being a hero doesn't mean you are invincible. It just means you're brave enough to stand up, and do what's needed.\n",
            "624                                                                                       Let's put a smile on that face!\n",
            "899                                       I used to think that my life was a tragedy. But now I realize, itâ€™s a comedy.\n",
            "161                                                                            Mrs. Robinson, you're trying to seduce me!\n",
            "901                                                                I used to use this little gun when I was a prostitude.\n",
            "Name: Movie Quote, dtype: object\n",
            "\n",
            "Quotes from cluster 8:\n",
            "398                                                                                                    You're taking all the caviar? That caviar is a garnish!\n",
            "544                                                                                       Didn't I tell you about the football field in the bathroom cupboard?\n",
            "810                        You're right, Aurora. But the drowning man will always try and drag somebody down with him. It ain't right, but the man's drowning.\n",
            "397                                                        A single grain of rice can tip the scale. One man may be the difference between victory and defeat.\n",
            "749                           We used to look up at the sky and wonder at our place in the stars, now we just look down and worry about our place in the dirt.\n",
            "220                                                                                                  Gentlemen, you can't fight in here! This is the War Room!\n",
            "431                                          Do you know what happens to a toad when it's struck by lightning? The same thing that happens to everything else.\n",
            "350    These walls are funny. First you hate â€™em. Then you get used to â€™em. Enough time passes, you get so you depend on them. Thatâ€™s institutionalized.\n",
            "714                                                          You have got to be willing to crash and burn. If you are afraid of failing, you wont get very far\n",
            "Name: Movie Quote, dtype: object\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Look at a couple example quotes from each cluster:\n",
        "K=8\n",
        "for cluster in range(0,K):\n",
        "  print(f'Quotes from cluster {cluster + 1}:')\n",
        "  print(train.loc[train['cluster_8']==cluster, 'Movie Quote'].sample(n=9, random_state = 11))\n",
        "  print()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iUhQ6cmdetWy",
      "metadata": {
        "id": "iUhQ6cmdetWy"
      },
      "source": [
        "**Summary and Interpretation**\n",
        "\n",
        "Based on the word analysis in the eight clusters, the following observations can be made:\n",
        "- Cluster 1 was seen previously and relates to **positive** themes like happiness, winning and love\n",
        "- Cluster 2 was also seen previously and relates to difficulty, pain and danger. In particular, the quotes reveal that a character tends to be undergoing some challenge or stressful situation. They often indicate some foreshadowing of danger ahead or have a foreboding feeling.\n",
        "- Cluster 3 is our largest cluster and it is difficult to discern a universal theme. It appears that many of the quotes are short, perhaps relating to a a catchphrase. This cluster may contain short quips that an audience can easily memorise and repeat.\n",
        "- Cluster 4 is our second-largest cluster. The common words are not particularly revealing so we look at some quotes in the cluster. We find that the quotes often refer to an aphorism, or are didactic in nature. They often try to impart some teaching or prompt action. For example, \"just keep swimming\" or \"you must strive to find your own voice\". There are exceptions to this however.\n",
        "- Cluster 5 is one of our smallest clusters. Previously, it was grouped with cluster 6 (see below). It appears all the quotes in it are quite long but do have a strong thematic connection. It appears to be splitting out the long quotes that are not about death and are perhaps have a more positive sentiment(see below).\n",
        "- Cluster 6 is our smallest cluster. These quotes also appear to be quite long but in contrast to cluster 5 the quotes appear to be about darker themes or perhaps have lower sentiment scores. Themes include death and insensitive themes but it is hard to draw broad observations with so few quotes.\n",
        "- Cluster 7 also appears to contain short quotes. It is difficult to discern a consistent theme in the quotes. They appear also to be short, memorable, well-known quotes similar to cluster 3, although are slightly longer in length.\n",
        "- Cluster 8 appears to relate to death and mortality. Often this relates to confronting this mortality."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "F_baiQ-YLViD",
      "metadata": {
        "id": "F_baiQ-YLViD"
      },
      "source": [
        "#### Q2b Step 2 - Manual Validation: summary statistics and plots\n",
        "**Outline**\n",
        "\n",
        "***What:***\n",
        "\n",
        "In this section we will use summary statistics and plots of each cluster to better discern the meaning and characteristics of each cluster. This will help to inform the executive summary to the board.\n",
        "\n",
        "***Why:***\n",
        "- Summary statistics will be used to understand the sentiment features and word count that were also used in our clustering. Plots will help us to visualise the distribution of these numeric features which may provide further insight than just summary statistics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 517,
      "id": "DTtKYuu-vQs_",
      "metadata": {
        "id": "DTtKYuu-vQs_"
      },
      "outputs": [],
      "source": [
        "# set up a function for calculate summary statistics:\n",
        "def cluster_summary_statistics(data, labels_vec, label_val):\n",
        "    cluster_data = data.loc[labels_vec == label_val]  # Filter data for the cluster\n",
        "    return cluster_data.describe(include='all')\n",
        "\n",
        "def get_cluster_means(data, cluster_labels):\n",
        "  data['cluster_label'] = cluster_labels+1\n",
        "  cluster_means = data.groupby('cluster_label').mean(numeric_only=True)\n",
        "  return cluster_means.transpose()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 518,
      "id": "k_or91ODt4a2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "k_or91ODt4a2",
        "outputId": "4eeb6db2-3c3c-485c-b2cd-f6b07f348e93"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 45260 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "### 4 clusters\n",
        "\n",
        "cluster_means = get_cluster_means(clustering_data.copy(), labels_4)  # Copy data to avoid modifying original\n",
        "\n",
        "# Create the plot\n",
        "fig, ax = plt.subplots(figsize=(12, 6))\n",
        "cluster_means.plot(kind='bar', ax=ax)\n",
        "\n",
        "# Customize plot\n",
        "ax.set_xticks(np.arange(cluster_means.shape[0]))\n",
        "ax.set_xticklabels(cluster_means.index)\n",
        "ax.set_xlabel('Column Name')\n",
        "ax.set_ylabel('Mean Value')\n",
        "ax.set_title('Mean of Numeric Columns by Cluster')\n",
        "plt.legend(title='Cluster Label')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for readability\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 519,
      "id": "cRxCAzGvdWMe",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 450
        },
        "id": "cRxCAzGvdWMe",
        "outputId": "402c5962-c5a8-4b60-c344-eeaa53431c1c"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 28816 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# we also create a heatmap\n",
        "sns.heatmap(cluster_means, cmap=\"plasma\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nBXSMdf19d01",
      "metadata": {
        "id": "nBXSMdf19d01"
      },
      "source": [
        "**Observations and Interpretation**\n",
        "\n",
        "- Cluster 1: this was our largest cluster for this configuration. We see that the mean for all the columns is relatively close to zero indicating that the cluster is fairly average and representative of our whole training data. This corroborates what we saw previously and how it was difficult to identify a clear common theme.\n",
        "- Cluster 2: has a very low value for PC1 and a very high value for word_count. The high word count matches what we observed previously. We are unsure what PC1 is associated with but a negative value for it may be associated with darker themes which is what we saw previously associated with this cluster.\n",
        "- Cluster 3: contains quotes with high sentiment values (both VADER and TextBlob). This was the category that we previously idenified as being associated with positive themes which is supported by the notion of having positive sentiment.\n",
        "- Cluster 4: characterised by negative sentiment scores and negative values for PC2, PC4 and PC5. The quotes also appear to be quite short. This appears to match what we observed previously where the quotes were associated with 'evil' and 'hatred'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 520,
      "id": "R6F8iYvG873k",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "R6F8iYvG873k",
        "outputId": "22bccf94-87f6-49e6-800f-ce8ab9ee97df"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 48064 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "### 8 clusters\n",
        "\n",
        "cluster_means = get_cluster_means(clustering_data.copy(), labels_8)  # Copy data to avoid modifying original\n",
        "\n",
        "# Create the plot\n",
        "fig, ax = plt.subplots(figsize=(12, 6))\n",
        "cluster_means.plot(kind='bar', ax=ax)\n",
        "\n",
        "# Customize plot\n",
        "ax.set_xticks(np.arange(cluster_means.shape[0]))\n",
        "ax.set_xticklabels(cluster_means.index)\n",
        "ax.set_xlabel('Column Name')\n",
        "ax.set_ylabel('Mean Value')\n",
        "ax.set_title('Mean of Numeric Columns by Cluster')\n",
        "plt.legend(title='Cluster Label')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for readability\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 521,
      "id": "t4Jbl26Wc1KP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 451
        },
        "id": "t4Jbl26Wc1KP",
        "outputId": "cd7c7582-34b3-463b-fc2a-e87a15fad150"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 26228 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# create a heatmap:\n",
        "sns.heatmap(cluster_means, cmap=\"plasma\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fuKiqGwE_5mO",
      "metadata": {
        "id": "fuKiqGwE_5mO"
      },
      "source": [
        "**Observations and Interpretation**\n",
        "Below are the observations from the heatmap and bar chart above. For a clearer, holistic interpretation of the clusters, please read the *executive summary*.\n",
        "\n",
        "- Cluster 1: characterised by positive sentiment scores and an average word count. We cannot draw much conclusion from the principal components without further analysis. This cluster is the same as cluster 3 above for when we looked at using four clusters.\n",
        "- Cluster 2: characterised by short quotes with negative sentiment scores. Corresponds to cluster 4 above when we used four clusters.\n",
        "- Cluster 3: characterised by *very* short quotes with relatively neutral sentiment scores. This appears to correspond to our word analysis where we identified this cluster as corresponding to short, memorable quips.\n",
        "- Cluster 4: characterised by high values for PC4, PC5 and PC6. The sentiment scores and word count appear to be relatively average. Previously, we thought that these quotes were didactic and about growth/learning; perhaps PCs 4-6 capture this theme somewhat.\n",
        "- Clusters 5 and 6: when using four clusters these two clusters are combined and were characterised by high word counts and low PC1 scores. This appears to still be the case when they are split out but the clusters have very different sentiment scores with cluster 5 being positive and cluster 6 negative. This corresponds with what we observed in prior analysis. The executives may be less interested in whether long quotes are positive or negative but just that long movie quotes exist.\n",
        "- Cluster 7: characterised by short quotes and high values for PC3 and negative values for PC6. The sentiment scores appear to disagree on average although may take fairly neutral values. We struggled to distinguish this cluster from cluster 3 previously however, it appears that PC3 and PC6 are different to cluster 3 and so there may be some semantic difference. This cluster's PCs tend to be opposite to cluster 8 which tends to be about death. Therefore, this cluster may relate to life and living life to the fullest.\n",
        "- Cluster 8: characterised by medium-length quotes with negative sentiment scores. Previously, we identifed this cluster to relate to mortality and death. The direction of the PCs and sentiment scores appear to match Cluster 6 but the key distinguishing trait is the length of these quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-ykdFebzWkQQ",
      "metadata": {
        "id": "-ykdFebzWkQQ"
      },
      "source": [
        "#### Q2b - Executive Summary: Different Quote Types\n",
        "\n",
        "**Outline**\n",
        "\n",
        "In this section we provide a clear summary of the key characteristics of different movie quotes to present to the executives of the movie company. We ensure to pitch it appropriately such as by not using technical jargon and by focusing on the **salient** results rather than unnecessary detail. While eight clusters were analysed, we had two small clusters that were similar so group them together when presenting to the executives.\n",
        "\n",
        "Examples of quotes are provided when needed to back up our points. The executives want to know characteristics that will help them sell movie tickets, win awards and create memorable movies; connections to these objectives will be highlighted."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ek_4yq6XHwpX",
      "metadata": {
        "id": "ek_4yq6XHwpX"
      },
      "source": [
        "**Executive Summary: Famous Movie Quote Characteristics**\n",
        "\n",
        "In today’s saturated film market, only a few films truly standout and create lasting cultural impact. Iconic quotes play a pivotal role in this distinction. Memorable lines spark conversation, promote ticket sales and solidify a film in cultural memory. This report highlights key characteristics of different types of famous movie quotes and provides actionable insights to help us create unforgettable moments in our own films. By strategically employing these insights, we can leave lasting impressions, boost film marketability, and place our films at the forefront of award recognition.\n",
        "\n",
        "We analysed several characteristics of famous movie quotes to identify several categories of famous movie quotes. The features used to distinguish quotes were:\n",
        "-\tSemantics and themes associated with a quote\n",
        "-\tSentiment towards a quote (positive vs negative)\n",
        "-\tThe length of quotes\n",
        "\n",
        "Overall, the analysis revealed the following key categories of famous quotes:\n",
        "-\t**Catchphrases** – short, neutral quotes with high memorability\n",
        "-\t**Proverbs** – impart wisdom in a creative and memorable way\n",
        "-\t**Feel-good quotes** – relate to positive themes to resonate with a large audience\n",
        "-\t**Hardship quotes** – tap into the universality of human struggle\n",
        "-\t**Dark humour quotes** – highlight the need for humour as a release from depressing situations\n",
        "-\t**Acerbic quotes** – short, negative quotes with high memorability\n",
        "-\t**Monologues** – long quotes that are powerful and emotionally-moving\n",
        "\n",
        "The largest group of quotes (30% of the quotes analysed) was characterised by being very short and neutral in sentiment. The brevity of these 'catchphrases' promotes memorability and quotability for the audience (O’Neill, Guo, Temple & Basu, 2012). This may explain why this cluster is the largest. Examples include James Bond’s *“Martini…shaken, not stirred”* and the Terminator’s *“Hasta la vista, baby”*. When writing movie scripts, we may want to prioritise creating a few short, impactful phrases that encapsulate a character’s essence or a film’s core message.  The memorability of these phrases helps to create pop culture references and enduring engagement with the movie which may allow us to better market sequels or franchises.\n",
        "\n",
        "The analysis also revealed a sizeable group (19%) containing 'proverbial' quotes. These memorable lines typically state a general truth or impart wisdom in a creative way. ‘Proverbs’ tend to be average in length and are non-provocative, with neutral sentiment scores. Examples include Yoda’s *“Do, or do not. There is no try”*, and *“Death cannot stop true love”* from *The Princess Bride*. The universal lessons in these quotes may resonate with a large audience. Therefore by incorporating these proverbial moments into our films, we can foster a long-lasting cultural impact and potentially an increase in ticket sales.\n",
        "\n",
        "Another notable cluster of quotes (14%) are typified by positive sentiment scores and themes of happiness, love, and achievement. The material size of this group highlights that audiences connect with these positive themes. An example is \"*Happiness is only real when shared*\". These ‘feel-good’ quotes appeal to a wide audience and are family-friendly. This suggests that integrating dialogue that celebrates positive themes can have mass-market appeal, increasing ticket sales.\n",
        "\n",
        "Our analysis identified a significant cluster of quotes (14%) centred on human resilience in the face of adversity. These quotes explore contrasting responses to challenges, with some advocating defiance (\"*Following's not really my style*\") and others capturing despair in the face of overwhelming odds (\"*Resistance is futile*\"). Despite these differing perspectives, a core theme emerges: the universality of human struggle. This suggests that in our films we should emphasise creating characters who face relatable challenges as this clearly resonates with audiences. A focus on the human experience contributes to the enduring impact and memorability of these ‘hardship’ quotes.\n",
        "\n",
        "Another collection of quotes refers to dark and morbid themes such as death, violence and evil. Dark and wry humour is commonly employed in these quotes to contrast the graveness of the situation and provide relief to the audience. For example, *“drop the gun, take the cannoli”* in *The Godfather*. The key insight from this group of quotes is that dark themes are popular but it is the juxtaposition with humour that makes these scenes iconic. This suggests the potential effectiveness of strategically employing humour to create memorable film moments.\n",
        "\n",
        "While ‘feel-good’ quotes comprised a large proportion of famous lines (14%), a smaller, yet noteworthy group (9%) showcased negative emotions. These ‘acerbic’ quotes, often shorter than their ‘feel-good’ counterparts, were characterised by their bluntness or reference to darker themes. This reiterates the effectiveness of brevity in delivering impact and creating quotable moments. A line that exemplifies this category is *“You’re going to need a bigger boat”* from *Jaws*. While these darker themes may have less mass-market appeal than ‘feel-good’ quotes or ‘catchphrases’, these short, dark quotes can still have a significant cultural impact. In darker films, scriptwriters should focus on creating short, blunt dialogue to boost the memorability of scenes.\n",
        "\n",
        "Finally, the smallest cluster of quotes (5%) are long quotes that span several sentences. The small size of the group suggests that it is difficult for these ‘monologues’ to achieve widespread recognition. However, these quotes have the potential to create more powerful emotional connections with viewers, as demonstrated by their extreme sentiment scores. This highlights that when trying to deliver profound, emotional moments we should focus on creating well-written monologues. While memorability may be a challenge due to their length, they have more potential to ignite audience conversation and increase engagement. Strong emotional connections can increase word-of-mouth marketing, boosting ticket sales and award recognition, aligning with our overall objectives.\n",
        "\n",
        "The analysis highlighted strategic insights relating to each of these categories. In the next section, we provide tangible strategies that leverage these insights.\n",
        "\n",
        "**Word Count: 928**\n",
        "\n",
        "#### References\n",
        "\n",
        "O’Neill, P., Guo, E., Temple, J., & Basu, T. (2012). The secret science of memorable quotes. Retrieved from https://www.technologyreview.com/2012/04/03/186934/the-secret-science-of-memorable-quotes/\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "v8cg_KciWac5",
      "metadata": {
        "id": "v8cg_KciWac5"
      },
      "source": [
        "**Overall summary**  \n",
        "\n",
        "- Manually validated the clustering outcomes using several techniques:\n",
        "  - Word clouds\n",
        "  - Summary statistics\n",
        "  - Plots such as heatmap and bar plot\n",
        "  - Look at individual quotes\n",
        "- Found that using three and four clusters, while statistically optimal, did not provide enough detailed insight to be useful to the board\n",
        "- Eight clusters was found to provide a good level of detail but due to the similarity of two clusters, only seven were presented in the executive summary\n",
        "- Described the key characteristics of each cluster to the executives"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LZptfNoUWtrT",
      "metadata": {
        "id": "LZptfNoUWtrT"
      },
      "source": [
        "### Q2c - Suggest ways the company can use your findings\n",
        "#### Answer:\n",
        "\n",
        "In this section we provide tangible strategies that can be employed by the company to leverage the insights about famous movie quotes. Three strategies are recommended:\n",
        "1.\tCreate short, impactful quotes for marketability and cultural impact.\n",
        "2.\tEmploy humour in dark moments to create viral quotes.\n",
        "3.\tUse powerful monologues to deliver emotional depth and increase award recognition.\n",
        "\n",
        "First, it is recommended that scriptwriters create short, punchy ‘catchphrases’ that encapsulate a film’s core message or a character’s essence. The analysis found that these types of quotes are the most likely to become famous and have an enduring cultural legacy because of their quotability. To manufacture these memorable catchphrases, scriptwriters should focus on expressing ideas in a simple way but with an unusual combination of words (O’Neill et al., 2012). The brevity of these phrases also means they are well-suited for usage in marketing materials and social media campaigns which can boost ticket sales. Additionally, by creating these enduring pop culture references, we may be able to better market future sequels and film merchandise (which can be more lucrative than the film itself).\n",
        "\n",
        "Secondly, the analysis revealed that humour is a powerful technique for creating memorability, especially when it is combined with darker themes. It is therefore recommended that during darker moments, scriptwriters consider adding dry humour to relieve audience tension and to help create an iconic quote. Notably dark humour is an effective way to connect with younger consumers and create internet virality (Mishra et al., 2023). Therefore, by further analysing current cultural trends, these quotes can be used to spark viral moments or memes on social media platforms like TikTok and Facebook. This can help generate interest in a film and potentially increase ticket sales, particularly amongst younger audiences. However, it is important not to force these moments in films or recycle jokes as this may have the unintended effect of seeming inauthentic and forgettable to Gen Z consumers (Marija, 2024).  \n",
        "\n",
        "Thirdly, to create profound emotional scenes, scriptwriters should focus on constructing well-written monologues. Our analysis found that this type of famous quote had the greatest sentiment scores, suggesting that they create strong emotional connections with audiences. Haus (2020) corroborates this idea that monologues can be used to convey complex emotions to audiences. Powerful moments leave a more lasting impact with audiences which may increase word-of-mouth marketing and boost ticket sales. Additionally, film award voters may look for emotional depth in scenes, which can be generated by these quotes, indicating that this strategy is particularly relevant for our films that are targeting award recognition.\n",
        "\n",
        "Based on our analysis, it is recommended that scriptwriters prioritise crafting marketable catchphrases, embrace dark humour and employ powerful monologues. By implementing these strategies, we can ensure our films leave a lasting impression, generate significant buzz, and achieve both commercial and critical success.\n",
        "\n",
        "**Word Count: 468**\n",
        "\n",
        "#### References\n",
        "\n",
        "O’Neill, P., Guo, E., Temple, J., & Basu, T. (2012). The secret science of memorable quotes. Retrieved from https://www.technologyreview.com/2012/04/03/186934/the-secret-science-of-memorable-quotes/\n",
        "\n",
        "Marija, M. (2024). The impact of memes and humor in gen Z and gen y marketing. Retrieved from https://phuconcepts.com/the-impact-of-memes-and-humor-in-gen-z-and-gen-y-marketing/\n",
        "\n",
        "Mishra, A., Getaneh, M., & Martin, K. (2023). How can you use humor to build brand loyalty with Gen z Consumers? Retrieved from https://www.linkedin.com/advice/0/how-can-you-use-humor-build-brand-loyalty-gen-z-consumers-eefbc\n",
        "\n",
        "Haus, T. (2020). The power of monologues in the theatre. Retrieved from https://www.theatrehaus.com/2020/04/the-power-of-monologues-in-the-theatre/"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mZv4K4RKYg7A",
      "metadata": {
        "id": "mZv4K4RKYg7A"
      },
      "source": [
        "## Q3 - Identify quotes that may be problematic"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ti6ZT0-SXyCX",
      "metadata": {
        "id": "ti6ZT0-SXyCX"
      },
      "source": [
        "### Q3a - Suggest strategies to address class imbalance\n",
        "#### Answer:\n",
        "\n",
        "Our data suffers from class imbalance, with only 13% of quotes labelled as 'problematic'. This can bias our model towards the majority class ('non-problematic'), leading to poor detection of problematic quotes and hence a useless model for our objective. We will highlight several strategies that can be used to address this.\n",
        "\n",
        "Firstly, we will employ resampling techniques to balance the data. One option is under-sampling which involves discarding data from the majority class to balance the data. Since we already lack training data, this method is not suitable as it throws away data and exacerbates the data scarcity issue. Alternatively, we could employ random over-sampling which increases the number of observations in the minority class by duplicating records. This may lead to overfitting on the ‘problematic’ quotes in our training data, since they are duplicated, and so the classifier may generalise poorly to unseen data. Instead, we can use an oversampling technique known as SMOTE (Synthetic Minority Oversampling Technique) which partially addresses this overfitting problem. Rather than duplicating minority points, this technique interpolates between them to create synthetic observations. While these synthetic samples will not refer to real quotes, they will represent plausible observations in the minority class. SMOTE will help to balance our data while limiting the impact of overfitting. Therefore, we will employ SMOTE on our training data.\n",
        "\n",
        "Next, we will consider using cost-sensitive learning to train our neural network. This involves assigning a higher penalty for misclassifying the minority class in our cost function. Consequently, our model will be forced to pay more attention to the minority class during the learning process, potentially improving the model’s ability to correctly identify problematic quotes. The weight assigned to each class can be tuned as a hyperparameter during our process. However, due to computing power constraints, we will instead set the weight to be the inverse of the class distribution in our training data. Notably, if we employ SMOTE, our two classes will make up the same proportion of the training data so would be assigned the same weight under this heuristic. Therefore, while cost-sensitive learning is an effective strategy for addressing class imbalance it may not be necessary if other techniques are employed.\n",
        "\n",
        "Finally, with imbalanced data we should choose appropriate evaluation metrics for comparing and selecting our models at each step. Accuracy should be avoided as an evaluation metric for models when data is imbalanced. This is because classifying all observations as the majority class would result in a high accuracy score (87% in our case) despite having no predictive power for the minority class. For our problem, false-negatives (FN) likely represent a greater potential cost than false-positives (FP). If we fail to identify problematic quotes (FN) we may breach our contractual obligations with actors, potentially leading to legal costs, expensive recasts, and delays in production. We therefore should focus on using recall and other confusion matrix metrics like precision and F1-score to compare our models. These metrics are more relevant as they focus on a model’s ability to correctly identify ‘problematic’ quotes and to distinguish the two classes. Therefore, to address class imbalance, we should prioritise these metrics that assess a model's ability to identify the underrepresented (problematic) quotes effectively.\n",
        "\n",
        "In summary, we will undertake resampling techniques (SMOTE), use appropriate evaluation metrics, and possibly employ cost-sensitive learning to address the class imbalance in our response variable. This should assist in improving our final classifier’s ability to accurately detect problematic quotes.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4GQUmODCmr6X",
      "metadata": {
        "id": "4GQUmODCmr6X"
      },
      "source": [
        "### Q3b - Construct a neural network classifier\n",
        "#### Purpose:\n",
        "\n",
        "In this section we fit several neural network classifiers to the training data, attempting to detect problematic quotes. The validation dataset will be used to evaluate and compare models, as well as to tune hyperparameters.\n",
        "\n",
        "#### References:\n",
        "Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.\n",
        "\n",
        "Bordelon, B., Canatar, A., & Pehlevan, C. (2021). How neural networks escape perils of overparameterization. Harvard Brain Science Initiative. https://brain.harvard.edu/hbi_news/how-neural-networks-escape-perils-of-overparameterization/\n",
        "\n",
        "Kukačka, J., Golkov, V., & Cremers, D. (2017). Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710.10686."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-JnY2X8wmr6g",
      "metadata": {
        "id": "-JnY2X8wmr6g"
      },
      "source": [
        "#### Q3b Step 1 -  Data preparation for neural network\n",
        "**Outline**\n",
        "\n",
        "In this step we setup the training, validation and test data that will be used directly for model training and evaluation.\n",
        "\n",
        "In previous steps, we cleaned the data and constructed a response variable. However, in this step we set up our data so that it is in the correct format for a neural network with Keras. This includes:\n",
        "\n",
        "- Selecting and justifying possible features we want to use and ensuring they are in our datasets\n",
        "- Scaling datasets (using training data to set up scaler); this is needed for neural networks so that one feature does not dominate\n",
        "- Applying SMOTE to the training data to combat class imbalance"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Z7jOLE1jNAoz",
      "metadata": {
        "id": "Z7jOLE1jNAoz"
      },
      "source": [
        "**1. Make copies of the datasets**\n",
        "\n",
        "We first make copies of our training, validation and test datasets. This is so we can transform these datasets for use in a neural network without disrupting the data used for prior analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 522,
      "id": "Zpz08V3b5O2J",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zpz08V3b5O2J",
        "outputId": "641a917a-20b6-4bef-db09-155d92e9a70e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Current train shape:\n",
            "(633, 38)\n",
            "Current validation shape:\n",
            "(80, 33)\n",
            "Current test shape:\n",
            "(80, 33)\n"
          ]
        }
      ],
      "source": [
        "# 1. make copies of datasets that we will transform and use for the neural network\n",
        "train_NN = train.copy()\n",
        "val_NN = val.copy()\n",
        "test_NN = test.copy()\n",
        "\n",
        "print('Current train shape:')\n",
        "print(train_NN.shape)\n",
        "print('Current validation shape:')\n",
        "print(val_NN.shape)\n",
        "print('Current test shape:')\n",
        "print(test_NN.shape)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1lUlZRNOOwcZ",
      "metadata": {
        "id": "1lUlZRNOOwcZ"
      },
      "source": [
        "**2. Select columns/features that are relevant to our model training**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d0RlfxYT6AWX",
      "metadata": {
        "id": "d0RlfxYT6AWX"
      },
      "source": [
        "The training data currently has more columns than the other datasets due to the working columns we created previously. We will remove these working columns as they are not needed for our model training. We also only select the columns/features we need for our model training.\n",
        "\n",
        "We will select the columns associated with the first 7 principal components from the BERT vectorisation (7 PCs was determined to provide a good balance between explainability and dimension reduction). Then we will add additional variables from the training data such as:\n",
        "\n",
        "- The 9 Genre flag columns\n",
        "- Exclamation and question mark flag (may be related to tone and hence whether a quote is problematic)\n",
        "- Sentiment score columns (we may expect lower sentiment scores to be associated with problematic quotes)\n",
        "\n",
        "We also extract out our response column `problematic`. In total, the maximum number of explanatory variables we will use is twenty (20). Not all of these features may be used in our final model.\n",
        "\n",
        "#####**Justifying Feature Selection**\n",
        "\n",
        "- *Principal components from BERT embedding*: these attempt to capture different dimensions of meaning in the quotes which is likely related to whether a quote is problematic or not. For example, a dimension may roughly capture whether a quote refers to negative themes that relate to it being rude. These PCs depend on the quote itself so will be available at deployment and can be used in our model.\n",
        "- *Genre columns*: we may expect different genres of film to have higher rates of problematic quotes. Furthermore, a neural network lets us capture interactions between explanatory variables and so the genre may interact with the interpretation of the embedding. For example, certain embedding values may take on a different meaning when we know the context that the quote is from a fantasy setting. We have argued previously that genre will be known at the time a script is finished and hence will be known at model deployment. Therefore, this variable can be used in our classifier.\n",
        "- *Exclamation and question mark flag*: the presence of these can communicate a particular tone especially when interacting with the above. Certain tones may help predict whether a quote is problematic or not. They are a characteristic of the quote itself and so will be available at deployment.\n",
        "- *Sentiment scores*: negative sentiment may be associated with a quote being problematic so we will test using these variables in our model. Our EDA revealed that the interaction between sentiment score and our response is likely non-linear which a neural network can help capture. These scores are dependent only on the quote and so will be available at deployment.\n",
        "\n",
        "Other columns were not used for several reasons:\n",
        "- **Not available at deployment**: columns like `Year` or `Actor` may not be known at the time of deployment. To avoid leakage we do not include many columns.\n",
        "- **High cardinality**: many columns had high cardinality, such as `Actor`, `Director` and `Movie Name`. Using these variables would add a significant number of parameters to our models. This would likely lead to an over-parameterised model and/or overfitting. Furthermore, we would be unable to account for new values of these variables that would appear in the future.\n",
        "- **Unrelated to response**: Based on our domain knowledge, many columns were unlikely to have a significant relationship with our response. For example, `Movie Name`. We therefore, exclude these columns from our analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 523,
      "id": "y9EzBXnj6dgd",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y9EzBXnj6dgd",
        "outputId": "cfb507f8-db4d-469c-af3d-10a615771b31"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The training data had 633 rows. The neural network training data has 633 rows.\n",
            "We expect to use up to 20 features for the classifier. The NN training data has 20 features and 1 response column.\n",
            "Current train shape:\n",
            "(633, 21)\n",
            "Current validation shape:\n",
            "(80, 21)\n",
            "Current test shape:\n",
            "(80, 21)\n"
          ]
        }
      ],
      "source": [
        "# 2. next we ensure that our datasets have the columns we will use as explanatory/response variables\n",
        "# We may not use all of these variables in our final model selected but ensure we have them all for now:\n",
        "\n",
        "features = ['Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance',\n",
        "            'exclamation_flag', 'question_flag',\n",
        "            'Sentiment_VADER', 'Sentiment_TextBlob',\n",
        "            'problematic']\n",
        "\n",
        "# We want the first 7 principal components from vectorising with BERT and join with our features\n",
        "train_NN = pd.concat([pd.DataFrame(pca_values_withSW), train_NN[features].reset_index(drop=True)], axis = 1, ignore_index=True)\n",
        "val_NN = pd.concat([pd.DataFrame(pca_values_withSW_val), val_NN[features].reset_index(drop=True)], axis = 1, ignore_index=True)\n",
        "test_NN = pd.concat([pd.DataFrame(pca_values_withSW_test), test_NN[features].reset_index(drop=True)], axis = 1, ignore_index=True)\n",
        "\n",
        "# check that concatentation and column selection is correct:\n",
        "print(f'The training data had {train.shape[0]} rows. The neural network training data has {train_NN.shape[0]} rows.')\n",
        "print(f'We expect to use up to {20} features for the classifier. The NN training data has {train_NN.shape[1]-1} features and 1 response column.')\n",
        "\n",
        "column_names = ['PC1','PC2','PC3','PC4','PC5','PC6', 'PC7',\n",
        "                'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance',\n",
        "            'exclamation_flag', 'question_flag',\n",
        "            'Sentiment_VADER', 'Sentiment_TextBlob',\n",
        "            'problematic']\n",
        "\n",
        "# rename columns\n",
        "train_NN.columns = column_names\n",
        "val_NN.columns = column_names\n",
        "test_NN.columns = column_names\n",
        "\n",
        "print('Current train shape:')\n",
        "print(train_NN.shape)\n",
        "print('Current validation shape:')\n",
        "print(val_NN.shape)\n",
        "print('Current test shape:')\n",
        "print(test_NN.shape)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zaHqBQnnAg_A",
      "metadata": {
        "id": "zaHqBQnnAg_A"
      },
      "source": [
        "We see that the three datasets now have the same number of columns which is what we were aiming for. This indicates that our extraneous columns have been removed. We now split the data into explanatory variables and response, then apply scaling."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nquJBe4CRqG4",
      "metadata": {
        "id": "nquJBe4CRqG4"
      },
      "source": [
        "**3. Split features and response**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 524,
      "id": "SSVr88oN_3OQ",
      "metadata": {
        "id": "SSVr88oN_3OQ"
      },
      "outputs": [],
      "source": [
        "# 3. split out features and response:\n",
        "\n",
        "# TRAIN\n",
        "train_NN_X_unscaled = train_NN.drop('problematic', axis = 1)\n",
        "train_y_binary = train_NN['problematic']\n",
        "\n",
        "# VALIDATION\n",
        "val_NN_X_unscaled = val_NN.drop('problematic', axis = 1)\n",
        "val_y_binary = val_NN['problematic']\n",
        "\n",
        "# TEST\n",
        "test_NN_X_unscaled = test_NN.drop('problematic', axis = 1)\n",
        "test_y_binary = test_NN['problematic']\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LNRhcINGRvXO",
      "metadata": {
        "id": "LNRhcINGRvXO"
      },
      "source": [
        "**4. Apply scaling to features**\n",
        "\n",
        "Our possible features have different scales (e.g. sentiment scores vs principal components). Therefore to avoid certain features dominating the training with gradient descent, we scale our features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 525,
      "id": "itFpxWGWB88a",
      "metadata": {
        "id": "itFpxWGWB88a"
      },
      "outputs": [],
      "source": [
        "# 4. Now apply scaling:\n",
        "\n",
        "NN_scaler = StandardScaler()\n",
        "train_NN_X_scaled = NN_scaler.fit_transform(train_NN_X_unscaled) # fit scaler to training data\n",
        "\n",
        "# apply scaler to validation and test data:\n",
        "val_x = NN_scaler.transform(val_NN_X_unscaled)\n",
        "test_x = NN_scaler.transform(test_NN_X_unscaled)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "NWiypmFgStq5",
      "metadata": {
        "id": "NWiypmFgStq5"
      },
      "source": [
        "**5. Address Class Imbalance**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5p67_00vqIsL",
      "metadata": {
        "id": "5p67_00vqIsL"
      },
      "source": [
        "##### Addressing Class Imbalance: SMOTE"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WNys698cSzDB",
      "metadata": {
        "id": "WNys698cSzDB"
      },
      "source": [
        "As discussed previously, we will employ SMOTE to address the class imbalance in the data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 526,
      "id": "NKs29UhEExQ0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NKs29UhEExQ0",
        "outputId": "c12c88b0-ffe7-4f3b-93af-534cef612a2d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.10/dist-packages (0.8.1)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.25.2)\n",
            "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.11.4)\n",
            "Requirement already satisfied: scikit-learn>=0.24 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.4.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24->imbalanced-learn) (3.4.0)\n",
            "Training set before SMOTE Counter({0: 552, 1: 81})\n",
            "Training set before SMOTE Counter({0: 552, 1: 552})\n"
          ]
        }
      ],
      "source": [
        "# 5. Apply SMOTE to help balance our data\n",
        "\n",
        "!pip install imbalanced-learn\n",
        "from imblearn.over_sampling import SMOTE\n",
        "\n",
        "counter = Counter(train_y_binary)\n",
        "\n",
        "print('Training set before SMOTE', counter)\n",
        "\n",
        "# now fit SMOTE to training data and apply the sampling\n",
        "oversample = SMOTE(random_state = random_seed)\n",
        "\n",
        "train_x, train_y_binary2 = oversample.fit_resample(train_NN_X_scaled, train_y_binary)\n",
        "\n",
        "counter = Counter(train_y_binary2)\n",
        "print('Training set before SMOTE', counter)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oy89gImwHScQ",
      "metadata": {
        "id": "oy89gImwHScQ"
      },
      "source": [
        "We can see based on the output that previously we only had 81 observations labelled 'problematic' but after applying SMOTE, we have a 50/50 split and 552 observations labelled 'problematic'. Therefore the class imbalance appears to have been addressed. Let's plot the training data labels vs. the first two PCs to see how the simulated observations look."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 527,
      "id": "Z7YQU1dmINb2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "Z7YQU1dmINb2",
        "outputId": "3d3095bb-b096-45d9-d0d8-1605ff818856"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 332824 characters]",
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# CREATE FIRST PLOT WITH LABELS BEFORE SMOTE:\n",
        "plt.figure(figsize=(12, 6))\n",
        "ax1 = plt.subplot(1,2,1)\n",
        "\n",
        "plt.scatter(x = train_NN_X_scaled[:, 0], y = train_NN_X_scaled[:, 1], c=train_y_binary)\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('PC1')\n",
        "plt.ylabel('PC2')\n",
        "plt.title('Labels before SMOTE')\n",
        "\n",
        "# CREATE SECOND PLOT WITH LABELS AFTER SMOTE:\n",
        "ax1 = plt.subplot(1,2,2)\n",
        "plt.scatter(x = train_x[:, 0], y = train_x[:, 1], c=train_y_binary2 )\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('PC1')\n",
        "plt.ylabel('PC2')\n",
        "plt.title('Labels after SMOTE')\n",
        "\n",
        "# Show the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CkPee9bsKx-t",
      "metadata": {
        "id": "CkPee9bsKx-t"
      },
      "source": [
        "The yellow points in our plots represent 'problematic' quotes. We can see significantly more yellow points after applying SMOTE and we see a slight distinction between the yellow and purple points based on the first two principal components. Obviously, in higher dimensions they may be more clearly separated but we cannot visualise this easily.\n",
        "\n",
        "Sometimes SMOTE can add significant noise to the data and cause worse results, but the distinction in the clusters suggests that it may have worked well in this case. This is a key step in handling the imbalance in our data."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OsTVEw3Ee2Uy",
      "metadata": {
        "id": "OsTVEw3Ee2Uy"
      },
      "source": [
        "**6. Ensure our response is in the correct format (one-hot encode)**\n",
        "\n",
        "We apply one-hot encoding to get our response in the correct format for Keras. This will convert our single response column into two columns, with one column representing 'non-problematic' and the other 'problematic'. While this isn't strictly necessary for a binary response we find that it makes our subsequent analysis easier as we can make our NN predictions represent probabilities of the quote being in either category."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 528,
      "id": "n7StqA3Uk77_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n7StqA3Uk77_",
        "outputId": "43795e70-a9cd-4cea-f646-63a2f559251a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Before:\n",
            "0    0\n",
            "1    0\n",
            "2    0\n",
            "3    0\n",
            "4    1\n",
            "Name: problematic, dtype: int64\n",
            "After:\n",
            "[[1. 0.]\n",
            " [1. 0.]\n",
            " [1. 0.]\n",
            " [1. 0.]\n",
            " [0. 1.]]\n",
            "Our encodings are:\n",
            "['non-problematic', 'problematic']\n"
          ]
        }
      ],
      "source": [
        "# 6. Now apply 'to_categorical' to our y_variables which applies one-hot_encoding\n",
        "# this is needed to get our data in a format for training a NN with Keras\n",
        "\n",
        "train_y = to_categorical(train_y_binary2)\n",
        "\n",
        "val_y = to_categorical(val_y_binary)\n",
        "\n",
        "test_y = to_categorical(test_y_binary)\n",
        "\n",
        "print('Before:')\n",
        "print(train_y_binary.head(5))\n",
        "print('After:')\n",
        "print(train_y[0:5])\n",
        "\n",
        "# Next we create data labels for our response variable\n",
        "# This is needed to make our outputs more readable\n",
        "label_encoder = LabelEncoder()\n",
        "label_encoder.fit(['non-problematic', 'problematic'])\n",
        "\n",
        "data_labels = list(label_encoder.inverse_transform([0, 1]))\n",
        "print('Our encodings are:')\n",
        "print(data_labels)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HFvh2gdrnx28",
      "metadata": {
        "id": "HFvh2gdrnx28"
      },
      "source": [
        "We see that our binary respose column has been converted to two columns. The first column is 1 if the quote is 'problematic' and the other column is 1, otherwise. This prepares our response to be in the correct format for Keras (which requires one-hot encoded matrices)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "txbghYDuMHBU",
      "metadata": {
        "id": "txbghYDuMHBU"
      },
      "source": [
        "**Summary of steps taken**\n",
        "\n",
        "- Selected possible features we would use in our neural network. Justification for the decisions was provided\n",
        "- Split features and response\n",
        "- Applied scaling to our features. This is needed so larger features do not dominate during gradient descent and improves the speed of the algorithm.\n",
        "- Applied SMOTE to simulate new observations and make our dataset more balanced.\n",
        "- Transformed our response to be in the correct format for Keras"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KHE0jggBSbQ2",
      "metadata": {
        "id": "KHE0jggBSbQ2"
      },
      "source": [
        "#### Q3b Step 2 -  Functions for building neural networks\n",
        "**Outline**\n",
        "\n",
        "We will train up several neural networks with different structures, different features and hyperparameters.\n",
        "\n",
        "Since we will be repeatedly training neural networks but with slight adjustments, it makes sense for us to create a function to quickly train up new neural networks.\n",
        "\n",
        "Furthermore, we will be calculating several metrics to assess and compare models at each step so it makes sense to create functions to make this easier.\n",
        "\n",
        "These functions will be defined in this step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 529,
      "id": "Hzq3itrUTkpH",
      "metadata": {
        "id": "Hzq3itrUTkpH"
      },
      "outputs": [],
      "source": [
        "#### FUNCTIONS FOR METRICS ####\n",
        "\n",
        "## Function to plot the confusion matrix for each Neural Network (NN)\n",
        "  # SOURCE: DSA course notes\n",
        "  # Comments are my own\n",
        "\n",
        "def plot_confusion_matrix(cm, classes,\n",
        "                          normalise=False,\n",
        "                          title='Confusion matrix',\n",
        "                          cmap=plt.cm.Blues):\n",
        "    \"\"\"\n",
        "    This function takes a confusion matrix and plots it\n",
        "\n",
        "    Args:\n",
        "        cm: a confusion matrix\n",
        "        classes: Vector of class names\n",
        "        normalise: Boolean for determing if class %s are calculated\n",
        "        title: plot title\n",
        "        cmap: colours to use in the confusion matrix\n",
        "\n",
        "    Returns:\n",
        "        Plots a confusion matrix\n",
        "    \"\"\"\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=75)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "\n",
        "    if normalise:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] * 100\n",
        "        cm = np.round(cm, 2)\n",
        "\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        if i == 0 and j == 0:\n",
        "            color = 'white'\n",
        "        else:\n",
        "            color = 'black' # set colours of the confusion matrix\n",
        "        if normalise:\n",
        "            plt.text(j, i, \"{:.2f}%\".format(cm[i, j]),\n",
        "                     horizontalalignment='center',\n",
        "                     color=color)\n",
        "        else:\n",
        "            plt.text(j, i, str(cm[i, j]),\n",
        "                     horizontalalignment='center',\n",
        "                     color=color)\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.ylabel('True response')\n",
        "    plt.xlabel('Predicted response')\n",
        "\n",
        "\n",
        "## Function to calculate success metrics for each NN model\n",
        "def success_metrics(y, y_pred):\n",
        "    \"\"\"\n",
        "    Calculate accuracy, precision, recall, F1 score and the confusion matrix and\n",
        "    returns the values\n",
        "\n",
        "    Args:\n",
        "        y_true: true values for response\n",
        "        y_pred: predicted values for response\n",
        "\n",
        "    Returns:\n",
        "        Dataframe of results\n",
        "    \"\"\"\n",
        "    scores = dict()\n",
        "    scores['Accuracy'] = metrics.accuracy_score(y, y_pred)\n",
        "    scores['Precision'] = metrics.precision_score(y, y_pred,  average='binary', pos_label=True)\n",
        "    scores['Recall'] = metrics.recall_score(y, y_pred, average='binary', pos_label=True)\n",
        "    scores['F1-score'] = metrics.f1_score(y, y_pred, average='binary', pos_label=True)\n",
        "    scores['Confusion Matrix'] = metrics.confusion_matrix(y, y_pred)\n",
        "\n",
        "    return scores\n",
        "\n",
        "## Function to print out the success metrics of a model\n",
        "def metrics_print(scores, confused = False):\n",
        "    \"\"\"\n",
        "    Takes in a dataframe of metric scores and prints them\n",
        "\n",
        "    Args:\n",
        "        scores: dataframe of metrics, as calculated by success_metrics\n",
        "        confused: Boolean that determine if the confusion matrix is plotted\n",
        "\n",
        "    Returns:\n",
        "        Dataframe of results\n",
        "    \"\"\"\n",
        "    print(('{:^9} '*4).format(*scores.keys()))\n",
        "    print(('{:^9.1%} '*4).format(*scores.values()))\n",
        "\n",
        "    if confused:\n",
        "        plt.figure(figsize=(12, 5))\n",
        "\n",
        "        # Plotting the non-normalized confusion matrix (count)\n",
        "        plt.subplot(1, 2, 1)\n",
        "        plot_confusion_matrix(scores['Confusion Matrix'], classes=data_labels, normalise=False, title='Confusion Matrix for the Validation Dataset (Count)')\n",
        "\n",
        "        # Plotting the normalized confusion matrix (proportion)\n",
        "        plt.subplot(1, 2, 2)\n",
        "        plot_confusion_matrix(scores['Confusion Matrix'], classes=data_labels, normalise=True, title='Confusion Matrix for the Validation Dataset (Proportion)')\n",
        "\n",
        "        plt.tight_layout()\n",
        "        plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 530,
      "id": "Xa3mcrZ3ZGiJ",
      "metadata": {
        "id": "Xa3mcrZ3ZGiJ"
      },
      "outputs": [],
      "source": [
        "## Function to force keras to behave deterministically\n",
        "def keras_deterministic(seed_value= 0):\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  #import tensorflow as tf\n",
        "  tf.compat.v1.set_random_seed(seed_value)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 531,
      "id": "QVtMBAAgZbL7",
      "metadata": {
        "id": "QVtMBAAgZbL7"
      },
      "outputs": [],
      "source": [
        "# Initialise a neural network dictionary to document the information for each version/iteration of the neural network\n",
        "nn_models = dict()\n",
        "\n",
        "# SOURCE: course notes\n",
        "  # I have made my own adjustments and comments to suit my needs\n",
        "\n",
        "# Define a function for training and assessing neural networks\n",
        "def nn_fit(ver,\n",
        "             train_x, train_y, test_x, test_y,\n",
        "             l1_neurons,l1_activation,\n",
        "             learn_rate, epochs, batch_size,\n",
        "             loss_function = 'categorical_crossentropy',\n",
        "             optimiser = 'SGD',\n",
        "             early_stop=False, dropout_rate = None,\n",
        "             verbose=1,\n",
        "             l2_neurons=None, l2_activation=None,\n",
        "             l1_reg_strength=0.0, l2_reg_strength=0.0, plot_toggle = True):\n",
        "\n",
        "    \"\"\"\n",
        "    Trains a neural network based on the data provided. Also calculates\n",
        "    evaluation metrics on the validation data set\n",
        "\n",
        "    Args:\n",
        "        ver: name or number of model, will be used to store in the dictionary\n",
        "        train_x: matrix of training feature data\n",
        "        train_y: vector of training responses\n",
        "        test_x: matrix of test/validation feature data\n",
        "        test_y: vector of test/validation responses\n",
        "\n",
        "        l1_neurons: number of neurons in the first hidden layer\n",
        "        l1_activation: activation function to use in the first hidden layer\n",
        "\n",
        "        learn_rate: learning rate to use in the optimisation algorithm\n",
        "        epochs: number of epochs\n",
        "        batch_size: number of observations in each batch\n",
        "        loss_function: the loss function to minimise with the optimisation\n",
        "          algorithm\n",
        "        optimiser: optimisation algorithm to use to minimise the loss function\n",
        "          only 'Adam' and 'SGD' are available\n",
        "        early_stop: Boolean, for determing if the optimiser should stop early\n",
        "        dropout_rate: can apply a dropout rate to the training neurons which may\n",
        "          help with overfitting (a form of regularisation).\n",
        "          Set to 'None' for no dropout_rate.\n",
        "        verbose: set to 0 for training to be silent, set to 1 to show progress bar\n",
        "\n",
        "        l2_neurons: number of neurons in the second hidden layer, set to 'None'\n",
        "          for no second hidden layer.\n",
        "        l2_activation: activation function to use in the second hidden layer\n",
        "\n",
        "        l1_reg_strength: hyperparameter for controlling strength of regularisation\n",
        "          for first hidden layer\n",
        "        l2_reg_strength: hyperparameter for controlling strength of regularisation\n",
        "          for second hidden layer\n",
        "\n",
        "        plot_toggle: Boolean, let's us toggle on and off the training output. Used\n",
        "          to make the report cleaner.\n",
        "\n",
        "    Returns:\n",
        "        Prints results and stores information in a dictionary called 'nn_models'\n",
        "    \"\"\"\n",
        "\n",
        "\n",
        "    # Use Adam optimiser as the use of adaptive learning rates allows ADAM to automatically adjust the step size of the optimization algorithm, which can help to\n",
        "    # avoid overshooting the global minimum and improve the convergence speed of the algorithm. It is suitable for most non-convex optimisation problems with large data sets and high dimensional space\n",
        "    if optimiser == 'Adam':\n",
        "      opt = Adam(learning_rate = learn_rate)\n",
        "    else:\n",
        "      opt = tf.keras.optimizers.SGD(learning_rate = learn_rate)\n",
        "\n",
        "    # Set up early stopping\n",
        "    if early_stop != False:\n",
        "        early_stopping_callback = EarlyStopping(monitor=\"val_loss\", # Quantity to be monitored is the validation loss\n",
        "                                                min_delta=0.001, # Minimum change in the monitored quality to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.\n",
        "                                                patience=30, # Number of epochs with no improvement after which training will be stopped\n",
        "                                                verbose=0, restore_best_weights=True)\n",
        "        callbacks = [early_stopping_callback]\n",
        "    else:\n",
        "        callbacks = None\n",
        "\n",
        "    # Set up class weights for handling imbalance\n",
        "      # NOTE: we have already partially handled the class imbalance using SMOTE\n",
        "      # After selecting model we may try fitting on data without SMOTE to see if it performs better (in that case will use class weights)\n",
        "    label_encoded_train_y = np.argmax(train_y, axis=1)\n",
        "    unique_classes = np.unique(label_encoded_train_y)\n",
        "    class_weights = compute_class_weight('balanced', classes=unique_classes, y=label_encoded_train_y)\n",
        "    class_weights_dict = {cls: weight for cls, weight in zip(unique_classes, class_weights)}\n",
        "\n",
        "    # Set up model\n",
        "    tf.random.set_seed(random_seed)\n",
        "    model = Sequential()\n",
        "\n",
        "    # Define first hidden layer\n",
        "    model.add(Dense(units = l1_neurons, input_dim = train_x.shape[1], activation = l1_activation, kernel_regularizer=l2(l1_reg_strength)))\n",
        "    if dropout_rate != None:\n",
        "      model.add(Dropout(dropout_rate, seed = 50)) # add dropout rate to prevent overfitting\n",
        "\n",
        "    # Second hidden layer\n",
        "    if l2_neurons != None:\n",
        "        model.add(Dense(units = l2_neurons, activation = l2_activation, kernel_regularizer=l2(l2_reg_strength)))\n",
        "\n",
        "    # Output layer\n",
        "      # we use softmax for our output layer activation function so that the response values represent probabilities\n",
        "    model.add(Dense(train_y.shape[1], activation='softmax'))\n",
        "\n",
        "    # Compile model\n",
        "      # loss function is what we are trying to minimise\n",
        "      # optimiser is how we minimise i.e. the algorithm\n",
        "      # metrics will be calculated on training and validation set to assess model\n",
        "    model.compile(loss = loss_function, optimizer = opt, metrics = ['accuracy'])\n",
        "\n",
        "\n",
        "    # Create control object to save results to\n",
        "    nn_models[ver]={}\n",
        "    start_time = time.time()\n",
        "\n",
        "    # Now start training model:\n",
        "    history = model.fit(train_x, train_y,\n",
        "                        validation_data = (test_x, test_y),\n",
        "                        epochs = epochs,\n",
        "                        batch_size = batch_size,\n",
        "                        class_weight = class_weights_dict,\n",
        "                        verbose = verbose,\n",
        "                        callbacks = callbacks)\n",
        "    nn_models[ver]['history'] = history\n",
        "    end_time = time.time()\n",
        "    print('Time taken to run this Neural Network model is: {:.0f} mins'.format(round((end_time - start_time)/60,2)))\n",
        "\n",
        "    # Plot learning curve (shows each metric vs number of epochs)\n",
        "      # will create subplots so can compare side-by-side\n",
        "\n",
        "    epochs = range(1, len(history.history['loss']) + 1)\n",
        "    metrics = ['accuracy', 'loss']\n",
        "\n",
        "    if plot_toggle:\n",
        "        fig, axes = plt.subplots(nrows=1, ncols=len(metrics), figsize=(20, 5))\n",
        "\n",
        "        for i, metric in enumerate(metrics):\n",
        "            train_metric = history.history[metric]\n",
        "            val_metric = history.history['val_'+metric]\n",
        "            ax = axes[i]\n",
        "\n",
        "            ax.plot(epochs, train_metric, label='Training')\n",
        "            ax.plot(epochs, val_metric,  label='Validation')\n",
        "\n",
        "            ax.set_title(metric)\n",
        "            ax.set_xlabel('Epochs')\n",
        "            ax.set_ylabel(metric)\n",
        "            ax.legend()\n",
        "\n",
        "        plt.show()\n",
        "\n",
        "    # Generate confusion matrix\n",
        "    nn_models[ver]['y_prediction_prob'] = model.predict(test_x) # get model predictions\n",
        "    nn_models[ver]['y_prediction_class'] = np.argmax(nn_models[ver]['y_prediction_prob'], axis=1) # get class with greatest probability as predicted class\n",
        "    nn_models[ver]['y_actual_class'] = np.argmax(test_y, axis=1)\n",
        "\n",
        "    # Print success matrix\n",
        "    nn_models[ver]['scores'] = success_metrics(nn_models[ver]['y_actual_class'],nn_models[ver]['y_prediction_class'])\n",
        "    metrics_print(nn_models[ver]['scores'], confused = plot_toggle)\n",
        "\n",
        "    # Print classification report\n",
        "    nn_models[ver]['classification_report'] = classification_report(nn_models[ver]['y_actual_class'], nn_models[ver]['y_prediction_class'], target_names=data_labels)\n",
        "\n",
        "    # Calculate AUC for the training set:\n",
        "    y_train_pred_prob = model.predict(train_x)\n",
        "    train_auc = roc_auc_score(train_y, y_train_pred_prob)\n",
        "    nn_models[ver]['scores']['train_AUC'] = train_auc\n",
        "\n",
        "    # Calculate AUC for validation/test set\n",
        "    test_auc = roc_auc_score(test_y, nn_models[ver]['y_prediction_prob'])\n",
        "    nn_models[ver]['scores']['val_AUC'] = test_auc\n",
        "\n",
        "    if plot_toggle:\n",
        "      print(nn_models[ver]['classification_report'])\n",
        "      print(\"Training AUC:\", train_auc)\n",
        "      print(\"Validation AUC:\", test_auc)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 532,
      "id": "pIwV_BqbaA6s",
      "metadata": {
        "id": "pIwV_BqbaA6s"
      },
      "outputs": [],
      "source": [
        "### FUNCTIONS FOR COMPARING OUTPUTS\n",
        "\n",
        "def compare_icon(new_val, old_val):\n",
        "  \"\"\"\n",
        "  This function compares two values and returns a symbol indicating their difference.\n",
        "\n",
        "  Args:\n",
        "      val1 (float): The first value.\n",
        "      val2 (float): The second value.\n",
        "\n",
        "  Returns:\n",
        "      str: '+' if val1 > val2, '-' if val1 < val2, '=' otherwise.\n",
        "  \"\"\"\n",
        "  difference = new_val - old_val\n",
        "  if difference > 0:\n",
        "    return '+'\n",
        "  elif difference < 0:\n",
        "    return '-'\n",
        "  else:\n",
        "    return '='\n",
        "\n",
        "def model_comparison(new_model, comparison_model):\n",
        "  \"\"\"\n",
        "  This function compares the metrics of two neural network models.\n",
        "\n",
        "  Args:\n",
        "      new_model: The name of the latest model\n",
        "      comparison_model: The name of the model you wish to compare to\n",
        "\n",
        "  \"\"\"\n",
        "\n",
        "  # create data frame to hold results\n",
        "  df_results = pd.DataFrame(columns=['Metric', 'New Model', 'Comparison Model', 'Change'])\n",
        "\n",
        "  # Now add metrics of the two models to compare:\n",
        "  df_results.loc[len(df_results)] =['Precision', nn_models[new_model]['scores']['Precision'],\n",
        "                                    nn_models[comparison_model]['scores']['Precision'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['Precision'], nn_models[comparison_model]['scores']['Precision'])]\n",
        "  df_results.loc[len(df_results)] =['Recall', nn_models[new_model]['scores']['Recall'],\n",
        "                                    nn_models[comparison_model]['scores']['Recall'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['Recall'], nn_models[comparison_model]['scores']['Recall'])]\n",
        "  df_results.loc[len(df_results)] =['F1-score', nn_models[new_model]['scores']['F1-score'],\n",
        "                                    nn_models[comparison_model]['scores']['F1-score'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['F1-score'], nn_models[comparison_model]['scores']['F1-score'])]\n",
        "  df_results.loc[len(df_results)] =['Accuracy', nn_models[new_model]['scores']['Accuracy'], nn_models[comparison_model]['scores']['Accuracy'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['Accuracy'], nn_models[comparison_model]['scores']['Accuracy'])]\n",
        "  df_results.loc[len(df_results)] =['Validation AUC', nn_models[new_model]['scores']['val_AUC'], nn_models[comparison_model]['scores']['val_AUC'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['val_AUC'], nn_models[comparison_model]['scores']['val_AUC'])]\n",
        "  df_results.loc[len(df_results)] =['Training AUC', nn_models[new_model]['scores']['train_AUC'], nn_models[comparison_model]['scores']['train_AUC'],\n",
        "                                    compare_icon(nn_models[new_model]['scores']['train_AUC'], nn_models[comparison_model]['scores']['train_AUC'])]\n",
        "\n",
        "  print('Metrics for model: '+ str(new_model) +' compared to model: '+ str(comparison_model))\n",
        "  print(df_results)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "G4jH4l6emr6i",
      "metadata": {
        "id": "G4jH4l6emr6i"
      },
      "source": [
        "#### Q3b Step 3 - Discussion of model output interpretation\n",
        "**Outline**\n",
        "\n",
        "We discuss here how we will compare models and the meaning of the model output in our problem context.\n",
        "\n",
        "**Output Plots and Overfitting**\n",
        "\n",
        "- After training a model, two plots are shown of the accuracy and cross-entropy (loss). These charts are used to identify overfitting and underfitting in the model training. If the charts show improvements in the training metrics but flat or degrading changes in the validation metrics, this likely indicates overfitting. This directs us to make adjustments to the model like reducing the model complexity.\n",
        "- Similarly, if the chart shows that the validation loss has not yet plateaued, it may indicate that the model has underfit.\n",
        "- Our small validation dataset results in the validation accuracy being quite volatile and so we tend to focus on the loss chart.\n",
        "\n",
        "**Metrics and considering class imbalance**\n",
        "\n",
        "As there is significant class imbalance in the validation dataset, **we do not use accuracy to assess and compare models** (see Q3a for a discussion of why). We therefore focus on recall, precision, F1-score and validation AUC to assess model performance. These metrics focus on a model's ability to correctly identify problematic quotes.\n",
        "- A higher recall suggests a lower false negative rate. In our context, false negatives are costly due to contractual risks with actors. Recall is therefore very important!\n",
        "- A higher precision results in less false positives. This is important as it may be costly for us to check lots of quotes that are incorrectly flagged as problematic. A few false positives is tolerable as a human can quickly verify the model label. We therefore slightly favour recall over precision.\n",
        "- F1-score provides a balance between recall and precision as the two metrics can be at odds with each other. This will hence be our primary comparison metric for training.\n",
        "- However, given that we have a small validation dataset, over-optimising the validation F1-score may make the model biased towards the particular quotes in the validation dataset. We hence use the validation AUC as a supplementary metric for comparing models. A moderate increase in AUC can indicate a genuine improvement in the model.\n",
        "\n",
        "These metrics will be used to select/compare models as we iteratively make adjustments. In some cases, selecting a model after a change requires professional judgment due to contrasting signals from the evaluation metrics. A justification of the choice will be provided in these cases."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MvoBmJ6TY1rQ",
      "metadata": {
        "id": "MvoBmJ6TY1rQ"
      },
      "source": [
        "#### Q3b Step 4 - Train Initial Model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AC_j9PuyprFO",
      "metadata": {
        "id": "AC_j9PuyprFO"
      },
      "source": [
        "\n",
        "**Outline**\n",
        "\n",
        "In this step, we train up an initial neural network classifier that will be used as the baseline for future comparisons.\n",
        "\n",
        "For this initial model, we will use:\n",
        "\n",
        "- **Features:** Only the first 7 PCs of our word embedding via BERT. It is better to start simpler and add complexity later hence why we only use a few features initially.\n",
        "- **Architecture:** One hidden layer with 20 neurons. Again we start simple and will add complexity incrementally to determine an optimal point.\n",
        "- The ReLU activation function for our hidden layer. Research suggests that this activation function performs well for the majority of tasks (Krizhevsky, Sutskever, Hinton, 2012) and so is a good starting point.\n",
        "- The softmax activation function for our output layer. This is needed as we want our output to represent probabilities that sum to one.\n",
        "- Use a learning rate of 0.01. An arbitrary initial pick that is neither too big or too small\n",
        "- 200 epochs. We choose a relatively later epoch initially as it is better to overfit initially and use regularisation etc. to simplify the model and reduce overfitting later.\n",
        "- Batch size of 64. A recommended default batch size for GPUs. We choose a batch size that is commensurate with the size of our training set data but also do not want to choose a value too small so that the training is impacted significantly by noise. This will be tuned later.\n",
        "- Use stochastic gradient descent (really mini-batch gradient descent). This is the standard algorithm for training neural networks; it is robust and reliable. Later we will consider another algorithm, Adam, which is also quite robust, fast and may help in converging to a global minimum.\n",
        "- No regularisation. We will test regularisation at later steps once we have tuned the features and architecture. This may result in our intial models suffering from overfitting.\n",
        "\n",
        "We will also use our simulated data from applying SMOTE to address imbalance in the dataset.\n",
        "\n",
        "**References**\n",
        "\n",
        "Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 533,
      "id": "kyBmZEMXte1x",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "kyBmZEMXte1x",
        "outputId": "7bbcc262-3e23-4f97-a902-38b490ece14b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 1 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66980 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 37ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     28.6%     46.2%     35.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51860 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.78      0.83        67\n",
            "    problematic       0.29      0.46      0.35        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.58      0.62      0.59        80\n",
            "   weighted avg       0.78      0.72      0.75        80\n",
            "\n",
            "Training AUC: 0.8437073356437723\n",
            "Validation AUC: 0.679678530424799\n"
          ]
        }
      ],
      "source": [
        "## Run initial model\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=1,\n",
        "         train_x=train_x[:,0:7], # only select first 7 features (the PCs)\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,0:7],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=20,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZyM9eO1Tp0ma",
      "metadata": {
        "id": "ZyM9eO1Tp0ma"
      },
      "source": [
        "#####**Observations and Summary**\n",
        "\n",
        "We see that our initial model has achieved validation accuracy of 72.5% compared to 76% on the training data. Although, accuracy may not be an appropriate metric for the validation data due to imbalance, the accuracy is lower than 84% which would be obtained if we classified all as 'non-problematic'.\n",
        "\n",
        "From the confusion matrix we see that that the majority of quotes are classified as non-problematic. This is expected given that the validation data is heaviliy imbalanced towards this class. Let us consider the validation recall and precision:\n",
        "- When a quote was actually problematic, the model only identified it correctly 46% of the time (recall)\n",
        "- Of the quotes identified as problematic, only 29% of them were actually problematic (precision).\n",
        "\n",
        "Therefore, the model is not very accurate at identifying problematic quotes and suggests significant room for improvement.\n",
        "\n",
        "The poor metric values indicate that the model may be underfitting and that additional model complexity may be needed to improve model performance.\n",
        "In the next step, we intend to do this by altering the features used and the architecture to improve the precision, recall and AUC (our main metrics for assessing model fit)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3w-5yT2YaWX2",
      "metadata": {
        "id": "3w-5yT2YaWX2"
      },
      "source": [
        "#### Q3b Step 5 - Initial Architecture Experimentation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Eep-a65rHF3Q",
      "metadata": {
        "id": "Eep-a65rHF3Q"
      },
      "source": [
        "**Outline**\n",
        "\n",
        "Next we intend to try different neural network architectures, which involves adjusting:\n",
        "\n",
        "- The number of hidden layers\n",
        "- The number of nodes in each hidden layer\n",
        "\n",
        "As well as feature selection, this is likely to have the greatest impact on our model performance. By increasing the number of neurons and considering more complex architectures we may be able to address the underfitting observed in the previous step. Increasing model complexity may result in a more complex decision boundary that allows the model to better identify problematic quotes.\n",
        "\n",
        "We choose to not use more than two hidden layers given that our training data is limited and because our task is not complex enough to warrant a deeper network. We use experimentation to test different architectures and compare their metrics to signal which architectures are preferable.\n",
        "\n",
        "Although, we look at architecture here, model improvement will be *iterative* and we will adjust the architecture again after exploring different features. This is because adjustments are not independent."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ThKdgyXpXRRC",
      "metadata": {
        "id": "ThKdgyXpXRRC"
      },
      "source": [
        "---\n",
        "**Version 2.1: 1 Layer, 12 neurons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 534,
      "id": "nsn6romjV5J2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "nsn6romjV5J2",
        "outputId": "f414b8e0-8040-47bb-c37e-dc5ffb36ec1d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 1 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 64304 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 38ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     30.4%     53.8%     38.9%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 54344 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.89      0.76      0.82        67\n",
            "    problematic       0.30      0.54      0.39        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.60      0.65      0.61        80\n",
            "   weighted avg       0.80      0.72      0.75        80\n",
            "\n",
            "Training AUC: 0.838909236504936\n",
            "Validation AUC: 0.6544202066590126\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 2.1: decrease number of neurons in first layer\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=2.1,\n",
        "         train_x=train_x[:,0:7], # only select first 7 features (the PCs)\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,0:7],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=12,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 535,
      "id": "A5Jw7dSx3HgM",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A5Jw7dSx3HgM",
        "outputId": "af4fb3e8-f76c-4ee0-ae70-57e90b227017"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 2.1 compared to model: 1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.304348          0.285714      +\n",
            "1          Recall   0.538462          0.461538      +\n",
            "2        F1-score   0.388889          0.352941      +\n",
            "3        Accuracy   0.725000          0.725000      =\n",
            "4  Validation AUC   0.654420          0.679679      -\n",
            "5    Training AUC   0.838909          0.843707      -\n"
          ]
        }
      ],
      "source": [
        "# compare model with initial model\n",
        "model_comparison(2.1, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "meJf6dB2GdMK",
      "metadata": {
        "id": "meJf6dB2GdMK"
      },
      "source": [
        "**Summary**  \n",
        "**What was changed and why?**\n",
        "\n",
        "The number of neurons was decreased to 12. Previously, 20 neurons was chosen arbitrarily. Therefore, as an experiment, we test model performance on fewer neurons.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Based on the above metrics, the model performed marginally worse than our baseline model for all metrics except recall. Also, we see that the validation loss has plateaued, indicating that longer training will not improve this model.\n",
        "\n",
        "Overall the model performance is **marginally worse than our baseline model**. This suggests that the model architecture does not need to be simpler than our baseline model.\n",
        "\n",
        "We now experiment with adding more complexity to the model by adding more neurons:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KUiZoTkKarWN",
      "metadata": {
        "id": "KUiZoTkKarWN"
      },
      "source": [
        "---\n",
        "**Version 2.2: 1 Layer, 60 neurons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 536,
      "id": "Ow1N_eoKarzk",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Ow1N_eoKarzk",
        "outputId": "28fb84b0-187c-411f-87b3-44bb15600cc4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 1 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 68540 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 17ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     26.3%     38.5%     31.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51656 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.79      0.83        67\n",
            "    problematic       0.26      0.38      0.31        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.57      0.59      0.57        80\n",
            "   weighted avg       0.77      0.72      0.74        80\n",
            "\n",
            "Training AUC: 0.8611701848351188\n",
            "Validation AUC: 0.7152698048220436\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 2.2: increase number of neurons in first layer\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=2.2,\n",
        "         train_x=train_x[:,0:7], # only select first 7 features (the PCs)\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,0:7],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 537,
      "id": "a8x0h7k_3VQK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a8x0h7k_3VQK",
        "outputId": "5bd8c961-3f2a-4e5e-e783-d5027ca23788"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 2.2 compared to model: 1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.263158          0.285714      -\n",
            "1          Recall   0.384615          0.461538      -\n",
            "2        F1-score   0.312500          0.352941      -\n",
            "3        Accuracy   0.725000          0.725000      =\n",
            "4  Validation AUC   0.715270          0.679679      +\n",
            "5    Training AUC   0.861170          0.843707      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with initial model\n",
        "model_comparison(2.2, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Pbm8d-V7cdBD",
      "metadata": {
        "id": "Pbm8d-V7cdBD"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We increased the number of neurons to 60. Our baseline model performed relatively poorly, suggesting that more complexity is needed. Therefore we considered adding more neurons.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Based on precision, recall and F1 score, this model performed in line with our baseline model, indicating it is not any better at identifying problematic quotes correctly. It appears that it is making the same predictions as our baseline model. However, the validation AUC is higher. This is likely a result of the higher probabilities assigned to each class (the model is more confident in its predictions)\n",
        "\n",
        "**Other observations**\n",
        "\n",
        "The model has likely overfit. We see that the validation accuracy does not change much after 100 epochs despite the training version improving. After 160 epochs the validation accuracy gets worse while the training gets better. Similarly for the loss, the training loss improves rapidly while the validation loss hardly changes and flattens out towards the end.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "This model is perhaps marginally better than our baseline model. We next experiment with different number of layers.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oyQdaZxkeX82",
      "metadata": {
        "id": "oyQdaZxkeX82"
      },
      "source": [
        "---\n",
        "**Version 2.3: 2 Layers, 30 + 30 neurons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 538,
      "id": "rloc0q50eX82",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "rloc0q50eX82",
        "outputId": "8cfb410e-f803-40f4-a1b2-355ad7df0f51"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 77752 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 23ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  76.2%     33.3%     46.2%     38.7%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51772 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.89      0.82      0.85        67\n",
            "    problematic       0.33      0.46      0.39        13\n",
            "\n",
            "       accuracy                           0.76        80\n",
            "      macro avg       0.61      0.64      0.62        80\n",
            "   weighted avg       0.80      0.76      0.78        80\n",
            "\n",
            "Training AUC: 0.8874153276622558\n",
            "Validation AUC: 0.7083811710677382\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 2.3: two layers\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=2.3,\n",
        "         train_x=train_x[:,0:7], # only select first 7 features (the PCs)\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,0:7],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=30,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=30,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 539,
      "id": "tV99LPf73aAb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tV99LPf73aAb",
        "outputId": "9cc49b8b-fc78-4c47-98b5-10ad2a5ebf88"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 2.3 compared to model: 1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.333333          0.285714      +\n",
            "1          Recall   0.461538          0.461538      =\n",
            "2        F1-score   0.387097          0.352941      +\n",
            "3        Accuracy   0.762500          0.725000      +\n",
            "4  Validation AUC   0.708381          0.679679      +\n",
            "5    Training AUC   0.887415          0.843707      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with initial model\n",
        "model_comparison(2.3, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "RZZc1Y00iy5u",
      "metadata": {
        "id": "RZZc1Y00iy5u"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "Using multiple layers allows us to create more complex decision boundaries than just a single layer. Given that a quote can be problematic based on very disparate concepts (swearing, politics, religion etc.), it suggests a more complex decision boundary may be appropriate. Furthermore, previous increases in the number of neurons for a single-layer NN have not yielded significant improvements in performance. Therefore, we try using two layers as it may due to more substantial gains in performance.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Based on all metrics, this model appears to be **better than our baseline model** although the difference is **marginal**. The model also has maringally higher precision than model 2.2, suggesting it is better at correctly identifying problematic quotes. However model 2.3's AUC is marginally worse than 2.2; the models therefore appear similar in performance.\n",
        "\n",
        "**Other observations**\n",
        "\n",
        "There appears to be overfitting given that the training accuracy and loss is improving rapidly in the final epochs while the validation versions get worse.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "Given this model is marginally better than model 2.2, it indicates that the more complex decision boundary from multiple layers is suited to this task. Given this, we next experiment with changing the number of nodes in the two layers.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "royasPOYk7_P",
      "metadata": {
        "id": "royasPOYk7_P"
      },
      "source": [
        "---\n",
        "**Version 2.4: 2 Layers, 40 + 20 neurons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 540,
      "id": "Cmo58mV7k7_X",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Cmo58mV7k7_X",
        "outputId": "4fe78436-667f-4fbf-a8fd-9d0dacf0f689"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 79064 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 29ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     26.3%     38.5%     31.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51656 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.79      0.83        67\n",
            "    problematic       0.26      0.38      0.31        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.57      0.59      0.57        80\n",
            "   weighted avg       0.77      0.72      0.74        80\n",
            "\n",
            "Training AUC: 0.9042972852341944\n",
            "Validation AUC: 0.6681974741676235\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 2.4: two layers, change allocation of neurons\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=2.4,\n",
        "         train_x=train_x[:,0:7], # only select first 7 features (the PCs)\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,0:7],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=40,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=20,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 541,
      "id": "kc-K23TS3gx_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kc-K23TS3gx_",
        "outputId": "52da8f9d-5525-4609-bb66-6926d95dd98c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 2.4 compared to model: 1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.263158          0.285714      -\n",
            "1          Recall   0.384615          0.461538      -\n",
            "2        F1-score   0.312500          0.352941      -\n",
            "3        Accuracy   0.725000          0.725000      =\n",
            "4  Validation AUC   0.668197          0.679679      -\n",
            "5    Training AUC   0.904297          0.843707      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with initial model\n",
        "model_comparison(2.4, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "z0LQxVwCk7_X",
      "metadata": {
        "id": "z0LQxVwCk7_X"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "The allocation of neurons between the first and second layer was changed to allocate more to the first layer. The first layer can uncover key features of the intial data so more neurons may be needed here.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "This model appears to perform marginally worse than our baseline model at correctly detecting problematic quotes. This is exemplified by a slightly lower F1-score and lower recall. Recall is a particularly important metric for our business context so the worse performance here suggests this model is unacceptable.\n",
        "\n",
        "Notably, the training metrics appear to be substantially better for this model. This is a result of overfitting. Although it may be the specific validation set that makes the model look worse (the validation set is quite small).\n",
        "\n",
        "Overall, we choose not to proceed with this model as better models have been trained.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_ibDcn1Cmr6j",
      "metadata": {
        "id": "_ibDcn1Cmr6j"
      },
      "source": [
        "#####**Section Summary**  \n",
        "\n",
        "In this step we experimented with different model architectures while keeping the features constant. There was not a clear 'winner' model from this step, with only marginal improvements over the baseline model.\n",
        "\n",
        "Overall, we found that **a single-layer neural network with 60 neurons (model 2.2)** and a **two-layer neural network with 60 neurons (model 2.3)** to be the architectures that provided the 'best' results. They were only marginally better than the baseline model however.\n",
        "\n",
        "- Model 2.2: Showed signs of overfitting, suggesting early stopping or regularisation might help.\n",
        "\n",
        "- Model 2.3: Had a more complex decision boundary due to two layers, potentially better suited for capturing the nuances of \"problematic\" quotes (e.g. religion, sex, rudeness). This may be even more relevant with additional features.\n",
        "\n",
        "Both models performed similarly and so both will be further explored as we refine features and hyperparameters. While the impact of architecture was less significant initially, it may become more prominent with new features.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "m02B7wFfcb_k",
      "metadata": {
        "id": "m02B7wFfcb_k"
      },
      "source": [
        "#### Q3b Step 6 - Feature Experimentation"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vSDf8gPus2DH",
      "metadata": {
        "id": "vSDf8gPus2DH"
      },
      "source": [
        "\n",
        "**Outline**\n",
        "\n",
        "Adjusting the architecture yielded marginal improvements in model performance. To enhance model complexity and potentially improve performance, we will explore incorporating additional features.\n",
        "\n",
        "We had two candidate models from the architecture experiment stage: Model 2.2 (1 hidden layer) and Model 2.3 (2 hidden layers). Typically we would evaluate which features are informative on both architectures as they may be different. However due to limits in computing power and for the sake of brevity we do our feature experiments mostly on model 2.2.\n",
        "\n",
        "We will sequentially add features in the following order, based on our ranking of importance using domain knowledge:\n",
        "\n",
        "- Sentiment scores\n",
        "- Exclamation and question mark flags\n",
        "- Genre\n",
        "\n",
        "This approach allows us to gradually increase model complexity and assess the impact of each feature. *See Q3b Step 1 for details on why these features were chosen and others were not*."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LDYZsPfKwBWM",
      "metadata": {
        "id": "LDYZsPfKwBWM"
      },
      "source": [
        "---\n",
        "**Version 3.1: 1 Layer, 60 neurons, add `Sentiment_VADER`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 542,
      "id": "d-OrSeuJxTtP",
      "metadata": {
        "id": "d-OrSeuJxTtP"
      },
      "outputs": [],
      "source": [
        "# create function for us to easily select the features for our NN\n",
        "def get_column_indices(column_names, selected_features):\n",
        "  return [column_names.index(name) for name in column_names if name in selected_features]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 543,
      "id": "3lbsBYMqwA6S",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3lbsBYMqwA6S",
        "outputId": "4e98d5b9-07ca-4565-e86e-de36fe53be5e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71272 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 29ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  75.0%     31.6%     46.2%     37.5%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51620 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.89      0.81      0.84        67\n",
            "    problematic       0.32      0.46      0.38        13\n",
            "\n",
            "       accuracy                           0.75        80\n",
            "      macro avg       0.60      0.63      0.61        80\n",
            "   weighted avg       0.79      0.75      0.77        80\n",
            "\n",
            "Training AUC: 0.8740252835538753\n",
            "Validation AUC: 0.6567164179104478\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.1: single layer, add Sentiment_VADER\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.1,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 544,
      "id": "lXWPuvqh3n_x",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lXWPuvqh3n_x",
        "outputId": "babadfbb-f98a-48b0-cc2e-84c26da3729a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.1 compared to model: 2.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.315789          0.263158      +\n",
            "1          Recall   0.461538          0.384615      +\n",
            "2        F1-score   0.375000          0.312500      +\n",
            "3        Accuracy   0.750000          0.725000      +\n",
            "4  Validation AUC   0.656716          0.715270      -\n",
            "5    Training AUC   0.874025          0.861170      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with chosen model at last step\n",
        "model_comparison(3.1, 2.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eaLC2SpIyYxU",
      "metadata": {
        "id": "eaLC2SpIyYxU"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "Based domain knowledge, `Sentiment_VADER` is likely related to whether a quote is problematic or not. Our EDA also indicated a possible non-linear relationship between this feature and our response. Therefore, we add it to our candidate model (model 2.2). `Sentiment_VADER` had a more volatile distribution than `Sentiment_TextBlob` so may be better at distinguishing quotes.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "There appears to be a marginal improvement in the model's metrics although it is incredibly minor. A slight improvement in precision without a loss of recall may actually represent a material cost-saving for the business as less problematic quotes have to be verified.\n",
        "\n",
        "Overall, this feature likely does improve model performance. Its inclusion in the model may become more important as it interacts with additional features added later.\n",
        "\n",
        "We now experiment with adding the feature to our two-layer NN (model 2.3) to see if a similar result is achieved."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JwnnQf-j1gyv",
      "metadata": {
        "id": "JwnnQf-j1gyv"
      },
      "source": [
        "---\n",
        "**Version 3.2: 2 Layers, 30+30 neurons, add `Sentiment_VADER`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 545,
      "id": "ylWud7DBzl1A",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ylWud7DBzl1A",
        "outputId": "2cd751ea-8874-4acd-9138-b2a30b473d29"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 77068 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     27.8%     38.5%     32.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51756 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.81      0.84        67\n",
            "    problematic       0.28      0.38      0.32        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.57      0.60      0.58        80\n",
            "   weighted avg       0.77      0.74      0.75        80\n",
            "\n",
            "Training AUC: 0.9020590474690191\n",
            "Validation AUC: 0.6326061997703789\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.2: two layer, add Sentiment_VADER\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.2,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=30,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=30,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 546,
      "id": "0rAg7E9I3vNI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0rAg7E9I3vNI",
        "outputId": "cf39c043-fb54-4e16-bfe9-b4fdbb95d004"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.2 compared to model: 2.3\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.277778          0.333333      -\n",
            "1          Recall   0.384615          0.461538      -\n",
            "2        F1-score   0.322581          0.387097      -\n",
            "3        Accuracy   0.737500          0.762500      -\n",
            "4  Validation AUC   0.632606          0.708381      -\n",
            "5    Training AUC   0.902059          0.887415      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with two-layer NN at last step\n",
        "model_comparison(3.2, 2.3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lPpDa2J3z0UZ",
      "metadata": {
        "id": "lPpDa2J3z0UZ"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "`Sentiment_VADER` was added as a feature to our two-layer NN (see model above for discussion of the why the variable was added). We are unsure of the correct architecture and hypothesise that a two-layer architecture may be more appropriate as we add additional features hence why we test both model structures. (Note this step is demonstrative and in the future we will only test additional features on the single layer network).\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "**This model appears to be worse than model 2.3** based on our main metrics (e.g. worse recall and precision). There is clear evidence of overfitting as we see the validation loss deteriorates in the final epochs while the training loss improves. Therefore, while the results appear worse, implementing techniques like early stopping may drastically improve this model and similarly indicate that `Sentiment_VADER` is an informative feature.\n",
        "\n",
        "Next, we experiment with using `Sentiment_TextBlob` instead of `Sentiment_VADER` (we restrict our analysis to just our single-hidden layer NN)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kIwAKKpx1kpl",
      "metadata": {
        "id": "kIwAKKpx1kpl"
      },
      "source": [
        "---\n",
        "**Version 3.3: 1 Layer, 60 neurons, add `Sentiment_TextBlob`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 547,
      "id": "zqzWC_eL1uve",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "zqzWC_eL1uve",
        "outputId": "890f554c-a4db-46d2-bd73-3a3841b349ab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 69400 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     30.0%     46.2%     36.4%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51588 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.79      0.83        67\n",
            "    problematic       0.30      0.46      0.36        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.59      0.63      0.60        80\n",
            "   weighted avg       0.79      0.74      0.76        80\n",
            "\n",
            "Training AUC: 0.8752428586431422\n",
            "Validation AUC: 0.669345579793341\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.3: single layer, add Sentiment_TextBlob\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.3,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_TextBlob'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_TextBlob'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 548,
      "id": "zt08SvTe4LM9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zt08SvTe4LM9",
        "outputId": "5f794923-ead4-4d23-9c86-34cea744489c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.3 compared to model: 2.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.300000          0.263158      +\n",
            "1          Recall   0.461538          0.384615      +\n",
            "2        F1-score   0.363636          0.312500      +\n",
            "3        Accuracy   0.737500          0.725000      +\n",
            "4  Validation AUC   0.669346          0.715270      -\n",
            "5    Training AUC   0.875243          0.861170      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with chosen model at last step\n",
        "model_comparison(3.3, 2.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1EdP43uT2v8b",
      "metadata": {
        "id": "1EdP43uT2v8b"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "Based on our EDA and domain knowledge, `Sentiment_TextBlob` is likely related to whether a quote is problematic or not. Therefore, we consider adding it to our model. `Sentiment_TextBlob` uses a different algorithm than `Sentiment_VADER` and tended to be more polarised. This difference may better distinguish problematic quotes.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "This model appears to be roughly similar in performance to model 3.1 which used `Sentiment_VADER`, although the validation AUC is marginally better. Compared to model 2.2 it has a marginally higher precision with no drop in validation recall. Again, this improvement in precision may be minor but lead to material cost-savings for the company if the model was deployed at scale.\n",
        "\n",
        "\n",
        "Overall, both sentiment scores appeared to individually improve model performance. Next we add both sentiment scores to our model as the interaction between them may improve predictions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yXn9sWkk3i10",
      "metadata": {
        "id": "yXn9sWkk3i10"
      },
      "source": [
        "---\n",
        "**Version 3.4: 1 Layer, 60 neurons, add both sentiment scores**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 549,
      "id": "ASRtIFdS3i1-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ASRtIFdS3i1-",
        "outputId": "44da4bcb-83bd-412a-8b06-575a1df04156"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70080 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     28.6%     46.2%     35.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51860 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.78      0.83        67\n",
            "    problematic       0.29      0.46      0.35        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.58      0.62      0.59        80\n",
            "   weighted avg       0.78      0.72      0.75        80\n",
            "\n",
            "Training AUC: 0.8736052037387103\n",
            "Validation AUC: 0.6142365097588978\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.3: single layer, add Sentiment_TextBlob\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.4,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER','Sentiment_TextBlob'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','Sentiment_VADER','Sentiment_TextBlob'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 550,
      "id": "qvCFlwUv4TVA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qvCFlwUv4TVA",
        "outputId": "dd01ef50-e85f-44ee-8ccc-32bebb15b6f5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.4 compared to model: 3.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.285714          0.315789      -\n",
            "1          Recall   0.461538          0.461538      =\n",
            "2        F1-score   0.352941          0.375000      -\n",
            "3        Accuracy   0.725000          0.750000      -\n",
            "4  Validation AUC   0.614237          0.656716      -\n",
            "5    Training AUC   0.873605          0.874025      -\n"
          ]
        }
      ],
      "source": [
        "# compare model with model that just has Sentiment_VADER\n",
        "model_comparison(3.4, 3.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "q0obhTf130py",
      "metadata": {
        "id": "q0obhTf130py"
      },
      "source": [
        "\n",
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "Added both sentiment scores as features since the interaction between them may be important in distinguishing problematic quotes.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Compared to our model with just `Sentiment_VADER`, there does not appear to be any change in the main model metrics when we use both sentiment scores (see output above). Notably, the validation AUC is lower when we use both sentiment scores, indicating that the model is worse overall.\n",
        "\n",
        "Therefore, it seems that using both sentiment scores does not improve the model's predictive power beyond what is captured with just `Sentiment_VADER`. We only use this sentiment score in future model iterations.\n",
        "\n",
        "By themselves, the sentiment scores do not appear to dramatically improve model performance although they may become more relevant when interacting with additional features.\n",
        "\n",
        "We now consider the exclamation flag variable which takes the value 1 if the quote contains an exclamation mark and 0 otherwise:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "q23k1LkV61YX",
      "metadata": {
        "id": "q23k1LkV61YX"
      },
      "source": [
        "---\n",
        "**Version 3.5: 1 Layer, 60 neurons, add exclamation mark flag**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 551,
      "id": "Wh7nkOPp61Yo",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Wh7nkOPp61Yo",
        "outputId": "d86e0438-0987-4c9e-a560-08d4f799a127"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70016 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  75.0%     33.3%     53.8%     41.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51592 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.79      0.84        67\n",
            "    problematic       0.33      0.54      0.41        13\n",
            "\n",
            "       accuracy                           0.75        80\n",
            "      macro avg       0.62      0.66      0.63        80\n",
            "   weighted avg       0.81      0.75      0.77        80\n",
            "\n",
            "Training AUC: 0.8653020636420921\n",
            "Validation AUC: 0.7497129735935706\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.5: single layer, add exclamation flag as feature\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.5,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 552,
      "id": "aHW67hws4a68",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aHW67hws4a68",
        "outputId": "7517f4e3-43fc-4fa9-8cdb-91f1caa271d0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.5 compared to model: 2.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.333333          0.263158      +\n",
            "1          Recall   0.538462          0.384615      +\n",
            "2        F1-score   0.411765          0.312500      +\n",
            "3        Accuracy   0.750000          0.725000      +\n",
            "4  Validation AUC   0.749713          0.715270      +\n",
            "5    Training AUC   0.865302          0.861170      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with chosen single layer model at last step\n",
        "model_comparison(3.5, 2.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lgQqyD0s7dEO",
      "metadata": {
        "id": "lgQqyD0s7dEO"
      },
      "source": [
        "**What was changed and why?**\n",
        "\n",
        "Previously we engineered a feature that flagged whether a quote contained an exclamation mark or not. Exclamation marks are expected to relate to tone and hence whether a quote may be rude hence why we consider it as a potentially informative variable.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Yes! The addition of `exclamation_flag` has led to a significant improvement in our model based on precision, recall and validation AUC. For the first time our model has a recall greater than 50% indicating that it correctly identifies problematic quotes more often than not. This is particularly important given our context as reductions in false negatives will reduce the risk of contract breaches and lead to cost savings.\n",
        "\n",
        "**Other observations**\n",
        "\n",
        "The model may be underfitting as we see that the validation loss is still decreasing with the training loss in the final epochs. Similarly, we saw a dip in accuracy just before model training ended which may be due to noise from the batch sizes. Therefore, playing around with other hyperparameters may further improve this model.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "We keep the exclamation mark flag as a feature as it clearly improves our model. We now add the question mark flag, without the exclamation flag included, to see if it has any explanatory power."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CblYKVZ59lF0",
      "metadata": {
        "id": "CblYKVZ59lF0"
      },
      "source": [
        "---\n",
        "**Version 3.6: 1 Layer, 60 neurons, add question mark flag**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 553,
      "id": "KWiORkbT9lF1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "KWiORkbT9lF1",
        "outputId": "fdee3641-2361-4fc1-8545-27aaf746e539"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 67308 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  72.5%     28.6%     46.2%     35.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51860 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.78      0.83        67\n",
            "    problematic       0.29      0.46      0.35        13\n",
            "\n",
            "       accuracy                           0.72        80\n",
            "      macro avg       0.58      0.62      0.59        80\n",
            "   weighted avg       0.78      0.72      0.75        80\n",
            "\n",
            "Training AUC: 0.8702183102289436\n",
            "Validation AUC: 0.6991963260619978\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.6: single layer, add question mark flag\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.6,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7', 'question_flag'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','question_flag'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 554,
      "id": "wJIB0lWo4d38",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wJIB0lWo4d38",
        "outputId": "16e3d875-ba08-44db-ab69-3d3004558db7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.6 compared to model: 2.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.285714          0.263158      +\n",
            "1          Recall   0.461538          0.384615      +\n",
            "2        F1-score   0.352941          0.312500      +\n",
            "3        Accuracy   0.725000          0.725000      =\n",
            "4  Validation AUC   0.699196          0.715270      -\n",
            "5    Training AUC   0.870218          0.861170      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with chosen single layer model at last step\n",
        "model_comparison(3.6, 2.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vNPY-rmAA5fd",
      "metadata": {
        "id": "vNPY-rmAA5fd"
      },
      "source": [
        "**What was changed and why?**\n",
        "\n",
        "Previously, we constructed a feature that flagged whether a quote contained a question mark. This may be related to the tone of a quote and hence whether it is problematic. To isolate its impact we add it without the exclamation flag.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Adding `question_flag` as a feature appears to result in a worse model than our model 2.2, across all metrics. The model appears to have overfit which may have contributed to its poor performance on the validation set. Even still it is unclear if early stopping would have improved the model as the validation loss is relatively flat.\n",
        "\n",
        "Overall we choose not to include this variable as a feature in our future iterations as it does not appear to have significant predictive power.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "Next we look at adding the genre columns as features. Given that `exclamation_flag` led to a noticeable improvement in the model, we will add the genre flags on top of a model with this feature."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_oNtRbNfD-Zx",
      "metadata": {
        "id": "_oNtRbNfD-Zx"
      },
      "source": [
        "---\n",
        "**Version 3.7: 1 Layer, 60 neurons, exclamation flag + Genre Columns**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 555,
      "id": "KrdxUmKBD-Z3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "KrdxUmKBD-Z3",
        "outputId": "d50788ed-3137-4640-e594-5a22601a6d7b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71216 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  80.0%     38.5%     38.5%     38.5%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52040 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.88      0.88        67\n",
            "    problematic       0.38      0.38      0.38        13\n",
            "\n",
            "       accuracy                           0.80        80\n",
            "      macro avg       0.63      0.63      0.63        80\n",
            "   weighted avg       0.80      0.80      0.80        80\n",
            "\n",
            "Training AUC: 0.9590093992858644\n",
            "Validation AUC: 0.7726750861079219\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.7: single layer, add exclamation flag and genre columns as features\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.7,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 556,
      "id": "vJe4NZ1h4h-e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vJe4NZ1h4h-e",
        "outputId": "55bb5b79-dac4-4dfa-eae4-64df77aa44b0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.7 compared to model: 3.5\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.384615          0.333333      +\n",
            "1          Recall   0.384615          0.538462      -\n",
            "2        F1-score   0.384615          0.411765      -\n",
            "3        Accuracy   0.800000          0.750000      +\n",
            "4  Validation AUC   0.772675          0.749713      +\n",
            "5    Training AUC   0.959009          0.865302      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with single layer model with exclamation flag added\n",
        "model_comparison(3.7, 3.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Iq5wHoByD-Z4",
      "metadata": {
        "id": "Iq5wHoByD-Z4"
      },
      "source": [
        "**What was changed and why?**\n",
        "\n",
        "Previously, we created features that flagged whether a quote was from a movie of a particular genre. This may interact with other dimensions of our quotes to better identify whether a quote is problematic. Alternatively, certain genres may be more likely to produce problematic quotes of a certain type. Also, the additional information from the genre may provide some context for the quote, helping the model discern whether a quote is problematic.\n",
        "\n",
        "We will add nine features at this step which is significant and may result in overfitting to our training data given our limited data.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Compared to model 3.5 (our model with exclamation flag added), this model has better precision and AUC score. However, the recall has fallen substantially and lower than model 2.2 (46%). There is an inherent trade-off between precision and recall which is reflected in the F1 score, however this metric stil indicates that the model is worse than 3.5.\n",
        "\n",
        "The model has overfit during training; we see the validation loss increases in the final epochs. Therefore, the model may benefit from early stopping or regularisation techniques which may result in the model performing similarly to model 3.5.\n",
        "\n",
        "It is therefore difficult to ascertain whether this is a model improvement. We experiment with dropping some of the genres to see if this helps.\n",
        "\n",
        "\n",
        "**What next?**\n",
        "\n",
        "To combat the observed overfitting, we consider dropping some of the genre columns, potentially improving the model's performance. For the sake of brevity, we do not show all our experimentation with the genre column:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xb-TveWpsivb",
      "metadata": {
        "id": "xb-TveWpsivb"
      },
      "source": [
        "---\n",
        "**Version 3.8: 1 Layer, 60 neurons, exclamation flag + 'Fantasy' column**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 557,
      "id": "RgGtvqaKrCg3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "RgGtvqaKrCg3",
        "outputId": "b3222040-b148-4bdc-efe1-884d225f1561"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71540 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  77.5%     35.3%     46.2%     40.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51640 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.89      0.84      0.86        67\n",
            "    problematic       0.35      0.46      0.40        13\n",
            "\n",
            "       accuracy                           0.78        80\n",
            "      macro avg       0.62      0.65      0.63        80\n",
            "   weighted avg       0.80      0.78      0.79        80\n",
            "\n",
            "Training AUC: 0.8833228313379542\n",
            "Validation AUC: 0.7841561423650976\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.8: single layer, add exclamation flag and 'Fantasy' as features\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.8,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag','Fantasy'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag','Fantasy'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 558,
      "id": "8qJYmBlt4vdo",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8qJYmBlt4vdo",
        "outputId": "15b35165-9fa0-432e-c927-584ba831df9a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.8 compared to model: 3.5\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.352941          0.333333      +\n",
            "1          Recall   0.461538          0.538462      -\n",
            "2        F1-score   0.400000          0.411765      -\n",
            "3        Accuracy   0.775000          0.750000      +\n",
            "4  Validation AUC   0.784156          0.749713      +\n",
            "5    Training AUC   0.883323          0.865302      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with single layer model with exclamation flag added\n",
        "model_comparison(3.8, 3.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gUE9vPkosrRf",
      "metadata": {
        "id": "gUE9vPkosrRf"
      },
      "source": [
        "**What was changed and why?**\n",
        "\n",
        "It is not shown here but we experimented with different combinations of Genre columns. Ultimately, it was found that if we just included the `Fantasy` column then we got the best validation performance. However, this feels like we are over-optimising for the validation metrics and the specific quotes in it. The model therefore may not generalise well for truly unseen data.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "As suggested before, this model performed substantially better than model 3.5 based on our validation metrics. However, this may be an over-optimisation for the validation dataset where this dataset just so happens to contain a few problematic quotes in the Fantasy genre. This is amplified by our validation dataset being rather small. This results in our model having very similar performance on the training and validation datasets.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "Therefore, **although this model has the best metrics so far we choose not to proceed with it** due to over-optimisation of the validation metrics.\n",
        "\n",
        "Given that model 3.5 (exclamation flag) and 3.7 (exclamation + all genres) performed similarly in terms of F1-score and AUC we try to refine these two further. We consider adding the sentiment scores to these models as its interaction with the additional columns may improve predictive power."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rfPLSI5yu6ak",
      "metadata": {
        "id": "rfPLSI5yu6ak"
      },
      "source": [
        "---\n",
        "**Version 3.9: 1 Layer, 60 neurons, exclamation flag + `Sentiment_VADER`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 559,
      "id": "cCfcTXG7uusj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "cCfcTXG7uusj",
        "outputId": "163d14a2-bfff-4fae-e5f1-151bb8726041"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 74560 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  78.8%     38.9%     53.8%     45.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51648 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.84      0.87        67\n",
            "    problematic       0.39      0.54      0.45        13\n",
            "\n",
            "       accuracy                           0.79        80\n",
            "      macro avg       0.65      0.69      0.66        80\n",
            "   weighted avg       0.82      0.79      0.80        80\n",
            "\n",
            "Training AUC: 0.8811797679059022\n",
            "Validation AUC: 0.7531572904707233\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.9: single layer, add exclamation flag as feature\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.9,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 560,
      "id": "_j8I3zpf41ms",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_j8I3zpf41ms",
        "outputId": "21441c65-72c7-4e8c-b577-26db4df808eb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.9 compared to model: 3.5\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.388889          0.333333      +\n",
            "1          Recall   0.538462          0.538462      =\n",
            "2        F1-score   0.451613          0.411765      +\n",
            "3        Accuracy   0.787500          0.750000      +\n",
            "4  Validation AUC   0.753157          0.749713      +\n",
            "5    Training AUC   0.881180          0.865302      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with single layer model with exclamation flag added\n",
        "model_comparison(3.9, 3.5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TZS36Cw6w1e5",
      "metadata": {
        "id": "TZS36Cw6w1e5"
      },
      "source": [
        "**What was changed and why?**\n",
        "\n",
        "After adding exclamation flag, we consider adding the sentiment scores as features again. We may expect an interaction between exclamation marks and sentiment scores since exclamation marks can indicate jovial tones or rude tones depending on the intended sentiment of a quote. We experimented with both VADER and TextBlob and found that only VADER improved the model performance.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "We see an improvement in precision and hence F1-score by adding `Sentiment_VADER` to our model with the exclamation mark flag. Furthermore, the recall which is the most critical metric for our business context remains the same.\n",
        "The model thus appears better.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "Next we consider adding the `Sentiment_VADER` to the model with the genre columns added to see what improvement this makes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EB1_7dHA8s0e",
      "metadata": {
        "id": "EB1_7dHA8s0e"
      },
      "source": [
        "---\n",
        "**Version 3.10: 1 Layer, 60 neurons, exclamation flag + `Sentiment_VADER` + Genre columns**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 561,
      "id": "pf3v48cTx3Vm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "pf3v48cTx3Vm",
        "outputId": "db52103e-70c2-4dc5-8a05-d04bebf8d760"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70900 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     46.2%     48.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52208 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.91      0.90        67\n",
            "    problematic       0.50      0.46      0.48        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.70      0.69      0.69        80\n",
            "   weighted avg       0.83      0.84      0.83        80\n",
            "\n",
            "Training AUC: 0.9605190611216131\n",
            "Validation AUC: 0.7944890929965556\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.10: single layer, add exclamation flag, genre columns and Sentiment_VADER as features\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.10,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 562,
      "id": "egpGCNG644ND",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "egpGCNG644ND",
        "outputId": "c54a16c0-0593-4bb6-f100-3d2fa8ad78a5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.1 compared to model: 3.7\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.384615      +\n",
            "1          Recall   0.461538          0.384615      +\n",
            "2        F1-score   0.480000          0.384615      +\n",
            "3        Accuracy   0.837500          0.800000      +\n",
            "4  Validation AUC   0.794489          0.772675      +\n",
            "5    Training AUC   0.960519          0.959009      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with single layer model with exclamation flag and genre added (same but not VADER)\n",
        "model_comparison(3.10, 3.7)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "w52AZ00n4nFe",
      "metadata": {
        "id": "w52AZ00n4nFe"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "In this step we experimented with adding the sentiment score columns as features to the model that includes the genres as features. We find that only `Sentiment_VADER` leads to a significant improvement in the model (vs model 3.7). While `Sentiment_VADER` by itself did not appear to improve model performance significantly, when it interacts with genre and the exclamation mark flag it appears to improve model performance considerably.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "Compared to model 3.7, we see that this model improves in all validation metrics. It achieves the highest validation F1-score and AUC that we have seen across all models so far.\n",
        "\n",
        "Compared to model 3.9 (exclamation flag and sentiment_VADER) this model appears to be better in all validation metrics except recall. Notably it has a moderately higher AUC than 3.9 which often does correlate to signficiant improvements in results. We therefore favour model 3.10 going into the next step, however, may still consider 3.9 with different architectures.\n",
        "\n",
        "**Observations**\n",
        "\n",
        "There is clear evidence of overfitting since the validation loss plateaus and then deteriorates slightly while the training loss continues to improve. The use of regularisation and early stopping may be useful for improving this model.\n",
        "\n",
        "**What next?**\n",
        "\n",
        "Overall, we seem to have settled on our model features and will again experiment with the model architecture using our findings from this stage. The features selected and the architecture are not independent and so we should test other architectures with our new features.\n",
        "\n",
        "Before this, we look at using only the first five principal components in our model. Previously when constructing the PCs, it was determined that using between 5 to 7 PCs was likely appropriate. We have not yet tested different numbers of PCs."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "k8uoAGN08ygW",
      "metadata": {
        "id": "k8uoAGN08ygW"
      },
      "source": [
        "---\n",
        "**Version 3.11: 1 Layer, 60 neurons, 5 PCs + exclamation flag + `Sentiment_VADER` + Genre columns**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 563,
      "id": "c-2F0Tq46nkm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "c-2F0Tq46nkm",
        "outputId": "72f4afa4-2c54-46be-8772-716692d834d6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70336 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  78.8%     40.0%     61.5%     48.5%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51868 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.82      0.87        67\n",
            "    problematic       0.40      0.62      0.48        13\n",
            "\n",
            "       accuracy                           0.79        80\n",
            "      macro avg       0.66      0.72      0.68        80\n",
            "   weighted avg       0.83      0.79      0.80        80\n",
            "\n",
            "Training AUC: 0.9466006353707204\n",
            "Validation AUC: 0.78300803673938\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 3.11: single layer, use only the first 5 PCs,add exclamation flag, genre columns and Sentiment_VADER as features\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=3.11,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5', 'exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5', 'exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 564,
      "id": "SL791zMA5QLZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SL791zMA5QLZ",
        "outputId": "8fd2a5ca-7106-4cc0-e67c-208747e82b40"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 3.11 compared to model: 3.7\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.400000          0.384615      +\n",
            "1          Recall   0.615385          0.384615      +\n",
            "2        F1-score   0.484848          0.384615      +\n",
            "3        Accuracy   0.787500          0.800000      -\n",
            "4  Validation AUC   0.783008          0.772675      +\n",
            "5    Training AUC   0.946601          0.959009      -\n"
          ]
        }
      ],
      "source": [
        "# compare model with single layer model with exclamation flag and genre added (same but not VADER)\n",
        "model_comparison(3.11, 3.7)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gUAyn2O19G5S",
      "metadata": {
        "id": "gUAyn2O19G5S"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We have used only the first 5 PCs rather than 7 PCs. When creating the principal components, between 5 and 7 PCs were considered appropriate via the 'elbow' method. Although additional PCs may explain more of the semantic variance in our quotes, the dimensions captured in the last two may be irrelevant to whether a quote is problematic or not. Given model 3.10 was overfitting, we attempt to remove two PCs to see if this improves the model.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "This model has the highest recall of any model we have created, indicating that it has a propensity to correctly classify actual problematic quotes. This has come at the cost of some precision compared to 3.10. Overall the model has a higher F1-score, relative to model 3.10, despite a marginally lower AUC.\n",
        "\n",
        "As stated previously, improving recall is critical in our context as it will lead to less breaches of legal contracts and so less costs incurred. This suggests this model may be the best so far.\n",
        "\n",
        "**Observations**\n",
        "\n",
        "There was a spike up in the validation accuracy in the final few epochs so the results we see may be 'lucky' given that the validation loss has plateaued. It is somewhat difficult to determine if the model is overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dXi88jag-upY",
      "metadata": {
        "id": "dXi88jag-upY"
      },
      "source": [
        "#####**Section Summary**  \n",
        "\n",
        "In this step we experimented with different model features,keeping the model architecture constant. We found that:\n",
        "\n",
        "- The addition of the exclamation mark flag feature signficantly improved model performance.\n",
        "- By themselves, the sentiment scores did not have high explanatory power but when combined with the exclamation flag there was a noticeable improvement in model performance. This is expected as exclamation marks can indicate excitement *or* rudeness, for example, and the sentiment helps distinguish this.\n",
        "- Question mark flag was unimportant.\n",
        "- The addition of genres also appeared to improve model performance although there was evidence of overfitting with the addition of this variable.\n",
        "- Using 5 PCs of the embedding rather than 7 may reduce overfitting and improve model performance. The results differed depending on the presence of other features.\n",
        "- Using just `Fantasy` as a genre column led to significant improvement in the validation set performance. However, we chose not to use the model as it was over-optimised for the validation dataset.\n",
        "\n",
        "Overall, we choose to continue with three candidate models into the next stage. These models performed similarly on the validation dataset and with further tuning it may be clearer which model is best. The candidate models at this stage are:\n",
        "\n",
        "- Model 3.9: 7 PCs, exclamation flag and Sentiment_VADER\n",
        "- Model 3.10: 7 PCs, exclamation flag, 10 genres and Sentiment_VADER\n",
        "- Model 3.11: 5 PCs, exclamation flag, 10 genres and Sentiment_VADER\n",
        "\n",
        "Next, we will again look at model architecture using the three models above. Features and architecture are not independent and so we believe it is appropriate to again experiment with different architectures."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BSvWA_E5eozp",
      "metadata": {
        "id": "BSvWA_E5eozp"
      },
      "source": [
        "#### Q3b Step 7 - Second-round of Architecture Adjustment"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LwNuq8E_Ap55",
      "metadata": {
        "id": "LwNuq8E_Ap55"
      },
      "source": [
        "\n",
        "**Outline**\n",
        "\n",
        "After getting more clarity on our features, we again test different model architectures. This is because adding features had changed the data complexity and so the optimal architecture may have changed. We will consider neural network structures with both one and two hidden layers. The architecture of the three candidate models identified previously will be tuned in this step."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MuAPkJFrHBxw",
      "metadata": {
        "id": "MuAPkJFrHBxw"
      },
      "source": [
        "---\n",
        "**Version 4.1: 2 Layer, 40 + 20 neurons, 7 PCs + exclamation flag + `Sentiment_VADER`**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 565,
      "id": "2rUMdDJMHKzm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "2rUMdDJMHKzm",
        "outputId": "065127bb-111a-41cb-cfca-b8c2fb8a431d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 83748 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     21.4%     23.1%     22.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52232 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.85      0.84      0.84        67\n",
            "    problematic       0.21      0.23      0.22        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.53      0.53      0.53        80\n",
            "   weighted avg       0.75      0.74      0.74        80\n",
            "\n",
            "Training AUC: 0.9228956626759084\n",
            "Validation AUC: 0.6727898966704937\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 4.1: two layers (exclamation + Sentiment_VADER )\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=4.1,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=40,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=20,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 566,
      "id": "bx73xpAh5Ye9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bx73xpAh5Ye9",
        "outputId": "efdd8421-215d-4862-96b1-8c730540f433"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 4.1 compared to model: 3.9\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.214286          0.388889      -\n",
            "1          Recall   0.230769          0.538462      -\n",
            "2        F1-score   0.222222          0.451613      -\n",
            "3        Accuracy   0.737500          0.787500      -\n",
            "4  Validation AUC   0.672790          0.753157      -\n",
            "5    Training AUC   0.922896          0.881180      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with 1 layer\n",
        "model_comparison(4.1, 3.9)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "v_M32466_smO",
      "metadata": {
        "id": "v_M32466_smO"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "Split the neurons between two hidden layers rather than just a single hidden layer. This change was made to model 3.9 (7 PCs + exclamation flag + `Sentiment_VADER`). We have added additional features since looking at a second hidden layer so we may uncover more complex relationships between the features and improve the performance of the model.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "We see that this model performed significantly worse across all validation metrics compared to model 3.9. We therefore reject this model architecture when using this set of features.\n",
        "\n",
        "We next consider other architectures with ourother models from the previous stage."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EUyv87cufcJE",
      "metadata": {
        "id": "EUyv87cufcJE"
      },
      "source": [
        "---\n",
        "**Version 4.2: 2 Layer, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 567,
      "id": "qQPkSJeTJEXN",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "qQPkSJeTJEXN",
        "outputId": "9f9a2cec-7cb9-4f62-c3c3-c041d6382f8f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 74876 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 20ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  82.5%     46.7%     53.8%     50.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51836 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.88      0.89        67\n",
            "    problematic       0.47      0.54      0.50        13\n",
            "\n",
            "       accuracy                           0.82        80\n",
            "      macro avg       0.69      0.71      0.70        80\n",
            "   weighted avg       0.84      0.82      0.83        80\n",
            "\n",
            "Training AUC: 0.9792880959882377\n",
            "Validation AUC: 0.7979334098737083\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 4.2: two layers (exclamation + Genre + Sentiment_VADER )\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=4.2,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 568,
      "id": "yogctAgE5hvB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yogctAgE5hvB",
        "outputId": "d3f9d550-8fb5-4c26-8729-40eacc16a833"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 4.2 compared to model: 3.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.466667          0.500000      -\n",
            "1          Recall   0.538462          0.461538      +\n",
            "2        F1-score   0.500000          0.480000      +\n",
            "3        Accuracy   0.825000          0.837500      -\n",
            "4  Validation AUC   0.797933          0.794489      +\n",
            "5    Training AUC   0.979288          0.960519      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with 1 layer\n",
        "model_comparison(4.2, 3.10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "__GKietsDeup",
      "metadata": {
        "id": "__GKietsDeup"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We alter the architecture of model 3.10 (7 PCs, exclamation, VADER, genre) so that it is has two hidden layers with 45 and 12 neurons each. We have decreased the total number of neurons in the model from 3.10 as there was evidence of overfitting in this model. Adding a second layer may exacerbate the overfitting so we reduce the number of neurons to balance this.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "This model has a substantially higher validation recall than model 3.10 and the same validation precision. This means it is better at correctly labelling actual problematic quotes. We expected this model to benefit from a second hidden layer as it contains the most features of our models and so there were potentially more complex relationships that could be modelled with two hidden layers.\n",
        "\n",
        "However, there is clear overfitting for this model. The AUC for training is extremely high and we see extremely high training accuracy while the validation accuracy does not improve. Despite this, the model seems to generalise well to the validation set. This model may require regularisation etc. to reduce the impact of overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xprKqcgiGsem",
      "metadata": {
        "id": "xprKqcgiGsem"
      },
      "source": [
        "---\n",
        "**Version 4.3: 2 Layer, 45 + 12 neurons, 5 PCs + exclamation flag + `Sentiment_VADER` + Genre**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 569,
      "id": "gJb6XLJBJhx8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "gJb6XLJBJhx8",
        "outputId": "eae6e839-98fb-4fc3-a26d-afb22756a9bc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 73860 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  78.8%     35.7%     38.5%     37.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52040 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.87      0.87        67\n",
            "    problematic       0.36      0.38      0.37        13\n",
            "\n",
            "       accuracy                           0.79        80\n",
            "      macro avg       0.62      0.63      0.62        80\n",
            "   weighted avg       0.79      0.79      0.79        80\n",
            "\n",
            "Training AUC: 0.9713525257298886\n",
            "Validation AUC: 0.69345579793341\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 4.3: two layers (5 PCS + exclamation + Genre + Sentiment_VADER )\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=4.3,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=40,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.01,\n",
        "         epochs=200,\n",
        "         batch_size=64,\n",
        "         optimiser = 'SGD',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 570,
      "id": "9ko-GZDC5k7s",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9ko-GZDC5k7s",
        "outputId": "4217e3ce-0f1a-49b2-e945-d4437629e943"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 4.3 compared to model: 3.11\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.357143          0.400000      -\n",
            "1          Recall   0.384615          0.615385      -\n",
            "2        F1-score   0.370370          0.484848      -\n",
            "3        Accuracy   0.787500          0.787500      =\n",
            "4  Validation AUC   0.693456          0.783008      -\n",
            "5    Training AUC   0.971353          0.946601      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with 1 layer\n",
        "model_comparison(4.3, 3.11)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7wk7octeG4JN",
      "metadata": {
        "id": "7wk7octeG4JN"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We alter the architecture of model 3.11 (5 PCs, exclamation, VADER, genre) so that it is has two hidden layers with 45 and 12 neurons each. We have decreased the total number of neurons in the model from 3.11 as there was evidence of overfitting in this model. Adding a second layer may exacerbate overfitting so we reduce the number of neurons to balance this.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "This model is clearly worse than model 3.11 based on all validation metrics. Furthermore, there is even greater overfitting, exemplified by the training metrics being high despite no improvement in validation loss. As the model is more complex it has likely led to a better fit on the training data and consequently overfitting (more variance error).\n",
        "\n",
        "It seems that with this set of features there is no need for a second hidden layer."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FVwaBQWvIPOS",
      "metadata": {
        "id": "FVwaBQWvIPOS"
      },
      "source": [
        "#####**Section Summary**  \n",
        "\n",
        "In this step we experimented with adding a second layer to the models selected as part of the feature experimentation.\n",
        "\n",
        "Ultimately it was found that in most cases a second hidden layer did not improve the model but rather excerbated the impact of overfitting.\n",
        "However, model 3.10, which contains the most features (7 PCs, exclamation flag, Genres and VADER), did see significant improvement in the validation metrics as a result of adding a second hidden layer, although overfitting was still a problem. This model will likely benefit from regularisation in future steps.\n",
        "\n",
        "For now we continue with the following candidate models:\n",
        "- Model 3.9: 7 PCs, exclamation flag and Sentiment_VADER\n",
        "- Model 3.10: 7 PCs, exclamation flag, 10 genres and Sentiment_VADER\n",
        "- Model 3.11: 5 PCs, exclamation flag, 10 genres and Sentiment_VADER\n",
        "- Model 4.2: 2 hidden layers (7 PCs, exclamation flag, 10 genres and Sentiment_VADER)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nTnUaVxffBDy",
      "metadata": {
        "id": "nTnUaVxffBDy"
      },
      "source": [
        "#### Q3b Step 8 - Optimiser"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kxzu6kyzJ0YH",
      "metadata": {
        "id": "kxzu6kyzJ0YH"
      },
      "source": [
        "\n",
        "**Outline**\n",
        "\n",
        "So far we have been using mini-batch gradient descent as our optimisation algorithm. In this step, we consider using the Adam optimiser. This optimiser may be better as it converges faster and adaptively changes the learning rate, leading to less dependence on the initial learning rate selection.\n",
        "\n",
        "In this step, we test our models with the Adam optimiser. For brevity, not all model tests are shown and the results are just summarised at the end.\n",
        "\n",
        "Due to Adam's faster convergence, we adjust the hyperparameters (learning rate and epoch) to prevent overfitting and to fairly appraise this algorithm. These hyperparameters will be tuned in a subsequent step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 571,
      "id": "lgupgfdwLQbb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "lgupgfdwLQbb",
        "outputId": "2200ad9d-220f-45c6-c5da-86c9fe9c606f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 69080 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     53.8%     51.9%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52552 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.90      0.90        67\n",
            "    problematic       0.50      0.54      0.52        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.70      0.72      0.71        80\n",
            "   weighted avg       0.84      0.84      0.84        80\n",
            "\n",
            "Training AUC: 0.9848049254358329\n",
            "Validation AUC: 0.791044776119403\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 5.1: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag, Sentiment_VADER, Genre\n",
        "    ### use Adam optimiser ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=5.1,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER', 'Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.001, # have decreased for Adam\n",
        "         epochs=50, # have decreased for Adam\n",
        "         batch_size=64,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 572,
      "id": "X4e5vT_Q5pLY",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X4e5vT_Q5pLY",
        "outputId": "ada6d5ed-9e59-4829-a783-d8c254f709c9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 5.1 compared to model: 3.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.500000      =\n",
            "1          Recall   0.538462          0.461538      +\n",
            "2        F1-score   0.518519          0.480000      +\n",
            "3        Accuracy   0.837500          0.837500      =\n",
            "4  Validation AUC   0.791045          0.794489      -\n",
            "5    Training AUC   0.984805          0.960519      +\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with 'SGD''\n",
        "model_comparison(5.1, 3.10)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BykR6oMlOZv6",
      "metadata": {
        "id": "BykR6oMlOZv6"
      },
      "source": [
        "#####**Section Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We use the Adam optimiser with what was previously model 3.10. We also reduced the learning rate and number of epochs since the rapid convergence of Adam, led to the model overfitting rapidly with our previous hyperparamter values.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "The model appears to be pretty much equivalent to model 3.10 in terms of the validation metrics. The model has overfit, exemplified by the validation loss increasing, but this can be resolved with regularisation later.\n",
        "\n",
        "Notably, the algorithm ran substantially *faster*. Therefore given the results were similar and the speed of this algorithm is more user-friendly, ***we choose to use the Adam optimiser.***"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pANuAixEgIDq",
      "metadata": {
        "id": "pANuAixEgIDq"
      },
      "source": [
        "#### Q3b Step 9 - Grid Search: Batch Size & Learning Rate"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JxLq1MuTS3EC",
      "metadata": {
        "id": "JxLq1MuTS3EC"
      },
      "source": [
        "\n",
        "**Outline**\n",
        "\n",
        "We have not yet tuned two of our hyperparameters: Batch Size and Learning Rate. Although Adam adaptively changes the learning rate, the initial selection still has an impact so should be tuned for optimal performance.\n",
        "\n",
        "Learning rate and batch size are interconnected. For example, a larger batch size might necessitate a lower learning rate to prevent large updates and potential overshooting of the minimum. We therefore perform a grid search with a prior model to identify the best combination of the two hyperparameters.\n",
        "\n",
        "We use the validation F1 score as the tuning metric. The F1 score balances precision and recall, aligning with our objective of accurately identifying problematic quotes. It therefore provides a good measure of how well the model generalises.\n",
        "\n",
        "We do not have the computing power to check a large grid of hyperparamter combinations. Therefore, we restrict our tuning to just three possible values for each hyperparameter. This should be mostly sufficient to get a good result."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 573,
      "id": "qTs7GMDLUthN",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qTs7GMDLUthN",
        "outputId": "52af70ed-c3e5-4cac-8e53-770a229c8817"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  76.2%     33.3%     46.2%     38.7%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  70.0%     17.6%     23.1%     20.0%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  81.2%     25.0%     7.7%      11.8%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 18ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  77.5%     36.8%     53.8%     43.8%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 20ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     25.0%     30.8%     27.6%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  82.5%     45.5%     38.5%     41.7%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  76.2%     35.0%     53.8%     42.4%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 17ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     27.8%     38.5%     32.3%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "Time taken to run this Neural Network model is: 0 mins\n",
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  77.5%     27.3%     23.1%     25.0%   \n",
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n"
          ]
        }
      ],
      "source": [
        "batch_size_vec = [32,64,128]\n",
        "learning_rate_vec = [0.0005, 0.001, 0.005]\n",
        "\n",
        "df_results = pd.DataFrame(columns=['batch_size', 'learning_rate', 'F1_score'])\n",
        "\n",
        "for batch_size in batch_size_vec:\n",
        "  for learning_rate in learning_rate_vec:\n",
        "    # fit temp model\n",
        "    # force keras to behave deterministically\n",
        "      with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "      nn_fit(ver='grid',\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=60,\n",
        "              l1_activation='relu',\n",
        "              learn_rate = learning_rate,\n",
        "              epochs=60,\n",
        "              batch_size=batch_size,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False,\n",
        "             plot_toggle = False)\n",
        "\n",
        "      df_results.loc[len(df_results)] =[batch_size, learning_rate, nn_models['grid']['scores']['F1-score']]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 574,
      "id": "wM62JDsIXk86",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "wM62JDsIXk86",
        "outputId": "a57ad4fe-cec8-470e-e2d1-910daa9218d6"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"df_results\",\n  \"rows\": 9,\n  \"fields\": [\n    {\n      \"column\": \"batch_size\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 42.332020977033444,\n        \"min\": 32.0,\n        \"max\": 128.0,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          32.0,\n          64.0,\n          128.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"learning_rate\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.002136000936329383,\n        \"min\": 0.0005,\n        \"max\": 0.005,\n        \"num_unique_values\": 3,\n        \"samples\": [\n          0.0005,\n          0.001,\n          0.005\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"F1_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.11206044835827525,\n        \"min\": 0.11764705882352941,\n        \"max\": 0.4375,\n        \"num_unique_values\": 9,\n        \"samples\": [\n          0.3225806451612903,\n          0.2,\n          0.4166666666666667\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "df_results"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-07c61408-1052-43f2-a665-7007ff41faed\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>batch_size</th>\n",
              "      <th>learning_rate</th>\n",
              "      <th>F1_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>32.0</td>\n",
              "      <td>0.0005</td>\n",
              "      <td>0.387097</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>32.0</td>\n",
              "      <td>0.0010</td>\n",
              "      <td>0.200000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>32.0</td>\n",
              "      <td>0.0050</td>\n",
              "      <td>0.117647</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>64.0</td>\n",
              "      <td>0.0005</td>\n",
              "      <td>0.437500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>64.0</td>\n",
              "      <td>0.0010</td>\n",
              "      <td>0.275862</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>64.0</td>\n",
              "      <td>0.0050</td>\n",
              "      <td>0.416667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>128.0</td>\n",
              "      <td>0.0005</td>\n",
              "      <td>0.424242</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>128.0</td>\n",
              "      <td>0.0010</td>\n",
              "      <td>0.322581</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>128.0</td>\n",
              "      <td>0.0050</td>\n",
              "      <td>0.250000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-07c61408-1052-43f2-a665-7007ff41faed')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-07c61408-1052-43f2-a665-7007ff41faed button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-07c61408-1052-43f2-a665-7007ff41faed');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-011d16a1-6e4e-44a4-a6c4-f4edd17e4567\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-011d16a1-6e4e-44a4-a6c4-f4edd17e4567')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-011d16a1-6e4e-44a4-a6c4-f4edd17e4567 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_00df375a-c673-4aa0-8e7b-daafd40bb599\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_results')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_00df375a-c673-4aa0-8e7b-daafd40bb599 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_results');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   batch_size  learning_rate  F1_score\n",
              "0        32.0         0.0005  0.387097\n",
              "1        32.0         0.0010  0.200000\n",
              "2        32.0         0.0050  0.117647\n",
              "3        64.0         0.0005  0.437500\n",
              "4        64.0         0.0010  0.275862\n",
              "5        64.0         0.0050  0.416667\n",
              "6       128.0         0.0005  0.424242\n",
              "7       128.0         0.0010  0.322581\n",
              "8       128.0         0.0050  0.250000"
            ]
          },
          "execution_count": 574,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Now we print the results from our grid search:\n",
        "df_results"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xf4671tWfWA4",
      "metadata": {
        "id": "xf4671tWfWA4"
      },
      "source": [
        "#####**Section Summary**\n",
        "\n",
        "The grid search identified a **learning rate of 0.0005** and a **batch size of 128** as 'optimal', resulting in the highest validation F1-score. The F1-score surpassed the model's previous best score, highlighting how tuning these hyperparameters can improve model performance.\n",
        "\n",
        "The models with the highest learning rate performed the worst. This may because they began to overfit and perform poorly, or may be a result of the algorithm struggling to converge. It is likely the former as we would expect that Adam would help the algorithm converge.\n",
        "\n",
        "We have ended up selecting the highest batch size of those tested. A higher batch size generally results in more accurate gradient estimates at each step. A smaller batch size is not necessary since our dataset is relatively small and so model training occurs fairly quickly.\n",
        "\n",
        "We expect the larger batch size and lower learning rate to promote smoother and potentially more accurate convergence for the algorithm in future steps.\n",
        "\n",
        "\n",
        "---\n",
        "Before moving on we retrain our candidate models using 'Adam' and our tuned batch size and learning rate. We see some drops in validation metrics in our models but this is expected; it is a result of the models training for too long due to the faster speed of Adam (see graphs below)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 575,
      "id": "ZLQiVfo4AADc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "ZLQiVfo4AADc",
        "outputId": "c436ca6b-e036-46c4-ba91-7ab28c4f6af8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 75320 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  75.0%     29.4%     38.5%     33.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51876 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.82      0.85        67\n",
            "    problematic       0.29      0.38      0.33        13\n",
            "\n",
            "       accuracy                           0.75        80\n",
            "      macro avg       0.58      0.60      0.59        80\n",
            "   weighted avg       0.78      0.75      0.76        80\n",
            "\n",
            "Training AUC: 0.9006839424490654\n",
            "Validation AUC: 0.7428243398392651\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 6.1: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER as features\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "nn_fit(ver=6.1,\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=60,\n",
        "              l1_activation='relu',\n",
        "              learn_rate = 0.0005,\n",
        "              epochs=100,\n",
        "              batch_size=128,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 576,
      "id": "BYmSseqXBTaf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "BYmSseqXBTaf",
        "outputId": "e71b905e-0c8e-4af8-a88e-112a756babf0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70164 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     61.5%     55.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52024 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.88      0.90        67\n",
            "    problematic       0.50      0.62      0.55        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.71      0.75      0.73        80\n",
            "   weighted avg       0.85      0.84      0.84        80\n",
            "\n",
            "Training AUC: 0.9714017538332284\n",
            "Validation AUC: 0.7956371986222732\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 6.2: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=6.2,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 577,
      "id": "NWCPbEyfBUT8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "NWCPbEyfBUT8",
        "outputId": "65f7f9ef-3501-4f9a-8321-3357d6d8e4aa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 72872 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  81.2%     44.4%     61.5%     51.6%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51736 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.85      0.88        67\n",
            "    problematic       0.44      0.62      0.52        13\n",
            "\n",
            "       accuracy                           0.81        80\n",
            "      macro avg       0.68      0.73      0.70        80\n",
            "   weighted avg       0.84      0.81      0.82        80\n",
            "\n",
            "Training AUC: 0.9518746061751734\n",
            "Validation AUC: 0.8025258323765787\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 6.3: single layer 60 neurons\n",
        "  # Features: 5 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=6.3,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 578,
      "id": "zlJuw5IXBZw-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "zlJuw5IXBZw-",
        "outputId": "cf2d7037-4741-480d-8a0d-23cc0e2e67cb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71524 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  87.5%     66.7%     46.2%     54.5%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52224 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.96      0.93        67\n",
            "    problematic       0.67      0.46      0.55        13\n",
            "\n",
            "       accuracy                           0.88        80\n",
            "      macro avg       0.78      0.71      0.74        80\n",
            "   weighted avg       0.86      0.88      0.87        80\n",
            "\n",
            "Training AUC: 0.9910470489392984\n",
            "Validation AUC: 0.8059701492537313\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 6.4: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=6.4,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "y9o9RUJQhfkH",
      "metadata": {
        "id": "y9o9RUJQhfkH"
      },
      "source": [
        "#### Q3b Step 10 - Adjust Activation Function\n",
        "**Outline**\n",
        "\n",
        "We have exclusively used the ReLU activation so far. The sigmoid activation function is a popular choice for binary classification tasks like ours. Sigmoid's output naturally ranges between 0 and 1, potentially aligning well with our binary classification task.\n",
        "\n",
        "We will incorporate the sigmoid activation function into some of our candidate models. This allows us to compare its performance against ReLU and potentially identify a better model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TMVdsF9_LHn5",
      "metadata": {
        "id": "TMVdsF9_LHn5"
      },
      "source": [
        "---\n",
        "**Version 7.1: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER`, sigmoid activation function**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 579,
      "id": "P6HqDawtEZzt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "P6HqDawtEZzt",
        "outputId": "3f64f681-67ac-46b4-cdde-46220cabfe69"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 58080 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  60.0%     17.2%     38.5%     23.8%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 53644 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.84      0.64      0.73        67\n",
            "    problematic       0.17      0.38      0.24        13\n",
            "\n",
            "       accuracy                           0.60        80\n",
            "      macro avg       0.51      0.51      0.48        80\n",
            "   weighted avg       0.73      0.60      0.65        80\n",
            "\n",
            "Training AUC: 0.7580504358328082\n",
            "Validation AUC: 0.5947187141216992\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 7.1: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER as features\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Use Sigmoid activation function ###\n",
        "with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "nn_fit(ver=7.1,\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=70,\n",
        "              l1_activation='sigmoid',\n",
        "              learn_rate = 0.0005,\n",
        "              epochs=100,\n",
        "              batch_size=128,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 580,
      "id": "Nfd2yetF58fr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nfd2yetF58fr",
        "outputId": "a49949bd-b8e8-4e39-98b8-c0184f188c14"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 7.1 compared to model: 6.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.172414          0.294118      -\n",
            "1          Recall   0.384615          0.384615      =\n",
            "2        F1-score   0.238095          0.333333      -\n",
            "3        Accuracy   0.600000          0.750000      -\n",
            "4  Validation AUC   0.594719          0.742824      -\n",
            "5    Training AUC   0.758050          0.900684      -\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with ReLU\n",
        "model_comparison(7.1, 6.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bK-xQuFdFPbv",
      "metadata": {
        "id": "bK-xQuFdFPbv"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We changed the activation function from ReLU to sigmoid. This is to tune the activation function where we believe the sigmoid function may be appropriate for a binary classification problem.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "We see that with a *sigmoid* activation function the model has performed significantly worse on all validation metrics, Furthermore, its training accuracy is much lower than the validation accuracy we were able to achieve on other models, indicating it is a bad fit.\n",
        "\n",
        "**This suggests that ReLU is a better choice for the activation function.**\n",
        "\n",
        "We do not show it here, for brevity's sake, but similar results were obtained when using sigmoid on our other single layer model candidates.\n",
        "\n",
        "Next we look at our two-layer neural network and the combinations of activation functions:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1GcQPUxcLTiL",
      "metadata": {
        "id": "1GcQPUxcLTiL"
      },
      "source": [
        "---\n",
        "**Version 7.2: 2 Layer, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre**, ReLU - Sigmoid combination"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 581,
      "id": "WjoFgofoHVZa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "WjoFgofoHVZa",
        "outputId": "3e18f8fe-a0f6-4b49-c894-2315d0e47f85"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 81020 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     46.2%     48.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52208 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.91      0.90        67\n",
            "    problematic       0.50      0.46      0.48        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.70      0.69      0.69        80\n",
            "   weighted avg       0.83      0.84      0.83        80\n",
            "\n",
            "Training AUC: 0.9441359483301827\n",
            "Validation AUC: 0.7921928817451206\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 7.2: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### ReLU - Sigmoid combination ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=7.2,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='sigmoid',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 582,
      "id": "6ECfZOjr6Eh0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6ECfZOjr6Eh0",
        "outputId": "97f67006-d5d1-464a-a9b2-9e321d10bf8e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 7.2 compared to model: 6.4\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.666667      -\n",
            "1          Recall   0.461538          0.461538      =\n",
            "2        F1-score   0.480000          0.545455      -\n",
            "3        Accuracy   0.837500          0.875000      -\n",
            "4  Validation AUC   0.792193          0.805970      -\n",
            "5    Training AUC   0.944136          0.991047      -\n"
          ]
        }
      ],
      "source": [
        "# compare model with same model but with ReLU\n",
        "model_comparison(7.2, 6.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SP36jytMLxZx",
      "metadata": {
        "id": "SP36jytMLxZx"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "**What was changed and why?**\n",
        "\n",
        "We used ReLU for the activation function in the first hidden layer and sigmoid in the second hidden layer. This is to experiment with activation functions and possibly achieve better model performance.\n",
        "\n",
        "**Did this improve the model?**\n",
        "\n",
        "We see that this activation function combination performs marginally worse than with just using ReLU, based on the F1-score. We therefore do not use this activation function setup.\n",
        "\n",
        "**We test other combinations of activation function with two hidden layers and see much worse results. For brevity, we do not discuss the results as the relevant points are similar to above.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jEqo3GA9O817",
      "metadata": {
        "id": "jEqo3GA9O817"
      },
      "source": [
        "---\n",
        "**Version 7.3: 2 Layer, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre, Sigmoid - Sigmoid combination**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 583,
      "id": "MNl-AlhfOSGX",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "MNl-AlhfOSGX",
        "outputId": "3bb2bea2-82c5-458c-ad73-09416f405e9d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 68764 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  65.0%     22.2%     46.2%     30.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 53084 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.69      0.77        67\n",
            "    problematic       0.22      0.46      0.30        13\n",
            "\n",
            "       accuracy                           0.65        80\n",
            "      macro avg       0.55      0.57      0.53        80\n",
            "   weighted avg       0.76      0.65      0.69        80\n",
            "\n",
            "Training AUC: 0.7958674648183155\n",
            "Validation AUC: 0.6509758897818599\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 7.2: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Sigmoid - Sigmoid combination ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=7.3,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='sigmoid',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='sigmoid',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ab58fXxWO_5u",
      "metadata": {
        "id": "Ab58fXxWO_5u"
      },
      "source": [
        "---\n",
        "**Version 7.4: 2 Layer, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre**, Sigmoid - ReLU combination"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 584,
      "id": "25r5BroVOsJk",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "25r5BroVOsJk",
        "outputId": "bb1b1bf7-72c2-4552-804a-650dad4674d2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 85484 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 21ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  71.2%     27.3%     46.2%     34.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 54336 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.88      0.76      0.82        67\n",
            "    problematic       0.27      0.46      0.34        13\n",
            "\n",
            "       accuracy                           0.71        80\n",
            "      macro avg       0.58      0.61      0.58        80\n",
            "   weighted avg       0.78      0.71      0.74        80\n",
            "\n",
            "Training AUC: 0.8756990390674229\n",
            "Validation AUC: 0.730195177956372\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 7.4: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### sigmoid - ReLU combination ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=7.4,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='sigmoid',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=100,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tZWWdyykPLeI",
      "metadata": {
        "id": "tZWWdyykPLeI"
      },
      "source": [
        "##### **Section Summary**\n",
        "\n",
        "In this step we tuned the activation function by experimenting with the *sigmoid* function rather than *ReLU*. In all of our tests, model performance was worse using *sigmoid*. Empirically, ReLU appears to work well for most problems as it does not suffer from a vanishing gradient (slow changes) compared to sigmoid (Krizhevsky, Sutskever, Hinton, 2012).\n",
        "\n",
        "**We therefore proceed with ReLU as our activation function.**\n",
        "\n",
        "**References**\n",
        "\n",
        "Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wK4vqAqyRZ8e",
      "metadata": {
        "id": "wK4vqAqyRZ8e"
      },
      "source": [
        "#### Q3b Step 11 - Early Stopping and Drop Out Rate\n",
        "**Outline**\n",
        "\n",
        "Our previous models exhibited signs of overfitting. To mitigate this we will employ two techniques: early stopping and neuron dropout.\n",
        "\n",
        "- **Early stopping**: Stops training once the validation loss fails to improve for a predefined number of epochs, preventing the model from memorising training data.\n",
        "- **Dropout**: randomly 'shut off' a proportion of neurons during training. Helps prevent the model from relying on any one node and can break-up overfitting. The rate of nodes dropping out is a hyperparameter that we will tune.\n",
        "\n",
        "We iteratively employ these techniques on our four candidate models to tune the relevant hyperparameters. After applying this tuning, we will narrow down our focus to a single candidate model. Future steps will then be applied only to this model before choosing a final model."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cd7-P7dDVU_A",
      "metadata": {
        "id": "cd7-P7dDVU_A"
      },
      "source": [
        "---\n",
        "\n",
        "***Candidate Model 1 Tuning***\n",
        "\n",
        "**Version 8.1: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER`, early stopping, no dropout**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 585,
      "id": "-_SZPKb3V4GS",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "-_SZPKb3V4GS",
        "outputId": "716db7b3-6f88-4cc6-a624-e36f10db10da"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70888 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 18ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     27.8%     38.5%     32.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51756 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.81      0.84        67\n",
            "    problematic       0.28      0.38      0.32        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.57      0.60      0.58        80\n",
            "   weighted avg       0.77      0.74      0.75        80\n",
            "\n",
            "Training AUC: 0.9270636420919975\n",
            "Validation AUC: 0.7359357060849598\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.1: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER as features\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early Stopping ###\n",
        "with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.1,\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=60,\n",
        "              l1_activation='relu',\n",
        "              learn_rate = 0.0005,\n",
        "              epochs=200,\n",
        "              batch_size=128,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False,\n",
        "              early_stop = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_OZ9OH5Yg7B8",
      "metadata": {
        "id": "_OZ9OH5Yg7B8"
      },
      "source": [
        "---\n",
        "**Version 8.2: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER`, early stopping, dropout rate = 0.1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 586,
      "id": "48AtDiZQhJTU",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "48AtDiZQhJTU",
        "outputId": "baead21a-a716-409e-919e-68c842418c22"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 95024 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  73.8%     27.8%     38.5%     32.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51756 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.81      0.84        67\n",
            "    problematic       0.28      0.38      0.32        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.57      0.60      0.58        80\n",
            "   weighted avg       0.77      0.74      0.75        80\n",
            "\n",
            "Training AUC: 0.9108347773576979\n",
            "Validation AUC: 0.7416762342135477\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.2: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER as features\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early Stopping ###\n",
        "with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.2,\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=60,\n",
        "              l1_activation='relu',\n",
        "              learn_rate = 0.0005,\n",
        "              epochs=200,\n",
        "              batch_size=128,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False,\n",
        "              early_stop = True,\n",
        "              dropout_rate = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7XIKLGywdGN3",
      "metadata": {
        "id": "7XIKLGywdGN3"
      },
      "source": [
        "---\n",
        "**Version 8.3: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER`, early stopping, dropout rate = 0.2**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 587,
      "id": "0pXnZZSIhmQj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "0pXnZZSIhmQj",
        "outputId": "7ec4f4a9-9be4-43b8-8f53-97ac813da341"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 103420 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 17ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  75.0%     29.4%     38.5%     33.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51876 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.87      0.82      0.85        67\n",
            "    problematic       0.29      0.38      0.33        13\n",
            "\n",
            "       accuracy                           0.75        80\n",
            "      macro avg       0.58      0.60      0.59        80\n",
            "   weighted avg       0.78      0.75      0.76        80\n",
            "\n",
            "Training AUC: 0.9152193604284814\n",
            "Validation AUC: 0.7313432835820896\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.3: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER as features\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early Stopping + dropout rate = 0.2 ###\n",
        "with tf.device(device_name):\n",
        "        keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.3,\n",
        "              train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              train_y=train_y,\n",
        "              test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])],\n",
        "              test_y=val_y,\n",
        "              l1_neurons=60,\n",
        "              l1_activation='relu',\n",
        "              learn_rate = 0.0005,\n",
        "              epochs=200,\n",
        "              batch_size=128,\n",
        "              optimiser = 'Adam',\n",
        "              verbose = False,\n",
        "              early_stop = True,\n",
        "              dropout_rate = 0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_zM13wPRdLCl",
      "metadata": {
        "id": "_zM13wPRdLCl"
      },
      "source": [
        "---\n",
        "**Candidate Model 1: Metric Comparisons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 588,
      "id": "O5hnIeLWWJnB",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O5hnIeLWWJnB",
        "outputId": "1b82e96d-4416-42f7-b83f-3d5f056e18bc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.1 compared to model: 6.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.277778          0.294118      -\n",
            "1          Recall   0.384615          0.384615      =\n",
            "2        F1-score   0.322581          0.333333      -\n",
            "3        Accuracy   0.737500          0.750000      -\n",
            "4  Validation AUC   0.735936          0.742824      -\n",
            "5    Training AUC   0.927064          0.900684      +\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping vs model with no early stopping\n",
        "model_comparison(8.1, 6.1)\n",
        "\n",
        "# we see that the training stopped after about 120 epochs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 589,
      "id": "p4Hjw0W1haqh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p4Hjw0W1haqh",
        "outputId": "2da2ddd5-fec7-40b5-bf5a-d3e69a426d38"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.2 compared to model: 6.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.277778          0.294118      -\n",
            "1          Recall   0.384615          0.384615      =\n",
            "2        F1-score   0.322581          0.333333      -\n",
            "3        Accuracy   0.737500          0.750000      -\n",
            "4  Validation AUC   0.741676          0.742824      -\n",
            "5    Training AUC   0.910835          0.900684      +\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.1 vs model with no early stopping\n",
        "model_comparison(8.2, 6.1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 590,
      "id": "NB6O-jlChrJp",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NB6O-jlChrJp",
        "outputId": "0cf34ec0-640b-4415-8901-4ff843221d6f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.3 compared to model: 6.1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.294118          0.294118      =\n",
            "1          Recall   0.384615          0.384615      =\n",
            "2        F1-score   0.333333          0.333333      =\n",
            "3        Accuracy   0.750000          0.750000      =\n",
            "4  Validation AUC   0.731343          0.742824      -\n",
            "5    Training AUC   0.915219          0.900684      +\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.2 vs model with no early stopping\n",
        "model_comparison(8.3, 6.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "B5Dj-KMLp6X0",
      "metadata": {
        "id": "B5Dj-KMLp6X0"
      },
      "source": [
        "##### Candidate Model 1 Summary\n",
        "\n",
        "In this step we experimented with early stopping and dropout for candidate model 1 (model with 7 embedding PCs, the exclamation flag and `Sentiment_VADER` as features)\n",
        "\n",
        "We see that after introducing early stopping the model stopped training after 120 epochs which is actually more epochs than model 6.1 (same model without early stopping). The model with just early stopping achieved considerably worse results compared to model 6.1. In particular, the recall dropped from 62% to 38%. This a material drop and this low recall level is not tolerable in our context as it leaves a significant number of problematic quotes undetected.\n",
        "\n",
        "After introducing a neuron dropout rate of 0.1, model performance on the validation set was similar to the model without dropout. As expected, the training took longer (150 epochs) with dropout as the model had to learn to adapt to possible dropouts.\n",
        "\n",
        "The model with a dropout rate of 0.2 led to similar results as the model with a dropout rate of 0.1.\n",
        "\n",
        "These tests suggest that early stopping and dropout do not improve this model's performance.\n",
        "\n",
        "**We proceed with model 6.1 (no early stopping or dropout rate) into our candidate model comparison at the end of this section.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "96hu_sWGof77",
      "metadata": {
        "id": "96hu_sWGof77"
      },
      "source": [
        "---\n",
        "***Candidate Model 2 Tuning***"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J5Yop6L9dSjQ",
      "metadata": {
        "id": "J5Yop6L9dSjQ"
      },
      "source": [
        "**Version 8.4: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 591,
      "id": "dNZ-1JU-iP83",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "dNZ-1JU-iP83",
        "outputId": "31a47a93-f114-4d7a-ffbe-284a02bbc99a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70164 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 20ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     61.5%     55.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52024 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.88      0.90        67\n",
            "    problematic       0.50      0.62      0.55        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.71      0.75      0.73        80\n",
            "   weighted avg       0.85      0.84      0.84        80\n",
            "\n",
            "Training AUC: 0.9522356122663307\n",
            "Validation AUC: 0.8105625717566016\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.4: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.4,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xi59AtX_dbLX",
      "metadata": {
        "id": "xi59AtX_dbLX"
      },
      "source": [
        "---\n",
        "**Version 8.5: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 592,
      "id": "Afk9P0vUi014",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Afk9P0vUi014",
        "outputId": "0f5d8d27-ee0b-4a9b-e1d8-cb88828cd1c9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 81140 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  82.5%     46.7%     53.8%     50.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51836 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.88      0.89        67\n",
            "    problematic       0.47      0.54      0.50        13\n",
            "\n",
            "       accuracy                           0.82        80\n",
            "      macro avg       0.69      0.71      0.70        80\n",
            "   weighted avg       0.84      0.82      0.83        80\n",
            "\n",
            "Training AUC: 0.951113211510187\n",
            "Validation AUC: 0.8048220436280138\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.5: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping + dropout rate = 0.1 ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.5,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "W7jzY86ldgZg",
      "metadata": {
        "id": "W7jzY86ldgZg"
      },
      "source": [
        "---\n",
        "**Version 8.6: 1 Layer, 60 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.2**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 593,
      "id": "bSiBES7Cjapr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "bSiBES7Cjapr",
        "outputId": "2b972338-d430-4c22-e0c9-d6966d4969dc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 85164 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     53.8%     51.9%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52552 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.90      0.90        67\n",
            "    problematic       0.50      0.54      0.52        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.70      0.72      0.71        80\n",
            "   weighted avg       0.84      0.84      0.84        80\n",
            "\n",
            "Training AUC: 0.9506898498214661\n",
            "Validation AUC: 0.799081515499426\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.6: single layer 60 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping + dropout rate = 0.2 ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.6,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8iBss_tvdm2R",
      "metadata": {
        "id": "8iBss_tvdm2R"
      },
      "source": [
        "---\n",
        "**Candidate Model 2: Metric comparisons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 594,
      "id": "rNK5zif8idCj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rNK5zif8idCj",
        "outputId": "7af034c3-6875-46a0-e2a0-ed42172c0dcd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.4 compared to model: 6.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.500000      =\n",
            "1          Recall   0.615385          0.615385      =\n",
            "2        F1-score   0.551724          0.551724      =\n",
            "3        Accuracy   0.837500          0.837500      =\n",
            "4  Validation AUC   0.810563          0.795637      +\n",
            "5    Training AUC   0.952236          0.971402      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping vs model with no early stopping\n",
        "model_comparison(8.4, 6.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 595,
      "id": "rdNYWa75i02B",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rdNYWa75i02B",
        "outputId": "0b07c15f-5dbb-4439-c62b-e952ac204389"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.5 compared to model: 6.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.466667          0.500000      -\n",
            "1          Recall   0.538462          0.615385      -\n",
            "2        F1-score   0.500000          0.551724      -\n",
            "3        Accuracy   0.825000          0.837500      -\n",
            "4  Validation AUC   0.804822          0.795637      +\n",
            "5    Training AUC   0.951113          0.971402      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.1 vs model with no early stopping\n",
        "model_comparison(8.5, 6.2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 596,
      "id": "4SItwNL0jap3",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4SItwNL0jap3",
        "outputId": "12db1b06-1af4-4633-a963-7e7d8a414031"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.6 compared to model: 6.2\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.500000      =\n",
            "1          Recall   0.538462          0.615385      -\n",
            "2        F1-score   0.518519          0.551724      -\n",
            "3        Accuracy   0.837500          0.837500      =\n",
            "4  Validation AUC   0.799082          0.795637      +\n",
            "5    Training AUC   0.950690          0.971402      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.2 vs model with no early stopping\n",
        "model_comparison(8.6, 6.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BL3mXaHIaZ31",
      "metadata": {
        "id": "BL3mXaHIaZ31"
      },
      "source": [
        "##### Candidate Model 2 Summary\n",
        "\n",
        "In this step we experimented with early stopping and dropout for candidate model 2 (model with 7 embedding PCs, the exclamation flag, `Sentiment_VADER` and the Genre columns as features, single hidden layer)\n",
        "\n",
        "We see that after introducing early stopping the first model stopped training after 100 epochs. The model with early stopping achieved similar results to the model without, although had a marginally lower F1-score despite a moderate jump in validation AUC.\n",
        "\n",
        "Introducing neuron dropout similarly resulted in a worse F1-score but a better validation AUC. As the neuron dropout rate increased the F1-score dropped further, indicating that neuron dropout resulted in a worse model. Therefore **we do not proceed with neuron dropout for this model**.\n",
        "\n",
        "The model with just early stopping had similar performance to the model without. Although the F1-score was marginally lower we believe the improvement in AUC to be more beneficial. The F1-score difference may be a result of our small validation set and over-optimising it here may result in worse performance on other unseen data. Furthermore, early stopping reduces the impact of overfitting which we believe to be important for good model performance on unseen data.\n",
        "\n",
        "**We proceed with model 8.4 (with early stopping) into our candidate model comparison at the end of this section.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MGE5aRhtnSSb",
      "metadata": {
        "id": "MGE5aRhtnSSb"
      },
      "source": [
        "---\n",
        "***Candidate Model 3 Tuning***\n",
        "\n",
        "**Version 8.7: 1 Layer, 60 neurons, 5 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 597,
      "id": "LVlLr_YIm4Cw",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "LVlLr_YIm4Cw",
        "outputId": "a50c30ea-f8be-49f7-97ce-9c69b9b1577c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71004 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 16ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  81.2%     44.4%     61.5%     51.6%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51736 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.85      0.88        67\n",
            "    problematic       0.44      0.62      0.52        13\n",
            "\n",
            "       accuracy                           0.81        80\n",
            "      macro avg       0.68      0.73      0.70        80\n",
            "   weighted avg       0.84      0.81      0.82        80\n",
            "\n",
            "Training AUC: 0.9496888783868935\n",
            "Validation AUC: 0.7967853042479909\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.7: single layer 60 neurons\n",
        "  # Features: 5 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.7,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qi22fpmHeqmc",
      "metadata": {
        "id": "qi22fpmHeqmc"
      },
      "source": [
        "---\n",
        "**Version 8.8: 1 Layer, 60 neurons, 5 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 598,
      "id": "qkPPbqXxnWXh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "qkPPbqXxnWXh",
        "outputId": "31bb8ca1-1b71-4b5f-bdf6-f3869236bf38"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 87192 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 15ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  82.5%     47.1%     61.5%     53.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52028 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.87      0.89        67\n",
            "    problematic       0.47      0.62      0.53        13\n",
            "\n",
            "       accuracy                           0.82        80\n",
            "      macro avg       0.70      0.74      0.71        80\n",
            "   weighted avg       0.85      0.82      0.83        80\n",
            "\n",
            "Training AUC: 0.9572404694391934\n",
            "Validation AUC: 0.801377726750861\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.8: single layer 60 neurons\n",
        "  # Features: 5 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping + dropout rate = 0.1 ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.8,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bIyvqcf9exwX",
      "metadata": {
        "id": "bIyvqcf9exwX"
      },
      "source": [
        "---\n",
        "**Version 8.9: 1 Layer, 60 neurons, 5 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.2**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 599,
      "id": "XTJxWYuKnvHs",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "XTJxWYuKnvHs",
        "outputId": "423fa474-6f0f-42ad-9aba-6545604e8520"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 91860 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 22ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  81.2%     44.4%     61.5%     51.6%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51736 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.85      0.88        67\n",
            "    problematic       0.44      0.62      0.52        13\n",
            "\n",
            "       accuracy                           0.81        80\n",
            "      macro avg       0.68      0.73      0.70        80\n",
            "   weighted avg       0.84      0.81      0.82        80\n",
            "\n",
            "Training AUC: 0.9501975687880697\n",
            "Validation AUC: 0.7944890929965558\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.9: single layer 60 neurons\n",
        "  # Features: 5 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping + dropout rate = 0.2 ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.9,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=60,\n",
        "         l1_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "j622R1KTe3zW",
      "metadata": {
        "id": "j622R1KTe3zW"
      },
      "source": [
        "---\n",
        "**Candidate Model 3: Metric comparisons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 600,
      "id": "rrSov--4m4C6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rrSov--4m4C6",
        "outputId": "004838bd-0c44-4a96-fee7-dba4818ecb6a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.7 compared to model: 6.3\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.444444          0.444444      =\n",
            "1          Recall   0.615385          0.615385      =\n",
            "2        F1-score   0.516129          0.516129      =\n",
            "3        Accuracy   0.812500          0.812500      =\n",
            "4  Validation AUC   0.796785          0.802526      -\n",
            "5    Training AUC   0.949689          0.951875      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping vs model with no early stopping\n",
        "model_comparison(8.7, 6.3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 601,
      "id": "hVbu8PK-nWXi",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hVbu8PK-nWXi",
        "outputId": "4f31c504-6116-4ac2-b02f-fa8ce99602c5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.8 compared to model: 6.3\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.470588          0.444444      +\n",
            "1          Recall   0.615385          0.615385      =\n",
            "2        F1-score   0.533333          0.516129      +\n",
            "3        Accuracy   0.825000          0.812500      +\n",
            "4  Validation AUC   0.801378          0.802526      -\n",
            "5    Training AUC   0.957240          0.951875      +\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.1 vs model with no early stopping\n",
        "model_comparison(8.8, 6.3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 602,
      "id": "pyDXVSYVnvHt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pyDXVSYVnvHt",
        "outputId": "43347f65-c268-4ebb-8287-652c96fb7015"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.9 compared to model: 6.3\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.444444          0.444444      =\n",
            "1          Recall   0.615385          0.615385      =\n",
            "2        F1-score   0.516129          0.516129      =\n",
            "3        Accuracy   0.812500          0.812500      =\n",
            "4  Validation AUC   0.794489          0.802526      -\n",
            "5    Training AUC   0.950198          0.951875      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.2 vs model with no early stopping\n",
        "model_comparison(8.9, 6.3)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wTwQdDxrfPCv",
      "metadata": {
        "id": "wTwQdDxrfPCv"
      },
      "source": [
        "##### Candidate Model 3 Summary\n",
        "\n",
        "In this step we experimented with early stopping and dropout for candidate model 3 (model with 5 embedding PCs, the exclamation flag, `Sentiment_VADER` and the Genre columns as features)\n",
        "\n",
        "We see that after introducing early stopping the first model stopped training after 120 epochs. The model with early stopping achieved the exact same precision, recall and hence F1-score as the model without. The validation AUC however was slightly better. This suggests that early stopping improved the model. Additionally, a lower training AUC indicates that early stopped has helped in reducing overfitting to the training dataset.\n",
        "\n",
        "Introducing neuron dropout similarly resulted in the same precision, recall and F1-score. However, lower validation and training AUC was achieved. Overall, this suggests that neuron dropout did not make much difference to the model.\n",
        "\n",
        "**We proceed with model 8.7 (with early stopping) into our candidate model comparison at the end of this section.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vdWF_QspoDod",
      "metadata": {
        "id": "vdWF_QspoDod"
      },
      "source": [
        "---\n",
        "***Candidate Model 4 Tuning***"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LDCleLN7hQTr",
      "metadata": {
        "id": "LDCleLN7hQTr"
      },
      "source": [
        "**Version 8.10: 2 Layers, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 603,
      "id": "_8ywxbRdoEwW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "_8ywxbRdoEwW",
        "outputId": "ee8812b8-82a3-43ce-e58a-a3cd00e6c489"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 71432 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  85.0%     53.8%     53.8%     53.8%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52232 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.91      0.91        67\n",
            "    problematic       0.54      0.54      0.54        13\n",
            "\n",
            "       accuracy                           0.85        80\n",
            "      macro avg       0.72      0.72      0.72        80\n",
            "   weighted avg       0.85      0.85      0.85        80\n",
            "\n",
            "Training AUC: 0.9751824721697122\n",
            "Validation AUC: 0.8105625717566016\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.10: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping ###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver='8.10',\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "N2oQSTSwh7CG",
      "metadata": {
        "id": "N2oQSTSwh7CG"
      },
      "source": [
        "**Version 8.11: 2 Layers, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 604,
      "id": "wBBedujQohNt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "wBBedujQohNt",
        "outputId": "a5e6141c-e8cd-4eb0-b908-a3dc80a23a1b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 81416 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 20ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  85.0%     53.3%     61.5%     57.1%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52656 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.90      0.91        67\n",
            "    problematic       0.53      0.62      0.57        13\n",
            "\n",
            "       accuracy                           0.85        80\n",
            "      macro avg       0.73      0.76      0.74        80\n",
            "   weighted avg       0.86      0.85      0.85        80\n",
            "\n",
            "Training AUC: 0.9655403276622558\n",
            "Validation AUC: 0.8025258323765787\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.11: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping and drop out rate = 0.1###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.11,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X1IMpgPMh_dX",
      "metadata": {
        "id": "X1IMpgPMh_dX"
      },
      "source": [
        "**Version 8.12: 2 Layers, 45 + 12 neurons, 7 PCs + exclamation flag + `Sentiment_VADER` + Genre columns, early stopping + dropout rate = 0.2**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 605,
      "id": "HZqr6XAIpH3f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "HZqr6XAIpH3f",
        "outputId": "6f0b4607-2ab7-4a5b-be57-0e5432324989"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 90052 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  83.8%     50.0%     61.5%     55.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52024 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.88      0.90        67\n",
            "    problematic       0.50      0.62      0.55        13\n",
            "\n",
            "       accuracy                           0.84        80\n",
            "      macro avg       0.71      0.75      0.73        80\n",
            "   weighted avg       0.85      0.84      0.84        80\n",
            "\n",
            "Training AUC: 0.9554518483511867\n",
            "Validation AUC: 0.7921928817451205\n"
          ]
        }
      ],
      "source": [
        "# Run iteration 8.12: two layers 45 + 12 neurons\n",
        "  # Features: 7 PCs, exclamation flag and Sentiment_VADER, genre columns\n",
        "  # Adam optimiser, 128 batch size, 0.0005 learning rate\n",
        "  ### Early stopping and drop out rate = 0.2###\n",
        "\n",
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "nn_fit(ver=8.12,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rO5IYdTHiWk9",
      "metadata": {
        "id": "rO5IYdTHiWk9"
      },
      "source": [
        "**Candidate Model 4: Metric comparisons**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 606,
      "id": "rYzfpF24oWy5",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rYzfpF24oWy5",
        "outputId": "80ac228f-4d31-4edc-9d36-1a086236e731"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.10 compared to model: 6.4\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.538462          0.666667      -\n",
            "1          Recall   0.538462          0.461538      +\n",
            "2        F1-score   0.538462          0.545455      -\n",
            "3        Accuracy   0.850000          0.875000      -\n",
            "4  Validation AUC   0.810563          0.805970      +\n",
            "5    Training AUC   0.975182          0.991047      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping vs model with no early stopping\n",
        "model_comparison('8.10', 6.4)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 607,
      "id": "959YjOxsohNu",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "959YjOxsohNu",
        "outputId": "c818acf2-3d88-4cd7-f8ed-fcc93a3f981d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.11 compared to model: 6.4\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.533333          0.666667      -\n",
            "1          Recall   0.615385          0.461538      +\n",
            "2        F1-score   0.571429          0.545455      +\n",
            "3        Accuracy   0.850000          0.875000      -\n",
            "4  Validation AUC   0.802526          0.805970      -\n",
            "5    Training AUC   0.965540          0.991047      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.1 vs model with no early stopping\n",
        "model_comparison(8.11, 6.4)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 608,
      "id": "Q3GO8XHapH3g",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q3GO8XHapH3g",
        "outputId": "7237d71f-4ab6-4935-e553-480c84cb775e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.12 compared to model: 6.4\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.666667      -\n",
            "1          Recall   0.615385          0.461538      +\n",
            "2        F1-score   0.551724          0.545455      +\n",
            "3        Accuracy   0.837500          0.875000      -\n",
            "4  Validation AUC   0.792193          0.805970      -\n",
            "5    Training AUC   0.955452          0.991047      -\n"
          ]
        }
      ],
      "source": [
        "# model with early stopping and drop out rate = 0.2 vs model with no early stopping\n",
        "model_comparison(8.12, 6.4)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "igQ7OgCpi-LZ",
      "metadata": {
        "id": "igQ7OgCpi-LZ"
      },
      "source": [
        "##### Candidate Model 4 Summary\n",
        "\n",
        "In this step we experimented with early stopping and dropout for candidate model 4 (model with 7 embedding PCs, the exclamation flag, `Sentiment_VADER` and the Genre columns as features, 2 hidden layer structure)\n",
        "\n",
        "After introducing early stopping the first model stopped training after 90 epochs. All three models tested led to:\n",
        "- A slight decrease in precision\n",
        "- A moderate jump in recall, meaning the model better categorised actual problematic quotes.\n",
        "- Consequently, there was a moderate jump in F1-score\n",
        "- Marginal improvement in validation AUC\n",
        "- Decrease in training AUC. This is significant given that the original model has extremely high training AUC and so was clearly overfitting.\n",
        "\n",
        "Therefore, using early stopping and neuron dropout led to an improvement in model performance on the validation dataset and a reduction in overfitting. Implementing these measures was hence successful.\n",
        "\n",
        "Overall model 8.11 (early stopping and dropout rate = 0.1) achieved the best results of the models considered (based on recall, F1-score and AUC).\n",
        "\n",
        "**We proceed with model 8.11 (with early stopping and neuron dropout) into our candidate model comparison at the end of this section.**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X8-slYCDlXd1",
      "metadata": {
        "id": "X8-slYCDlXd1"
      },
      "source": [
        "##### Candidate Model Comparisons and Summary\n",
        "\n",
        "We now compare the four candidate models after tuning the early stopping and dropout rate. We will narrow down to one candidate model that we apply additional tuning to and ultimately select as our final model.\n",
        "\n",
        "Let us look at the validation metrics of the candidate models:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 609,
      "id": "RqruDctAl5IQ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RqruDctAl5IQ",
        "outputId": "c4977044-5020-4745-8b25-7d5b40b3172b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 6.1 compared to model: 8.7\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.294118          0.444444      -\n",
            "1          Recall   0.384615          0.615385      -\n",
            "2        F1-score   0.333333          0.516129      -\n",
            "3        Accuracy   0.750000          0.812500      -\n",
            "4  Validation AUC   0.742824          0.796785      -\n",
            "5    Training AUC   0.900684          0.949689      -\n"
          ]
        }
      ],
      "source": [
        "# Compare model 6.1 and 8.7\n",
        "  # 6.1: 1 hidden layer. Features: 7 PCs, exclamation flag, sentiment_VADER\n",
        "  # 8.7: 1 hidden layer. Features: 5 PCs, exclamation flag, sentiment_VADER, Genre columns\n",
        "model_comparison(6.1, 8.7)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 610,
      "id": "Z6oUnVSol83Y",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z6oUnVSol83Y",
        "outputId": "7573af17-fb19-400b-96b1-5bbb6957bd0e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.4 compared to model: 8.11\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.500000          0.533333      -\n",
            "1          Recall   0.615385          0.615385      =\n",
            "2        F1-score   0.551724          0.571429      -\n",
            "3        Accuracy   0.837500          0.850000      -\n",
            "4  Validation AUC   0.810563          0.802526      +\n",
            "5    Training AUC   0.952236          0.965540      -\n"
          ]
        }
      ],
      "source": [
        "# Compare model 8.4 and 8.11\n",
        "  # 8.4: 1 hidden layer. Features: 7 PCs, exclamation flag, sentiment_VADER, Genre columns\n",
        "  # 8.11: 2 hidden layers. Features: 7 PCs, exclamation flag, sentiment_VADER, Genre columns\n",
        "model_comparison(8.4, 8.11)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "P7C3RaByoFa6",
      "metadata": {
        "id": "P7C3RaByoFa6"
      },
      "source": [
        "Recall is the most critical metric in our context due to the high cost of false negatives (expensive legal breaches). However, all the candidate models achieved the same recall so we use other metrics to narrow down our selection.\n",
        "\n",
        "Model 6.1 is clearly the worse as it has the lowest precision (29%), another key metric for this context. A low precision is not tolerable as it has a cost in the form of time wasted verifying model flags. Therefore, we reject model 6.1.\n",
        "\n",
        "Model 8.7 similarly has a low precision and hence F1-score compared to models 8.4 and 8.11. Model 8.7 is therefore also inferior to the other models and is rejected.\n",
        "\n",
        "Model 8.4 and 8.11 are similar as they have the same features but a different architecture. Both models obtained the same recall so they cannot be distinguished based on this metric. However, 8.4 has higher validation AUC while 8.11 has a higher F1-score. Ultimately, we decide to go with model 8.11 because of its higher precision and because it contains neuron dropout which may make it more robust on unseen data.\n",
        "\n",
        "**Model 8.11 is selected as our candidate model at this step**. Model 8.11 has the following characteristics:\n",
        "- Features:  7 embedding PCs, exclamation flag, sentiment_VADER, Genre columns\n",
        "- Two hidden-layer architecture\n",
        "- Early stopping and dropout rate of 0.1\n",
        "- Used the Adam optimiser to train\n",
        "\n",
        "Now as this model has two-hidden layers and a high number of features, it may still suffer from overfitting to the training data despite early stopping and neuron dropout. We therefore consider adding regularisation to the neural network."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "20KERDtXsGqW",
      "metadata": {
        "id": "20KERDtXsGqW"
      },
      "source": [
        "#### Q3b Step 12 - Regularisation\n",
        "**Outline**\n",
        "\n",
        "\n",
        "Our candidate model has two-hidden layers and a high number of features, so it is susceptible to overfitting despite early stopping and neuron dropout. This is liekly due to its high number of parameters. While neural networks can be over-parameterised and still generalise well (Bordelon, Canatar & Pehlevan, 2021), research suggests that regularisation can further improve model generalisability (Kukačka, Golkov & Cremers, 2017). We therefore consider adding regularisation to our neural network.\n",
        "\n",
        "Regularisation penalises large parameter values in the loss function, effectively simplifying the model. We experiment with several values for the regularisation hyperparameters but show only one example below.\n",
        "\n",
        "**References**\n",
        "\n",
        "Bordelon, B., Canatar, A., & Pehlevan, C. (2021). How neural networks escape perils of overparameterization. Harvard Brain Science Initiative. https://brain.harvard.edu/hbi_news/how-neural-networks-escape-perils-of-overparameterization/\n",
        "\n",
        "Kukačka, J., Golkov, V., & Cremers, D. (2017). Regularization for deep learning: A taxonomy. arXiv preprint arXiv:1710.10686."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 611,
      "id": "HoIqN24aupS4",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "HoIqN24aupS4",
        "outputId": "c479f936-4982-4b09-df28-42195ce32f2a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this Neural Network model is: 0 mins\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 85316 characters]",
            "text/plain": [
              "<Figure size 2000x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 19ms/step\n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  85.0%     54.5%     46.2%     50.0%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51980 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m35/35\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.90      0.93      0.91        67\n",
            "    problematic       0.55      0.46      0.50        13\n",
            "\n",
            "       accuracy                           0.85        80\n",
            "      macro avg       0.72      0.69      0.71        80\n",
            "   weighted avg       0.84      0.85      0.84        80\n",
            "\n",
            "Training AUC: 0.9560163306028145\n",
            "Validation AUC: 0.7876004592422503\n"
          ]
        }
      ],
      "source": [
        "# force keras to behave deterministically\n",
        "with tf.device(device_name):\n",
        "  keras_deterministic()\n",
        "\n",
        "# add regularisation to model 8.11\n",
        "nn_fit(ver=9.1,\n",
        "         train_x=train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         train_y=train_y,\n",
        "         test_x=val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])],\n",
        "         test_y=val_y,\n",
        "         l1_neurons=45,\n",
        "         l1_activation='relu',\n",
        "         l2_neurons=12,\n",
        "         l2_activation='relu',\n",
        "         learn_rate = 0.0005,\n",
        "         epochs=200,\n",
        "         batch_size=128,\n",
        "         optimiser = 'Adam',\n",
        "         verbose = False,\n",
        "         early_stop = True,\n",
        "         dropout_rate = 0.1,\n",
        "         l1_reg_strength = 0.01,\n",
        "         l2_reg_strength = 0.01)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 612,
      "id": "1e-VA9XyvCPQ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1e-VA9XyvCPQ",
        "outputId": "9a4a76c4-fad8-413e-9ecc-744df268365f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 9.1 compared to model: 8.11\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.545455          0.533333      +\n",
            "1          Recall   0.461538          0.615385      -\n",
            "2        F1-score   0.500000          0.571429      -\n",
            "3        Accuracy   0.850000          0.850000      =\n",
            "4  Validation AUC   0.787600          0.802526      -\n",
            "5    Training AUC   0.956016          0.965540      -\n"
          ]
        }
      ],
      "source": [
        "# compare results before and after regularisation:\n",
        "\n",
        "model_comparison(9.1, 8.11)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YEQ0ETCvwi3z",
      "metadata": {
        "id": "YEQ0ETCvwi3z"
      },
      "source": [
        "##### Section Summary\n",
        "\n",
        "After experimenting with different values for the regularisation hyperparameter we find that adding regularisation tends to lead to **marginally worse results**. There is a sizeable drop in recall which will lead to our model having more false negatives, which is costly for the movie producer. This suggests that our model may not suffer from signficiant overfitting after introducing early stopping and dropout.\n",
        "\n",
        "Therefore, we proceed with model ***8.11 as our final selected model*** (no regularisation)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dfQYEerNyHlZ",
      "metadata": {
        "id": "dfQYEerNyHlZ"
      },
      "source": [
        "#### Q3b Step 13 - Final Model vs. Initial Model\n",
        "\n",
        "We now compare our initial neural network model with our final model, 8.11\n",
        "\n",
        "---\n",
        "**Initial Model**\n",
        "- *Features*: 7 Principal Components of the the BERT embedding\n",
        "- *Architecture*: 1 hidden layer, 20 neurons\n",
        "- *Learning Rate*: 0.01\n",
        "- *Batch Size*: 64\n",
        "- *Optimiser*: Mini-batch gradient descent\n",
        "- No early stopping, neuron dropout or regularisation\n",
        "\n",
        "---\n",
        "**Final Model**\n",
        "- *Features*: 7 Principal Components of the the BERT embedding, exclamation mark flag, `Sentiment_VADER`, 10 genre columns\n",
        "- *Architecture*: 2 hidden layers, 45 + 12 neurons\n",
        "- *Learning Rate*: 0.0005\n",
        "- *Batch Size*: 128\n",
        "- *Optimiser*: Adam\n",
        "- Early stopping, neuron dropout of 0.1 and no regularisation\n",
        "\n",
        "---\n",
        "\n",
        "**Initial vs Final Model: Metric Comparison and Interpretation**\n",
        "\n",
        "We see a significant improvement in our model performance on the validation datasets, as shown by the metrics below. In particular, the precision and recall are both greater than 50% for our final model. This indicates that the 'problematic' quotes identified by the model, more often than not are actual problematic quotes (precision). Similarly, it suggests that actual problematic quotes are correctly identified most of the time (recall). This performance will be verified later when we look at the test set.\n",
        "\n",
        "It is clear that our iterative adjustment of the model has resulted in a better modelling outcome:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 613,
      "id": "jqQIFE2bz-uL",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jqQIFE2bz-uL",
        "outputId": "9d944183-aa3b-4931-dfa9-a3ad578d920b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metrics for model: 8.11 compared to model: 1\n",
            "           Metric  New Model  Comparison Model Change\n",
            "0       Precision   0.533333          0.285714      +\n",
            "1          Recall   0.615385          0.461538      +\n",
            "2        F1-score   0.571429          0.352941      +\n",
            "3        Accuracy   0.850000          0.725000      +\n",
            "4  Validation AUC   0.802526          0.679679      +\n",
            "5    Training AUC   0.965540          0.843707      +\n"
          ]
        }
      ],
      "source": [
        "model_comparison(8.11, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "61iOfBevOcEh",
      "metadata": {
        "id": "61iOfBevOcEh"
      },
      "source": [
        "#### Q3b Step 14 - Checks on Final Model Outcomes\n",
        "\n",
        "We do some final checks on the outcomes of our final model to ensure it is working as intended.\n",
        "\n",
        "- Check 1: Outputs two columns corresponding to the two classes: 'non-problematic' and 'problematic'\n",
        "\n",
        "- Check 2: Two output values should sum to 1 as they are probabilities\n",
        "\n",
        "- Check 3: Predicted probabilities should lie between 0 and 1\n",
        "\n",
        "- Check 4: Confusion matrix looks reasonable and is labelled correctly\n",
        "\n",
        "As the results below indicate, our final model passes all these checks."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "j6xizjw2PevE",
      "metadata": {
        "id": "j6xizjw2PevE"
      },
      "source": [
        "***Check 1: Outputs two columns***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 614,
      "id": "snzeiCS-PeS-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "snzeiCS-PeS-",
        "outputId": "b310d92c-64bd-4267-ff05-91a9cf498b2c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "execution_count": 614,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Check number of response classes predicted\n",
        "nn_models[8.11]['y_prediction_prob'].shape[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "A2V1QFTtRy5Q",
      "metadata": {
        "id": "A2V1QFTtRy5Q"
      },
      "source": [
        "We expect that our model outputs two predicted values for each observation, corresponding to 'non-problematic' and 'problematic'. The output above gives a value of 2 as expected, indicating that the model passes this check."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "UHHOyO4sP9_P",
      "metadata": {
        "id": "UHHOyO4sP9_P"
      },
      "source": [
        "***Check 2: Predicted probabilities for each observation sum to 100%***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 615,
      "id": "qwO3ch4fQbc8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qwO3ch4fQbc8",
        "outputId": "f5c65515-0654-4869-8d72-aa9134120a0d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The minimum value of the sum of probabilities is \n",
            "1.00\n",
            "\n",
            "\n",
            "The maximum value of the sum of probabilities is \n",
            "1.00\n"
          ]
        }
      ],
      "source": [
        "# Check probabilities in the final prediction matrix add up to 1\n",
        "print(\"The minimum value of the sum of probabilities is \")\n",
        "print(\"{:.2f}\".format(nn_models[8.11]['y_prediction_prob'].sum(axis = 1).min()))\n",
        "\n",
        "print('\\n')\n",
        "print(\"The maximum value of the sum of probabilities is \")\n",
        "print(\"{:.2f}\".format(nn_models[8.11]['y_prediction_prob'].sum(axis = 1).max()))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6lkFUKNTSY_f",
      "metadata": {
        "id": "6lkFUKNTSY_f"
      },
      "source": [
        "Our two output values represent probabilities and so they should sum to one. We see that the minimum and maximum value of the probability sums are both one (after rounding). Therefore, the model's outputs pass this sense check."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "JFPIsmtNQkX9",
      "metadata": {
        "id": "JFPIsmtNQkX9"
      },
      "source": [
        "***Check 3: All predicted probabilities are between 0 and 1***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 616,
      "id": "WpNPafh1RCmz",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "WpNPafh1RCmz",
        "outputId": "66e154e3-9eae-4b1f-fc1e-5c04406ef9a2"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": 0,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 28.05527607161119,\n        \"min\": 0.14421355724334717,\n        \"max\": 80.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          0.7509790658950806,\n          0.8363313674926758,\n          80.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": 1,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 28.188201245644947,\n        \"min\": 0.0028581346850842237,\n        \"max\": 80.0,\n        \"num_unique_values\": 8,\n        \"samples\": [\n          0.24902090430259705,\n          0.16366863995790482,\n          80.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-c9d7927e-54fc-4608-915f-3ec2bdd24624\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>80.000000</td>\n",
              "      <td>80.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.750979</td>\n",
              "      <td>0.249021</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.247283</td>\n",
              "      <td>0.247283</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.144214</td>\n",
              "      <td>0.002858</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.635168</td>\n",
              "      <td>0.043076</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.836331</td>\n",
              "      <td>0.163669</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.956924</td>\n",
              "      <td>0.364832</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>0.997142</td>\n",
              "      <td>0.855786</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c9d7927e-54fc-4608-915f-3ec2bdd24624')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c9d7927e-54fc-4608-915f-3ec2bdd24624 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c9d7927e-54fc-4608-915f-3ec2bdd24624');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-b2d8eac0-84a0-4f4c-a0e5-b41b92f5bea2\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-b2d8eac0-84a0-4f4c-a0e5-b41b92f5bea2')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-b2d8eac0-84a0-4f4c-a0e5-b41b92f5bea2 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "               0          1\n",
              "count  80.000000  80.000000\n",
              "mean    0.750979   0.249021\n",
              "std     0.247283   0.247283\n",
              "min     0.144214   0.002858\n",
              "25%     0.635168   0.043076\n",
              "50%     0.836331   0.163669\n",
              "75%     0.956924   0.364832\n",
              "max     0.997142   0.855786"
            ]
          },
          "execution_count": 616,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pd.DataFrame(nn_models[8.11]['y_prediction_prob']).describe()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uv50RtzlTTcu",
      "metadata": {
        "id": "uv50RtzlTTcu"
      },
      "source": [
        "As probabilities, we also expect the outputs to be between zero and one. It is possible they could sum to one but one of the values is negative.\n",
        "\n",
        "The output above gives the maximum and minimum values of each of the probability columns. The first column relates to 'non-problematic' probability while the second column relates to the 'problematic' probability. The maximum and minimum values for both columns are between zero and one as expected so the model passes this sense check."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XRh6_ceKQ7_R",
      "metadata": {
        "id": "XRh6_ceKQ7_R"
      },
      "source": [
        "***Check 4: Confusion matrix results look reasonable***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 617,
      "id": "Gwrcz0kURJU1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 593
        },
        "id": "Gwrcz0kURJU1",
        "outputId": "bffdba43-464e-4a13-c098-727a3cb38d1a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Counts of quote categories in the validation dataset:\n",
            "problematic\n",
            "0    67\n",
            "1    13\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 39484 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "print('Counts of quote categories in the validation dataset:')\n",
        "print(val_y_binary.value_counts())\n",
        "\n",
        "plot_confusion_matrix(metrics.confusion_matrix(nn_models[8.11]['y_actual_class'],nn_models[8.11]['y_prediction_class']), classes=data_labels, normalise=False, title='Confusion Matrix for the Validation Dataset (Count)')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "L9LUUAYdT2SH",
      "metadata": {
        "id": "L9LUUAYdT2SH"
      },
      "source": [
        "In this check we verify that the counts in the confusion matrix match with what we have in the validation data. We also check that the labels are correct.\n",
        "\n",
        "The first row in the confusion matrix represents actual non-problematic quotes. The row sums to 67 = 60 + 7 which matches the sum we see in the raw validation data.\n",
        "\n",
        "The second row in the confusion matrix represents actual problematic quotes. The row sums to 13 = 5 + 8 which matches what we see the in the raw validation date.\n",
        "\n",
        "Therefore, the raw data appears to match the labels and breakdown in the confusion matrix. Therefore the metrics we used to evaluate models during training are also correct as they are based off this confusion matrix. The final model therefore passes this check.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zvUxfDCtcnbl",
      "metadata": {
        "id": "zvUxfDCtcnbl"
      },
      "source": [
        "#### Q3b Step 15 - Tune Threshold Probability\n",
        "\n",
        "As till now we have used a threshold probability of 50% to decide if a quote is classified as problematic or not problematic. However, in our context the cost of false negatives is greater than the cost of false positives (false negatives may lead to breaches of legal contracts whereas FPs can be verified somewhat easily). We therefore look at using different threshold values to achieve better modelling outcomes.\n",
        "\n",
        "We are limited in time and computing power so will look at the outcomes of our model on the validation dataset using a small set of different threshold values.\n",
        "\n",
        "As we decrease the threshold values, we should expect the recall to increase as more problematic quotes are correctly identified. However, the precision may decrease as also incorrectly identify more non-problematic quotes. We test threshold probabilities of 20%, 30% and 40%."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 618,
      "id": "tqE_VDk6cm95",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tqE_VDk6cm95",
        "outputId": "5bcbddcb-7236-4acf-e1fd-861a0d80e229"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 9ms/step \n"
          ]
        }
      ],
      "source": [
        "final_model = nn_models[8.11]['history'].model\n",
        "final_val_x = val_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])]\n",
        "\n",
        "# Apply model to validation set\n",
        "y_val_pred_prob = final_model.predict(final_val_x)\n",
        "\n",
        "# Calculate class allocation at different thresholds:\n",
        "y_val_pred_class_20 = np.where(y_val_pred_prob[:,1] > 0.2, 1, 0)\n",
        "y_val_pred_class_30 = np.where(y_val_pred_prob[:,1] > 0.3, 1, 0)\n",
        "y_val_pred_class_40 = np.where(y_val_pred_prob[:,1] > 0.4, 1, 0)\n",
        "y_val_pred_class_50 = np.argmax(y_val_pred_prob, axis=1)\n",
        "\n",
        "# Get actual class:\n",
        "y_val_actual_class = np.argmax(val_y, axis=1)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 619,
      "id": "CfiP3ytvfG1U",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "CfiP3ytvfG1U",
        "outputId": "92ce29b5-4f7d-484f-fb5a-5d88e97b24a6"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 45808 characters]",
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Calculate ROC curve metrics (fpr, tpr, thresholds)\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "fpr, tpr, thresholds = roc_curve(y_val_actual_class, y_val_pred_prob[:,1])\n",
        "\n",
        "# Calculate Area Under the ROC Curve (AUC)\n",
        "roc_auc = auc(fpr, tpr)\n",
        "\n",
        "# Plot the ROC curve\n",
        "plt.figure(figsize=(8, 6))\n",
        "plt.plot(fpr, tpr, label='ROC Curve (area = %0.2f)' % roc_auc)\n",
        "plt.plot([0, 1], [0, 1], 'k--', label='No Skill')\n",
        "plt.xlim([0.0, 1.0])\n",
        "plt.ylim([0.0, 1.05])\n",
        "plt.xlabel('False Positive Rate (FPR)')\n",
        "plt.ylabel('True Positive Rate (TPR)')\n",
        "plt.title('ROC Curve')\n",
        "plt.legend(loc=\"lower right\")\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 620,
      "id": "K8v3aWT2gA5n",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K8v3aWT2gA5n",
        "outputId": "24e39093-c6a1-4b44-c4ba-06d513e2acc6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "20% Threshold results:\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.93      0.61      0.74        67\n",
            "    problematic       0.28      0.77      0.41        13\n",
            "\n",
            "       accuracy                           0.64        80\n",
            "      macro avg       0.60      0.69      0.57        80\n",
            "   weighted avg       0.83      0.64      0.69        80\n",
            "\n",
            "\n",
            "30% Threshold results:\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.93      0.75      0.83        67\n",
            "    problematic       0.35      0.69      0.46        13\n",
            "\n",
            "       accuracy                           0.74        80\n",
            "      macro avg       0.64      0.72      0.64        80\n",
            "   weighted avg       0.83      0.74      0.77        80\n",
            "\n",
            "\n",
            "40% Threshold results:\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.85      0.88        67\n",
            "    problematic       0.44      0.62      0.52        13\n",
            "\n",
            "       accuracy                           0.81        80\n",
            "      macro avg       0.68      0.73      0.70        80\n",
            "   weighted avg       0.84      0.81      0.82        80\n",
            "\n",
            "\n",
            "50% Threshold results:\n",
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.90      0.91        67\n",
            "    problematic       0.53      0.62      0.57        13\n",
            "\n",
            "       accuracy                           0.85        80\n",
            "      macro avg       0.73      0.76      0.74        80\n",
            "   weighted avg       0.86      0.85      0.85        80\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Print success metrics for each threshold value\n",
        "print('20% Threshold results:')\n",
        "print(classification_report(y_val_actual_class, y_val_pred_class_20, target_names=data_labels))\n",
        "print()\n",
        "print('30% Threshold results:')\n",
        "print(classification_report(y_val_actual_class, y_val_pred_class_30, target_names=data_labels))\n",
        "print()\n",
        "print('40% Threshold results:')\n",
        "print(classification_report(y_val_actual_class, y_val_pred_class_40, target_names=data_labels))\n",
        "print()\n",
        "print('50% Threshold results:')\n",
        "print(classification_report(y_val_actual_class, y_val_pred_class_50, target_names=data_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OxqR_0j2g1Aj",
      "metadata": {
        "id": "OxqR_0j2g1Aj"
      },
      "source": [
        "We see that as hypothesised, as we decrease the threshold probability the recall increased but the precision decreased.\n",
        "\n",
        "Using a threshold probability of 20% attained the highest recall (77%) but the lowest precision and F1-score (27% and 46%)\n",
        "\n",
        "Using a threshold probability of 40% and 50% attained the lowest recall (62%). The highest F1-score was obtained with a threshold of 50%.\n",
        "\n",
        "It is therefore difficult to decide which threshold probability is best. However, we choose to consider our business context and the relative cost of false positives vs false negatives. As mentioned previously, false negatives bear a much higher cost due to the breach of legal contracts and the risk of souring actor relationships. We therefore believe that the moderate increase in recall under a 20% threshold outweighs the loss of precision. So even though it obtained the lowest F1-score of the thresholds we tested, **we recommend using a threshold of 20%** due to the criticality of recall in our context."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "UXG_N0_eV2U7",
      "metadata": {
        "id": "UXG_N0_eV2U7"
      },
      "source": [
        "### Q3c - Calculate measures of success\n",
        "#### Purpose:\n",
        "In this section, we use our final model to make predictions on the test dataset. We will then calculate metrics of the model's performance on this test set and compare it to two appropriate benchmark models. This will allow us to assess how good the model will be at deployment and whether it is useful for the movie producer at identifying problematic quotes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mown4CzpC19E",
      "metadata": {
        "id": "mown4CzpC19E"
      },
      "source": [
        "#### Q3c Step 1 - Metric Justification and Calculations\n",
        "**Outline**\n",
        "\n",
        "***What and Why?***\n",
        "\n",
        "In this step, we use our selected model to predict the label of the test set quotes. We also calculate several test set metrics to evaluate model performance. We do this to understand how our final model will likely perform at deployment on truly unseen data.\n",
        "\n",
        "***Which metrics and justification for using them***\n",
        "\n",
        "We choose success metrics that are aligned with the movie producer's goal of identifying problematic quotes in scripts and will be easy for the board to understand.\n",
        "\n",
        "Accuracy is not aligned with our objective as a model classifying all quotes as 'non-problematic' would have high accuracy but be useless for flagging problematic quotes.\n",
        "For the evaluation we focus on precision, recall, and F1-score as success measures for the model. These metrics consider the cost of misclassifying problematic quotes and so are highly relevant to our problem context.\n",
        "\n",
        "**Recall** is a very important success measure for our context. A high recall indicates a low false negative rate. False negatives are risky for the movie producer as it may result in scripts being passed to actors that break their contracts. This can lead to expensive reshoots, delays and soured relationships. Therefore it is imperative our model has a high recall, making it our key success metric.\n",
        "\n",
        "**Precision** measures the number of predicted positives that are true positives. A low precision means that most of the model's positive predictions are irrelevant. In our context this is costly as it will waste resources verifying that quotes are problematic. Some false positives are tolerable as a human can quickly verify the model output for a few wrong samples. Therefore, having a good level of precision is important for our context, although recall is more critical.\n",
        "\n",
        "**F1-score** is a balance between the two metrics as often pursuing one can be at the cost of the other. For example, a model can achieve high recall by predicting all quotes as problematic, but the precision would be low and so the model would be useless. A high F1-score is hence a good overall measure of model performance.\n",
        "\n",
        "Highlighting both the recall and precision can communicate what the model can do well but also possibly what may go wrong, which is important to know when deploying a model. Furthermore, **these metrics are easy to understand and easy to communicate to the board of directors**. They can easily be used to assess whether the model is suitable for the intended purpose.\n",
        "\n",
        "Additionally, we use the **test AUC** to compare our selected model to a model that just randomly guesses the class. A model that just randomly guessed whether a quote was problematic or not would be expected to achieve an AUC of 0.5. The AUC will not be directly communicated with the executives but its interpretation may be (i.e. whether the model is better than a random classifier).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X5tQERuD4shg",
      "metadata": {
        "id": "X5tQERuD4shg"
      },
      "source": [
        "---\n",
        "The test set is meant to represent unseen data, similar to what the model would be used on if it was deployed. We therefore use the model to predict the label of quotes in the test data and calculate our success metrics.\n",
        "\n",
        "In a previous step, we recommended that a threshold of 20% be used when deploying our model due to the criticality of recall in our context. We will therefore use this threshold when predicting the classes of our test set and evaluating our model performance so as to mimic our recommended deployment environment. However, as a point of comparison we first look at model performance using a 50% threshold (the default threshold). This may help us determine if the threshold recommended was appropriate.\n",
        "\n",
        "We test model performance using a 50% threshold:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 621,
      "id": "907bd4ee-2af5-4d41-861d-2643bb9af2d0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 733
        },
        "id": "907bd4ee-2af5-4d41-861d-2643bb9af2d0",
        "outputId": "18f9c250-1ab2-4d71-d3c5-b2f70d33671c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1m3/3\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 5ms/step \n",
            "Accuracy  Precision  Recall   F1-score  \n",
            "  77.5%     14.3%     25.0%     18.2%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52716 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.83      0.87        72\n",
            "    problematic       0.14      0.25      0.18         8\n",
            "\n",
            "       accuracy                           0.78        80\n",
            "      macro avg       0.53      0.54      0.53        80\n",
            "   weighted avg       0.83      0.78      0.80        80\n",
            "\n",
            "Test AUC: 0.5017361111111112\n"
          ]
        }
      ],
      "source": [
        "### Make predictions on the test set and calculate success metrics ###\n",
        "\n",
        "# Select the final NN model\n",
        "final_model = nn_models[8.11]['history'].model\n",
        "final_test_x = test_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])]\n",
        "\n",
        "# Apply model to test set\n",
        "y_test_pred_prob = final_model.predict(final_test_x)\n",
        "y_test_pred_class = np.argmax(y_test_pred_prob, axis=1)\n",
        "y_test_actual_class = np.argmax(test_y, axis=1)\n",
        "\n",
        "# Print success metrics\n",
        "test_scores = success_metrics(y_test_actual_class, y_test_pred_class)\n",
        "metrics_print(test_scores, confused=True)\n",
        "\n",
        "# Print classification report\n",
        "print(classification_report(y_test_actual_class, y_test_pred_class, target_names=data_labels))\n",
        "\n",
        "# Calculate AUC for the test set\n",
        "test_auc = roc_auc_score(test_y, y_test_pred_prob)\n",
        "print(\"Test AUC:\", test_auc)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VlzNjeeQ3fUR",
      "metadata": {
        "id": "VlzNjeeQ3fUR"
      },
      "source": [
        "***Summary of Test Metrics using 50% threshold***\n",
        "\n",
        "On the test set our model achieved:\n",
        "\n",
        "- **Precision**: 14%\n",
        "- **Recall**: 25%\n",
        "- **F1-score**: 18%\n",
        "- **Test AUC**: 0.502\n",
        "\n",
        "The model appears to have performed substantially worse on the test set than was achieved on the validation and training sets. It has an AUC of 0.502, indicating the model may be only marginally better than a random classifier. Of the 8 problematic quotes in the test dataset, only 2 of them were correctly identified (recall = 25%). Recall is our most critical success metric and it is difficult to justify a 75% false negative rate, given the cost of FNs in our context.\n",
        "\n",
        "---\n",
        "\n",
        "These metrics were calculated using a threshold probability of 50%. Let us experiment with using our recommended lower threshold probability of 20% and see how the success metrics compare. In particular, we should expect the recall to increase but the precision to possibly decrease. Let us test using a threshold probability of 20%:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 622,
      "id": "2p5pt2oPpCcF",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 698
        },
        "id": "2p5pt2oPpCcF",
        "outputId": "9f096081-63ae-4195-d0a3-bab7b861ba94"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy  Precision  Recall   F1-score  \n",
            "  62.5%     13.3%     50.0%     21.1%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52136 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.92      0.64      0.75        72\n",
            "    problematic       0.13      0.50      0.21         8\n",
            "\n",
            "       accuracy                           0.62        80\n",
            "      macro avg       0.53      0.57      0.48        80\n",
            "   weighted avg       0.84      0.62      0.70        80\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# use threshold probability of 20%\n",
        "y_test_pred_class_20 = np.where(y_test_pred_prob[:,1] > 0.2, 1, 0)\n",
        "\n",
        "# Print success metrics\n",
        "test_scores_20 = success_metrics(y_test_actual_class, y_test_pred_class_20)\n",
        "metrics_print(test_scores_20, confused=True)\n",
        "\n",
        "# Print classification report\n",
        "print(classification_report(y_test_actual_class, y_test_pred_class_20, target_names=data_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XCCkevFFq63p",
      "metadata": {
        "id": "XCCkevFFq63p"
      },
      "source": [
        "***Summary of Test Metrics using 20% threshold***\n",
        "\n",
        "Using a threhsold of 20% our model achieved the following on the test set:\n",
        "\n",
        "- **Precision**: 14%\n",
        "- **Recall**: 50%\n",
        "- **F1-score**: 22%\n",
        "- **Test AUC**: 0.502\n",
        "\n",
        "We see that decreasing the threshold has resulted in the recall increasing from 25% to 50%. While the number of false positives increased the precision rate remained unchanged overall, leading to an increase in F1-score compared to using a 50% threshold. Given the significant cost of false negatives, a recall of 50% may still not be high enough for deployment to be recommended.\n",
        "\n",
        "This suggest that using a 20% threshold was more suitable than 50% as it resulted in an increase in recall (our main success metric) at no cost to precision (another important success metric)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 623,
      "id": "1FH3nAjV0yuo",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "1FH3nAjV0yuo",
        "outputId": "ca715e10-6fc8-4c2f-a043-44af095f13f9"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 45796 characters]",
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Calculate ROC curve metrics (fpr, tpr, thresholds)\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "fpr, tpr, thresholds = roc_curve(y_test_actual_class, y_test_pred_prob[:,1])\n",
        "\n",
        "# Calculate Area Under the ROC Curve (AUC)\n",
        "roc_auc = auc(fpr, tpr)\n",
        "\n",
        "# Plot the ROC curve\n",
        "plt.figure(figsize=(8, 6))\n",
        "plt.plot(fpr, tpr, label='ROC Curve (area = %0.2f)' % roc_auc)\n",
        "plt.plot([0, 1], [0, 1], 'k--', label='No Skill')\n",
        "plt.xlim([0.0, 1.0])\n",
        "plt.ylim([0.0, 1.05])\n",
        "plt.xlabel('False Positive Rate (FPR)')\n",
        "plt.ylabel('True Positive Rate (TPR)')\n",
        "plt.title('ROC Curve')\n",
        "plt.legend(loc=\"lower right\")\n",
        "plt.grid(True)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mP3bDKRV7vVQ",
      "metadata": {
        "id": "mP3bDKRV7vVQ"
      },
      "source": [
        "The plot of the ROC curve confirms that the model is not much better than a random classifier at identifying problematic quotes (the curve approximately follows the diagonal line). Looking at the top-right of the graph highlights that there are a few problematic quotes that were predicted to have a low probability of being problematic by the model. This suggests the model had essentially no predictive power to identify these few quotes as problematic. Overall, the ROC curve highlights that our model does not have high predictive power for problematic quotes.\n",
        "\n",
        "Next let us look at the quotes in the test set that the model was able to correctly identify:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 624,
      "id": "DOyEdj1jNbHA",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 192
        },
        "id": "DOyEdj1jNbHA",
        "outputId": "9245add9-c3e7-4d75-97d2-f4a59074c295"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"TP_quotes\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"That light saber was luke's, and his father's before him, and now it calls to you!\",\n          \"Get away from her, you BITCH!!!\",\n          \"They\\u00e2\\u20ac\\u2122ve done studies, you know. 60% of the time, it works every time.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT_reliable\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"4. rude\",\n          \"1. sex\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "TP_quotes"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-d93f9f83-7adb-4164-90f6-acc9731f2119\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>label_ChatGPT_reliable</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>524</th>\n",
              "      <td>Theyâ€™ve done studies, you know. 60% of the time, it works every time.</td>\n",
              "      <td>1. sex</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>798</th>\n",
              "      <td>That light saber was luke's, and his father's before him, and now it calls to you!</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Get away from her, you BITCH!!!</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d93f9f83-7adb-4164-90f6-acc9731f2119')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d93f9f83-7adb-4164-90f6-acc9731f2119 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d93f9f83-7adb-4164-90f6-acc9731f2119');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3d82cbbf-2a94-43fa-956a-17f72f1a54b1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3d82cbbf-2a94-43fa-956a-17f72f1a54b1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3d82cbbf-2a94-43fa-956a-17f72f1a54b1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_a07fb6b3-cc77-42b5-b4f2-0f314754d273\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('TP_quotes')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_a07fb6b3-cc77-42b5-b4f2-0f314754d273 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('TP_quotes');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                 Movie Quote  \\\n",
              "524                                                                                                                                                                                                                  Theyâ€™ve done studies, you know. 60% of the time, it works every time.   \n",
              "798                                                                                                                                                                                                       That light saber was luke's, and his father's before him, and now it calls to you!   \n",
              "45   I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.   \n",
              "89                                                                                                                                                                                                                                                           Get away from her, you BITCH!!!   \n",
              "\n",
              "    label_ChatGPT_reliable  \n",
              "524                 1. sex  \n",
              "798                4. rude  \n",
              "45                 4. rude  \n",
              "89                 4. rude  "
            ]
          },
          "execution_count": 624,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# First get problematic quotes that were correctly identified:\n",
        "TP_indices = np.where((y_test_pred_class_20 == 1) & (y_test_actual_class == 1))[0]\n",
        "\n",
        "TP_quotes = test.iloc[TP_indices, ][['Movie Quote', 'label_ChatGPT_reliable']]\n",
        "\n",
        "TP_quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "G341FMxENskC",
      "metadata": {
        "id": "G341FMxENskC"
      },
      "source": [
        "Above are the test quotes that were correctly predicted as 'problematic'. Most of the identified quotes belonged to the 'rude' category, with one belonging to the 'sex' category. This indiciates that our model may be able to discern different problem categories. There is however no flagged quotes from 'politics' or 'religion'. This highlights a potential limitation of our model: it struggles with the more nuanced taboo topics. Alternatively there may not be good representation of these quotes in the test dataset.\n",
        "\n",
        "Also, it is questionable if the second true positive quote is actually rude which highlights a major limitation of our model: poor quality data. Further discussion of this is left to Q4."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ilSRek8i-G7t",
      "metadata": {
        "id": "ilSRek8i-G7t"
      },
      "source": [
        "#### Q3c Step 2 - Comparison to Benchmark Models\n",
        "**Outline**\n",
        "\n",
        "To assess the fit of our final model, we compare its results on the test set to what would be achieved with three simple benchmark models. This will allow us to evaluate how useful the neural network is and understand whether a simpler model may have been appropriate. We compare the model to:\n",
        "\n",
        "- A Random classifier where the probability of labelling a quote as problematic is based on the proportion of problematic quotes in the training data. Our classifier should at minimum beat this model to be considered useful.\n",
        "- A Rules-based classifier: searches for keywords such as swear words in quotes to flag quotes as problematic. This is an appropriate benchmark as it may indicate whether a rules-based approach could be more useful than a statistical model.\n",
        "- A simple GBM: a less complex statistical model may be able to achieve better results than a neural network indicating that simpler modelling approaches should be considered."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MbFOVdLB7g_b",
      "metadata": {
        "id": "MbFOVdLB7g_b"
      },
      "source": [
        "---\n",
        "##### Benchmark 1: Random Classifier"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 625,
      "id": "MTGEy-9a-JNK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MTGEy-9a-JNK",
        "outputId": "f728d668-6ce3-44b2-de3d-8f312b046c79"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Simple Benchmark Model 1: predict a quote as problematic with a probability of 0.13\n",
            "\n",
            "Prediction Summary:\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0    74\n",
              "1     6\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 625,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# First calculate proportion of 'problematic' quotes in the training dataset to use for the benchmark's guessing probability\n",
        "\n",
        "problematic_prob = train_y_binary.mean()\n",
        "print(f'Simple Benchmark Model 1: predict a quote as problematic with a probability of {problematic_prob:.2f}')\n",
        "\n",
        "np.random.seed(random_seed)\n",
        "benchmark_y_1 = np.random.choice([0, 1], size=len(test_y), p=[1-problematic_prob, problematic_prob])\n",
        "\n",
        "print()\n",
        "print('Prediction Summary:')\n",
        "pd.DataFrame(benchmark_y_1).value_counts()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 626,
      "id": "yWBgX0VM-Szq",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 715
        },
        "id": "yWBgX0VM-Szq",
        "outputId": "714ecbe5-6616-4705-d1b3-5efb1a5900d1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy  Precision  Recall   F1-score  \n",
            "  85.0%     16.7%     12.5%     14.3%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 52052 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.93      0.92        72\n",
            "    problematic       0.17      0.12      0.14         8\n",
            "\n",
            "       accuracy                           0.85        80\n",
            "      macro avg       0.54      0.53      0.53        80\n",
            "   weighted avg       0.83      0.85      0.84        80\n",
            "\n",
            "Test AUC: 0.5277777777777778\n"
          ]
        }
      ],
      "source": [
        "### Now create the confusion matrix and success metrics for benchmark model 1\n",
        "\n",
        "# Print success metrics\n",
        "test_scores = success_metrics(y_test_actual_class, benchmark_y_1)\n",
        "metrics_print(test_scores, confused=True)\n",
        "\n",
        "# Print classification report\n",
        "print(classification_report(y_test_actual_class, benchmark_y_1, target_names=data_labels))\n",
        "\n",
        "# Calculate AUC for the test set\n",
        "test_auc = roc_auc_score(test_y, to_categorical(benchmark_y_1))\n",
        "print(\"Test AUC:\", test_auc)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VtxRMVdz94vD",
      "metadata": {
        "id": "VtxRMVdz94vD"
      },
      "source": [
        "***Success Metrics for Benchmark 1***:\n",
        "\n",
        "- **Precision**: 17% (vs. 14% for final model with 20% threshold)\n",
        "- **Recall**: 12% (vs. 50% for final model)\n",
        "- **F1-score**: 14% (vs. 22% for final model)\n",
        "- **Test AUC**: 0.528 (vs. 0.502 for final model)\n",
        "\n",
        "Our final model's F1-score and Recall are higher than the random classifier, although the precision is marginally worse. Based on recall, the metric most critical to our context, **our model does outperform a random classifier** at identifying problematic quotes. This however is not a particularly meaningful comparison as a random classifier has no actual predictive power. We therefore look at two more benchmark models."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_MiMC1vdCrRC",
      "metadata": {
        "id": "_MiMC1vdCrRC"
      },
      "source": [
        "---\n",
        "##### Benchmark 2: Rules-based classifier\n",
        "\n",
        "We now consider if we had used a model that labels quotes as problematic based on intuitive rules rather than a statistical model. We will apply the following rules:\n",
        "\n",
        "- If a quote contains profanity it will be classified as 'problematic'. We use the profanity package for this.\n",
        "- If a quote contains 'God', 'sex', or 'religion' then it will also be labelled as 'problematic'.\n",
        "\n",
        "This is a very simple rules-based approach and does not capture many of the dimensions of a quote being problematic such as 'politics' or being insensitive. It is not a very complex model either with fairly simple word searchs. We would therefore hope that our model is more effective but if not it may highlight that a keyword-based approach is more appropriate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 627,
      "id": "dchvkhPGGI41",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dchvkhPGGI41",
        "outputId": "ab998699-c8ef-419d-826b-b611fa211501"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Benchmark 2 Prediction Summary:\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0    74\n",
              "1     6\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 627,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Use rules-based approach to classify quotes:\n",
        "\n",
        "def create_rules_based_response(data):\n",
        "\n",
        "  words_to_find = ['god', 'sex', 'religion']\n",
        "  response = data.apply(lambda row: 1 if profanity.contains_profanity(row['Quote_clean']) or any(word in row['Quote_clean'] for word in words_to_find) else 0, axis = 1)\n",
        "\n",
        "  return response\n",
        "\n",
        "benchmark_y_2 = create_rules_based_response(test)\n",
        "\n",
        "print()\n",
        "print('Benchmark 2 Prediction Summary:')\n",
        "pd.DataFrame(benchmark_y_1).value_counts()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 628,
      "id": "C9Kx08hiIEVv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 715
        },
        "id": "C9Kx08hiIEVv",
        "outputId": "dff1c3c4-0451-4036-ab06-d1ec57298b88"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy  Precision  Recall   F1-score  \n",
            "  95.0%    100.0%     50.0%     66.7%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 51328 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.95      1.00      0.97        72\n",
            "    problematic       1.00      0.50      0.67         8\n",
            "\n",
            "       accuracy                           0.95        80\n",
            "      macro avg       0.97      0.75      0.82        80\n",
            "   weighted avg       0.95      0.95      0.94        80\n",
            "\n",
            "Test AUC: 0.75\n"
          ]
        }
      ],
      "source": [
        "# Now create confusion matrix and success metrics.\n",
        "\n",
        "# Print success metrics\n",
        "test_scores = success_metrics(y_test_actual_class, benchmark_y_2)\n",
        "metrics_print(test_scores, confused=True)\n",
        "\n",
        "# Print classification report\n",
        "print(classification_report(y_test_actual_class, benchmark_y_2, target_names=data_labels))\n",
        "\n",
        "# Calculate AUC for the test set\n",
        "  # we assume the predicted probability is 1 for the category predicted using the rules based approach\n",
        "test_auc = roc_auc_score(test_y, to_categorical(benchmark_y_2))\n",
        "print(\"Test AUC:\", test_auc)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 629,
      "id": "LQPKupwA-Xe_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 192
        },
        "id": "LQPKupwA-Xe_",
        "outputId": "59f6877e-9b9f-4ab1-a3f3-ab66ea142d4c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"TP_quotes\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"So many times in my life I screwed up: I've talked when I should've listened, I've been harsh when I should've been tender.\",\n          \"Get away from her, you BITCH!!!\",\n          \"I'm not a big fat panda. I'm THE big fat panda.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT_reliable\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"4. rude\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "TP_quotes"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-8dbfdbad-bf6f-4cdd-87dd-09e2ffe4fbed\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>label_ChatGPT_reliable</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>622</th>\n",
              "      <td>I'm not a big fat panda. I'm THE big fat panda.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>743</th>\n",
              "      <td>So many times in my life I screwed up: I've talked when I should've listened, I've been harsh when I should've been tender.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Get away from her, you BITCH!!!</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8dbfdbad-bf6f-4cdd-87dd-09e2ffe4fbed')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8dbfdbad-bf6f-4cdd-87dd-09e2ffe4fbed button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8dbfdbad-bf6f-4cdd-87dd-09e2ffe4fbed');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6a29ec24-9a5a-4b8b-8e0e-bab2acd4ca97\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6a29ec24-9a5a-4b8b-8e0e-bab2acd4ca97')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6a29ec24-9a5a-4b8b-8e0e-bab2acd4ca97 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_22eddf06-1fc1-4108-9d3e-f381231ef7cd\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('TP_quotes')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_22eddf06-1fc1-4108-9d3e-f381231ef7cd button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('TP_quotes');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                 Movie Quote  \\\n",
              "622                                                                                                                                                                                                                                          I'm not a big fat panda. I'm THE big fat panda.   \n",
              "743                                                                                                                                                              So many times in my life I screwed up: I've talked when I should've listened, I've been harsh when I should've been tender.   \n",
              "45   I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.   \n",
              "89                                                                                                                                                                                                                                                           Get away from her, you BITCH!!!   \n",
              "\n",
              "    label_ChatGPT_reliable  \n",
              "622                4. rude  \n",
              "743                4. rude  \n",
              "45                 4. rude  \n",
              "89                 4. rude  "
            ]
          },
          "execution_count": 629,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# look at quotes that were TPs from this keyword-based classifier\n",
        "\n",
        "# First get problematic quotes that were correctly identified:\n",
        "TP_indices = np.where((benchmark_y_2 == 1) & (y_test_actual_class == 1))[0]\n",
        "\n",
        "TP_quotes = test.iloc[TP_indices, ][['Movie Quote', 'label_ChatGPT_reliable']]\n",
        "\n",
        "TP_quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QIEyVb5XIiok",
      "metadata": {
        "id": "QIEyVb5XIiok"
      },
      "source": [
        "***Success Metrics for Benchmark 2***:\n",
        "\n",
        "- **Precision**: 100% (vs. 14% for final model)\n",
        "- **Recall**: 50% (vs. 50% for final model)\n",
        "- **F1-score**: 67% (vs. 22% for final model)\n",
        "- **Test AUC**: 0.75 (vs. 0.502 for final model)\n",
        "\n",
        "We see that this rules-based metric has performed better than our final classifier (higher F1-score). While the recall was the same for the two models (50%), the better results for the rules-based classifier is mostly a result of it achieving a precision of 100%. We expect this model to achieve a high precision as the method was very mechanical and played it safe with its predictions.\n",
        "\n",
        "Furthermore, we see that the quotes the keyword-based classifier only identifies quotes in the 'rude' category as problematic and so this model is not effective for other categories. The quotes it identifies as problematic are slightly different to the quotes flagged by our model (see step 4 below), indicating that the two models may be useful for different dimensions of what makes a quote problematic.\n",
        "\n",
        "It could be argued that there is some leakage in the keyword-based model since the response was also constructed by identifying if a quote contained profanity. However this comparison highlights that our neural network has not been able to effectively learn whether a quote contains profanity. It indicates that a rules based approach may be more effective or at least could be useful to supplement our neural network model.\n",
        "\n",
        "Overall, the two models do achieve the same recall rate indicating they are similar in their predictive power."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "esRsVe-QJoah",
      "metadata": {
        "id": "esRsVe-QJoah"
      },
      "source": [
        "---\n",
        "##### Benchmark 3: Simple GBM\n",
        "\n",
        "We now train a simple GBM model to investigate whether a simpler modelling approach would have achieved better results than our final model. If our model cannot outperform this model then we may want to consider simpler models in the future. This may also identify that our neural network has overfit noise in the training data.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 630,
      "id": "hESS0YlmLnji",
      "metadata": {
        "id": "hESS0YlmLnji"
      },
      "outputs": [],
      "source": [
        "# Prepare training data for GBM\n",
        "\n",
        "# We do not use SMOTE here, rather we add extra weight to misclassifying 'problematic' quotes to handle imbalanced data\n",
        "unique_classes = np.unique(train_y_binary)\n",
        "class_weights = compute_class_weight('balanced', classes=unique_classes, y=train_y_binary)\n",
        "class_weights_dict = {cls: weight for cls, weight in zip(unique_classes, class_weights)}\n",
        "\n",
        "# Convert class weights to sample weights for each instance in the training set\n",
        "sample_weights = [class_weights_dict[label] for label in train_y_binary]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 631,
      "id": "B2bNtioTXBPy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B2bNtioTXBPy",
        "outputId": "13be80b0-2d05-4ad6-cdfc-f6d91e66ec72"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Time taken to run this model is: 0 mins\n"
          ]
        }
      ],
      "source": [
        "# Fit simple GBM:\n",
        "from sklearn.ensemble import GradientBoostingClassifier\n",
        "gbm_model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3, loss='log_loss', random_state=random_seed)\n",
        "\n",
        "start_time = time.time()\n",
        "gbm_model.fit(train_NN_X_scaled[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])], train_y_binary, sample_weight=sample_weights)\n",
        "end_time = time.time()\n",
        "\n",
        "print('Time taken to run this model is: {:.0f} mins'.format(round((end_time - start_time)/60, 2)))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 632,
      "id": "Vg7PNr4hK96g",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 715
        },
        "id": "Vg7PNr4hK96g",
        "outputId": "e112ee21-d363-458e-9d77-9d0874cd2f71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy  Precision  Recall   F1-score  \n",
            "  53.8%     10.8%     50.0%     17.8%   \n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 50540 characters]",
            "text/plain": [
              "<Figure size 1200x500 with 3 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "                 precision    recall  f1-score   support\n",
            "\n",
            "non-problematic       0.91      0.54      0.68        72\n",
            "    problematic       0.11      0.50      0.18         8\n",
            "\n",
            "       accuracy                           0.54        80\n",
            "      macro avg       0.51      0.52      0.43        80\n",
            "   weighted avg       0.83      0.54      0.63        80\n",
            "\n",
            "AUC: 0.48611111111111116\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Print evaluation metrics\n",
        "y_pred_prob = gbm_model.predict_proba(test_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER'])])\n",
        "y_pred_class = np.where(y_pred_prob[:,1] > 0.2, 1, 0)\n",
        "y_actual_class = np.argmax(test_y, axis=1)\n",
        "\n",
        "# Print success matrix and confusion matrix\n",
        "scores = success_metrics(y_actual_class, y_pred_class)\n",
        "metrics_print(scores, confused=True)\n",
        "\n",
        "# Print classification report\n",
        "print(classification_report(y_actual_class, y_pred_class, target_names=data_labels))\n",
        "\n",
        "# Calculate AUC\n",
        "auc = roc_auc_score(test_y, y_pred_prob)\n",
        "print(\"AUC:\", auc)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DZj8LcWrXkYF",
      "metadata": {
        "id": "DZj8LcWrXkYF"
      },
      "source": [
        "***Success Metrics for Benchmark 3***:\n",
        "\n",
        "- **Precision**: 9% (vs. 14% for final model)\n",
        "- **Recall**: 38% (vs. 50% for final model)\n",
        "- **F1-score**: 15% (vs. 22% for final model)\n",
        "- **Test AUC**: 0.438 (vs. 0.502 for final model)\n",
        "\n",
        "Based on all our success metrics, **our final classifier outperformed the benchmark GBM**. This suggests that the decision boundary for problematic quotes is quite complex and so a more complex model (i.e. a neural network) is likely needed. The benchmark GBM performed worse than the random classifier. This may also highlight that the test set may have been quite different to our training dataset, resulting in poor model performance for our NN."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AaMBOXcIADNw",
      "metadata": {
        "id": "AaMBOXcIADNw"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "The benchmark comparisons revealed several insights about out final model outcomes.\n",
        "\n",
        "- Our model outperformed random selection suggesting that our final model has some predictive power.\n",
        "- A simple rules-based classifier performed similarly to our final model, based on recall. However, they captured different types of quotes so a combination of the two approaches could be explored in the future.\n",
        "- A simple GBM performed worse than our NN model. This indicates that a more complex model is needed to effectively predict problematic quotes. In the future, the business may want to investigate a deeper neural network for this problem."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7-Esze1qdEZB",
      "metadata": {
        "id": "7-Esze1qdEZB"
      },
      "source": [
        "#### Q3c Step 3 - Feature Importance\n",
        "\n",
        "In this section we create some plots to better understand our final model. This includes a SHAP plot to understand the most important model features. We do this to aid our communication of the model's limitations and abilities to the executives in Q4."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LB8RGfKceDR8",
      "metadata": {
        "id": "LB8RGfKceDR8"
      },
      "source": [
        "---\n",
        "\n",
        "##### Feature Importance SHAP"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 633,
      "id": "yc3p3QhieO6-",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "1d4cd5b95d4245d2a5ecd54bcbec526c",
            "f8e6684340334b3f83274abc2bde73fc",
            "9be42af1ec70421c96a6d07f94fe4309",
            "57d619e4da874bb4b351e26e53021625",
            "172c86288ee64c77b750b26c0abe3f15",
            "d5123101af044321b6cc92793aeecc94",
            "3886cd94444c4779abd58d6ac64c27af",
            "371bc66b36204a50af02f67ef34d4601",
            "8767387f3f99442b895f0b4c13085cd9",
            "c1d813e6f89548769657995e998d4220",
            "332321a2620e440fbd1b22a99f812397"
          ]
        },
        "id": "yc3p3QhieO6-",
        "outputId": "45a2fa3b-a5c0-4f87-ab89-d9d5f5e64b13"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1d4cd5b95d4245d2a5ecd54bcbec526c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/80 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Use SHAP explainer object to explain model predictions on the test data:\n",
        "final_model = nn_models[8.11]['history'].model\n",
        "\n",
        "# only use first 100 samples due to long run time\n",
        "iRow1 = np.random.randint(train_x.shape[0], size=100)\n",
        "train_SHAP = train_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])][iRow1, :]\n",
        "explainer = shap.KernelExplainer(final_model, train_SHAP)\n",
        "\n",
        "test_SHAP = test_x[:,get_column_indices(column_names, ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance'])]\n",
        "shap_values = explainer.shap_values(test_SHAP)[:, :, 0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 634,
      "id": "aXdz9osQw4JZ",
      "metadata": {
        "id": "aXdz9osQw4JZ"
      },
      "outputs": [],
      "source": [
        "# Save the samples used in the SHAP predictions above for the force plot later\n",
        "df = pd.DataFrame(columns=['Non-Problematic', 'Problematic'])\n",
        "for sublist in nn_models[8.11]['y_prediction_prob']:\n",
        "  data = {'Non-Problematic': sublist[0], 'Problematic': sublist[1]}\n",
        "  df = pd.concat([df, pd.DataFrame([data])], ignore_index=True)\n",
        "\n",
        "predictions = df.reset_index()\n",
        "shap_pred = predictions.loc[predictions['index']]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 635,
      "id": "SOALQLTleWzV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 900
        },
        "id": "SOALQLTleWzV",
        "outputId": "6c49fd64-23aa-4cc3-835e-4e72fd97376f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:matplotlib.legend:No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 70740 characters]",
            "text/plain": [
              "<Figure size 1500x800 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Check the SHAP feature importance for the overall model\n",
        "feature_names = ['PC1','PC2','PC3','PC4','PC5','PC6','PC7','exclamation_flag', 'Sentiment_VADER','Drama','Comedy', 'ActionAdventure', 'Crime', 'ThrillerHorror', 'Fantasy','Sci-Fi', 'Animation', 'Romance']\n",
        "\n",
        "shap.summary_plot(shap_values, features = test_SHAP, plot_type = 'bar',\n",
        "                  feature_names = feature_names,\n",
        "                  max_display = 50,\n",
        "                  show=False\n",
        "                 )\n",
        "\n",
        "fig = plt.gcf()\n",
        "fig.set_size_inches(15, 8)\n",
        "leg = plt.legend()\n",
        "ax = plt.gca() #gca means \"get current axes\"\n",
        "ax.set_title('Feature Importance - Final Model', fontsize=20, pad=20)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "euJIbGbdlMMt",
      "metadata": {
        "id": "euJIbGbdlMMt"
      },
      "source": [
        "In our classifier, the most important features are:\n",
        "- the first principal component for the BERT embedding\n",
        "- the seventh principal component for the BERT embedding\n",
        "- whether a quote is from an animated movie\n",
        "- whether a quote is from a romance movie\n",
        "\n",
        "Interestingly, the principal component directions that explain the most variance do not necessarily correspond with the directions most related to the response. For example, we see the 7th principal component ranked highly in importance, while the 2nd component does not. We do not necessarily know what these PCs correspond to in terms of quote meaning and further exploration would be needed to find out (this is outside our scope)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lOgoAf2n8pWt",
      "metadata": {
        "id": "lOgoAf2n8pWt"
      },
      "source": [
        "#### Q3c Step 4 - Analysis of True Positive and False Negative Quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "GfgD6PnRnAaR",
      "metadata": {
        "id": "GfgD6PnRnAaR"
      },
      "source": [
        "---\n",
        "\n",
        "We now look at which problematic quotes the model was able to correctly identify and which it failed to identify. This will better inform us on the model's limitations. For example, we want to understand which aspects of being problematic was the model able to learn/not learn (e.g. rude, sex, religion). This will help us understand the model's limitation to communicate to the executives in Q4.\n",
        "\n",
        "**True Positive Quotes**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 636,
      "id": "hmj30ZNtn-3R",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 192
        },
        "id": "hmj30ZNtn-3R",
        "outputId": "57539e67-25e4-45e8-e36e-a40d3a096de8"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"TP_quotes\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"That light saber was luke's, and his father's before him, and now it calls to you!\",\n          \"Get away from her, you BITCH!!!\",\n          \"They\\u00e2\\u20ac\\u2122ve done studies, you know. 60% of the time, it works every time.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT_reliable\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"4. rude\",\n          \"1. sex\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "TP_quotes"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-80a9f585-4262-444e-baa9-7c777ecbfc46\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>label_ChatGPT_reliable</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>524</th>\n",
              "      <td>Theyâ€™ve done studies, you know. 60% of the time, it works every time.</td>\n",
              "      <td>1. sex</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>798</th>\n",
              "      <td>That light saber was luke's, and his father's before him, and now it calls to you!</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>89</th>\n",
              "      <td>Get away from her, you BITCH!!!</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-80a9f585-4262-444e-baa9-7c777ecbfc46')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-80a9f585-4262-444e-baa9-7c777ecbfc46 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-80a9f585-4262-444e-baa9-7c777ecbfc46');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2d4bc481-cd00-46bf-8015-24a33ff6895a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2d4bc481-cd00-46bf-8015-24a33ff6895a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2d4bc481-cd00-46bf-8015-24a33ff6895a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_4efd07c2-615d-48fa-b4a0-234a9264acc8\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('TP_quotes')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_4efd07c2-615d-48fa-b4a0-234a9264acc8 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('TP_quotes');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                                                                                                                                                                                 Movie Quote  \\\n",
              "524                                                                                                                                                                                                                  Theyâ€™ve done studies, you know. 60% of the time, it works every time.   \n",
              "798                                                                                                                                                                                                       That light saber was luke's, and his father's before him, and now it calls to you!   \n",
              "45   I love the smell of napalm in the morning. You know, one time we had a hill\\n. bombed, for 12 hours. When it was all over, I walked up. We didn't find one\\n. of 'em, not one stinkin' dink body. The smell, you know that gasoline smell,\\n. the whole hill. Smelled like ... victory.   \n",
              "89                                                                                                                                                                                                                                                           Get away from her, you BITCH!!!   \n",
              "\n",
              "    label_ChatGPT_reliable  \n",
              "524                 1. sex  \n",
              "798                4. rude  \n",
              "45                 4. rude  \n",
              "89                 4. rude  "
            ]
          },
          "execution_count": 636,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# First get problematic quotes that were correctly identified:\n",
        "TP_indices = np.where((y_test_pred_class_20 == 1) & (y_test_actual_class == 1))[0]\n",
        "\n",
        "TP_quotes = test.iloc[TP_indices, ][['Movie Quote', 'label_ChatGPT_reliable']]\n",
        "\n",
        "TP_quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jpGzBHU51jIY",
      "metadata": {
        "id": "jpGzBHU51jIY"
      },
      "source": [
        "Above are the test quotes that were correctly predicted as 'problematic'. We already discussed these quotes in Q3c Step 1 so do not repeat our analysis here.\n",
        "\n",
        "Let us look at a decision plot of the true positive quote with the highest predicted probability in the test dataset. We see that the values for `Animation` and `PC1` had the greatest impact on our prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 637,
      "id": "22bIqvMCrfRr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "22bIqvMCrfRr",
        "outputId": "1d19d779-1027-4cf3-d606-fd3390468d6c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div align='center'><img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABkAAAAWCAYAAAA1vze2AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAdxJREFUeNq0Vt1Rg0AQJjcpgBJiBWIFkgoMFYhPPAIVECogPuYpdJBYgXQQrMCUkA50V7+d2ZwXuXPGm9khHLu3f9+3l1nkWNvtNqfHLgpfQ1EUS3tz5nAQ0+NIsiAZSc6eDlI8M3J00B/mDuUKDk6kfOebAgW3pkdD0pFcODGW4gKKvOrAUm04MA4QDt1OEIXU9hDigfS5rC1eS5T90gltck1Xrizo257kgySZcNRzgCSxCvgiE9nckPJo2b/B2AcEkk2OwL8bD8gmOKR1GPbaCUqxEgTq0tLvgb6zfo7+DgYGkkWL2tqLDV4RSITfbHPPfJKIrWz4nJQTMPAWA7IbD6imcNaDeDfgk+4No+wZr40BL3g9eQJJCFqRQ54KiSt72lsLpE3o3MCBSxDuq4yOckU2hKXRuwBH3OyMR4g1UpyTYw6mlmBqNdUXRM1NfyF5EPI6JkcpIDBIX8jX6DR/6ckAZJ0wEAdLR8DEk6OfC1Pp8BKo6TQIwPJbvJ6toK5lmuvJoRtfK6Ym1iRYIarRo2UyYHvRN5qpakR3yoizWrouoyuXXQqI185LCw07op5ZyCRGL99h24InP0e9xdQukEKVmhzrqZuRIfwISB//cP3Wk3f8f/yR+BRgAHu00HjLcEQBAAAAAElFTkSuQmCC' /></div><script charset='utf-8'>/*! For license information please see bundle.js.LICENSE.txt */\n",
              "(()=>{var e={486:function(e,t,n){var r;e=n.nmd(e),function(){var a,i=\"Expected a function\",o=\"__lodash_hash_undefined__\",u=\"__lodash_placeholder__\",l=32,s=128,c=1/0,f=9007199254740991,p=NaN,d=4294967295,h=[[\"ary\",s],[\"bind\",1],[\"bindKey\",2],[\"curry\",8],[\"curryRight\",16],[\"flip\",512],[\"partial\",l],[\"partialRight\",64],[\"rearg\",256]],v=\"[object Arguments]\",g=\"[object Array]\",y=\"[object Boolean]\",m=\"[object Date]\",b=\"[object Error]\",_=\"[object Function]\",w=\"[object GeneratorFunction]\",x=\"[object Map]\",k=\"[object Number]\",S=\"[object Object]\",E=\"[object Promise]\",C=\"[object RegExp]\",T=\"[object Set]\",M=\"[object String]\",N=\"[object Symbol]\",P=\"[object WeakMap]\",z=\"[object ArrayBuffer]\",L=\"[object DataView]\",O=\"[object Float32Array]\",A=\"[object Float64Array]\",F=\"[object Int8Array]\",D=\"[object Int16Array]\",R=\"[object Int32Array]\",j=\"[object Uint8Array]\",U=\"[object Uint8ClampedArray]\",I=\"[object Uint16Array]\",$=\"[object Uint32Array]\",B=/\\b__p \\+= '';/g,W=/\\b(__p \\+=) '' \\+/g,V=/(__e\\(.*?\\)|\\b__t\\)) \\+\\n'';/g,H=/&(?:amp|lt|gt|quot|#39);/g,q=/[&<>\"']/g,Q=RegExp(H.source),Y=RegExp(q.source),G=/<%-([\\s\\S]+?)%>/g,K=/<%([\\s\\S]+?)%>/g,Z=/<%=([\\s\\S]+?)%>/g,X=/\\.|\\[(?:[^[\\]]*|([\"'])(?:(?!\\1)[^\\\\]|\\\\.)*?\\1)\\]/,J=/^\\w*$/,ee=/[^.[\\]]+|\\[(?:(-?\\d+(?:\\.\\d+)?)|([\"'])((?:(?!\\2)[^\\\\]|\\\\.)*?)\\2)\\]|(?=(?:\\.|\\[\\])(?:\\.|\\[\\]|$))/g,te=/[\\\\^$.*+?()[\\]{}|]/g,ne=RegExp(te.source),re=/^\\s+/,ae=/\\s/,ie=/\\{(?:\\n\\/\\* \\[wrapped with .+\\] \\*\\/)?\\n?/,oe=/\\{\\n\\/\\* \\[wrapped with (.+)\\] \\*/,ue=/,? & /,le=/[^\\x00-\\x2f\\x3a-\\x40\\x5b-\\x60\\x7b-\\x7f]+/g,se=/[()=,{}\\[\\]\\/\\s]/,ce=/\\\\(\\\\)?/g,fe=/\\$\\{([^\\\\}]*(?:\\\\.[^\\\\}]*)*)\\}/g,pe=/\\w*$/,de=/^[-+]0x[0-9a-f]+$/i,he=/^0b[01]+$/i,ve=/^\\[object .+?Constructor\\]$/,ge=/^0o[0-7]+$/i,ye=/^(?:0|[1-9]\\d*)$/,me=/[\\xc0-\\xd6\\xd8-\\xf6\\xf8-\\xff\\u0100-\\u017f]/g,be=/($^)/,_e=/['\\n\\r\\u2028\\u2029\\\\]/g,we=\"\\\\ud800-\\\\udfff\",xe=\"\\\\u0300-\\\\u036f\\\\ufe20-\\\\ufe2f\\\\u20d0-\\\\u20ff\",ke=\"\\\\u2700-\\\\u27bf\",Se=\"a-z\\\\xdf-\\\\xf6\\\\xf8-\\\\xff\",Ee=\"A-Z\\\\xc0-\\\\xd6\\\\xd8-\\\\xde\",Ce=\"\\\\ufe0e\\\\ufe0f\",Te=\"\\\\xac\\\\xb1\\\\xd7\\\\xf7\\\\x00-\\\\x2f\\\\x3a-\\\\x40\\\\x5b-\\\\x60\\\\x7b-\\\\xbf\\\\u2000-\\\\u206f \\\\t\\\\x0b\\\\f\\\\xa0\\\\ufeff\\\\n\\\\r\\\\u2028\\\\u2029\\\\u1680\\\\u180e\\\\u2000\\\\u2001\\\\u2002\\\\u2003\\\\u2004\\\\u2005\\\\u2006\\\\u2007\\\\u2008\\\\u2009\\\\u200a\\\\u202f\\\\u205f\\\\u3000\",Me=\"[\"+we+\"]\",Ne=\"[\"+Te+\"]\",Pe=\"[\"+xe+\"]\",ze=\"\\\\d+\",Le=\"[\"+ke+\"]\",Oe=\"[\"+Se+\"]\",Ae=\"[^\"+we+Te+ze+ke+Se+Ee+\"]\",Fe=\"\\\\ud83c[\\\\udffb-\\\\udfff]\",De=\"[^\"+we+\"]\",Re=\"(?:\\\\ud83c[\\\\udde6-\\\\uddff]){2}\",je=\"[\\\\ud800-\\\\udbff][\\\\udc00-\\\\udfff]\",Ue=\"[\"+Ee+\"]\",Ie=\"\\\\u200d\",$e=\"(?:\"+Oe+\"|\"+Ae+\")\",Be=\"(?:\"+Ue+\"|\"+Ae+\")\",We=\"(?:['’](?:d|ll|m|re|s|t|ve))?\",Ve=\"(?:['’](?:D|LL|M|RE|S|T|VE))?\",He=\"(?:\"+Pe+\"|\"+Fe+\")?\",qe=\"[\"+Ce+\"]?\",Qe=qe+He+\"(?:\"+Ie+\"(?:\"+[De,Re,je].join(\"|\")+\")\"+qe+He+\")*\",Ye=\"(?:\"+[Le,Re,je].join(\"|\")+\")\"+Qe,Ge=\"(?:\"+[De+Pe+\"?\",Pe,Re,je,Me].join(\"|\")+\")\",Ke=RegExp(\"['’]\",\"g\"),Ze=RegExp(Pe,\"g\"),Xe=RegExp(Fe+\"(?=\"+Fe+\")|\"+Ge+Qe,\"g\"),Je=RegExp([Ue+\"?\"+Oe+\"+\"+We+\"(?=\"+[Ne,Ue,\"$\"].join(\"|\")+\")\",Be+\"+\"+Ve+\"(?=\"+[Ne,Ue+$e,\"$\"].join(\"|\")+\")\",Ue+\"?\"+$e+\"+\"+We,Ue+\"+\"+Ve,\"\\\\d*(?:1ST|2ND|3RD|(?![123])\\\\dTH)(?=\\\\b|[a-z_])\",\"\\\\d*(?:1st|2nd|3rd|(?![123])\\\\dth)(?=\\\\b|[A-Z_])\",ze,Ye].join(\"|\"),\"g\"),et=RegExp(\"[\"+Ie+we+xe+Ce+\"]\"),tt=/[a-z][A-Z]|[A-Z]{2}[a-z]|[0-9][a-zA-Z]|[a-zA-Z][0-9]|[^a-zA-Z0-9 ]/,nt=[\"Array\",\"Buffer\",\"DataView\",\"Date\",\"Error\",\"Float32Array\",\"Float64Array\",\"Function\",\"Int8Array\",\"Int16Array\",\"Int32Array\",\"Map\",\"Math\",\"Object\",\"Promise\",\"RegExp\",\"Set\",\"String\",\"Symbol\",\"TypeError\",\"Uint8Array\",\"Uint8ClampedArray\",\"Uint16Array\",\"Uint32Array\",\"WeakMap\",\"_\",\"clearTimeout\",\"isFinite\",\"parseInt\",\"setTimeout\"],rt=-1,at={};at[O]=at[A]=at[F]=at[D]=at[R]=at[j]=at[U]=at[I]=at[$]=!0,at[v]=at[g]=at[z]=at[y]=at[L]=at[m]=at[b]=at[_]=at[x]=at[k]=at[S]=at[C]=at[T]=at[M]=at[P]=!1;var it={};it[v]=it[g]=it[z]=it[L]=it[y]=it[m]=it[O]=it[A]=it[F]=it[D]=it[R]=it[x]=it[k]=it[S]=it[C]=it[T]=it[M]=it[N]=it[j]=it[U]=it[I]=it[$]=!0,it[b]=it[_]=it[P]=!1;var ot={\"\\\\\":\"\\\\\",\"'\":\"'\",\"\\n\":\"n\",\"\\r\":\"r\",\"\\u2028\":\"u2028\",\"\\u2029\":\"u2029\"},ut=parseFloat,lt=parseInt,st=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,ct=\"object\"==typeof self&&self&&self.Object===Object&&self,ft=st||ct||Function(\"return this\")(),pt=t&&!t.nodeType&&t,dt=pt&&e&&!e.nodeType&&e,ht=dt&&dt.exports===pt,vt=ht&&st.process,gt=function(){try{return dt&&dt.require&&dt.require(\"util\").types||vt&&vt.binding&&vt.binding(\"util\")}catch(e){}}(),yt=gt&&gt.isArrayBuffer,mt=gt&&gt.isDate,bt=gt&&gt.isMap,_t=gt&&gt.isRegExp,wt=gt&&gt.isSet,xt=gt&&gt.isTypedArray;function kt(e,t,n){switch(n.length){case 0:return e.call(t);case 1:return e.call(t,n[0]);case 2:return e.call(t,n[0],n[1]);case 3:return e.call(t,n[0],n[1],n[2])}return e.apply(t,n)}function St(e,t,n,r){for(var a=-1,i=null==e?0:e.length;++a<i;){var o=e[a];t(r,o,n(o),e)}return r}function Et(e,t){for(var n=-1,r=null==e?0:e.length;++n<r&&!1!==t(e[n],n,e););return e}function Ct(e,t){for(var n=null==e?0:e.length;n--&&!1!==t(e[n],n,e););return e}function Tt(e,t){for(var n=-1,r=null==e?0:e.length;++n<r;)if(!t(e[n],n,e))return!1;return!0}function Mt(e,t){for(var n=-1,r=null==e?0:e.length,a=0,i=[];++n<r;){var o=e[n];t(o,n,e)&&(i[a++]=o)}return i}function Nt(e,t){return!(null==e||!e.length)&&Ut(e,t,0)>-1}function Pt(e,t,n){for(var r=-1,a=null==e?0:e.length;++r<a;)if(n(t,e[r]))return!0;return!1}function zt(e,t){for(var n=-1,r=null==e?0:e.length,a=Array(r);++n<r;)a[n]=t(e[n],n,e);return a}function Lt(e,t){for(var n=-1,r=t.length,a=e.length;++n<r;)e[a+n]=t[n];return e}function Ot(e,t,n,r){var a=-1,i=null==e?0:e.length;for(r&&i&&(n=e[++a]);++a<i;)n=t(n,e[a],a,e);return n}function At(e,t,n,r){var a=null==e?0:e.length;for(r&&a&&(n=e[--a]);a--;)n=t(n,e[a],a,e);return n}function Ft(e,t){for(var n=-1,r=null==e?0:e.length;++n<r;)if(t(e[n],n,e))return!0;return!1}var Dt=Wt(\"length\");function Rt(e,t,n){var r;return n(e,(function(e,n,a){if(t(e,n,a))return r=n,!1})),r}function jt(e,t,n,r){for(var a=e.length,i=n+(r?1:-1);r?i--:++i<a;)if(t(e[i],i,e))return i;return-1}function Ut(e,t,n){return t==t?function(e,t,n){for(var r=n-1,a=e.length;++r<a;)if(e[r]===t)return r;return-1}(e,t,n):jt(e,$t,n)}function It(e,t,n,r){for(var a=n-1,i=e.length;++a<i;)if(r(e[a],t))return a;return-1}function $t(e){return e!=e}function Bt(e,t){var n=null==e?0:e.length;return n?qt(e,t)/n:p}function Wt(e){return function(t){return null==t?a:t[e]}}function Vt(e){return function(t){return null==e?a:e[t]}}function Ht(e,t,n,r,a){return a(e,(function(e,a,i){n=r?(r=!1,e):t(n,e,a,i)})),n}function qt(e,t){for(var n,r=-1,i=e.length;++r<i;){var o=t(e[r]);o!==a&&(n=n===a?o:n+o)}return n}function Qt(e,t){for(var n=-1,r=Array(e);++n<e;)r[n]=t(n);return r}function Yt(e){return e?e.slice(0,pn(e)+1).replace(re,\"\"):e}function Gt(e){return function(t){return e(t)}}function Kt(e,t){return zt(t,(function(t){return e[t]}))}function Zt(e,t){return e.has(t)}function Xt(e,t){for(var n=-1,r=e.length;++n<r&&Ut(t,e[n],0)>-1;);return n}function Jt(e,t){for(var n=e.length;n--&&Ut(t,e[n],0)>-1;);return n}var en=Vt({À:\"A\",Á:\"A\",Â:\"A\",Ã:\"A\",Ä:\"A\",Å:\"A\",à:\"a\",á:\"a\",â:\"a\",ã:\"a\",ä:\"a\",å:\"a\",Ç:\"C\",ç:\"c\",Ð:\"D\",ð:\"d\",È:\"E\",É:\"E\",Ê:\"E\",Ë:\"E\",è:\"e\",é:\"e\",ê:\"e\",ë:\"e\",Ì:\"I\",Í:\"I\",Î:\"I\",Ï:\"I\",ì:\"i\",í:\"i\",î:\"i\",ï:\"i\",Ñ:\"N\",ñ:\"n\",Ò:\"O\",Ó:\"O\",Ô:\"O\",Õ:\"O\",Ö:\"O\",Ø:\"O\",ò:\"o\",ó:\"o\",ô:\"o\",õ:\"o\",ö:\"o\",ø:\"o\",Ù:\"U\",Ú:\"U\",Û:\"U\",Ü:\"U\",ù:\"u\",ú:\"u\",û:\"u\",ü:\"u\",Ý:\"Y\",ý:\"y\",ÿ:\"y\",Æ:\"Ae\",æ:\"ae\",Þ:\"Th\",þ:\"th\",ß:\"ss\",Ā:\"A\",Ă:\"A\",Ą:\"A\",ā:\"a\",ă:\"a\",ą:\"a\",Ć:\"C\",Ĉ:\"C\",Ċ:\"C\",Č:\"C\",ć:\"c\",ĉ:\"c\",ċ:\"c\",č:\"c\",Ď:\"D\",Đ:\"D\",ď:\"d\",đ:\"d\",Ē:\"E\",Ĕ:\"E\",Ė:\"E\",Ę:\"E\",Ě:\"E\",ē:\"e\",ĕ:\"e\",ė:\"e\",ę:\"e\",ě:\"e\",Ĝ:\"G\",Ğ:\"G\",Ġ:\"G\",Ģ:\"G\",ĝ:\"g\",ğ:\"g\",ġ:\"g\",ģ:\"g\",Ĥ:\"H\",Ħ:\"H\",ĥ:\"h\",ħ:\"h\",Ĩ:\"I\",Ī:\"I\",Ĭ:\"I\",Į:\"I\",İ:\"I\",ĩ:\"i\",ī:\"i\",ĭ:\"i\",į:\"i\",ı:\"i\",Ĵ:\"J\",ĵ:\"j\",Ķ:\"K\",ķ:\"k\",ĸ:\"k\",Ĺ:\"L\",Ļ:\"L\",Ľ:\"L\",Ŀ:\"L\",Ł:\"L\",ĺ:\"l\",ļ:\"l\",ľ:\"l\",ŀ:\"l\",ł:\"l\",Ń:\"N\",Ņ:\"N\",Ň:\"N\",Ŋ:\"N\",ń:\"n\",ņ:\"n\",ň:\"n\",ŋ:\"n\",Ō:\"O\",Ŏ:\"O\",Ő:\"O\",ō:\"o\",ŏ:\"o\",ő:\"o\",Ŕ:\"R\",Ŗ:\"R\",Ř:\"R\",ŕ:\"r\",ŗ:\"r\",ř:\"r\",Ś:\"S\",Ŝ:\"S\",Ş:\"S\",Š:\"S\",ś:\"s\",ŝ:\"s\",ş:\"s\",š:\"s\",Ţ:\"T\",Ť:\"T\",Ŧ:\"T\",ţ:\"t\",ť:\"t\",ŧ:\"t\",Ũ:\"U\",Ū:\"U\",Ŭ:\"U\",Ů:\"U\",Ű:\"U\",Ų:\"U\",ũ:\"u\",ū:\"u\",ŭ:\"u\",ů:\"u\",ű:\"u\",ų:\"u\",Ŵ:\"W\",ŵ:\"w\",Ŷ:\"Y\",ŷ:\"y\",Ÿ:\"Y\",Ź:\"Z\",Ż:\"Z\",Ž:\"Z\",ź:\"z\",ż:\"z\",ž:\"z\",Ĳ:\"IJ\",ĳ:\"ij\",Œ:\"Oe\",œ:\"oe\",ŉ:\"'n\",ſ:\"s\"}),tn=Vt({\"&\":\"&amp;\",\"<\":\"&lt;\",\">\":\"&gt;\",'\"':\"&quot;\",\"'\":\"&#39;\"});function nn(e){return\"\\\\\"+ot[e]}function rn(e){return et.test(e)}function an(e){var t=-1,n=Array(e.size);return e.forEach((function(e,r){n[++t]=[r,e]})),n}function on(e,t){return function(n){return e(t(n))}}function un(e,t){for(var n=-1,r=e.length,a=0,i=[];++n<r;){var o=e[n];o!==t&&o!==u||(e[n]=u,i[a++]=n)}return i}function ln(e){var t=-1,n=Array(e.size);return e.forEach((function(e){n[++t]=e})),n}function sn(e){var t=-1,n=Array(e.size);return e.forEach((function(e){n[++t]=[e,e]})),n}function cn(e){return rn(e)?function(e){for(var t=Xe.lastIndex=0;Xe.test(e);)++t;return t}(e):Dt(e)}function fn(e){return rn(e)?function(e){return e.match(Xe)||[]}(e):function(e){return e.split(\"\")}(e)}function pn(e){for(var t=e.length;t--&&ae.test(e.charAt(t)););return t}var dn=Vt({\"&amp;\":\"&\",\"&lt;\":\"<\",\"&gt;\":\">\",\"&quot;\":'\"',\"&#39;\":\"'\"}),hn=function e(t){var n,r=(t=null==t?ft:hn.defaults(ft.Object(),t,hn.pick(ft,nt))).Array,ae=t.Date,we=t.Error,xe=t.Function,ke=t.Math,Se=t.Object,Ee=t.RegExp,Ce=t.String,Te=t.TypeError,Me=r.prototype,Ne=xe.prototype,Pe=Se.prototype,ze=t[\"__core-js_shared__\"],Le=Ne.toString,Oe=Pe.hasOwnProperty,Ae=0,Fe=(n=/[^.]+$/.exec(ze&&ze.keys&&ze.keys.IE_PROTO||\"\"))?\"Symbol(src)_1.\"+n:\"\",De=Pe.toString,Re=Le.call(Se),je=ft._,Ue=Ee(\"^\"+Le.call(Oe).replace(te,\"\\\\$&\").replace(/hasOwnProperty|(function).*?(?=\\\\\\()| for .+?(?=\\\\\\])/g,\"$1.*?\")+\"$\"),Ie=ht?t.Buffer:a,$e=t.Symbol,Be=t.Uint8Array,We=Ie?Ie.allocUnsafe:a,Ve=on(Se.getPrototypeOf,Se),He=Se.create,qe=Pe.propertyIsEnumerable,Qe=Me.splice,Ye=$e?$e.isConcatSpreadable:a,Ge=$e?$e.iterator:a,Xe=$e?$e.toStringTag:a,et=function(){try{var e=li(Se,\"defineProperty\");return e({},\"\",{}),e}catch(e){}}(),ot=t.clearTimeout!==ft.clearTimeout&&t.clearTimeout,st=ae&&ae.now!==ft.Date.now&&ae.now,ct=t.setTimeout!==ft.setTimeout&&t.setTimeout,pt=ke.ceil,dt=ke.floor,vt=Se.getOwnPropertySymbols,gt=Ie?Ie.isBuffer:a,Dt=t.isFinite,Vt=Me.join,vn=on(Se.keys,Se),gn=ke.max,yn=ke.min,mn=ae.now,bn=t.parseInt,_n=ke.random,wn=Me.reverse,xn=li(t,\"DataView\"),kn=li(t,\"Map\"),Sn=li(t,\"Promise\"),En=li(t,\"Set\"),Cn=li(t,\"WeakMap\"),Tn=li(Se,\"create\"),Mn=Cn&&new Cn,Nn={},Pn=Di(xn),zn=Di(kn),Ln=Di(Sn),On=Di(En),An=Di(Cn),Fn=$e?$e.prototype:a,Dn=Fn?Fn.valueOf:a,Rn=Fn?Fn.toString:a;function jn(e){if(eu(e)&&!Wo(e)&&!(e instanceof Bn)){if(e instanceof $n)return e;if(Oe.call(e,\"__wrapped__\"))return Ri(e)}return new $n(e)}var Un=function(){function e(){}return function(t){if(!Jo(t))return{};if(He)return He(t);e.prototype=t;var n=new e;return e.prototype=a,n}}();function In(){}function $n(e,t){this.__wrapped__=e,this.__actions__=[],this.__chain__=!!t,this.__index__=0,this.__values__=a}function Bn(e){this.__wrapped__=e,this.__actions__=[],this.__dir__=1,this.__filtered__=!1,this.__iteratees__=[],this.__takeCount__=d,this.__views__=[]}function Wn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function Vn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function Hn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function qn(e){var t=-1,n=null==e?0:e.length;for(this.__data__=new Hn;++t<n;)this.add(e[t])}function Qn(e){var t=this.__data__=new Vn(e);this.size=t.size}function Yn(e,t){var n=Wo(e),r=!n&&Bo(e),a=!n&&!r&&Qo(e),i=!n&&!r&&!a&&lu(e),o=n||r||a||i,u=o?Qt(e.length,Ce):[],l=u.length;for(var s in e)!t&&!Oe.call(e,s)||o&&(\"length\"==s||a&&(\"offset\"==s||\"parent\"==s)||i&&(\"buffer\"==s||\"byteLength\"==s||\"byteOffset\"==s)||vi(s,l))||u.push(s);return u}function Gn(e){var t=e.length;return t?e[Hr(0,t-1)]:a}function Kn(e,t){return zi(Ca(e),ir(t,0,e.length))}function Zn(e){return zi(Ca(e))}function Xn(e,t,n){(n!==a&&!Uo(e[t],n)||n===a&&!(t in e))&&rr(e,t,n)}function Jn(e,t,n){var r=e[t];Oe.call(e,t)&&Uo(r,n)&&(n!==a||t in e)||rr(e,t,n)}function er(e,t){for(var n=e.length;n--;)if(Uo(e[n][0],t))return n;return-1}function tr(e,t,n,r){return cr(e,(function(e,a,i){t(r,e,n(e),i)})),r}function nr(e,t){return e&&Ta(t,Pu(t),e)}function rr(e,t,n){\"__proto__\"==t&&et?et(e,t,{configurable:!0,enumerable:!0,value:n,writable:!0}):e[t]=n}function ar(e,t){for(var n=-1,i=t.length,o=r(i),u=null==e;++n<i;)o[n]=u?a:Eu(e,t[n]);return o}function ir(e,t,n){return e==e&&(n!==a&&(e=e<=n?e:n),t!==a&&(e=e>=t?e:t)),e}function or(e,t,n,r,i,o){var u,l=1&t,s=2&t,c=4&t;if(n&&(u=i?n(e,r,i,o):n(e)),u!==a)return u;if(!Jo(e))return e;var f=Wo(e);if(f){if(u=function(e){var t=e.length,n=new e.constructor(t);return t&&\"string\"==typeof e[0]&&Oe.call(e,\"index\")&&(n.index=e.index,n.input=e.input),n}(e),!l)return Ca(e,u)}else{var p=fi(e),d=p==_||p==w;if(Qo(e))return _a(e,l);if(p==S||p==v||d&&!i){if(u=s||d?{}:di(e),!l)return s?function(e,t){return Ta(e,ci(e),t)}(e,function(e,t){return e&&Ta(t,zu(t),e)}(u,e)):function(e,t){return Ta(e,si(e),t)}(e,nr(u,e))}else{if(!it[p])return i?e:{};u=function(e,t,n){var r,a=e.constructor;switch(t){case z:return wa(e);case y:case m:return new a(+e);case L:return function(e,t){var n=t?wa(e.buffer):e.buffer;return new e.constructor(n,e.byteOffset,e.byteLength)}(e,n);case O:case A:case F:case D:case R:case j:case U:case I:case $:return xa(e,n);case x:return new a;case k:case M:return new a(e);case C:return function(e){var t=new e.constructor(e.source,pe.exec(e));return t.lastIndex=e.lastIndex,t}(e);case T:return new a;case N:return r=e,Dn?Se(Dn.call(r)):{}}}(e,p,l)}}o||(o=new Qn);var h=o.get(e);if(h)return h;o.set(e,u),iu(e)?e.forEach((function(r){u.add(or(r,t,n,r,e,o))})):tu(e)&&e.forEach((function(r,a){u.set(a,or(r,t,n,a,e,o))}));var g=f?a:(c?s?ti:ei:s?zu:Pu)(e);return Et(g||e,(function(r,a){g&&(r=e[a=r]),Jn(u,a,or(r,t,n,a,e,o))})),u}function ur(e,t,n){var r=n.length;if(null==e)return!r;for(e=Se(e);r--;){var i=n[r],o=t[i],u=e[i];if(u===a&&!(i in e)||!o(u))return!1}return!0}function lr(e,t,n){if(\"function\"!=typeof e)throw new Te(i);return Ti((function(){e.apply(a,n)}),t)}function sr(e,t,n,r){var a=-1,i=Nt,o=!0,u=e.length,l=[],s=t.length;if(!u)return l;n&&(t=zt(t,Gt(n))),r?(i=Pt,o=!1):t.length>=200&&(i=Zt,o=!1,t=new qn(t));e:for(;++a<u;){var c=e[a],f=null==n?c:n(c);if(c=r||0!==c?c:0,o&&f==f){for(var p=s;p--;)if(t[p]===f)continue e;l.push(c)}else i(t,f,r)||l.push(c)}return l}jn.templateSettings={escape:G,evaluate:K,interpolate:Z,variable:\"\",imports:{_:jn}},jn.prototype=In.prototype,jn.prototype.constructor=jn,$n.prototype=Un(In.prototype),$n.prototype.constructor=$n,Bn.prototype=Un(In.prototype),Bn.prototype.constructor=Bn,Wn.prototype.clear=function(){this.__data__=Tn?Tn(null):{},this.size=0},Wn.prototype.delete=function(e){var t=this.has(e)&&delete this.__data__[e];return this.size-=t?1:0,t},Wn.prototype.get=function(e){var t=this.__data__;if(Tn){var n=t[e];return n===o?a:n}return Oe.call(t,e)?t[e]:a},Wn.prototype.has=function(e){var t=this.__data__;return Tn?t[e]!==a:Oe.call(t,e)},Wn.prototype.set=function(e,t){var n=this.__data__;return this.size+=this.has(e)?0:1,n[e]=Tn&&t===a?o:t,this},Vn.prototype.clear=function(){this.__data__=[],this.size=0},Vn.prototype.delete=function(e){var t=this.__data__,n=er(t,e);return!(n<0||(n==t.length-1?t.pop():Qe.call(t,n,1),--this.size,0))},Vn.prototype.get=function(e){var t=this.__data__,n=er(t,e);return n<0?a:t[n][1]},Vn.prototype.has=function(e){return er(this.__data__,e)>-1},Vn.prototype.set=function(e,t){var n=this.__data__,r=er(n,e);return r<0?(++this.size,n.push([e,t])):n[r][1]=t,this},Hn.prototype.clear=function(){this.size=0,this.__data__={hash:new Wn,map:new(kn||Vn),string:new Wn}},Hn.prototype.delete=function(e){var t=oi(this,e).delete(e);return this.size-=t?1:0,t},Hn.prototype.get=function(e){return oi(this,e).get(e)},Hn.prototype.has=function(e){return oi(this,e).has(e)},Hn.prototype.set=function(e,t){var n=oi(this,e),r=n.size;return n.set(e,t),this.size+=n.size==r?0:1,this},qn.prototype.add=qn.prototype.push=function(e){return this.__data__.set(e,o),this},qn.prototype.has=function(e){return this.__data__.has(e)},Qn.prototype.clear=function(){this.__data__=new Vn,this.size=0},Qn.prototype.delete=function(e){var t=this.__data__,n=t.delete(e);return this.size=t.size,n},Qn.prototype.get=function(e){return this.__data__.get(e)},Qn.prototype.has=function(e){return this.__data__.has(e)},Qn.prototype.set=function(e,t){var n=this.__data__;if(n instanceof Vn){var r=n.__data__;if(!kn||r.length<199)return r.push([e,t]),this.size=++n.size,this;n=this.__data__=new Hn(r)}return n.set(e,t),this.size=n.size,this};var cr=Pa(mr),fr=Pa(br,!0);function pr(e,t){var n=!0;return cr(e,(function(e,r,a){return n=!!t(e,r,a)})),n}function dr(e,t,n){for(var r=-1,i=e.length;++r<i;){var o=e[r],u=t(o);if(null!=u&&(l===a?u==u&&!uu(u):n(u,l)))var l=u,s=o}return s}function hr(e,t){var n=[];return cr(e,(function(e,r,a){t(e,r,a)&&n.push(e)})),n}function vr(e,t,n,r,a){var i=-1,o=e.length;for(n||(n=hi),a||(a=[]);++i<o;){var u=e[i];t>0&&n(u)?t>1?vr(u,t-1,n,r,a):Lt(a,u):r||(a[a.length]=u)}return a}var gr=za(),yr=za(!0);function mr(e,t){return e&&gr(e,t,Pu)}function br(e,t){return e&&yr(e,t,Pu)}function _r(e,t){return Mt(t,(function(t){return Ko(e[t])}))}function wr(e,t){for(var n=0,r=(t=ga(t,e)).length;null!=e&&n<r;)e=e[Fi(t[n++])];return n&&n==r?e:a}function xr(e,t,n){var r=t(e);return Wo(e)?r:Lt(r,n(e))}function kr(e){return null==e?e===a?\"[object Undefined]\":\"[object Null]\":Xe&&Xe in Se(e)?function(e){var t=Oe.call(e,Xe),n=e[Xe];try{e[Xe]=a;var r=!0}catch(e){}var i=De.call(e);return r&&(t?e[Xe]=n:delete e[Xe]),i}(e):function(e){return De.call(e)}(e)}function Sr(e,t){return e>t}function Er(e,t){return null!=e&&Oe.call(e,t)}function Cr(e,t){return null!=e&&t in Se(e)}function Tr(e,t,n){for(var i=n?Pt:Nt,o=e[0].length,u=e.length,l=u,s=r(u),c=1/0,f=[];l--;){var p=e[l];l&&t&&(p=zt(p,Gt(t))),c=yn(p.length,c),s[l]=!n&&(t||o>=120&&p.length>=120)?new qn(l&&p):a}p=e[0];var d=-1,h=s[0];e:for(;++d<o&&f.length<c;){var v=p[d],g=t?t(v):v;if(v=n||0!==v?v:0,!(h?Zt(h,g):i(f,g,n))){for(l=u;--l;){var y=s[l];if(!(y?Zt(y,g):i(e[l],g,n)))continue e}h&&h.push(g),f.push(v)}}return f}function Mr(e,t,n){var r=null==(e=Si(e,t=ga(t,e)))?e:e[Fi(Yi(t))];return null==r?a:kt(r,e,n)}function Nr(e){return eu(e)&&kr(e)==v}function Pr(e,t,n,r,i){return e===t||(null==e||null==t||!eu(e)&&!eu(t)?e!=e&&t!=t:function(e,t,n,r,i,o){var u=Wo(e),l=Wo(t),s=u?g:fi(e),c=l?g:fi(t),f=(s=s==v?S:s)==S,p=(c=c==v?S:c)==S,d=s==c;if(d&&Qo(e)){if(!Qo(t))return!1;u=!0,f=!1}if(d&&!f)return o||(o=new Qn),u||lu(e)?Xa(e,t,n,r,i,o):function(e,t,n,r,a,i,o){switch(n){case L:if(e.byteLength!=t.byteLength||e.byteOffset!=t.byteOffset)return!1;e=e.buffer,t=t.buffer;case z:return!(e.byteLength!=t.byteLength||!i(new Be(e),new Be(t)));case y:case m:case k:return Uo(+e,+t);case b:return e.name==t.name&&e.message==t.message;case C:case M:return e==t+\"\";case x:var u=an;case T:var l=1&r;if(u||(u=ln),e.size!=t.size&&!l)return!1;var s=o.get(e);if(s)return s==t;r|=2,o.set(e,t);var c=Xa(u(e),u(t),r,a,i,o);return o.delete(e),c;case N:if(Dn)return Dn.call(e)==Dn.call(t)}return!1}(e,t,s,n,r,i,o);if(!(1&n)){var h=f&&Oe.call(e,\"__wrapped__\"),_=p&&Oe.call(t,\"__wrapped__\");if(h||_){var w=h?e.value():e,E=_?t.value():t;return o||(o=new Qn),i(w,E,n,r,o)}}return!!d&&(o||(o=new Qn),function(e,t,n,r,i,o){var u=1&n,l=ei(e),s=l.length;if(s!=ei(t).length&&!u)return!1;for(var c=s;c--;){var f=l[c];if(!(u?f in t:Oe.call(t,f)))return!1}var p=o.get(e),d=o.get(t);if(p&&d)return p==t&&d==e;var h=!0;o.set(e,t),o.set(t,e);for(var v=u;++c<s;){var g=e[f=l[c]],y=t[f];if(r)var m=u?r(y,g,f,t,e,o):r(g,y,f,e,t,o);if(!(m===a?g===y||i(g,y,n,r,o):m)){h=!1;break}v||(v=\"constructor\"==f)}if(h&&!v){var b=e.constructor,_=t.constructor;b==_||!(\"constructor\"in e)||!(\"constructor\"in t)||\"function\"==typeof b&&b instanceof b&&\"function\"==typeof _&&_ instanceof _||(h=!1)}return o.delete(e),o.delete(t),h}(e,t,n,r,i,o))}(e,t,n,r,Pr,i))}function zr(e,t,n,r){var i=n.length,o=i,u=!r;if(null==e)return!o;for(e=Se(e);i--;){var l=n[i];if(u&&l[2]?l[1]!==e[l[0]]:!(l[0]in e))return!1}for(;++i<o;){var s=(l=n[i])[0],c=e[s],f=l[1];if(u&&l[2]){if(c===a&&!(s in e))return!1}else{var p=new Qn;if(r)var d=r(c,f,s,e,t,p);if(!(d===a?Pr(f,c,3,r,p):d))return!1}}return!0}function Lr(e){return!(!Jo(e)||(t=e,Fe&&Fe in t))&&(Ko(e)?Ue:ve).test(Di(e));var t}function Or(e){return\"function\"==typeof e?e:null==e?nl:\"object\"==typeof e?Wo(e)?jr(e[0],e[1]):Rr(e):fl(e)}function Ar(e){if(!_i(e))return vn(e);var t=[];for(var n in Se(e))Oe.call(e,n)&&\"constructor\"!=n&&t.push(n);return t}function Fr(e,t){return e<t}function Dr(e,t){var n=-1,a=Ho(e)?r(e.length):[];return cr(e,(function(e,r,i){a[++n]=t(e,r,i)})),a}function Rr(e){var t=ui(e);return 1==t.length&&t[0][2]?xi(t[0][0],t[0][1]):function(n){return n===e||zr(n,e,t)}}function jr(e,t){return yi(e)&&wi(t)?xi(Fi(e),t):function(n){var r=Eu(n,e);return r===a&&r===t?Cu(n,e):Pr(t,r,3)}}function Ur(e,t,n,r,i){e!==t&&gr(t,(function(o,u){if(i||(i=new Qn),Jo(o))!function(e,t,n,r,i,o,u){var l=Ei(e,n),s=Ei(t,n),c=u.get(s);if(c)Xn(e,n,c);else{var f=o?o(l,s,n+\"\",e,t,u):a,p=f===a;if(p){var d=Wo(s),h=!d&&Qo(s),v=!d&&!h&&lu(s);f=s,d||h||v?Wo(l)?f=l:qo(l)?f=Ca(l):h?(p=!1,f=_a(s,!0)):v?(p=!1,f=xa(s,!0)):f=[]:ru(s)||Bo(s)?(f=l,Bo(l)?f=gu(l):Jo(l)&&!Ko(l)||(f=di(s))):p=!1}p&&(u.set(s,f),i(f,s,r,o,u),u.delete(s)),Xn(e,n,f)}}(e,t,u,n,Ur,r,i);else{var l=r?r(Ei(e,u),o,u+\"\",e,t,i):a;l===a&&(l=o),Xn(e,u,l)}}),zu)}function Ir(e,t){var n=e.length;if(n)return vi(t+=t<0?n:0,n)?e[t]:a}function $r(e,t,n){t=t.length?zt(t,(function(e){return Wo(e)?function(t){return wr(t,1===e.length?e[0]:e)}:e})):[nl];var r=-1;t=zt(t,Gt(ii()));var a=Dr(e,(function(e,n,a){var i=zt(t,(function(t){return t(e)}));return{criteria:i,index:++r,value:e}}));return function(e,t){var r=e.length;for(e.sort((function(e,t){return function(e,t,n){for(var r=-1,a=e.criteria,i=t.criteria,o=a.length,u=n.length;++r<o;){var l=ka(a[r],i[r]);if(l)return r>=u?l:l*(\"desc\"==n[r]?-1:1)}return e.index-t.index}(e,t,n)}));r--;)e[r]=e[r].value;return e}(a)}function Br(e,t,n){for(var r=-1,a=t.length,i={};++r<a;){var o=t[r],u=wr(e,o);n(u,o)&&Kr(i,ga(o,e),u)}return i}function Wr(e,t,n,r){var a=r?It:Ut,i=-1,o=t.length,u=e;for(e===t&&(t=Ca(t)),n&&(u=zt(e,Gt(n)));++i<o;)for(var l=0,s=t[i],c=n?n(s):s;(l=a(u,c,l,r))>-1;)u!==e&&Qe.call(u,l,1),Qe.call(e,l,1);return e}function Vr(e,t){for(var n=e?t.length:0,r=n-1;n--;){var a=t[n];if(n==r||a!==i){var i=a;vi(a)?Qe.call(e,a,1):la(e,a)}}return e}function Hr(e,t){return e+dt(_n()*(t-e+1))}function qr(e,t){var n=\"\";if(!e||t<1||t>f)return n;do{t%2&&(n+=e),(t=dt(t/2))&&(e+=e)}while(t);return n}function Qr(e,t){return Mi(ki(e,t,nl),e+\"\")}function Yr(e){return Gn(Uu(e))}function Gr(e,t){var n=Uu(e);return zi(n,ir(t,0,n.length))}function Kr(e,t,n,r){if(!Jo(e))return e;for(var i=-1,o=(t=ga(t,e)).length,u=o-1,l=e;null!=l&&++i<o;){var s=Fi(t[i]),c=n;if(\"__proto__\"===s||\"constructor\"===s||\"prototype\"===s)return e;if(i!=u){var f=l[s];(c=r?r(f,s,l):a)===a&&(c=Jo(f)?f:vi(t[i+1])?[]:{})}Jn(l,s,c),l=l[s]}return e}var Zr=Mn?function(e,t){return Mn.set(e,t),e}:nl,Xr=et?function(e,t){return et(e,\"toString\",{configurable:!0,enumerable:!1,value:Ju(t),writable:!0})}:nl;function Jr(e){return zi(Uu(e))}function ea(e,t,n){var a=-1,i=e.length;t<0&&(t=-t>i?0:i+t),(n=n>i?i:n)<0&&(n+=i),i=t>n?0:n-t>>>0,t>>>=0;for(var o=r(i);++a<i;)o[a]=e[a+t];return o}function ta(e,t){var n;return cr(e,(function(e,r,a){return!(n=t(e,r,a))})),!!n}function na(e,t,n){var r=0,a=null==e?r:e.length;if(\"number\"==typeof t&&t==t&&a<=2147483647){for(;r<a;){var i=r+a>>>1,o=e[i];null!==o&&!uu(o)&&(n?o<=t:o<t)?r=i+1:a=i}return a}return ra(e,t,nl,n)}function ra(e,t,n,r){var i=0,o=null==e?0:e.length;if(0===o)return 0;for(var u=(t=n(t))!=t,l=null===t,s=uu(t),c=t===a;i<o;){var f=dt((i+o)/2),p=n(e[f]),d=p!==a,h=null===p,v=p==p,g=uu(p);if(u)var y=r||v;else y=c?v&&(r||d):l?v&&d&&(r||!h):s?v&&d&&!h&&(r||!g):!h&&!g&&(r?p<=t:p<t);y?i=f+1:o=f}return yn(o,4294967294)}function aa(e,t){for(var n=-1,r=e.length,a=0,i=[];++n<r;){var o=e[n],u=t?t(o):o;if(!n||!Uo(u,l)){var l=u;i[a++]=0===o?0:o}}return i}function ia(e){return\"number\"==typeof e?e:uu(e)?p:+e}function oa(e){if(\"string\"==typeof e)return e;if(Wo(e))return zt(e,oa)+\"\";if(uu(e))return Rn?Rn.call(e):\"\";var t=e+\"\";return\"0\"==t&&1/e==-1/0?\"-0\":t}function ua(e,t,n){var r=-1,a=Nt,i=e.length,o=!0,u=[],l=u;if(n)o=!1,a=Pt;else if(i>=200){var s=t?null:qa(e);if(s)return ln(s);o=!1,a=Zt,l=new qn}else l=t?[]:u;e:for(;++r<i;){var c=e[r],f=t?t(c):c;if(c=n||0!==c?c:0,o&&f==f){for(var p=l.length;p--;)if(l[p]===f)continue e;t&&l.push(f),u.push(c)}else a(l,f,n)||(l!==u&&l.push(f),u.push(c))}return u}function la(e,t){return null==(e=Si(e,t=ga(t,e)))||delete e[Fi(Yi(t))]}function sa(e,t,n,r){return Kr(e,t,n(wr(e,t)),r)}function ca(e,t,n,r){for(var a=e.length,i=r?a:-1;(r?i--:++i<a)&&t(e[i],i,e););return n?ea(e,r?0:i,r?i+1:a):ea(e,r?i+1:0,r?a:i)}function fa(e,t){var n=e;return n instanceof Bn&&(n=n.value()),Ot(t,(function(e,t){return t.func.apply(t.thisArg,Lt([e],t.args))}),n)}function pa(e,t,n){var a=e.length;if(a<2)return a?ua(e[0]):[];for(var i=-1,o=r(a);++i<a;)for(var u=e[i],l=-1;++l<a;)l!=i&&(o[i]=sr(o[i]||u,e[l],t,n));return ua(vr(o,1),t,n)}function da(e,t,n){for(var r=-1,i=e.length,o=t.length,u={};++r<i;){var l=r<o?t[r]:a;n(u,e[r],l)}return u}function ha(e){return qo(e)?e:[]}function va(e){return\"function\"==typeof e?e:nl}function ga(e,t){return Wo(e)?e:yi(e,t)?[e]:Ai(yu(e))}var ya=Qr;function ma(e,t,n){var r=e.length;return n=n===a?r:n,!t&&n>=r?e:ea(e,t,n)}var ba=ot||function(e){return ft.clearTimeout(e)};function _a(e,t){if(t)return e.slice();var n=e.length,r=We?We(n):new e.constructor(n);return e.copy(r),r}function wa(e){var t=new e.constructor(e.byteLength);return new Be(t).set(new Be(e)),t}function xa(e,t){var n=t?wa(e.buffer):e.buffer;return new e.constructor(n,e.byteOffset,e.length)}function ka(e,t){if(e!==t){var n=e!==a,r=null===e,i=e==e,o=uu(e),u=t!==a,l=null===t,s=t==t,c=uu(t);if(!l&&!c&&!o&&e>t||o&&u&&s&&!l&&!c||r&&u&&s||!n&&s||!i)return 1;if(!r&&!o&&!c&&e<t||c&&n&&i&&!r&&!o||l&&n&&i||!u&&i||!s)return-1}return 0}function Sa(e,t,n,a){for(var i=-1,o=e.length,u=n.length,l=-1,s=t.length,c=gn(o-u,0),f=r(s+c),p=!a;++l<s;)f[l]=t[l];for(;++i<u;)(p||i<o)&&(f[n[i]]=e[i]);for(;c--;)f[l++]=e[i++];return f}function Ea(e,t,n,a){for(var i=-1,o=e.length,u=-1,l=n.length,s=-1,c=t.length,f=gn(o-l,0),p=r(f+c),d=!a;++i<f;)p[i]=e[i];for(var h=i;++s<c;)p[h+s]=t[s];for(;++u<l;)(d||i<o)&&(p[h+n[u]]=e[i++]);return p}function Ca(e,t){var n=-1,a=e.length;for(t||(t=r(a));++n<a;)t[n]=e[n];return t}function Ta(e,t,n,r){var i=!n;n||(n={});for(var o=-1,u=t.length;++o<u;){var l=t[o],s=r?r(n[l],e[l],l,n,e):a;s===a&&(s=e[l]),i?rr(n,l,s):Jn(n,l,s)}return n}function Ma(e,t){return function(n,r){var a=Wo(n)?St:tr,i=t?t():{};return a(n,e,ii(r,2),i)}}function Na(e){return Qr((function(t,n){var r=-1,i=n.length,o=i>1?n[i-1]:a,u=i>2?n[2]:a;for(o=e.length>3&&\"function\"==typeof o?(i--,o):a,u&&gi(n[0],n[1],u)&&(o=i<3?a:o,i=1),t=Se(t);++r<i;){var l=n[r];l&&e(t,l,r,o)}return t}))}function Pa(e,t){return function(n,r){if(null==n)return n;if(!Ho(n))return e(n,r);for(var a=n.length,i=t?a:-1,o=Se(n);(t?i--:++i<a)&&!1!==r(o[i],i,o););return n}}function za(e){return function(t,n,r){for(var a=-1,i=Se(t),o=r(t),u=o.length;u--;){var l=o[e?u:++a];if(!1===n(i[l],l,i))break}return t}}function La(e){return function(t){var n=rn(t=yu(t))?fn(t):a,r=n?n[0]:t.charAt(0),i=n?ma(n,1).join(\"\"):t.slice(1);return r[e]()+i}}function Oa(e){return function(t){return Ot(Ku(Bu(t).replace(Ke,\"\")),e,\"\")}}function Aa(e){return function(){var t=arguments;switch(t.length){case 0:return new e;case 1:return new e(t[0]);case 2:return new e(t[0],t[1]);case 3:return new e(t[0],t[1],t[2]);case 4:return new e(t[0],t[1],t[2],t[3]);case 5:return new e(t[0],t[1],t[2],t[3],t[4]);case 6:return new e(t[0],t[1],t[2],t[3],t[4],t[5]);case 7:return new e(t[0],t[1],t[2],t[3],t[4],t[5],t[6])}var n=Un(e.prototype),r=e.apply(n,t);return Jo(r)?r:n}}function Fa(e){return function(t,n,r){var i=Se(t);if(!Ho(t)){var o=ii(n,3);t=Pu(t),n=function(e){return o(i[e],e,i)}}var u=e(t,n,r);return u>-1?i[o?t[u]:u]:a}}function Da(e){return Ja((function(t){var n=t.length,r=n,o=$n.prototype.thru;for(e&&t.reverse();r--;){var u=t[r];if(\"function\"!=typeof u)throw new Te(i);if(o&&!l&&\"wrapper\"==ri(u))var l=new $n([],!0)}for(r=l?r:n;++r<n;){var s=ri(u=t[r]),c=\"wrapper\"==s?ni(u):a;l=c&&mi(c[0])&&424==c[1]&&!c[4].length&&1==c[9]?l[ri(c[0])].apply(l,c[3]):1==u.length&&mi(u)?l[s]():l.thru(u)}return function(){var e=arguments,r=e[0];if(l&&1==e.length&&Wo(r))return l.plant(r).value();for(var a=0,i=n?t[a].apply(this,e):r;++a<n;)i=t[a].call(this,i);return i}}))}function Ra(e,t,n,i,o,u,l,c,f,p){var d=t&s,h=1&t,v=2&t,g=24&t,y=512&t,m=v?a:Aa(e);return function s(){for(var b=arguments.length,_=r(b),w=b;w--;)_[w]=arguments[w];if(g)var x=ai(s),k=function(e,t){for(var n=e.length,r=0;n--;)e[n]===t&&++r;return r}(_,x);if(i&&(_=Sa(_,i,o,g)),u&&(_=Ea(_,u,l,g)),b-=k,g&&b<p){var S=un(_,x);return Va(e,t,Ra,s.placeholder,n,_,S,c,f,p-b)}var E=h?n:this,C=v?E[e]:e;return b=_.length,c?_=function(e,t){for(var n=e.length,r=yn(t.length,n),i=Ca(e);r--;){var o=t[r];e[r]=vi(o,n)?i[o]:a}return e}(_,c):y&&b>1&&_.reverse(),d&&f<b&&(_.length=f),this&&this!==ft&&this instanceof s&&(C=m||Aa(C)),C.apply(E,_)}}function ja(e,t){return function(n,r){return function(e,t,n,r){return mr(e,(function(e,a,i){t(r,n(e),a,i)})),r}(n,e,t(r),{})}}function Ua(e,t){return function(n,r){var i;if(n===a&&r===a)return t;if(n!==a&&(i=n),r!==a){if(i===a)return r;\"string\"==typeof n||\"string\"==typeof r?(n=oa(n),r=oa(r)):(n=ia(n),r=ia(r)),i=e(n,r)}return i}}function Ia(e){return Ja((function(t){return t=zt(t,Gt(ii())),Qr((function(n){var r=this;return e(t,(function(e){return kt(e,r,n)}))}))}))}function $a(e,t){var n=(t=t===a?\" \":oa(t)).length;if(n<2)return n?qr(t,e):t;var r=qr(t,pt(e/cn(t)));return rn(t)?ma(fn(r),0,e).join(\"\"):r.slice(0,e)}function Ba(e){return function(t,n,i){return i&&\"number\"!=typeof i&&gi(t,n,i)&&(n=i=a),t=pu(t),n===a?(n=t,t=0):n=pu(n),function(e,t,n,a){for(var i=-1,o=gn(pt((t-e)/(n||1)),0),u=r(o);o--;)u[a?o:++i]=e,e+=n;return u}(t,n,i=i===a?t<n?1:-1:pu(i),e)}}function Wa(e){return function(t,n){return\"string\"==typeof t&&\"string\"==typeof n||(t=vu(t),n=vu(n)),e(t,n)}}function Va(e,t,n,r,i,o,u,s,c,f){var p=8&t;t|=p?l:64,4&(t&=~(p?64:l))||(t&=-4);var d=[e,t,i,p?o:a,p?u:a,p?a:o,p?a:u,s,c,f],h=n.apply(a,d);return mi(e)&&Ci(h,d),h.placeholder=r,Ni(h,e,t)}function Ha(e){var t=ke[e];return function(e,n){if(e=vu(e),(n=null==n?0:yn(du(n),292))&&Dt(e)){var r=(yu(e)+\"e\").split(\"e\");return+((r=(yu(t(r[0]+\"e\"+(+r[1]+n)))+\"e\").split(\"e\"))[0]+\"e\"+(+r[1]-n))}return t(e)}}var qa=En&&1/ln(new En([,-0]))[1]==c?function(e){return new En(e)}:ul;function Qa(e){return function(t){var n=fi(t);return n==x?an(t):n==T?sn(t):function(e,t){return zt(t,(function(t){return[t,e[t]]}))}(t,e(t))}}function Ya(e,t,n,o,c,f,p,d){var h=2&t;if(!h&&\"function\"!=typeof e)throw new Te(i);var v=o?o.length:0;if(v||(t&=-97,o=c=a),p=p===a?p:gn(du(p),0),d=d===a?d:du(d),v-=c?c.length:0,64&t){var g=o,y=c;o=c=a}var m=h?a:ni(e),b=[e,t,n,o,c,g,y,f,p,d];if(m&&function(e,t){var n=e[1],r=t[1],a=n|r,i=a<131,o=r==s&&8==n||r==s&&256==n&&e[7].length<=t[8]||384==r&&t[7].length<=t[8]&&8==n;if(!i&&!o)return e;1&r&&(e[2]=t[2],a|=1&n?0:4);var l=t[3];if(l){var c=e[3];e[3]=c?Sa(c,l,t[4]):l,e[4]=c?un(e[3],u):t[4]}(l=t[5])&&(c=e[5],e[5]=c?Ea(c,l,t[6]):l,e[6]=c?un(e[5],u):t[6]),(l=t[7])&&(e[7]=l),r&s&&(e[8]=null==e[8]?t[8]:yn(e[8],t[8])),null==e[9]&&(e[9]=t[9]),e[0]=t[0],e[1]=a}(b,m),e=b[0],t=b[1],n=b[2],o=b[3],c=b[4],!(d=b[9]=b[9]===a?h?0:e.length:gn(b[9]-v,0))&&24&t&&(t&=-25),t&&1!=t)_=8==t||16==t?function(e,t,n){var i=Aa(e);return function o(){for(var u=arguments.length,l=r(u),s=u,c=ai(o);s--;)l[s]=arguments[s];var f=u<3&&l[0]!==c&&l[u-1]!==c?[]:un(l,c);return(u-=f.length)<n?Va(e,t,Ra,o.placeholder,a,l,f,a,a,n-u):kt(this&&this!==ft&&this instanceof o?i:e,this,l)}}(e,t,d):t!=l&&33!=t||c.length?Ra.apply(a,b):function(e,t,n,a){var i=1&t,o=Aa(e);return function t(){for(var u=-1,l=arguments.length,s=-1,c=a.length,f=r(c+l),p=this&&this!==ft&&this instanceof t?o:e;++s<c;)f[s]=a[s];for(;l--;)f[s++]=arguments[++u];return kt(p,i?n:this,f)}}(e,t,n,o);else var _=function(e,t,n){var r=1&t,a=Aa(e);return function t(){return(this&&this!==ft&&this instanceof t?a:e).apply(r?n:this,arguments)}}(e,t,n);return Ni((m?Zr:Ci)(_,b),e,t)}function Ga(e,t,n,r){return e===a||Uo(e,Pe[n])&&!Oe.call(r,n)?t:e}function Ka(e,t,n,r,i,o){return Jo(e)&&Jo(t)&&(o.set(t,e),Ur(e,t,a,Ka,o),o.delete(t)),e}function Za(e){return ru(e)?a:e}function Xa(e,t,n,r,i,o){var u=1&n,l=e.length,s=t.length;if(l!=s&&!(u&&s>l))return!1;var c=o.get(e),f=o.get(t);if(c&&f)return c==t&&f==e;var p=-1,d=!0,h=2&n?new qn:a;for(o.set(e,t),o.set(t,e);++p<l;){var v=e[p],g=t[p];if(r)var y=u?r(g,v,p,t,e,o):r(v,g,p,e,t,o);if(y!==a){if(y)continue;d=!1;break}if(h){if(!Ft(t,(function(e,t){if(!Zt(h,t)&&(v===e||i(v,e,n,r,o)))return h.push(t)}))){d=!1;break}}else if(v!==g&&!i(v,g,n,r,o)){d=!1;break}}return o.delete(e),o.delete(t),d}function Ja(e){return Mi(ki(e,a,Wi),e+\"\")}function ei(e){return xr(e,Pu,si)}function ti(e){return xr(e,zu,ci)}var ni=Mn?function(e){return Mn.get(e)}:ul;function ri(e){for(var t=e.name+\"\",n=Nn[t],r=Oe.call(Nn,t)?n.length:0;r--;){var a=n[r],i=a.func;if(null==i||i==e)return a.name}return t}function ai(e){return(Oe.call(jn,\"placeholder\")?jn:e).placeholder}function ii(){var e=jn.iteratee||rl;return e=e===rl?Or:e,arguments.length?e(arguments[0],arguments[1]):e}function oi(e,t){var n,r,a=e.__data__;return(\"string\"==(r=typeof(n=t))||\"number\"==r||\"symbol\"==r||\"boolean\"==r?\"__proto__\"!==n:null===n)?a[\"string\"==typeof t?\"string\":\"hash\"]:a.map}function ui(e){for(var t=Pu(e),n=t.length;n--;){var r=t[n],a=e[r];t[n]=[r,a,wi(a)]}return t}function li(e,t){var n=function(e,t){return null==e?a:e[t]}(e,t);return Lr(n)?n:a}var si=vt?function(e){return null==e?[]:(e=Se(e),Mt(vt(e),(function(t){return qe.call(e,t)})))}:hl,ci=vt?function(e){for(var t=[];e;)Lt(t,si(e)),e=Ve(e);return t}:hl,fi=kr;function pi(e,t,n){for(var r=-1,a=(t=ga(t,e)).length,i=!1;++r<a;){var o=Fi(t[r]);if(!(i=null!=e&&n(e,o)))break;e=e[o]}return i||++r!=a?i:!!(a=null==e?0:e.length)&&Xo(a)&&vi(o,a)&&(Wo(e)||Bo(e))}function di(e){return\"function\"!=typeof e.constructor||_i(e)?{}:Un(Ve(e))}function hi(e){return Wo(e)||Bo(e)||!!(Ye&&e&&e[Ye])}function vi(e,t){var n=typeof e;return!!(t=null==t?f:t)&&(\"number\"==n||\"symbol\"!=n&&ye.test(e))&&e>-1&&e%1==0&&e<t}function gi(e,t,n){if(!Jo(n))return!1;var r=typeof t;return!!(\"number\"==r?Ho(n)&&vi(t,n.length):\"string\"==r&&t in n)&&Uo(n[t],e)}function yi(e,t){if(Wo(e))return!1;var n=typeof e;return!(\"number\"!=n&&\"symbol\"!=n&&\"boolean\"!=n&&null!=e&&!uu(e))||J.test(e)||!X.test(e)||null!=t&&e in Se(t)}function mi(e){var t=ri(e),n=jn[t];if(\"function\"!=typeof n||!(t in Bn.prototype))return!1;if(e===n)return!0;var r=ni(n);return!!r&&e===r[0]}(xn&&fi(new xn(new ArrayBuffer(1)))!=L||kn&&fi(new kn)!=x||Sn&&fi(Sn.resolve())!=E||En&&fi(new En)!=T||Cn&&fi(new Cn)!=P)&&(fi=function(e){var t=kr(e),n=t==S?e.constructor:a,r=n?Di(n):\"\";if(r)switch(r){case Pn:return L;case zn:return x;case Ln:return E;case On:return T;case An:return P}return t});var bi=ze?Ko:vl;function _i(e){var t=e&&e.constructor;return e===(\"function\"==typeof t&&t.prototype||Pe)}function wi(e){return e==e&&!Jo(e)}function xi(e,t){return function(n){return null!=n&&n[e]===t&&(t!==a||e in Se(n))}}function ki(e,t,n){return t=gn(t===a?e.length-1:t,0),function(){for(var a=arguments,i=-1,o=gn(a.length-t,0),u=r(o);++i<o;)u[i]=a[t+i];i=-1;for(var l=r(t+1);++i<t;)l[i]=a[i];return l[t]=n(u),kt(e,this,l)}}function Si(e,t){return t.length<2?e:wr(e,ea(t,0,-1))}function Ei(e,t){if((\"constructor\"!==t||\"function\"!=typeof e[t])&&\"__proto__\"!=t)return e[t]}var Ci=Pi(Zr),Ti=ct||function(e,t){return ft.setTimeout(e,t)},Mi=Pi(Xr);function Ni(e,t,n){var r=t+\"\";return Mi(e,function(e,t){var n=t.length;if(!n)return e;var r=n-1;return t[r]=(n>1?\"& \":\"\")+t[r],t=t.join(n>2?\", \":\" \"),e.replace(ie,\"{\\n/* [wrapped with \"+t+\"] */\\n\")}(r,function(e,t){return Et(h,(function(n){var r=\"_.\"+n[0];t&n[1]&&!Nt(e,r)&&e.push(r)})),e.sort()}(function(e){var t=e.match(oe);return t?t[1].split(ue):[]}(r),n)))}function Pi(e){var t=0,n=0;return function(){var r=mn(),i=16-(r-n);if(n=r,i>0){if(++t>=800)return arguments[0]}else t=0;return e.apply(a,arguments)}}function zi(e,t){var n=-1,r=e.length,i=r-1;for(t=t===a?r:t;++n<t;){var o=Hr(n,i),u=e[o];e[o]=e[n],e[n]=u}return e.length=t,e}var Li,Oi,Ai=(Li=Oo((function(e){var t=[];return 46===e.charCodeAt(0)&&t.push(\"\"),e.replace(ee,(function(e,n,r,a){t.push(r?a.replace(ce,\"$1\"):n||e)})),t}),(function(e){return 500===Oi.size&&Oi.clear(),e})),Oi=Li.cache,Li);function Fi(e){if(\"string\"==typeof e||uu(e))return e;var t=e+\"\";return\"0\"==t&&1/e==-1/0?\"-0\":t}function Di(e){if(null!=e){try{return Le.call(e)}catch(e){}try{return e+\"\"}catch(e){}}return\"\"}function Ri(e){if(e instanceof Bn)return e.clone();var t=new $n(e.__wrapped__,e.__chain__);return t.__actions__=Ca(e.__actions__),t.__index__=e.__index__,t.__values__=e.__values__,t}var ji=Qr((function(e,t){return qo(e)?sr(e,vr(t,1,qo,!0)):[]})),Ui=Qr((function(e,t){var n=Yi(t);return qo(n)&&(n=a),qo(e)?sr(e,vr(t,1,qo,!0),ii(n,2)):[]})),Ii=Qr((function(e,t){var n=Yi(t);return qo(n)&&(n=a),qo(e)?sr(e,vr(t,1,qo,!0),a,n):[]}));function $i(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var a=null==n?0:du(n);return a<0&&(a=gn(r+a,0)),jt(e,ii(t,3),a)}function Bi(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var i=r-1;return n!==a&&(i=du(n),i=n<0?gn(r+i,0):yn(i,r-1)),jt(e,ii(t,3),i,!0)}function Wi(e){return null!=e&&e.length?vr(e,1):[]}function Vi(e){return e&&e.length?e[0]:a}var Hi=Qr((function(e){var t=zt(e,ha);return t.length&&t[0]===e[0]?Tr(t):[]})),qi=Qr((function(e){var t=Yi(e),n=zt(e,ha);return t===Yi(n)?t=a:n.pop(),n.length&&n[0]===e[0]?Tr(n,ii(t,2)):[]})),Qi=Qr((function(e){var t=Yi(e),n=zt(e,ha);return(t=\"function\"==typeof t?t:a)&&n.pop(),n.length&&n[0]===e[0]?Tr(n,a,t):[]}));function Yi(e){var t=null==e?0:e.length;return t?e[t-1]:a}var Gi=Qr(Ki);function Ki(e,t){return e&&e.length&&t&&t.length?Wr(e,t):e}var Zi=Ja((function(e,t){var n=null==e?0:e.length,r=ar(e,t);return Vr(e,zt(t,(function(e){return vi(e,n)?+e:e})).sort(ka)),r}));function Xi(e){return null==e?e:wn.call(e)}var Ji=Qr((function(e){return ua(vr(e,1,qo,!0))})),eo=Qr((function(e){var t=Yi(e);return qo(t)&&(t=a),ua(vr(e,1,qo,!0),ii(t,2))})),to=Qr((function(e){var t=Yi(e);return t=\"function\"==typeof t?t:a,ua(vr(e,1,qo,!0),a,t)}));function no(e){if(!e||!e.length)return[];var t=0;return e=Mt(e,(function(e){if(qo(e))return t=gn(e.length,t),!0})),Qt(t,(function(t){return zt(e,Wt(t))}))}function ro(e,t){if(!e||!e.length)return[];var n=no(e);return null==t?n:zt(n,(function(e){return kt(t,a,e)}))}var ao=Qr((function(e,t){return qo(e)?sr(e,t):[]})),io=Qr((function(e){return pa(Mt(e,qo))})),oo=Qr((function(e){var t=Yi(e);return qo(t)&&(t=a),pa(Mt(e,qo),ii(t,2))})),uo=Qr((function(e){var t=Yi(e);return t=\"function\"==typeof t?t:a,pa(Mt(e,qo),a,t)})),lo=Qr(no),so=Qr((function(e){var t=e.length,n=t>1?e[t-1]:a;return n=\"function\"==typeof n?(e.pop(),n):a,ro(e,n)}));function co(e){var t=jn(e);return t.__chain__=!0,t}function fo(e,t){return t(e)}var po=Ja((function(e){var t=e.length,n=t?e[0]:0,r=this.__wrapped__,i=function(t){return ar(t,e)};return!(t>1||this.__actions__.length)&&r instanceof Bn&&vi(n)?((r=r.slice(n,+n+(t?1:0))).__actions__.push({func:fo,args:[i],thisArg:a}),new $n(r,this.__chain__).thru((function(e){return t&&!e.length&&e.push(a),e}))):this.thru(i)})),ho=Ma((function(e,t,n){Oe.call(e,n)?++e[n]:rr(e,n,1)})),vo=Fa($i),go=Fa(Bi);function yo(e,t){return(Wo(e)?Et:cr)(e,ii(t,3))}function mo(e,t){return(Wo(e)?Ct:fr)(e,ii(t,3))}var bo=Ma((function(e,t,n){Oe.call(e,n)?e[n].push(t):rr(e,n,[t])})),_o=Qr((function(e,t,n){var a=-1,i=\"function\"==typeof t,o=Ho(e)?r(e.length):[];return cr(e,(function(e){o[++a]=i?kt(t,e,n):Mr(e,t,n)})),o})),wo=Ma((function(e,t,n){rr(e,n,t)}));function xo(e,t){return(Wo(e)?zt:Dr)(e,ii(t,3))}var ko=Ma((function(e,t,n){e[n?0:1].push(t)}),(function(){return[[],[]]})),So=Qr((function(e,t){if(null==e)return[];var n=t.length;return n>1&&gi(e,t[0],t[1])?t=[]:n>2&&gi(t[0],t[1],t[2])&&(t=[t[0]]),$r(e,vr(t,1),[])})),Eo=st||function(){return ft.Date.now()};function Co(e,t,n){return t=n?a:t,t=e&&null==t?e.length:t,Ya(e,s,a,a,a,a,t)}function To(e,t){var n;if(\"function\"!=typeof t)throw new Te(i);return e=du(e),function(){return--e>0&&(n=t.apply(this,arguments)),e<=1&&(t=a),n}}var Mo=Qr((function(e,t,n){var r=1;if(n.length){var a=un(n,ai(Mo));r|=l}return Ya(e,r,t,n,a)})),No=Qr((function(e,t,n){var r=3;if(n.length){var a=un(n,ai(No));r|=l}return Ya(t,r,e,n,a)}));function Po(e,t,n){var r,o,u,l,s,c,f=0,p=!1,d=!1,h=!0;if(\"function\"!=typeof e)throw new Te(i);function v(t){var n=r,i=o;return r=o=a,f=t,l=e.apply(i,n)}function g(e){var n=e-c;return c===a||n>=t||n<0||d&&e-f>=u}function y(){var e=Eo();if(g(e))return m(e);s=Ti(y,function(e){var n=t-(e-c);return d?yn(n,u-(e-f)):n}(e))}function m(e){return s=a,h&&r?v(e):(r=o=a,l)}function b(){var e=Eo(),n=g(e);if(r=arguments,o=this,c=e,n){if(s===a)return function(e){return f=e,s=Ti(y,t),p?v(e):l}(c);if(d)return ba(s),s=Ti(y,t),v(c)}return s===a&&(s=Ti(y,t)),l}return t=vu(t)||0,Jo(n)&&(p=!!n.leading,u=(d=\"maxWait\"in n)?gn(vu(n.maxWait)||0,t):u,h=\"trailing\"in n?!!n.trailing:h),b.cancel=function(){s!==a&&ba(s),f=0,r=c=o=s=a},b.flush=function(){return s===a?l:m(Eo())},b}var zo=Qr((function(e,t){return lr(e,1,t)})),Lo=Qr((function(e,t,n){return lr(e,vu(t)||0,n)}));function Oo(e,t){if(\"function\"!=typeof e||null!=t&&\"function\"!=typeof t)throw new Te(i);var n=function(){var r=arguments,a=t?t.apply(this,r):r[0],i=n.cache;if(i.has(a))return i.get(a);var o=e.apply(this,r);return n.cache=i.set(a,o)||i,o};return n.cache=new(Oo.Cache||Hn),n}function Ao(e){if(\"function\"!=typeof e)throw new Te(i);return function(){var t=arguments;switch(t.length){case 0:return!e.call(this);case 1:return!e.call(this,t[0]);case 2:return!e.call(this,t[0],t[1]);case 3:return!e.call(this,t[0],t[1],t[2])}return!e.apply(this,t)}}Oo.Cache=Hn;var Fo=ya((function(e,t){var n=(t=1==t.length&&Wo(t[0])?zt(t[0],Gt(ii())):zt(vr(t,1),Gt(ii()))).length;return Qr((function(r){for(var a=-1,i=yn(r.length,n);++a<i;)r[a]=t[a].call(this,r[a]);return kt(e,this,r)}))})),Do=Qr((function(e,t){var n=un(t,ai(Do));return Ya(e,l,a,t,n)})),Ro=Qr((function(e,t){var n=un(t,ai(Ro));return Ya(e,64,a,t,n)})),jo=Ja((function(e,t){return Ya(e,256,a,a,a,t)}));function Uo(e,t){return e===t||e!=e&&t!=t}var Io=Wa(Sr),$o=Wa((function(e,t){return e>=t})),Bo=Nr(function(){return arguments}())?Nr:function(e){return eu(e)&&Oe.call(e,\"callee\")&&!qe.call(e,\"callee\")},Wo=r.isArray,Vo=yt?Gt(yt):function(e){return eu(e)&&kr(e)==z};function Ho(e){return null!=e&&Xo(e.length)&&!Ko(e)}function qo(e){return eu(e)&&Ho(e)}var Qo=gt||vl,Yo=mt?Gt(mt):function(e){return eu(e)&&kr(e)==m};function Go(e){if(!eu(e))return!1;var t=kr(e);return t==b||\"[object DOMException]\"==t||\"string\"==typeof e.message&&\"string\"==typeof e.name&&!ru(e)}function Ko(e){if(!Jo(e))return!1;var t=kr(e);return t==_||t==w||\"[object AsyncFunction]\"==t||\"[object Proxy]\"==t}function Zo(e){return\"number\"==typeof e&&e==du(e)}function Xo(e){return\"number\"==typeof e&&e>-1&&e%1==0&&e<=f}function Jo(e){var t=typeof e;return null!=e&&(\"object\"==t||\"function\"==t)}function eu(e){return null!=e&&\"object\"==typeof e}var tu=bt?Gt(bt):function(e){return eu(e)&&fi(e)==x};function nu(e){return\"number\"==typeof e||eu(e)&&kr(e)==k}function ru(e){if(!eu(e)||kr(e)!=S)return!1;var t=Ve(e);if(null===t)return!0;var n=Oe.call(t,\"constructor\")&&t.constructor;return\"function\"==typeof n&&n instanceof n&&Le.call(n)==Re}var au=_t?Gt(_t):function(e){return eu(e)&&kr(e)==C},iu=wt?Gt(wt):function(e){return eu(e)&&fi(e)==T};function ou(e){return\"string\"==typeof e||!Wo(e)&&eu(e)&&kr(e)==M}function uu(e){return\"symbol\"==typeof e||eu(e)&&kr(e)==N}var lu=xt?Gt(xt):function(e){return eu(e)&&Xo(e.length)&&!!at[kr(e)]},su=Wa(Fr),cu=Wa((function(e,t){return e<=t}));function fu(e){if(!e)return[];if(Ho(e))return ou(e)?fn(e):Ca(e);if(Ge&&e[Ge])return function(e){for(var t,n=[];!(t=e.next()).done;)n.push(t.value);return n}(e[Ge]());var t=fi(e);return(t==x?an:t==T?ln:Uu)(e)}function pu(e){return e?(e=vu(e))===c||e===-1/0?17976931348623157e292*(e<0?-1:1):e==e?e:0:0===e?e:0}function du(e){var t=pu(e),n=t%1;return t==t?n?t-n:t:0}function hu(e){return e?ir(du(e),0,d):0}function vu(e){if(\"number\"==typeof e)return e;if(uu(e))return p;if(Jo(e)){var t=\"function\"==typeof e.valueOf?e.valueOf():e;e=Jo(t)?t+\"\":t}if(\"string\"!=typeof e)return 0===e?e:+e;e=Yt(e);var n=he.test(e);return n||ge.test(e)?lt(e.slice(2),n?2:8):de.test(e)?p:+e}function gu(e){return Ta(e,zu(e))}function yu(e){return null==e?\"\":oa(e)}var mu=Na((function(e,t){if(_i(t)||Ho(t))Ta(t,Pu(t),e);else for(var n in t)Oe.call(t,n)&&Jn(e,n,t[n])})),bu=Na((function(e,t){Ta(t,zu(t),e)})),_u=Na((function(e,t,n,r){Ta(t,zu(t),e,r)})),wu=Na((function(e,t,n,r){Ta(t,Pu(t),e,r)})),xu=Ja(ar),ku=Qr((function(e,t){e=Se(e);var n=-1,r=t.length,i=r>2?t[2]:a;for(i&&gi(t[0],t[1],i)&&(r=1);++n<r;)for(var o=t[n],u=zu(o),l=-1,s=u.length;++l<s;){var c=u[l],f=e[c];(f===a||Uo(f,Pe[c])&&!Oe.call(e,c))&&(e[c]=o[c])}return e})),Su=Qr((function(e){return e.push(a,Ka),kt(Ou,a,e)}));function Eu(e,t,n){var r=null==e?a:wr(e,t);return r===a?n:r}function Cu(e,t){return null!=e&&pi(e,t,Cr)}var Tu=ja((function(e,t,n){null!=t&&\"function\"!=typeof t.toString&&(t=De.call(t)),e[t]=n}),Ju(nl)),Mu=ja((function(e,t,n){null!=t&&\"function\"!=typeof t.toString&&(t=De.call(t)),Oe.call(e,t)?e[t].push(n):e[t]=[n]}),ii),Nu=Qr(Mr);function Pu(e){return Ho(e)?Yn(e):Ar(e)}function zu(e){return Ho(e)?Yn(e,!0):function(e){if(!Jo(e))return function(e){var t=[];if(null!=e)for(var n in Se(e))t.push(n);return t}(e);var t=_i(e),n=[];for(var r in e)(\"constructor\"!=r||!t&&Oe.call(e,r))&&n.push(r);return n}(e)}var Lu=Na((function(e,t,n){Ur(e,t,n)})),Ou=Na((function(e,t,n,r){Ur(e,t,n,r)})),Au=Ja((function(e,t){var n={};if(null==e)return n;var r=!1;t=zt(t,(function(t){return t=ga(t,e),r||(r=t.length>1),t})),Ta(e,ti(e),n),r&&(n=or(n,7,Za));for(var a=t.length;a--;)la(n,t[a]);return n})),Fu=Ja((function(e,t){return null==e?{}:function(e,t){return Br(e,t,(function(t,n){return Cu(e,n)}))}(e,t)}));function Du(e,t){if(null==e)return{};var n=zt(ti(e),(function(e){return[e]}));return t=ii(t),Br(e,n,(function(e,n){return t(e,n[0])}))}var Ru=Qa(Pu),ju=Qa(zu);function Uu(e){return null==e?[]:Kt(e,Pu(e))}var Iu=Oa((function(e,t,n){return t=t.toLowerCase(),e+(n?$u(t):t)}));function $u(e){return Gu(yu(e).toLowerCase())}function Bu(e){return(e=yu(e))&&e.replace(me,en).replace(Ze,\"\")}var Wu=Oa((function(e,t,n){return e+(n?\"-\":\"\")+t.toLowerCase()})),Vu=Oa((function(e,t,n){return e+(n?\" \":\"\")+t.toLowerCase()})),Hu=La(\"toLowerCase\"),qu=Oa((function(e,t,n){return e+(n?\"_\":\"\")+t.toLowerCase()})),Qu=Oa((function(e,t,n){return e+(n?\" \":\"\")+Gu(t)})),Yu=Oa((function(e,t,n){return e+(n?\" \":\"\")+t.toUpperCase()})),Gu=La(\"toUpperCase\");function Ku(e,t,n){return e=yu(e),(t=n?a:t)===a?function(e){return tt.test(e)}(e)?function(e){return e.match(Je)||[]}(e):function(e){return e.match(le)||[]}(e):e.match(t)||[]}var Zu=Qr((function(e,t){try{return kt(e,a,t)}catch(e){return Go(e)?e:new we(e)}})),Xu=Ja((function(e,t){return Et(t,(function(t){t=Fi(t),rr(e,t,Mo(e[t],e))})),e}));function Ju(e){return function(){return e}}var el=Da(),tl=Da(!0);function nl(e){return e}function rl(e){return Or(\"function\"==typeof e?e:or(e,1))}var al=Qr((function(e,t){return function(n){return Mr(n,e,t)}})),il=Qr((function(e,t){return function(n){return Mr(e,n,t)}}));function ol(e,t,n){var r=Pu(t),a=_r(t,r);null!=n||Jo(t)&&(a.length||!r.length)||(n=t,t=e,e=this,a=_r(t,Pu(t)));var i=!(Jo(n)&&\"chain\"in n&&!n.chain),o=Ko(e);return Et(a,(function(n){var r=t[n];e[n]=r,o&&(e.prototype[n]=function(){var t=this.__chain__;if(i||t){var n=e(this.__wrapped__);return(n.__actions__=Ca(this.__actions__)).push({func:r,args:arguments,thisArg:e}),n.__chain__=t,n}return r.apply(e,Lt([this.value()],arguments))})})),e}function ul(){}var ll=Ia(zt),sl=Ia(Tt),cl=Ia(Ft);function fl(e){return yi(e)?Wt(Fi(e)):function(e){return function(t){return wr(t,e)}}(e)}var pl=Ba(),dl=Ba(!0);function hl(){return[]}function vl(){return!1}var gl,yl=Ua((function(e,t){return e+t}),0),ml=Ha(\"ceil\"),bl=Ua((function(e,t){return e/t}),1),_l=Ha(\"floor\"),wl=Ua((function(e,t){return e*t}),1),xl=Ha(\"round\"),kl=Ua((function(e,t){return e-t}),0);return jn.after=function(e,t){if(\"function\"!=typeof t)throw new Te(i);return e=du(e),function(){if(--e<1)return t.apply(this,arguments)}},jn.ary=Co,jn.assign=mu,jn.assignIn=bu,jn.assignInWith=_u,jn.assignWith=wu,jn.at=xu,jn.before=To,jn.bind=Mo,jn.bindAll=Xu,jn.bindKey=No,jn.castArray=function(){if(!arguments.length)return[];var e=arguments[0];return Wo(e)?e:[e]},jn.chain=co,jn.chunk=function(e,t,n){t=(n?gi(e,t,n):t===a)?1:gn(du(t),0);var i=null==e?0:e.length;if(!i||t<1)return[];for(var o=0,u=0,l=r(pt(i/t));o<i;)l[u++]=ea(e,o,o+=t);return l},jn.compact=function(e){for(var t=-1,n=null==e?0:e.length,r=0,a=[];++t<n;){var i=e[t];i&&(a[r++]=i)}return a},jn.concat=function(){var e=arguments.length;if(!e)return[];for(var t=r(e-1),n=arguments[0],a=e;a--;)t[a-1]=arguments[a];return Lt(Wo(n)?Ca(n):[n],vr(t,1))},jn.cond=function(e){var t=null==e?0:e.length,n=ii();return e=t?zt(e,(function(e){if(\"function\"!=typeof e[1])throw new Te(i);return[n(e[0]),e[1]]})):[],Qr((function(n){for(var r=-1;++r<t;){var a=e[r];if(kt(a[0],this,n))return kt(a[1],this,n)}}))},jn.conforms=function(e){return function(e){var t=Pu(e);return function(n){return ur(n,e,t)}}(or(e,1))},jn.constant=Ju,jn.countBy=ho,jn.create=function(e,t){var n=Un(e);return null==t?n:nr(n,t)},jn.curry=function e(t,n,r){var i=Ya(t,8,a,a,a,a,a,n=r?a:n);return i.placeholder=e.placeholder,i},jn.curryRight=function e(t,n,r){var i=Ya(t,16,a,a,a,a,a,n=r?a:n);return i.placeholder=e.placeholder,i},jn.debounce=Po,jn.defaults=ku,jn.defaultsDeep=Su,jn.defer=zo,jn.delay=Lo,jn.difference=ji,jn.differenceBy=Ui,jn.differenceWith=Ii,jn.drop=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,(t=n||t===a?1:du(t))<0?0:t,r):[]},jn.dropRight=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,0,(t=r-(t=n||t===a?1:du(t)))<0?0:t):[]},jn.dropRightWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!0,!0):[]},jn.dropWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!0):[]},jn.fill=function(e,t,n,r){var i=null==e?0:e.length;return i?(n&&\"number\"!=typeof n&&gi(e,t,n)&&(n=0,r=i),function(e,t,n,r){var i=e.length;for((n=du(n))<0&&(n=-n>i?0:i+n),(r=r===a||r>i?i:du(r))<0&&(r+=i),r=n>r?0:hu(r);n<r;)e[n++]=t;return e}(e,t,n,r)):[]},jn.filter=function(e,t){return(Wo(e)?Mt:hr)(e,ii(t,3))},jn.flatMap=function(e,t){return vr(xo(e,t),1)},jn.flatMapDeep=function(e,t){return vr(xo(e,t),c)},jn.flatMapDepth=function(e,t,n){return n=n===a?1:du(n),vr(xo(e,t),n)},jn.flatten=Wi,jn.flattenDeep=function(e){return null!=e&&e.length?vr(e,c):[]},jn.flattenDepth=function(e,t){return null!=e&&e.length?vr(e,t=t===a?1:du(t)):[]},jn.flip=function(e){return Ya(e,512)},jn.flow=el,jn.flowRight=tl,jn.fromPairs=function(e){for(var t=-1,n=null==e?0:e.length,r={};++t<n;){var a=e[t];r[a[0]]=a[1]}return r},jn.functions=function(e){return null==e?[]:_r(e,Pu(e))},jn.functionsIn=function(e){return null==e?[]:_r(e,zu(e))},jn.groupBy=bo,jn.initial=function(e){return null!=e&&e.length?ea(e,0,-1):[]},jn.intersection=Hi,jn.intersectionBy=qi,jn.intersectionWith=Qi,jn.invert=Tu,jn.invertBy=Mu,jn.invokeMap=_o,jn.iteratee=rl,jn.keyBy=wo,jn.keys=Pu,jn.keysIn=zu,jn.map=xo,jn.mapKeys=function(e,t){var n={};return t=ii(t,3),mr(e,(function(e,r,a){rr(n,t(e,r,a),e)})),n},jn.mapValues=function(e,t){var n={};return t=ii(t,3),mr(e,(function(e,r,a){rr(n,r,t(e,r,a))})),n},jn.matches=function(e){return Rr(or(e,1))},jn.matchesProperty=function(e,t){return jr(e,or(t,1))},jn.memoize=Oo,jn.merge=Lu,jn.mergeWith=Ou,jn.method=al,jn.methodOf=il,jn.mixin=ol,jn.negate=Ao,jn.nthArg=function(e){return e=du(e),Qr((function(t){return Ir(t,e)}))},jn.omit=Au,jn.omitBy=function(e,t){return Du(e,Ao(ii(t)))},jn.once=function(e){return To(2,e)},jn.orderBy=function(e,t,n,r){return null==e?[]:(Wo(t)||(t=null==t?[]:[t]),Wo(n=r?a:n)||(n=null==n?[]:[n]),$r(e,t,n))},jn.over=ll,jn.overArgs=Fo,jn.overEvery=sl,jn.overSome=cl,jn.partial=Do,jn.partialRight=Ro,jn.partition=ko,jn.pick=Fu,jn.pickBy=Du,jn.property=fl,jn.propertyOf=function(e){return function(t){return null==e?a:wr(e,t)}},jn.pull=Gi,jn.pullAll=Ki,jn.pullAllBy=function(e,t,n){return e&&e.length&&t&&t.length?Wr(e,t,ii(n,2)):e},jn.pullAllWith=function(e,t,n){return e&&e.length&&t&&t.length?Wr(e,t,a,n):e},jn.pullAt=Zi,jn.range=pl,jn.rangeRight=dl,jn.rearg=jo,jn.reject=function(e,t){return(Wo(e)?Mt:hr)(e,Ao(ii(t,3)))},jn.remove=function(e,t){var n=[];if(!e||!e.length)return n;var r=-1,a=[],i=e.length;for(t=ii(t,3);++r<i;){var o=e[r];t(o,r,e)&&(n.push(o),a.push(r))}return Vr(e,a),n},jn.rest=function(e,t){if(\"function\"!=typeof e)throw new Te(i);return Qr(e,t=t===a?t:du(t))},jn.reverse=Xi,jn.sampleSize=function(e,t,n){return t=(n?gi(e,t,n):t===a)?1:du(t),(Wo(e)?Kn:Gr)(e,t)},jn.set=function(e,t,n){return null==e?e:Kr(e,t,n)},jn.setWith=function(e,t,n,r){return r=\"function\"==typeof r?r:a,null==e?e:Kr(e,t,n,r)},jn.shuffle=function(e){return(Wo(e)?Zn:Jr)(e)},jn.slice=function(e,t,n){var r=null==e?0:e.length;return r?(n&&\"number\"!=typeof n&&gi(e,t,n)?(t=0,n=r):(t=null==t?0:du(t),n=n===a?r:du(n)),ea(e,t,n)):[]},jn.sortBy=So,jn.sortedUniq=function(e){return e&&e.length?aa(e):[]},jn.sortedUniqBy=function(e,t){return e&&e.length?aa(e,ii(t,2)):[]},jn.split=function(e,t,n){return n&&\"number\"!=typeof n&&gi(e,t,n)&&(t=n=a),(n=n===a?d:n>>>0)?(e=yu(e))&&(\"string\"==typeof t||null!=t&&!au(t))&&!(t=oa(t))&&rn(e)?ma(fn(e),0,n):e.split(t,n):[]},jn.spread=function(e,t){if(\"function\"!=typeof e)throw new Te(i);return t=null==t?0:gn(du(t),0),Qr((function(n){var r=n[t],a=ma(n,0,t);return r&&Lt(a,r),kt(e,this,a)}))},jn.tail=function(e){var t=null==e?0:e.length;return t?ea(e,1,t):[]},jn.take=function(e,t,n){return e&&e.length?ea(e,0,(t=n||t===a?1:du(t))<0?0:t):[]},jn.takeRight=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,(t=r-(t=n||t===a?1:du(t)))<0?0:t,r):[]},jn.takeRightWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!1,!0):[]},jn.takeWhile=function(e,t){return e&&e.length?ca(e,ii(t,3)):[]},jn.tap=function(e,t){return t(e),e},jn.throttle=function(e,t,n){var r=!0,a=!0;if(\"function\"!=typeof e)throw new Te(i);return Jo(n)&&(r=\"leading\"in n?!!n.leading:r,a=\"trailing\"in n?!!n.trailing:a),Po(e,t,{leading:r,maxWait:t,trailing:a})},jn.thru=fo,jn.toArray=fu,jn.toPairs=Ru,jn.toPairsIn=ju,jn.toPath=function(e){return Wo(e)?zt(e,Fi):uu(e)?[e]:Ca(Ai(yu(e)))},jn.toPlainObject=gu,jn.transform=function(e,t,n){var r=Wo(e),a=r||Qo(e)||lu(e);if(t=ii(t,4),null==n){var i=e&&e.constructor;n=a?r?new i:[]:Jo(e)&&Ko(i)?Un(Ve(e)):{}}return(a?Et:mr)(e,(function(e,r,a){return t(n,e,r,a)})),n},jn.unary=function(e){return Co(e,1)},jn.union=Ji,jn.unionBy=eo,jn.unionWith=to,jn.uniq=function(e){return e&&e.length?ua(e):[]},jn.uniqBy=function(e,t){return e&&e.length?ua(e,ii(t,2)):[]},jn.uniqWith=function(e,t){return t=\"function\"==typeof t?t:a,e&&e.length?ua(e,a,t):[]},jn.unset=function(e,t){return null==e||la(e,t)},jn.unzip=no,jn.unzipWith=ro,jn.update=function(e,t,n){return null==e?e:sa(e,t,va(n))},jn.updateWith=function(e,t,n,r){return r=\"function\"==typeof r?r:a,null==e?e:sa(e,t,va(n),r)},jn.values=Uu,jn.valuesIn=function(e){return null==e?[]:Kt(e,zu(e))},jn.without=ao,jn.words=Ku,jn.wrap=function(e,t){return Do(va(t),e)},jn.xor=io,jn.xorBy=oo,jn.xorWith=uo,jn.zip=lo,jn.zipObject=function(e,t){return da(e||[],t||[],Jn)},jn.zipObjectDeep=function(e,t){return da(e||[],t||[],Kr)},jn.zipWith=so,jn.entries=Ru,jn.entriesIn=ju,jn.extend=bu,jn.extendWith=_u,ol(jn,jn),jn.add=yl,jn.attempt=Zu,jn.camelCase=Iu,jn.capitalize=$u,jn.ceil=ml,jn.clamp=function(e,t,n){return n===a&&(n=t,t=a),n!==a&&(n=(n=vu(n))==n?n:0),t!==a&&(t=(t=vu(t))==t?t:0),ir(vu(e),t,n)},jn.clone=function(e){return or(e,4)},jn.cloneDeep=function(e){return or(e,5)},jn.cloneDeepWith=function(e,t){return or(e,5,t=\"function\"==typeof t?t:a)},jn.cloneWith=function(e,t){return or(e,4,t=\"function\"==typeof t?t:a)},jn.conformsTo=function(e,t){return null==t||ur(e,t,Pu(t))},jn.deburr=Bu,jn.defaultTo=function(e,t){return null==e||e!=e?t:e},jn.divide=bl,jn.endsWith=function(e,t,n){e=yu(e),t=oa(t);var r=e.length,i=n=n===a?r:ir(du(n),0,r);return(n-=t.length)>=0&&e.slice(n,i)==t},jn.eq=Uo,jn.escape=function(e){return(e=yu(e))&&Y.test(e)?e.replace(q,tn):e},jn.escapeRegExp=function(e){return(e=yu(e))&&ne.test(e)?e.replace(te,\"\\\\$&\"):e},jn.every=function(e,t,n){var r=Wo(e)?Tt:pr;return n&&gi(e,t,n)&&(t=a),r(e,ii(t,3))},jn.find=vo,jn.findIndex=$i,jn.findKey=function(e,t){return Rt(e,ii(t,3),mr)},jn.findLast=go,jn.findLastIndex=Bi,jn.findLastKey=function(e,t){return Rt(e,ii(t,3),br)},jn.floor=_l,jn.forEach=yo,jn.forEachRight=mo,jn.forIn=function(e,t){return null==e?e:gr(e,ii(t,3),zu)},jn.forInRight=function(e,t){return null==e?e:yr(e,ii(t,3),zu)},jn.forOwn=function(e,t){return e&&mr(e,ii(t,3))},jn.forOwnRight=function(e,t){return e&&br(e,ii(t,3))},jn.get=Eu,jn.gt=Io,jn.gte=$o,jn.has=function(e,t){return null!=e&&pi(e,t,Er)},jn.hasIn=Cu,jn.head=Vi,jn.identity=nl,jn.includes=function(e,t,n,r){e=Ho(e)?e:Uu(e),n=n&&!r?du(n):0;var a=e.length;return n<0&&(n=gn(a+n,0)),ou(e)?n<=a&&e.indexOf(t,n)>-1:!!a&&Ut(e,t,n)>-1},jn.indexOf=function(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var a=null==n?0:du(n);return a<0&&(a=gn(r+a,0)),Ut(e,t,a)},jn.inRange=function(e,t,n){return t=pu(t),n===a?(n=t,t=0):n=pu(n),function(e,t,n){return e>=yn(t,n)&&e<gn(t,n)}(e=vu(e),t,n)},jn.invoke=Nu,jn.isArguments=Bo,jn.isArray=Wo,jn.isArrayBuffer=Vo,jn.isArrayLike=Ho,jn.isArrayLikeObject=qo,jn.isBoolean=function(e){return!0===e||!1===e||eu(e)&&kr(e)==y},jn.isBuffer=Qo,jn.isDate=Yo,jn.isElement=function(e){return eu(e)&&1===e.nodeType&&!ru(e)},jn.isEmpty=function(e){if(null==e)return!0;if(Ho(e)&&(Wo(e)||\"string\"==typeof e||\"function\"==typeof e.splice||Qo(e)||lu(e)||Bo(e)))return!e.length;var t=fi(e);if(t==x||t==T)return!e.size;if(_i(e))return!Ar(e).length;for(var n in e)if(Oe.call(e,n))return!1;return!0},jn.isEqual=function(e,t){return Pr(e,t)},jn.isEqualWith=function(e,t,n){var r=(n=\"function\"==typeof n?n:a)?n(e,t):a;return r===a?Pr(e,t,a,n):!!r},jn.isError=Go,jn.isFinite=function(e){return\"number\"==typeof e&&Dt(e)},jn.isFunction=Ko,jn.isInteger=Zo,jn.isLength=Xo,jn.isMap=tu,jn.isMatch=function(e,t){return e===t||zr(e,t,ui(t))},jn.isMatchWith=function(e,t,n){return n=\"function\"==typeof n?n:a,zr(e,t,ui(t),n)},jn.isNaN=function(e){return nu(e)&&e!=+e},jn.isNative=function(e){if(bi(e))throw new we(\"Unsupported core-js use. Try https://npms.io/search?q=ponyfill.\");return Lr(e)},jn.isNil=function(e){return null==e},jn.isNull=function(e){return null===e},jn.isNumber=nu,jn.isObject=Jo,jn.isObjectLike=eu,jn.isPlainObject=ru,jn.isRegExp=au,jn.isSafeInteger=function(e){return Zo(e)&&e>=-9007199254740991&&e<=f},jn.isSet=iu,jn.isString=ou,jn.isSymbol=uu,jn.isTypedArray=lu,jn.isUndefined=function(e){return e===a},jn.isWeakMap=function(e){return eu(e)&&fi(e)==P},jn.isWeakSet=function(e){return eu(e)&&\"[object WeakSet]\"==kr(e)},jn.join=function(e,t){return null==e?\"\":Vt.call(e,t)},jn.kebabCase=Wu,jn.last=Yi,jn.lastIndexOf=function(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var i=r;return n!==a&&(i=(i=du(n))<0?gn(r+i,0):yn(i,r-1)),t==t?function(e,t,n){for(var r=n+1;r--;)if(e[r]===t)return r;return r}(e,t,i):jt(e,$t,i,!0)},jn.lowerCase=Vu,jn.lowerFirst=Hu,jn.lt=su,jn.lte=cu,jn.max=function(e){return e&&e.length?dr(e,nl,Sr):a},jn.maxBy=function(e,t){return e&&e.length?dr(e,ii(t,2),Sr):a},jn.mean=function(e){return Bt(e,nl)},jn.meanBy=function(e,t){return Bt(e,ii(t,2))},jn.min=function(e){return e&&e.length?dr(e,nl,Fr):a},jn.minBy=function(e,t){return e&&e.length?dr(e,ii(t,2),Fr):a},jn.stubArray=hl,jn.stubFalse=vl,jn.stubObject=function(){return{}},jn.stubString=function(){return\"\"},jn.stubTrue=function(){return!0},jn.multiply=wl,jn.nth=function(e,t){return e&&e.length?Ir(e,du(t)):a},jn.noConflict=function(){return ft._===this&&(ft._=je),this},jn.noop=ul,jn.now=Eo,jn.pad=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;if(!t||r>=t)return e;var a=(t-r)/2;return $a(dt(a),n)+e+$a(pt(a),n)},jn.padEnd=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;return t&&r<t?e+$a(t-r,n):e},jn.padStart=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;return t&&r<t?$a(t-r,n)+e:e},jn.parseInt=function(e,t,n){return n||null==t?t=0:t&&(t=+t),bn(yu(e).replace(re,\"\"),t||0)},jn.random=function(e,t,n){if(n&&\"boolean\"!=typeof n&&gi(e,t,n)&&(t=n=a),n===a&&(\"boolean\"==typeof t?(n=t,t=a):\"boolean\"==typeof e&&(n=e,e=a)),e===a&&t===a?(e=0,t=1):(e=pu(e),t===a?(t=e,e=0):t=pu(t)),e>t){var r=e;e=t,t=r}if(n||e%1||t%1){var i=_n();return yn(e+i*(t-e+ut(\"1e-\"+((i+\"\").length-1))),t)}return Hr(e,t)},jn.reduce=function(e,t,n){var r=Wo(e)?Ot:Ht,a=arguments.length<3;return r(e,ii(t,4),n,a,cr)},jn.reduceRight=function(e,t,n){var r=Wo(e)?At:Ht,a=arguments.length<3;return r(e,ii(t,4),n,a,fr)},jn.repeat=function(e,t,n){return t=(n?gi(e,t,n):t===a)?1:du(t),qr(yu(e),t)},jn.replace=function(){var e=arguments,t=yu(e[0]);return e.length<3?t:t.replace(e[1],e[2])},jn.result=function(e,t,n){var r=-1,i=(t=ga(t,e)).length;for(i||(i=1,e=a);++r<i;){var o=null==e?a:e[Fi(t[r])];o===a&&(r=i,o=n),e=Ko(o)?o.call(e):o}return e},jn.round=xl,jn.runInContext=e,jn.sample=function(e){return(Wo(e)?Gn:Yr)(e)},jn.size=function(e){if(null==e)return 0;if(Ho(e))return ou(e)?cn(e):e.length;var t=fi(e);return t==x||t==T?e.size:Ar(e).length},jn.snakeCase=qu,jn.some=function(e,t,n){var r=Wo(e)?Ft:ta;return n&&gi(e,t,n)&&(t=a),r(e,ii(t,3))},jn.sortedIndex=function(e,t){return na(e,t)},jn.sortedIndexBy=function(e,t,n){return ra(e,t,ii(n,2))},jn.sortedIndexOf=function(e,t){var n=null==e?0:e.length;if(n){var r=na(e,t);if(r<n&&Uo(e[r],t))return r}return-1},jn.sortedLastIndex=function(e,t){return na(e,t,!0)},jn.sortedLastIndexBy=function(e,t,n){return ra(e,t,ii(n,2),!0)},jn.sortedLastIndexOf=function(e,t){if(null!=e&&e.length){var n=na(e,t,!0)-1;if(Uo(e[n],t))return n}return-1},jn.startCase=Qu,jn.startsWith=function(e,t,n){return e=yu(e),n=null==n?0:ir(du(n),0,e.length),t=oa(t),e.slice(n,n+t.length)==t},jn.subtract=kl,jn.sum=function(e){return e&&e.length?qt(e,nl):0},jn.sumBy=function(e,t){return e&&e.length?qt(e,ii(t,2)):0},jn.template=function(e,t,n){var r=jn.templateSettings;n&&gi(e,t,n)&&(t=a),e=yu(e),t=_u({},t,r,Ga);var i,o,u=_u({},t.imports,r.imports,Ga),l=Pu(u),s=Kt(u,l),c=0,f=t.interpolate||be,p=\"__p += '\",d=Ee((t.escape||be).source+\"|\"+f.source+\"|\"+(f===Z?fe:be).source+\"|\"+(t.evaluate||be).source+\"|$\",\"g\"),h=\"//# sourceURL=\"+(Oe.call(t,\"sourceURL\")?(t.sourceURL+\"\").replace(/\\s/g,\" \"):\"lodash.templateSources[\"+ ++rt+\"]\")+\"\\n\";e.replace(d,(function(t,n,r,a,u,l){return r||(r=a),p+=e.slice(c,l).replace(_e,nn),n&&(i=!0,p+=\"' +\\n__e(\"+n+\") +\\n'\"),u&&(o=!0,p+=\"';\\n\"+u+\";\\n__p += '\"),r&&(p+=\"' +\\n((__t = (\"+r+\")) == null ? '' : __t) +\\n'\"),c=l+t.length,t})),p+=\"';\\n\";var v=Oe.call(t,\"variable\")&&t.variable;if(v){if(se.test(v))throw new we(\"Invalid `variable` option passed into `_.template`\")}else p=\"with (obj) {\\n\"+p+\"\\n}\\n\";p=(o?p.replace(B,\"\"):p).replace(W,\"$1\").replace(V,\"$1;\"),p=\"function(\"+(v||\"obj\")+\") {\\n\"+(v?\"\":\"obj || (obj = {});\\n\")+\"var __t, __p = ''\"+(i?\", __e = _.escape\":\"\")+(o?\", __j = Array.prototype.join;\\nfunction print() { __p += __j.call(arguments, '') }\\n\":\";\\n\")+p+\"return __p\\n}\";var g=Zu((function(){return xe(l,h+\"return \"+p).apply(a,s)}));if(g.source=p,Go(g))throw g;return g},jn.times=function(e,t){if((e=du(e))<1||e>f)return[];var n=d,r=yn(e,d);t=ii(t),e-=d;for(var a=Qt(r,t);++n<e;)t(n);return a},jn.toFinite=pu,jn.toInteger=du,jn.toLength=hu,jn.toLower=function(e){return yu(e).toLowerCase()},jn.toNumber=vu,jn.toSafeInteger=function(e){return e?ir(du(e),-9007199254740991,f):0===e?e:0},jn.toString=yu,jn.toUpper=function(e){return yu(e).toUpperCase()},jn.trim=function(e,t,n){if((e=yu(e))&&(n||t===a))return Yt(e);if(!e||!(t=oa(t)))return e;var r=fn(e),i=fn(t);return ma(r,Xt(r,i),Jt(r,i)+1).join(\"\")},jn.trimEnd=function(e,t,n){if((e=yu(e))&&(n||t===a))return e.slice(0,pn(e)+1);if(!e||!(t=oa(t)))return e;var r=fn(e);return ma(r,0,Jt(r,fn(t))+1).join(\"\")},jn.trimStart=function(e,t,n){if((e=yu(e))&&(n||t===a))return e.replace(re,\"\");if(!e||!(t=oa(t)))return e;var r=fn(e);return ma(r,Xt(r,fn(t))).join(\"\")},jn.truncate=function(e,t){var n=30,r=\"...\";if(Jo(t)){var i=\"separator\"in t?t.separator:i;n=\"length\"in t?du(t.length):n,r=\"omission\"in t?oa(t.omission):r}var o=(e=yu(e)).length;if(rn(e)){var u=fn(e);o=u.length}if(n>=o)return e;var l=n-cn(r);if(l<1)return r;var s=u?ma(u,0,l).join(\"\"):e.slice(0,l);if(i===a)return s+r;if(u&&(l+=s.length-l),au(i)){if(e.slice(l).search(i)){var c,f=s;for(i.global||(i=Ee(i.source,yu(pe.exec(i))+\"g\")),i.lastIndex=0;c=i.exec(f);)var p=c.index;s=s.slice(0,p===a?l:p)}}else if(e.indexOf(oa(i),l)!=l){var d=s.lastIndexOf(i);d>-1&&(s=s.slice(0,d))}return s+r},jn.unescape=function(e){return(e=yu(e))&&Q.test(e)?e.replace(H,dn):e},jn.uniqueId=function(e){var t=++Ae;return yu(e)+t},jn.upperCase=Yu,jn.upperFirst=Gu,jn.each=yo,jn.eachRight=mo,jn.first=Vi,ol(jn,(gl={},mr(jn,(function(e,t){Oe.call(jn.prototype,t)||(gl[t]=e)})),gl),{chain:!1}),jn.VERSION=\"4.17.21\",Et([\"bind\",\"bindKey\",\"curry\",\"curryRight\",\"partial\",\"partialRight\"],(function(e){jn[e].placeholder=jn})),Et([\"drop\",\"take\"],(function(e,t){Bn.prototype[e]=function(n){n=n===a?1:gn(du(n),0);var r=this.__filtered__&&!t?new Bn(this):this.clone();return r.__filtered__?r.__takeCount__=yn(n,r.__takeCount__):r.__views__.push({size:yn(n,d),type:e+(r.__dir__<0?\"Right\":\"\")}),r},Bn.prototype[e+\"Right\"]=function(t){return this.reverse()[e](t).reverse()}})),Et([\"filter\",\"map\",\"takeWhile\"],(function(e,t){var n=t+1,r=1==n||3==n;Bn.prototype[e]=function(e){var t=this.clone();return t.__iteratees__.push({iteratee:ii(e,3),type:n}),t.__filtered__=t.__filtered__||r,t}})),Et([\"head\",\"last\"],(function(e,t){var n=\"take\"+(t?\"Right\":\"\");Bn.prototype[e]=function(){return this[n](1).value()[0]}})),Et([\"initial\",\"tail\"],(function(e,t){var n=\"drop\"+(t?\"\":\"Right\");Bn.prototype[e]=function(){return this.__filtered__?new Bn(this):this[n](1)}})),Bn.prototype.compact=function(){return this.filter(nl)},Bn.prototype.find=function(e){return this.filter(e).head()},Bn.prototype.findLast=function(e){return this.reverse().find(e)},Bn.prototype.invokeMap=Qr((function(e,t){return\"function\"==typeof e?new Bn(this):this.map((function(n){return Mr(n,e,t)}))})),Bn.prototype.reject=function(e){return this.filter(Ao(ii(e)))},Bn.prototype.slice=function(e,t){e=du(e);var n=this;return n.__filtered__&&(e>0||t<0)?new Bn(n):(e<0?n=n.takeRight(-e):e&&(n=n.drop(e)),t!==a&&(n=(t=du(t))<0?n.dropRight(-t):n.take(t-e)),n)},Bn.prototype.takeRightWhile=function(e){return this.reverse().takeWhile(e).reverse()},Bn.prototype.toArray=function(){return this.take(d)},mr(Bn.prototype,(function(e,t){var n=/^(?:filter|find|map|reject)|While$/.test(t),r=/^(?:head|last)$/.test(t),i=jn[r?\"take\"+(\"last\"==t?\"Right\":\"\"):t],o=r||/^find/.test(t);i&&(jn.prototype[t]=function(){var t=this.__wrapped__,u=r?[1]:arguments,l=t instanceof Bn,s=u[0],c=l||Wo(t),f=function(e){var t=i.apply(jn,Lt([e],u));return r&&p?t[0]:t};c&&n&&\"function\"==typeof s&&1!=s.length&&(l=c=!1);var p=this.__chain__,d=!!this.__actions__.length,h=o&&!p,v=l&&!d;if(!o&&c){t=v?t:new Bn(this);var g=e.apply(t,u);return g.__actions__.push({func:fo,args:[f],thisArg:a}),new $n(g,p)}return h&&v?e.apply(this,u):(g=this.thru(f),h?r?g.value()[0]:g.value():g)})})),Et([\"pop\",\"push\",\"shift\",\"sort\",\"splice\",\"unshift\"],(function(e){var t=Me[e],n=/^(?:push|sort|unshift)$/.test(e)?\"tap\":\"thru\",r=/^(?:pop|shift)$/.test(e);jn.prototype[e]=function(){var e=arguments;if(r&&!this.__chain__){var a=this.value();return t.apply(Wo(a)?a:[],e)}return this[n]((function(n){return t.apply(Wo(n)?n:[],e)}))}})),mr(Bn.prototype,(function(e,t){var n=jn[t];if(n){var r=n.name+\"\";Oe.call(Nn,r)||(Nn[r]=[]),Nn[r].push({name:t,func:n})}})),Nn[Ra(a,2).name]=[{name:\"wrapper\",func:a}],Bn.prototype.clone=function(){var e=new Bn(this.__wrapped__);return e.__actions__=Ca(this.__actions__),e.__dir__=this.__dir__,e.__filtered__=this.__filtered__,e.__iteratees__=Ca(this.__iteratees__),e.__takeCount__=this.__takeCount__,e.__views__=Ca(this.__views__),e},Bn.prototype.reverse=function(){if(this.__filtered__){var e=new Bn(this);e.__dir__=-1,e.__filtered__=!0}else(e=this.clone()).__dir__*=-1;return e},Bn.prototype.value=function(){var e=this.__wrapped__.value(),t=this.__dir__,n=Wo(e),r=t<0,a=n?e.length:0,i=function(e,t,n){for(var r=-1,a=n.length;++r<a;){var i=n[r],o=i.size;switch(i.type){case\"drop\":e+=o;break;case\"dropRight\":t-=o;break;case\"take\":t=yn(t,e+o);break;case\"takeRight\":e=gn(e,t-o)}}return{start:e,end:t}}(0,a,this.__views__),o=i.start,u=i.end,l=u-o,s=r?u:o-1,c=this.__iteratees__,f=c.length,p=0,d=yn(l,this.__takeCount__);if(!n||!r&&a==l&&d==l)return fa(e,this.__actions__);var h=[];e:for(;l--&&p<d;){for(var v=-1,g=e[s+=t];++v<f;){var y=c[v],m=y.iteratee,b=y.type,_=m(g);if(2==b)g=_;else if(!_){if(1==b)continue e;break e}}h[p++]=g}return h},jn.prototype.at=po,jn.prototype.chain=function(){return co(this)},jn.prototype.commit=function(){return new $n(this.value(),this.__chain__)},jn.prototype.next=function(){this.__values__===a&&(this.__values__=fu(this.value()));var e=this.__index__>=this.__values__.length;return{done:e,value:e?a:this.__values__[this.__index__++]}},jn.prototype.plant=function(e){for(var t,n=this;n instanceof In;){var r=Ri(n);r.__index__=0,r.__values__=a,t?i.__wrapped__=r:t=r;var i=r;n=n.__wrapped__}return i.__wrapped__=e,t},jn.prototype.reverse=function(){var e=this.__wrapped__;if(e instanceof Bn){var t=e;return this.__actions__.length&&(t=new Bn(this)),(t=t.reverse()).__actions__.push({func:fo,args:[Xi],thisArg:a}),new $n(t,this.__chain__)}return this.thru(Xi)},jn.prototype.toJSON=jn.prototype.valueOf=jn.prototype.value=function(){return fa(this.__wrapped__,this.__actions__)},jn.prototype.first=jn.prototype.head,Ge&&(jn.prototype[Ge]=function(){return this}),jn}();ft._=hn,(r=function(){return hn}.call(t,n,t,e))===a||(e.exports=r)}.call(this)},448:(e,t,n)=>{\"use strict\";var r=n(294),a=n(840);function i(e){for(var t=\"https://reactjs.org/docs/error-decoder.html?invariant=\"+e,n=1;n<arguments.length;n++)t+=\"&args[]=\"+encodeURIComponent(arguments[n]);return\"Minified React error #\"+e+\"; visit \"+t+\" for the full message or use the non-minified dev environment for full errors and additional helpful warnings.\"}var o=new Set,u={};function l(e,t){s(e,t),s(e+\"Capture\",t)}function s(e,t){for(u[e]=t,e=0;e<t.length;e++)o.add(t[e])}var c=!(\"undefined\"==typeof window||void 0===window.document||void 0===window.document.createElement),f=Object.prototype.hasOwnProperty,p=/^[:A-Z_a-z\\u00C0-\\u00D6\\u00D8-\\u00F6\\u00F8-\\u02FF\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF\\uF900-\\uFDCF\\uFDF0-\\uFFFD][:A-Z_a-z\\u00C0-\\u00D6\\u00D8-\\u00F6\\u00F8-\\u02FF\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF\\uF900-\\uFDCF\\uFDF0-\\uFFFD\\-.0-9\\u00B7\\u0300-\\u036F\\u203F-\\u2040]*$/,d={},h={};function v(e,t,n,r,a,i,o){this.acceptsBooleans=2===t||3===t||4===t,this.attributeName=r,this.attributeNamespace=a,this.mustUseProperty=n,this.propertyName=e,this.type=t,this.sanitizeURL=i,this.removeEmptyString=o}var g={};\"children dangerouslySetInnerHTML defaultValue defaultChecked innerHTML suppressContentEditableWarning suppressHydrationWarning style\".split(\" \").forEach((function(e){g[e]=new v(e,0,!1,e,null,!1,!1)})),[[\"acceptCharset\",\"accept-charset\"],[\"className\",\"class\"],[\"htmlFor\",\"for\"],[\"httpEquiv\",\"http-equiv\"]].forEach((function(e){var t=e[0];g[t]=new v(t,1,!1,e[1],null,!1,!1)})),[\"contentEditable\",\"draggable\",\"spellCheck\",\"value\"].forEach((function(e){g[e]=new v(e,2,!1,e.toLowerCase(),null,!1,!1)})),[\"autoReverse\",\"externalResourcesRequired\",\"focusable\",\"preserveAlpha\"].forEach((function(e){g[e]=new v(e,2,!1,e,null,!1,!1)})),\"allowFullScreen async autoFocus autoPlay controls default defer disabled disablePictureInPicture disableRemotePlayback formNoValidate hidden loop noModule noValidate open playsInline readOnly required reversed scoped seamless itemScope\".split(\" \").forEach((function(e){g[e]=new v(e,3,!1,e.toLowerCase(),null,!1,!1)})),[\"checked\",\"multiple\",\"muted\",\"selected\"].forEach((function(e){g[e]=new v(e,3,!0,e,null,!1,!1)})),[\"capture\",\"download\"].forEach((function(e){g[e]=new v(e,4,!1,e,null,!1,!1)})),[\"cols\",\"rows\",\"size\",\"span\"].forEach((function(e){g[e]=new v(e,6,!1,e,null,!1,!1)})),[\"rowSpan\",\"start\"].forEach((function(e){g[e]=new v(e,5,!1,e.toLowerCase(),null,!1,!1)}));var y=/[\\-:]([a-z])/g;function m(e){return e[1].toUpperCase()}function b(e,t,n,r){var a=g.hasOwnProperty(t)?g[t]:null;(null!==a?0!==a.type:r||!(2<t.length)||\"o\"!==t[0]&&\"O\"!==t[0]||\"n\"!==t[1]&&\"N\"!==t[1])&&(function(e,t,n,r){if(null==t||function(e,t,n,r){if(null!==n&&0===n.type)return!1;switch(typeof t){case\"function\":case\"symbol\":return!0;case\"boolean\":return!r&&(null!==n?!n.acceptsBooleans:\"data-\"!==(e=e.toLowerCase().slice(0,5))&&\"aria-\"!==e);default:return!1}}(e,t,n,r))return!0;if(r)return!1;if(null!==n)switch(n.type){case 3:return!t;case 4:return!1===t;case 5:return isNaN(t);case 6:return isNaN(t)||1>t}return!1}(t,n,a,r)&&(n=null),r||null===a?function(e){return!!f.call(h,e)||!f.call(d,e)&&(p.test(e)?h[e]=!0:(d[e]=!0,!1))}(t)&&(null===n?e.removeAttribute(t):e.setAttribute(t,\"\"+n)):a.mustUseProperty?e[a.propertyName]=null===n?3!==a.type&&\"\":n:(t=a.attributeName,r=a.attributeNamespace,null===n?e.removeAttribute(t):(n=3===(a=a.type)||4===a&&!0===n?\"\":\"\"+n,r?e.setAttributeNS(r,t,n):e.setAttribute(t,n))))}\"accent-height alignment-baseline arabic-form baseline-shift cap-height clip-path clip-rule color-interpolation color-interpolation-filters color-profile color-rendering dominant-baseline enable-background fill-opacity fill-rule flood-color flood-opacity font-family font-size font-size-adjust font-stretch font-style font-variant font-weight glyph-name glyph-orientation-horizontal glyph-orientation-vertical horiz-adv-x horiz-origin-x image-rendering letter-spacing lighting-color marker-end marker-mid marker-start overline-position overline-thickness paint-order panose-1 pointer-events rendering-intent shape-rendering stop-color stop-opacity strikethrough-position strikethrough-thickness stroke-dasharray stroke-dashoffset stroke-linecap stroke-linejoin stroke-miterlimit stroke-opacity stroke-width text-anchor text-decoration text-rendering underline-position underline-thickness unicode-bidi unicode-range units-per-em v-alphabetic v-hanging v-ideographic v-mathematical vector-effect vert-adv-y vert-origin-x vert-origin-y word-spacing writing-mode xmlns:xlink x-height\".split(\" \").forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,null,!1,!1)})),\"xlink:actuate xlink:arcrole xlink:role xlink:show xlink:title xlink:type\".split(\" \").forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,\"http://www.w3.org/1999/xlink\",!1,!1)})),[\"xml:base\",\"xml:lang\",\"xml:space\"].forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,\"http://www.w3.org/XML/1998/namespace\",!1,!1)})),[\"tabIndex\",\"crossOrigin\"].forEach((function(e){g[e]=new v(e,1,!1,e.toLowerCase(),null,!1,!1)})),g.xlinkHref=new v(\"xlinkHref\",1,!1,\"xlink:href\",\"http://www.w3.org/1999/xlink\",!0,!1),[\"src\",\"href\",\"action\",\"formAction\"].forEach((function(e){g[e]=new v(e,1,!1,e.toLowerCase(),null,!0,!0)}));var _=r.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED,w=Symbol.for(\"react.element\"),x=Symbol.for(\"react.portal\"),k=Symbol.for(\"react.fragment\"),S=Symbol.for(\"react.strict_mode\"),E=Symbol.for(\"react.profiler\"),C=Symbol.for(\"react.provider\"),T=Symbol.for(\"react.context\"),M=Symbol.for(\"react.forward_ref\"),N=Symbol.for(\"react.suspense\"),P=Symbol.for(\"react.suspense_list\"),z=Symbol.for(\"react.memo\"),L=Symbol.for(\"react.lazy\");Symbol.for(\"react.scope\"),Symbol.for(\"react.debug_trace_mode\");var O=Symbol.for(\"react.offscreen\");Symbol.for(\"react.legacy_hidden\"),Symbol.for(\"react.cache\"),Symbol.for(\"react.tracing_marker\");var A=Symbol.iterator;function F(e){return null===e||\"object\"!=typeof e?null:\"function\"==typeof(e=A&&e[A]||e[\"@@iterator\"])?e:null}var D,R=Object.assign;function j(e){if(void 0===D)try{throw Error()}catch(e){var t=e.stack.trim().match(/\\n( *(at )?)/);D=t&&t[1]||\"\"}return\"\\n\"+D+e}var U=!1;function I(e,t){if(!e||U)return\"\";U=!0;var n=Error.prepareStackTrace;Error.prepareStackTrace=void 0;try{if(t)if(t=function(){throw Error()},Object.defineProperty(t.prototype,\"props\",{set:function(){throw Error()}}),\"object\"==typeof Reflect&&Reflect.construct){try{Reflect.construct(t,[])}catch(e){var r=e}Reflect.construct(e,[],t)}else{try{t.call()}catch(e){r=e}e.call(t.prototype)}else{try{throw Error()}catch(e){r=e}e()}}catch(t){if(t&&r&&\"string\"==typeof t.stack){for(var a=t.stack.split(\"\\n\"),i=r.stack.split(\"\\n\"),o=a.length-1,u=i.length-1;1<=o&&0<=u&&a[o]!==i[u];)u--;for(;1<=o&&0<=u;o--,u--)if(a[o]!==i[u]){if(1!==o||1!==u)do{if(o--,0>--u||a[o]!==i[u]){var l=\"\\n\"+a[o].replace(\" at new \",\" at \");return e.displayName&&l.includes(\"<anonymous>\")&&(l=l.replace(\"<anonymous>\",e.displayName)),l}}while(1<=o&&0<=u);break}}}finally{U=!1,Error.prepareStackTrace=n}return(e=e?e.displayName||e.name:\"\")?j(e):\"\"}function $(e){switch(e.tag){case 5:return j(e.type);case 16:return j(\"Lazy\");case 13:return j(\"Suspense\");case 19:return j(\"SuspenseList\");case 0:case 2:case 15:return I(e.type,!1);case 11:return I(e.type.render,!1);case 1:return I(e.type,!0);default:return\"\"}}function B(e){if(null==e)return null;if(\"function\"==typeof e)return e.displayName||e.name||null;if(\"string\"==typeof e)return e;switch(e){case k:return\"Fragment\";case x:return\"Portal\";case E:return\"Profiler\";case S:return\"StrictMode\";case N:return\"Suspense\";case P:return\"SuspenseList\"}if(\"object\"==typeof e)switch(e.$$typeof){case T:return(e.displayName||\"Context\")+\".Consumer\";case C:return(e._context.displayName||\"Context\")+\".Provider\";case M:var t=e.render;return(e=e.displayName)||(e=\"\"!==(e=t.displayName||t.name||\"\")?\"ForwardRef(\"+e+\")\":\"ForwardRef\"),e;case z:return null!==(t=e.displayName||null)?t:B(e.type)||\"Memo\";case L:t=e._payload,e=e._init;try{return B(e(t))}catch(e){}}return null}function W(e){var t=e.type;switch(e.tag){case 24:return\"Cache\";case 9:return(t.displayName||\"Context\")+\".Consumer\";case 10:return(t._context.displayName||\"Context\")+\".Provider\";case 18:return\"DehydratedFragment\";case 11:return e=(e=t.render).displayName||e.name||\"\",t.displayName||(\"\"!==e?\"ForwardRef(\"+e+\")\":\"ForwardRef\");case 7:return\"Fragment\";case 5:return t;case 4:return\"Portal\";case 3:return\"Root\";case 6:return\"Text\";case 16:return B(t);case 8:return t===S?\"StrictMode\":\"Mode\";case 22:return\"Offscreen\";case 12:return\"Profiler\";case 21:return\"Scope\";case 13:return\"Suspense\";case 19:return\"SuspenseList\";case 25:return\"TracingMarker\";case 1:case 0:case 17:case 2:case 14:case 15:if(\"function\"==typeof t)return t.displayName||t.name||null;if(\"string\"==typeof t)return t}return null}function V(e){switch(typeof e){case\"boolean\":case\"number\":case\"string\":case\"undefined\":case\"object\":return e;default:return\"\"}}function H(e){var t=e.type;return(e=e.nodeName)&&\"input\"===e.toLowerCase()&&(\"checkbox\"===t||\"radio\"===t)}function q(e){e._valueTracker||(e._valueTracker=function(e){var t=H(e)?\"checked\":\"value\",n=Object.getOwnPropertyDescriptor(e.constructor.prototype,t),r=\"\"+e[t];if(!e.hasOwnProperty(t)&&void 0!==n&&\"function\"==typeof n.get&&\"function\"==typeof n.set){var a=n.get,i=n.set;return Object.defineProperty(e,t,{configurable:!0,get:function(){return a.call(this)},set:function(e){r=\"\"+e,i.call(this,e)}}),Object.defineProperty(e,t,{enumerable:n.enumerable}),{getValue:function(){return r},setValue:function(e){r=\"\"+e},stopTracking:function(){e._valueTracker=null,delete e[t]}}}}(e))}function Q(e){if(!e)return!1;var t=e._valueTracker;if(!t)return!0;var n=t.getValue(),r=\"\";return e&&(r=H(e)?e.checked?\"true\":\"false\":e.value),(e=r)!==n&&(t.setValue(e),!0)}function Y(e){if(void 0===(e=e||(\"undefined\"!=typeof document?document:void 0)))return null;try{return e.activeElement||e.body}catch(t){return e.body}}function G(e,t){var n=t.checked;return R({},t,{defaultChecked:void 0,defaultValue:void 0,value:void 0,checked:null!=n?n:e._wrapperState.initialChecked})}function K(e,t){var n=null==t.defaultValue?\"\":t.defaultValue,r=null!=t.checked?t.checked:t.defaultChecked;n=V(null!=t.value?t.value:n),e._wrapperState={initialChecked:r,initialValue:n,controlled:\"checkbox\"===t.type||\"radio\"===t.type?null!=t.checked:null!=t.value}}function Z(e,t){null!=(t=t.checked)&&b(e,\"checked\",t,!1)}function X(e,t){Z(e,t);var n=V(t.value),r=t.type;if(null!=n)\"number\"===r?(0===n&&\"\"===e.value||e.value!=n)&&(e.value=\"\"+n):e.value!==\"\"+n&&(e.value=\"\"+n);else if(\"submit\"===r||\"reset\"===r)return void e.removeAttribute(\"value\");t.hasOwnProperty(\"value\")?ee(e,t.type,n):t.hasOwnProperty(\"defaultValue\")&&ee(e,t.type,V(t.defaultValue)),null==t.checked&&null!=t.defaultChecked&&(e.defaultChecked=!!t.defaultChecked)}function J(e,t,n){if(t.hasOwnProperty(\"value\")||t.hasOwnProperty(\"defaultValue\")){var r=t.type;if(!(\"submit\"!==r&&\"reset\"!==r||void 0!==t.value&&null!==t.value))return;t=\"\"+e._wrapperState.initialValue,n||t===e.value||(e.value=t),e.defaultValue=t}\"\"!==(n=e.name)&&(e.name=\"\"),e.defaultChecked=!!e._wrapperState.initialChecked,\"\"!==n&&(e.name=n)}function ee(e,t,n){\"number\"===t&&Y(e.ownerDocument)===e||(null==n?e.defaultValue=\"\"+e._wrapperState.initialValue:e.defaultValue!==\"\"+n&&(e.defaultValue=\"\"+n))}var te=Array.isArray;function ne(e,t,n,r){if(e=e.options,t){t={};for(var a=0;a<n.length;a++)t[\"$\"+n[a]]=!0;for(n=0;n<e.length;n++)a=t.hasOwnProperty(\"$\"+e[n].value),e[n].selected!==a&&(e[n].selected=a),a&&r&&(e[n].defaultSelected=!0)}else{for(n=\"\"+V(n),t=null,a=0;a<e.length;a++){if(e[a].value===n)return e[a].selected=!0,void(r&&(e[a].defaultSelected=!0));null!==t||e[a].disabled||(t=e[a])}null!==t&&(t.selected=!0)}}function re(e,t){if(null!=t.dangerouslySetInnerHTML)throw Error(i(91));return R({},t,{value:void 0,defaultValue:void 0,children:\"\"+e._wrapperState.initialValue})}function ae(e,t){var n=t.value;if(null==n){if(n=t.children,t=t.defaultValue,null!=n){if(null!=t)throw Error(i(92));if(te(n)){if(1<n.length)throw Error(i(93));n=n[0]}t=n}null==t&&(t=\"\"),n=t}e._wrapperState={initialValue:V(n)}}function ie(e,t){var n=V(t.value),r=V(t.defaultValue);null!=n&&((n=\"\"+n)!==e.value&&(e.value=n),null==t.defaultValue&&e.defaultValue!==n&&(e.defaultValue=n)),null!=r&&(e.defaultValue=\"\"+r)}function oe(e){var t=e.textContent;t===e._wrapperState.initialValue&&\"\"!==t&&null!==t&&(e.value=t)}function ue(e){switch(e){case\"svg\":return\"http://www.w3.org/2000/svg\";case\"math\":return\"http://www.w3.org/1998/Math/MathML\";default:return\"http://www.w3.org/1999/xhtml\"}}function le(e,t){return null==e||\"http://www.w3.org/1999/xhtml\"===e?ue(t):\"http://www.w3.org/2000/svg\"===e&&\"foreignObject\"===t?\"http://www.w3.org/1999/xhtml\":e}var se,ce,fe=(ce=function(e,t){if(\"http://www.w3.org/2000/svg\"!==e.namespaceURI||\"innerHTML\"in e)e.innerHTML=t;else{for((se=se||document.createElement(\"div\")).innerHTML=\"<svg>\"+t.valueOf().toString()+\"</svg>\",t=se.firstChild;e.firstChild;)e.removeChild(e.firstChild);for(;t.firstChild;)e.appendChild(t.firstChild)}},\"undefined\"!=typeof MSApp&&MSApp.execUnsafeLocalFunction?function(e,t,n,r){MSApp.execUnsafeLocalFunction((function(){return ce(e,t)}))}:ce);function pe(e,t){if(t){var n=e.firstChild;if(n&&n===e.lastChild&&3===n.nodeType)return void(n.nodeValue=t)}e.textContent=t}var de={animationIterationCount:!0,aspectRatio:!0,borderImageOutset:!0,borderImageSlice:!0,borderImageWidth:!0,boxFlex:!0,boxFlexGroup:!0,boxOrdinalGroup:!0,columnCount:!0,columns:!0,flex:!0,flexGrow:!0,flexPositive:!0,flexShrink:!0,flexNegative:!0,flexOrder:!0,gridArea:!0,gridRow:!0,gridRowEnd:!0,gridRowSpan:!0,gridRowStart:!0,gridColumn:!0,gridColumnEnd:!0,gridColumnSpan:!0,gridColumnStart:!0,fontWeight:!0,lineClamp:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,tabSize:!0,widows:!0,zIndex:!0,zoom:!0,fillOpacity:!0,floodOpacity:!0,stopOpacity:!0,strokeDasharray:!0,strokeDashoffset:!0,strokeMiterlimit:!0,strokeOpacity:!0,strokeWidth:!0},he=[\"Webkit\",\"ms\",\"Moz\",\"O\"];function ve(e,t,n){return null==t||\"boolean\"==typeof t||\"\"===t?\"\":n||\"number\"!=typeof t||0===t||de.hasOwnProperty(e)&&de[e]?(\"\"+t).trim():t+\"px\"}function ge(e,t){for(var n in e=e.style,t)if(t.hasOwnProperty(n)){var r=0===n.indexOf(\"--\"),a=ve(n,t[n],r);\"float\"===n&&(n=\"cssFloat\"),r?e.setProperty(n,a):e[n]=a}}Object.keys(de).forEach((function(e){he.forEach((function(t){t=t+e.charAt(0).toUpperCase()+e.substring(1),de[t]=de[e]}))}));var ye=R({menuitem:!0},{area:!0,base:!0,br:!0,col:!0,embed:!0,hr:!0,img:!0,input:!0,keygen:!0,link:!0,meta:!0,param:!0,source:!0,track:!0,wbr:!0});function me(e,t){if(t){if(ye[e]&&(null!=t.children||null!=t.dangerouslySetInnerHTML))throw Error(i(137,e));if(null!=t.dangerouslySetInnerHTML){if(null!=t.children)throw Error(i(60));if(\"object\"!=typeof t.dangerouslySetInnerHTML||!(\"__html\"in t.dangerouslySetInnerHTML))throw Error(i(61))}if(null!=t.style&&\"object\"!=typeof t.style)throw Error(i(62))}}function be(e,t){if(-1===e.indexOf(\"-\"))return\"string\"==typeof t.is;switch(e){case\"annotation-xml\":case\"color-profile\":case\"font-face\":case\"font-face-src\":case\"font-face-uri\":case\"font-face-format\":case\"font-face-name\":case\"missing-glyph\":return!1;default:return!0}}var _e=null;function we(e){return(e=e.target||e.srcElement||window).correspondingUseElement&&(e=e.correspondingUseElement),3===e.nodeType?e.parentNode:e}var xe=null,ke=null,Se=null;function Ee(e){if(e=ba(e)){if(\"function\"!=typeof xe)throw Error(i(280));var t=e.stateNode;t&&(t=wa(t),xe(e.stateNode,e.type,t))}}function Ce(e){ke?Se?Se.push(e):Se=[e]:ke=e}function Te(){if(ke){var e=ke,t=Se;if(Se=ke=null,Ee(e),t)for(e=0;e<t.length;e++)Ee(t[e])}}function Me(e,t){return e(t)}function Ne(){}var Pe=!1;function ze(e,t,n){if(Pe)return e(t,n);Pe=!0;try{return Me(e,t,n)}finally{Pe=!1,(null!==ke||null!==Se)&&(Ne(),Te())}}function Le(e,t){var n=e.stateNode;if(null===n)return null;var r=wa(n);if(null===r)return null;n=r[t];e:switch(t){case\"onClick\":case\"onClickCapture\":case\"onDoubleClick\":case\"onDoubleClickCapture\":case\"onMouseDown\":case\"onMouseDownCapture\":case\"onMouseMove\":case\"onMouseMoveCapture\":case\"onMouseUp\":case\"onMouseUpCapture\":case\"onMouseEnter\":(r=!r.disabled)||(r=!(\"button\"===(e=e.type)||\"input\"===e||\"select\"===e||\"textarea\"===e)),e=!r;break e;default:e=!1}if(e)return null;if(n&&\"function\"!=typeof n)throw Error(i(231,t,typeof n));return n}var Oe=!1;if(c)try{var Ae={};Object.defineProperty(Ae,\"passive\",{get:function(){Oe=!0}}),window.addEventListener(\"test\",Ae,Ae),window.removeEventListener(\"test\",Ae,Ae)}catch(ce){Oe=!1}function Fe(e,t,n,r,a,i,o,u,l){var s=Array.prototype.slice.call(arguments,3);try{t.apply(n,s)}catch(e){this.onError(e)}}var De=!1,Re=null,je=!1,Ue=null,Ie={onError:function(e){De=!0,Re=e}};function $e(e,t,n,r,a,i,o,u,l){De=!1,Re=null,Fe.apply(Ie,arguments)}function Be(e){var t=e,n=e;if(e.alternate)for(;t.return;)t=t.return;else{e=t;do{0!=(4098&(t=e).flags)&&(n=t.return),e=t.return}while(e)}return 3===t.tag?n:null}function We(e){if(13===e.tag){var t=e.memoizedState;if(null===t&&null!==(e=e.alternate)&&(t=e.memoizedState),null!==t)return t.dehydrated}return null}function Ve(e){if(Be(e)!==e)throw Error(i(188))}function He(e){return null!==(e=function(e){var t=e.alternate;if(!t){if(null===(t=Be(e)))throw Error(i(188));return t!==e?null:e}for(var n=e,r=t;;){var a=n.return;if(null===a)break;var o=a.alternate;if(null===o){if(null!==(r=a.return)){n=r;continue}break}if(a.child===o.child){for(o=a.child;o;){if(o===n)return Ve(a),e;if(o===r)return Ve(a),t;o=o.sibling}throw Error(i(188))}if(n.return!==r.return)n=a,r=o;else{for(var u=!1,l=a.child;l;){if(l===n){u=!0,n=a,r=o;break}if(l===r){u=!0,r=a,n=o;break}l=l.sibling}if(!u){for(l=o.child;l;){if(l===n){u=!0,n=o,r=a;break}if(l===r){u=!0,r=o,n=a;break}l=l.sibling}if(!u)throw Error(i(189))}}if(n.alternate!==r)throw Error(i(190))}if(3!==n.tag)throw Error(i(188));return n.stateNode.current===n?e:t}(e))?qe(e):null}function qe(e){if(5===e.tag||6===e.tag)return e;for(e=e.child;null!==e;){var t=qe(e);if(null!==t)return t;e=e.sibling}return null}var Qe=a.unstable_scheduleCallback,Ye=a.unstable_cancelCallback,Ge=a.unstable_shouldYield,Ke=a.unstable_requestPaint,Ze=a.unstable_now,Xe=a.unstable_getCurrentPriorityLevel,Je=a.unstable_ImmediatePriority,et=a.unstable_UserBlockingPriority,tt=a.unstable_NormalPriority,nt=a.unstable_LowPriority,rt=a.unstable_IdlePriority,at=null,it=null,ot=Math.clz32?Math.clz32:function(e){return 0===(e>>>=0)?32:31-(ut(e)/lt|0)|0},ut=Math.log,lt=Math.LN2,st=64,ct=4194304;function ft(e){switch(e&-e){case 1:return 1;case 2:return 2;case 4:return 4;case 8:return 8;case 16:return 16;case 32:return 32;case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return 4194240&e;case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:return 130023424&e;case 134217728:return 134217728;case 268435456:return 268435456;case 536870912:return 536870912;case 1073741824:return 1073741824;default:return e}}function pt(e,t){var n=e.pendingLanes;if(0===n)return 0;var r=0,a=e.suspendedLanes,i=e.pingedLanes,o=268435455&n;if(0!==o){var u=o&~a;0!==u?r=ft(u):0!=(i&=o)&&(r=ft(i))}else 0!=(o=n&~a)?r=ft(o):0!==i&&(r=ft(i));if(0===r)return 0;if(0!==t&&t!==r&&0==(t&a)&&((a=r&-r)>=(i=t&-t)||16===a&&0!=(4194240&i)))return t;if(0!=(4&r)&&(r|=16&n),0!==(t=e.entangledLanes))for(e=e.entanglements,t&=r;0<t;)a=1<<(n=31-ot(t)),r|=e[n],t&=~a;return r}function dt(e,t){switch(e){case 1:case 2:case 4:return t+250;case 8:case 16:case 32:case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return t+5e3;default:return-1}}function ht(e){return 0!=(e=-1073741825&e.pendingLanes)?e:1073741824&e?1073741824:0}function vt(){var e=st;return 0==(4194240&(st<<=1))&&(st=64),e}function gt(e){for(var t=[],n=0;31>n;n++)t.push(e);return t}function yt(e,t,n){e.pendingLanes|=t,536870912!==t&&(e.suspendedLanes=0,e.pingedLanes=0),(e=e.eventTimes)[t=31-ot(t)]=n}function mt(e,t){var n=e.entangledLanes|=t;for(e=e.entanglements;n;){var r=31-ot(n),a=1<<r;a&t|e[r]&t&&(e[r]|=t),n&=~a}}var bt=0;function _t(e){return 1<(e&=-e)?4<e?0!=(268435455&e)?16:536870912:4:1}var wt,xt,kt,St,Et,Ct=!1,Tt=[],Mt=null,Nt=null,Pt=null,zt=new Map,Lt=new Map,Ot=[],At=\"mousedown mouseup touchcancel touchend touchstart auxclick dblclick pointercancel pointerdown pointerup dragend dragstart drop compositionend compositionstart keydown keypress keyup input textInput copy cut paste click change contextmenu reset submit\".split(\" \");function Ft(e,t){switch(e){case\"focusin\":case\"focusout\":Mt=null;break;case\"dragenter\":case\"dragleave\":Nt=null;break;case\"mouseover\":case\"mouseout\":Pt=null;break;case\"pointerover\":case\"pointerout\":zt.delete(t.pointerId);break;case\"gotpointercapture\":case\"lostpointercapture\":Lt.delete(t.pointerId)}}function Dt(e,t,n,r,a,i){return null===e||e.nativeEvent!==i?(e={blockedOn:t,domEventName:n,eventSystemFlags:r,nativeEvent:i,targetContainers:[a]},null!==t&&null!==(t=ba(t))&&xt(t),e):(e.eventSystemFlags|=r,t=e.targetContainers,null!==a&&-1===t.indexOf(a)&&t.push(a),e)}function Rt(e){var t=ma(e.target);if(null!==t){var n=Be(t);if(null!==n)if(13===(t=n.tag)){if(null!==(t=We(n)))return e.blockedOn=t,void Et(e.priority,(function(){kt(n)}))}else if(3===t&&n.stateNode.current.memoizedState.isDehydrated)return void(e.blockedOn=3===n.tag?n.stateNode.containerInfo:null)}e.blockedOn=null}function jt(e){if(null!==e.blockedOn)return!1;for(var t=e.targetContainers;0<t.length;){var n=Gt(e.domEventName,e.eventSystemFlags,t[0],e.nativeEvent);if(null!==n)return null!==(t=ba(n))&&xt(t),e.blockedOn=n,!1;var r=new(n=e.nativeEvent).constructor(n.type,n);_e=r,n.target.dispatchEvent(r),_e=null,t.shift()}return!0}function Ut(e,t,n){jt(e)&&n.delete(t)}function It(){Ct=!1,null!==Mt&&jt(Mt)&&(Mt=null),null!==Nt&&jt(Nt)&&(Nt=null),null!==Pt&&jt(Pt)&&(Pt=null),zt.forEach(Ut),Lt.forEach(Ut)}function $t(e,t){e.blockedOn===t&&(e.blockedOn=null,Ct||(Ct=!0,a.unstable_scheduleCallback(a.unstable_NormalPriority,It)))}function Bt(e){function t(t){return $t(t,e)}if(0<Tt.length){$t(Tt[0],e);for(var n=1;n<Tt.length;n++){var r=Tt[n];r.blockedOn===e&&(r.blockedOn=null)}}for(null!==Mt&&$t(Mt,e),null!==Nt&&$t(Nt,e),null!==Pt&&$t(Pt,e),zt.forEach(t),Lt.forEach(t),n=0;n<Ot.length;n++)(r=Ot[n]).blockedOn===e&&(r.blockedOn=null);for(;0<Ot.length&&null===(n=Ot[0]).blockedOn;)Rt(n),null===n.blockedOn&&Ot.shift()}var Wt=_.ReactCurrentBatchConfig,Vt=!0;function Ht(e,t,n,r){var a=bt,i=Wt.transition;Wt.transition=null;try{bt=1,Qt(e,t,n,r)}finally{bt=a,Wt.transition=i}}function qt(e,t,n,r){var a=bt,i=Wt.transition;Wt.transition=null;try{bt=4,Qt(e,t,n,r)}finally{bt=a,Wt.transition=i}}function Qt(e,t,n,r){if(Vt){var a=Gt(e,t,n,r);if(null===a)Vr(e,t,r,Yt,n),Ft(e,r);else if(function(e,t,n,r,a){switch(t){case\"focusin\":return Mt=Dt(Mt,e,t,n,r,a),!0;case\"dragenter\":return Nt=Dt(Nt,e,t,n,r,a),!0;case\"mouseover\":return Pt=Dt(Pt,e,t,n,r,a),!0;case\"pointerover\":var i=a.pointerId;return zt.set(i,Dt(zt.get(i)||null,e,t,n,r,a)),!0;case\"gotpointercapture\":return i=a.pointerId,Lt.set(i,Dt(Lt.get(i)||null,e,t,n,r,a)),!0}return!1}(a,e,t,n,r))r.stopPropagation();else if(Ft(e,r),4&t&&-1<At.indexOf(e)){for(;null!==a;){var i=ba(a);if(null!==i&&wt(i),null===(i=Gt(e,t,n,r))&&Vr(e,t,r,Yt,n),i===a)break;a=i}null!==a&&r.stopPropagation()}else Vr(e,t,r,null,n)}}var Yt=null;function Gt(e,t,n,r){if(Yt=null,null!==(e=ma(e=we(r))))if(null===(t=Be(e)))e=null;else if(13===(n=t.tag)){if(null!==(e=We(t)))return e;e=null}else if(3===n){if(t.stateNode.current.memoizedState.isDehydrated)return 3===t.tag?t.stateNode.containerInfo:null;e=null}else t!==e&&(e=null);return Yt=e,null}function Kt(e){switch(e){case\"cancel\":case\"click\":case\"close\":case\"contextmenu\":case\"copy\":case\"cut\":case\"auxclick\":case\"dblclick\":case\"dragend\":case\"dragstart\":case\"drop\":case\"focusin\":case\"focusout\":case\"input\":case\"invalid\":case\"keydown\":case\"keypress\":case\"keyup\":case\"mousedown\":case\"mouseup\":case\"paste\":case\"pause\":case\"play\":case\"pointercancel\":case\"pointerdown\":case\"pointerup\":case\"ratechange\":case\"reset\":case\"resize\":case\"seeked\":case\"submit\":case\"touchcancel\":case\"touchend\":case\"touchstart\":case\"volumechange\":case\"change\":case\"selectionchange\":case\"textInput\":case\"compositionstart\":case\"compositionend\":case\"compositionupdate\":case\"beforeblur\":case\"afterblur\":case\"beforeinput\":case\"blur\":case\"fullscreenchange\":case\"focus\":case\"hashchange\":case\"popstate\":case\"select\":case\"selectstart\":return 1;case\"drag\":case\"dragenter\":case\"dragexit\":case\"dragleave\":case\"dragover\":case\"mousemove\":case\"mouseout\":case\"mouseover\":case\"pointermove\":case\"pointerout\":case\"pointerover\":case\"scroll\":case\"toggle\":case\"touchmove\":case\"wheel\":case\"mouseenter\":case\"mouseleave\":case\"pointerenter\":case\"pointerleave\":return 4;case\"message\":switch(Xe()){case Je:return 1;case et:return 4;case tt:case nt:return 16;case rt:return 536870912;default:return 16}default:return 16}}var Zt=null,Xt=null,Jt=null;function en(){if(Jt)return Jt;var e,t,n=Xt,r=n.length,a=\"value\"in Zt?Zt.value:Zt.textContent,i=a.length;for(e=0;e<r&&n[e]===a[e];e++);var o=r-e;for(t=1;t<=o&&n[r-t]===a[i-t];t++);return Jt=a.slice(e,1<t?1-t:void 0)}function tn(e){var t=e.keyCode;return\"charCode\"in e?0===(e=e.charCode)&&13===t&&(e=13):e=t,10===e&&(e=13),32<=e||13===e?e:0}function nn(){return!0}function rn(){return!1}function an(e){function t(t,n,r,a,i){for(var o in this._reactName=t,this._targetInst=r,this.type=n,this.nativeEvent=a,this.target=i,this.currentTarget=null,e)e.hasOwnProperty(o)&&(t=e[o],this[o]=t?t(a):a[o]);return this.isDefaultPrevented=(null!=a.defaultPrevented?a.defaultPrevented:!1===a.returnValue)?nn:rn,this.isPropagationStopped=rn,this}return R(t.prototype,{preventDefault:function(){this.defaultPrevented=!0;var e=this.nativeEvent;e&&(e.preventDefault?e.preventDefault():\"unknown\"!=typeof e.returnValue&&(e.returnValue=!1),this.isDefaultPrevented=nn)},stopPropagation:function(){var e=this.nativeEvent;e&&(e.stopPropagation?e.stopPropagation():\"unknown\"!=typeof e.cancelBubble&&(e.cancelBubble=!0),this.isPropagationStopped=nn)},persist:function(){},isPersistent:nn}),t}var on,un,ln,sn={eventPhase:0,bubbles:0,cancelable:0,timeStamp:function(e){return e.timeStamp||Date.now()},defaultPrevented:0,isTrusted:0},cn=an(sn),fn=R({},sn,{view:0,detail:0}),pn=an(fn),dn=R({},fn,{screenX:0,screenY:0,clientX:0,clientY:0,pageX:0,pageY:0,ctrlKey:0,shiftKey:0,altKey:0,metaKey:0,getModifierState:En,button:0,buttons:0,relatedTarget:function(e){return void 0===e.relatedTarget?e.fromElement===e.srcElement?e.toElement:e.fromElement:e.relatedTarget},movementX:function(e){return\"movementX\"in e?e.movementX:(e!==ln&&(ln&&\"mousemove\"===e.type?(on=e.screenX-ln.screenX,un=e.screenY-ln.screenY):un=on=0,ln=e),on)},movementY:function(e){return\"movementY\"in e?e.movementY:un}}),hn=an(dn),vn=an(R({},dn,{dataTransfer:0})),gn=an(R({},fn,{relatedTarget:0})),yn=an(R({},sn,{animationName:0,elapsedTime:0,pseudoElement:0})),mn=R({},sn,{clipboardData:function(e){return\"clipboardData\"in e?e.clipboardData:window.clipboardData}}),bn=an(mn),_n=an(R({},sn,{data:0})),wn={Esc:\"Escape\",Spacebar:\" \",Left:\"ArrowLeft\",Up:\"ArrowUp\",Right:\"ArrowRight\",Down:\"ArrowDown\",Del:\"Delete\",Win:\"OS\",Menu:\"ContextMenu\",Apps:\"ContextMenu\",Scroll:\"ScrollLock\",MozPrintableKey:\"Unidentified\"},xn={8:\"Backspace\",9:\"Tab\",12:\"Clear\",13:\"Enter\",16:\"Shift\",17:\"Control\",18:\"Alt\",19:\"Pause\",20:\"CapsLock\",27:\"Escape\",32:\" \",33:\"PageUp\",34:\"PageDown\",35:\"End\",36:\"Home\",37:\"ArrowLeft\",38:\"ArrowUp\",39:\"ArrowRight\",40:\"ArrowDown\",45:\"Insert\",46:\"Delete\",112:\"F1\",113:\"F2\",114:\"F3\",115:\"F4\",116:\"F5\",117:\"F6\",118:\"F7\",119:\"F8\",120:\"F9\",121:\"F10\",122:\"F11\",123:\"F12\",144:\"NumLock\",145:\"ScrollLock\",224:\"Meta\"},kn={Alt:\"altKey\",Control:\"ctrlKey\",Meta:\"metaKey\",Shift:\"shiftKey\"};function Sn(e){var t=this.nativeEvent;return t.getModifierState?t.getModifierState(e):!!(e=kn[e])&&!!t[e]}function En(){return Sn}var Cn=R({},fn,{key:function(e){if(e.key){var t=wn[e.key]||e.key;if(\"Unidentified\"!==t)return t}return\"keypress\"===e.type?13===(e=tn(e))?\"Enter\":String.fromCharCode(e):\"keydown\"===e.type||\"keyup\"===e.type?xn[e.keyCode]||\"Unidentified\":\"\"},code:0,location:0,ctrlKey:0,shiftKey:0,altKey:0,metaKey:0,repeat:0,locale:0,getModifierState:En,charCode:function(e){return\"keypress\"===e.type?tn(e):0},keyCode:function(e){return\"keydown\"===e.type||\"keyup\"===e.type?e.keyCode:0},which:function(e){return\"keypress\"===e.type?tn(e):\"keydown\"===e.type||\"keyup\"===e.type?e.keyCode:0}}),Tn=an(Cn),Mn=an(R({},dn,{pointerId:0,width:0,height:0,pressure:0,tangentialPressure:0,tiltX:0,tiltY:0,twist:0,pointerType:0,isPrimary:0})),Nn=an(R({},fn,{touches:0,targetTouches:0,changedTouches:0,altKey:0,metaKey:0,ctrlKey:0,shiftKey:0,getModifierState:En})),Pn=an(R({},sn,{propertyName:0,elapsedTime:0,pseudoElement:0})),zn=R({},dn,{deltaX:function(e){return\"deltaX\"in e?e.deltaX:\"wheelDeltaX\"in e?-e.wheelDeltaX:0},deltaY:function(e){return\"deltaY\"in e?e.deltaY:\"wheelDeltaY\"in e?-e.wheelDeltaY:\"wheelDelta\"in e?-e.wheelDelta:0},deltaZ:0,deltaMode:0}),Ln=an(zn),On=[9,13,27,32],An=c&&\"CompositionEvent\"in window,Fn=null;c&&\"documentMode\"in document&&(Fn=document.documentMode);var Dn=c&&\"TextEvent\"in window&&!Fn,Rn=c&&(!An||Fn&&8<Fn&&11>=Fn),jn=String.fromCharCode(32),Un=!1;function In(e,t){switch(e){case\"keyup\":return-1!==On.indexOf(t.keyCode);case\"keydown\":return 229!==t.keyCode;case\"keypress\":case\"mousedown\":case\"focusout\":return!0;default:return!1}}function $n(e){return\"object\"==typeof(e=e.detail)&&\"data\"in e?e.data:null}var Bn=!1,Wn={color:!0,date:!0,datetime:!0,\"datetime-local\":!0,email:!0,month:!0,number:!0,password:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0};function Vn(e){var t=e&&e.nodeName&&e.nodeName.toLowerCase();return\"input\"===t?!!Wn[e.type]:\"textarea\"===t}function Hn(e,t,n,r){Ce(r),0<(t=qr(t,\"onChange\")).length&&(n=new cn(\"onChange\",\"change\",null,n,r),e.push({event:n,listeners:t}))}var qn=null,Qn=null;function Yn(e){jr(e,0)}function Gn(e){if(Q(_a(e)))return e}function Kn(e,t){if(\"change\"===e)return t}var Zn=!1;if(c){var Xn;if(c){var Jn=\"oninput\"in document;if(!Jn){var er=document.createElement(\"div\");er.setAttribute(\"oninput\",\"return;\"),Jn=\"function\"==typeof er.oninput}Xn=Jn}else Xn=!1;Zn=Xn&&(!document.documentMode||9<document.documentMode)}function tr(){qn&&(qn.detachEvent(\"onpropertychange\",nr),Qn=qn=null)}function nr(e){if(\"value\"===e.propertyName&&Gn(Qn)){var t=[];Hn(t,Qn,e,we(e)),ze(Yn,t)}}function rr(e,t,n){\"focusin\"===e?(tr(),Qn=n,(qn=t).attachEvent(\"onpropertychange\",nr)):\"focusout\"===e&&tr()}function ar(e){if(\"selectionchange\"===e||\"keyup\"===e||\"keydown\"===e)return Gn(Qn)}function ir(e,t){if(\"click\"===e)return Gn(t)}function or(e,t){if(\"input\"===e||\"change\"===e)return Gn(t)}var ur=\"function\"==typeof Object.is?Object.is:function(e,t){return e===t&&(0!==e||1/e==1/t)||e!=e&&t!=t};function lr(e,t){if(ur(e,t))return!0;if(\"object\"!=typeof e||null===e||\"object\"!=typeof t||null===t)return!1;var n=Object.keys(e),r=Object.keys(t);if(n.length!==r.length)return!1;for(r=0;r<n.length;r++){var a=n[r];if(!f.call(t,a)||!ur(e[a],t[a]))return!1}return!0}function sr(e){for(;e&&e.firstChild;)e=e.firstChild;return e}function cr(e,t){var n,r=sr(e);for(e=0;r;){if(3===r.nodeType){if(n=e+r.textContent.length,e<=t&&n>=t)return{node:r,offset:t-e};e=n}e:{for(;r;){if(r.nextSibling){r=r.nextSibling;break e}r=r.parentNode}r=void 0}r=sr(r)}}function fr(e,t){return!(!e||!t)&&(e===t||(!e||3!==e.nodeType)&&(t&&3===t.nodeType?fr(e,t.parentNode):\"contains\"in e?e.contains(t):!!e.compareDocumentPosition&&!!(16&e.compareDocumentPosition(t))))}function pr(){for(var e=window,t=Y();t instanceof e.HTMLIFrameElement;){try{var n=\"string\"==typeof t.contentWindow.location.href}catch(e){n=!1}if(!n)break;t=Y((e=t.contentWindow).document)}return t}function dr(e){var t=e&&e.nodeName&&e.nodeName.toLowerCase();return t&&(\"input\"===t&&(\"text\"===e.type||\"search\"===e.type||\"tel\"===e.type||\"url\"===e.type||\"password\"===e.type)||\"textarea\"===t||\"true\"===e.contentEditable)}function hr(e){var t=pr(),n=e.focusedElem,r=e.selectionRange;if(t!==n&&n&&n.ownerDocument&&fr(n.ownerDocument.documentElement,n)){if(null!==r&&dr(n))if(t=r.start,void 0===(e=r.end)&&(e=t),\"selectionStart\"in n)n.selectionStart=t,n.selectionEnd=Math.min(e,n.value.length);else if((e=(t=n.ownerDocument||document)&&t.defaultView||window).getSelection){e=e.getSelection();var a=n.textContent.length,i=Math.min(r.start,a);r=void 0===r.end?i:Math.min(r.end,a),!e.extend&&i>r&&(a=r,r=i,i=a),a=cr(n,i);var o=cr(n,r);a&&o&&(1!==e.rangeCount||e.anchorNode!==a.node||e.anchorOffset!==a.offset||e.focusNode!==o.node||e.focusOffset!==o.offset)&&((t=t.createRange()).setStart(a.node,a.offset),e.removeAllRanges(),i>r?(e.addRange(t),e.extend(o.node,o.offset)):(t.setEnd(o.node,o.offset),e.addRange(t)))}for(t=[],e=n;e=e.parentNode;)1===e.nodeType&&t.push({element:e,left:e.scrollLeft,top:e.scrollTop});for(\"function\"==typeof n.focus&&n.focus(),n=0;n<t.length;n++)(e=t[n]).element.scrollLeft=e.left,e.element.scrollTop=e.top}}var vr=c&&\"documentMode\"in document&&11>=document.documentMode,gr=null,yr=null,mr=null,br=!1;function _r(e,t,n){var r=n.window===n?n.document:9===n.nodeType?n:n.ownerDocument;br||null==gr||gr!==Y(r)||(r=\"selectionStart\"in(r=gr)&&dr(r)?{start:r.selectionStart,end:r.selectionEnd}:{anchorNode:(r=(r.ownerDocument&&r.ownerDocument.defaultView||window).getSelection()).anchorNode,anchorOffset:r.anchorOffset,focusNode:r.focusNode,focusOffset:r.focusOffset},mr&&lr(mr,r)||(mr=r,0<(r=qr(yr,\"onSelect\")).length&&(t=new cn(\"onSelect\",\"select\",null,t,n),e.push({event:t,listeners:r}),t.target=gr)))}function wr(e,t){var n={};return n[e.toLowerCase()]=t.toLowerCase(),n[\"Webkit\"+e]=\"webkit\"+t,n[\"Moz\"+e]=\"moz\"+t,n}var xr={animationend:wr(\"Animation\",\"AnimationEnd\"),animationiteration:wr(\"Animation\",\"AnimationIteration\"),animationstart:wr(\"Animation\",\"AnimationStart\"),transitionend:wr(\"Transition\",\"TransitionEnd\")},kr={},Sr={};function Er(e){if(kr[e])return kr[e];if(!xr[e])return e;var t,n=xr[e];for(t in n)if(n.hasOwnProperty(t)&&t in Sr)return kr[e]=n[t];return e}c&&(Sr=document.createElement(\"div\").style,\"AnimationEvent\"in window||(delete xr.animationend.animation,delete xr.animationiteration.animation,delete xr.animationstart.animation),\"TransitionEvent\"in window||delete xr.transitionend.transition);var Cr=Er(\"animationend\"),Tr=Er(\"animationiteration\"),Mr=Er(\"animationstart\"),Nr=Er(\"transitionend\"),Pr=new Map,zr=\"abort auxClick cancel canPlay canPlayThrough click close contextMenu copy cut drag dragEnd dragEnter dragExit dragLeave dragOver dragStart drop durationChange emptied encrypted ended error gotPointerCapture input invalid keyDown keyPress keyUp load loadedData loadedMetadata loadStart lostPointerCapture mouseDown mouseMove mouseOut mouseOver mouseUp paste pause play playing pointerCancel pointerDown pointerMove pointerOut pointerOver pointerUp progress rateChange reset resize seeked seeking stalled submit suspend timeUpdate touchCancel touchEnd touchStart volumeChange scroll toggle touchMove waiting wheel\".split(\" \");function Lr(e,t){Pr.set(e,t),l(t,[e])}for(var Or=0;Or<zr.length;Or++){var Ar=zr[Or];Lr(Ar.toLowerCase(),\"on\"+(Ar[0].toUpperCase()+Ar.slice(1)))}Lr(Cr,\"onAnimationEnd\"),Lr(Tr,\"onAnimationIteration\"),Lr(Mr,\"onAnimationStart\"),Lr(\"dblclick\",\"onDoubleClick\"),Lr(\"focusin\",\"onFocus\"),Lr(\"focusout\",\"onBlur\"),Lr(Nr,\"onTransitionEnd\"),s(\"onMouseEnter\",[\"mouseout\",\"mouseover\"]),s(\"onMouseLeave\",[\"mouseout\",\"mouseover\"]),s(\"onPointerEnter\",[\"pointerout\",\"pointerover\"]),s(\"onPointerLeave\",[\"pointerout\",\"pointerover\"]),l(\"onChange\",\"change click focusin focusout input keydown keyup selectionchange\".split(\" \")),l(\"onSelect\",\"focusout contextmenu dragend focusin keydown keyup mousedown mouseup selectionchange\".split(\" \")),l(\"onBeforeInput\",[\"compositionend\",\"keypress\",\"textInput\",\"paste\"]),l(\"onCompositionEnd\",\"compositionend focusout keydown keypress keyup mousedown\".split(\" \")),l(\"onCompositionStart\",\"compositionstart focusout keydown keypress keyup mousedown\".split(\" \")),l(\"onCompositionUpdate\",\"compositionupdate focusout keydown keypress keyup mousedown\".split(\" \"));var Fr=\"abort canplay canplaythrough durationchange emptied encrypted ended error loadeddata loadedmetadata loadstart pause play playing progress ratechange resize seeked seeking stalled suspend timeupdate volumechange waiting\".split(\" \"),Dr=new Set(\"cancel close invalid load scroll toggle\".split(\" \").concat(Fr));function Rr(e,t,n){var r=e.type||\"unknown-event\";e.currentTarget=n,function(e,t,n,r,a,o,u,l,s){if($e.apply(this,arguments),De){if(!De)throw Error(i(198));var c=Re;De=!1,Re=null,je||(je=!0,Ue=c)}}(r,t,void 0,e),e.currentTarget=null}function jr(e,t){t=0!=(4&t);for(var n=0;n<e.length;n++){var r=e[n],a=r.event;r=r.listeners;e:{var i=void 0;if(t)for(var o=r.length-1;0<=o;o--){var u=r[o],l=u.instance,s=u.currentTarget;if(u=u.listener,l!==i&&a.isPropagationStopped())break e;Rr(a,u,s),i=l}else for(o=0;o<r.length;o++){if(l=(u=r[o]).instance,s=u.currentTarget,u=u.listener,l!==i&&a.isPropagationStopped())break e;Rr(a,u,s),i=l}}}if(je)throw e=Ue,je=!1,Ue=null,e}function Ur(e,t){var n=t[va];void 0===n&&(n=t[va]=new Set);var r=e+\"__bubble\";n.has(r)||(Wr(t,e,2,!1),n.add(r))}function Ir(e,t,n){var r=0;t&&(r|=4),Wr(n,e,r,t)}var $r=\"_reactListening\"+Math.random().toString(36).slice(2);function Br(e){if(!e[$r]){e[$r]=!0,o.forEach((function(t){\"selectionchange\"!==t&&(Dr.has(t)||Ir(t,!1,e),Ir(t,!0,e))}));var t=9===e.nodeType?e:e.ownerDocument;null===t||t[$r]||(t[$r]=!0,Ir(\"selectionchange\",!1,t))}}function Wr(e,t,n,r){switch(Kt(t)){case 1:var a=Ht;break;case 4:a=qt;break;default:a=Qt}n=a.bind(null,t,n,e),a=void 0,!Oe||\"touchstart\"!==t&&\"touchmove\"!==t&&\"wheel\"!==t||(a=!0),r?void 0!==a?e.addEventListener(t,n,{capture:!0,passive:a}):e.addEventListener(t,n,!0):void 0!==a?e.addEventListener(t,n,{passive:a}):e.addEventListener(t,n,!1)}function Vr(e,t,n,r,a){var i=r;if(0==(1&t)&&0==(2&t)&&null!==r)e:for(;;){if(null===r)return;var o=r.tag;if(3===o||4===o){var u=r.stateNode.containerInfo;if(u===a||8===u.nodeType&&u.parentNode===a)break;if(4===o)for(o=r.return;null!==o;){var l=o.tag;if((3===l||4===l)&&((l=o.stateNode.containerInfo)===a||8===l.nodeType&&l.parentNode===a))return;o=o.return}for(;null!==u;){if(null===(o=ma(u)))return;if(5===(l=o.tag)||6===l){r=i=o;continue e}u=u.parentNode}}r=r.return}ze((function(){var r=i,a=we(n),o=[];e:{var u=Pr.get(e);if(void 0!==u){var l=cn,s=e;switch(e){case\"keypress\":if(0===tn(n))break e;case\"keydown\":case\"keyup\":l=Tn;break;case\"focusin\":s=\"focus\",l=gn;break;case\"focusout\":s=\"blur\",l=gn;break;case\"beforeblur\":case\"afterblur\":l=gn;break;case\"click\":if(2===n.button)break e;case\"auxclick\":case\"dblclick\":case\"mousedown\":case\"mousemove\":case\"mouseup\":case\"mouseout\":case\"mouseover\":case\"contextmenu\":l=hn;break;case\"drag\":case\"dragend\":case\"dragenter\":case\"dragexit\":case\"dragleave\":case\"dragover\":case\"dragstart\":case\"drop\":l=vn;break;case\"touchcancel\":case\"touchend\":case\"touchmove\":case\"touchstart\":l=Nn;break;case Cr:case Tr:case Mr:l=yn;break;case Nr:l=Pn;break;case\"scroll\":l=pn;break;case\"wheel\":l=Ln;break;case\"copy\":case\"cut\":case\"paste\":l=bn;break;case\"gotpointercapture\":case\"lostpointercapture\":case\"pointercancel\":case\"pointerdown\":case\"pointermove\":case\"pointerout\":case\"pointerover\":case\"pointerup\":l=Mn}var c=0!=(4&t),f=!c&&\"scroll\"===e,p=c?null!==u?u+\"Capture\":null:u;c=[];for(var d,h=r;null!==h;){var v=(d=h).stateNode;if(5===d.tag&&null!==v&&(d=v,null!==p&&null!=(v=Le(h,p))&&c.push(Hr(h,v,d))),f)break;h=h.return}0<c.length&&(u=new l(u,s,null,n,a),o.push({event:u,listeners:c}))}}if(0==(7&t)){if(l=\"mouseout\"===e||\"pointerout\"===e,(!(u=\"mouseover\"===e||\"pointerover\"===e)||n===_e||!(s=n.relatedTarget||n.fromElement)||!ma(s)&&!s[ha])&&(l||u)&&(u=a.window===a?a:(u=a.ownerDocument)?u.defaultView||u.parentWindow:window,l?(l=r,null!==(s=(s=n.relatedTarget||n.toElement)?ma(s):null)&&(s!==(f=Be(s))||5!==s.tag&&6!==s.tag)&&(s=null)):(l=null,s=r),l!==s)){if(c=hn,v=\"onMouseLeave\",p=\"onMouseEnter\",h=\"mouse\",\"pointerout\"!==e&&\"pointerover\"!==e||(c=Mn,v=\"onPointerLeave\",p=\"onPointerEnter\",h=\"pointer\"),f=null==l?u:_a(l),d=null==s?u:_a(s),(u=new c(v,h+\"leave\",l,n,a)).target=f,u.relatedTarget=d,v=null,ma(a)===r&&((c=new c(p,h+\"enter\",s,n,a)).target=d,c.relatedTarget=f,v=c),f=v,l&&s)e:{for(p=s,h=0,d=c=l;d;d=Qr(d))h++;for(d=0,v=p;v;v=Qr(v))d++;for(;0<h-d;)c=Qr(c),h--;for(;0<d-h;)p=Qr(p),d--;for(;h--;){if(c===p||null!==p&&c===p.alternate)break e;c=Qr(c),p=Qr(p)}c=null}else c=null;null!==l&&Yr(o,u,l,c,!1),null!==s&&null!==f&&Yr(o,f,s,c,!0)}if(\"select\"===(l=(u=r?_a(r):window).nodeName&&u.nodeName.toLowerCase())||\"input\"===l&&\"file\"===u.type)var g=Kn;else if(Vn(u))if(Zn)g=or;else{g=ar;var y=rr}else(l=u.nodeName)&&\"input\"===l.toLowerCase()&&(\"checkbox\"===u.type||\"radio\"===u.type)&&(g=ir);switch(g&&(g=g(e,r))?Hn(o,g,n,a):(y&&y(e,u,r),\"focusout\"===e&&(y=u._wrapperState)&&y.controlled&&\"number\"===u.type&&ee(u,\"number\",u.value)),y=r?_a(r):window,e){case\"focusin\":(Vn(y)||\"true\"===y.contentEditable)&&(gr=y,yr=r,mr=null);break;case\"focusout\":mr=yr=gr=null;break;case\"mousedown\":br=!0;break;case\"contextmenu\":case\"mouseup\":case\"dragend\":br=!1,_r(o,n,a);break;case\"selectionchange\":if(vr)break;case\"keydown\":case\"keyup\":_r(o,n,a)}var m;if(An)e:{switch(e){case\"compositionstart\":var b=\"onCompositionStart\";break e;case\"compositionend\":b=\"onCompositionEnd\";break e;case\"compositionupdate\":b=\"onCompositionUpdate\";break e}b=void 0}else Bn?In(e,n)&&(b=\"onCompositionEnd\"):\"keydown\"===e&&229===n.keyCode&&(b=\"onCompositionStart\");b&&(Rn&&\"ko\"!==n.locale&&(Bn||\"onCompositionStart\"!==b?\"onCompositionEnd\"===b&&Bn&&(m=en()):(Xt=\"value\"in(Zt=a)?Zt.value:Zt.textContent,Bn=!0)),0<(y=qr(r,b)).length&&(b=new _n(b,e,null,n,a),o.push({event:b,listeners:y}),(m||null!==(m=$n(n)))&&(b.data=m))),(m=Dn?function(e,t){switch(e){case\"compositionend\":return $n(t);case\"keypress\":return 32!==t.which?null:(Un=!0,jn);case\"textInput\":return(e=t.data)===jn&&Un?null:e;default:return null}}(e,n):function(e,t){if(Bn)return\"compositionend\"===e||!An&&In(e,t)?(e=en(),Jt=Xt=Zt=null,Bn=!1,e):null;switch(e){case\"paste\":default:return null;case\"keypress\":if(!(t.ctrlKey||t.altKey||t.metaKey)||t.ctrlKey&&t.altKey){if(t.char&&1<t.char.length)return t.char;if(t.which)return String.fromCharCode(t.which)}return null;case\"compositionend\":return Rn&&\"ko\"!==t.locale?null:t.data}}(e,n))&&0<(r=qr(r,\"onBeforeInput\")).length&&(a=new _n(\"onBeforeInput\",\"beforeinput\",null,n,a),o.push({event:a,listeners:r}),a.data=m)}jr(o,t)}))}function Hr(e,t,n){return{instance:e,listener:t,currentTarget:n}}function qr(e,t){for(var n=t+\"Capture\",r=[];null!==e;){var a=e,i=a.stateNode;5===a.tag&&null!==i&&(a=i,null!=(i=Le(e,n))&&r.unshift(Hr(e,i,a)),null!=(i=Le(e,t))&&r.push(Hr(e,i,a))),e=e.return}return r}function Qr(e){if(null===e)return null;do{e=e.return}while(e&&5!==e.tag);return e||null}function Yr(e,t,n,r,a){for(var i=t._reactName,o=[];null!==n&&n!==r;){var u=n,l=u.alternate,s=u.stateNode;if(null!==l&&l===r)break;5===u.tag&&null!==s&&(u=s,a?null!=(l=Le(n,i))&&o.unshift(Hr(n,l,u)):a||null!=(l=Le(n,i))&&o.push(Hr(n,l,u))),n=n.return}0!==o.length&&e.push({event:t,listeners:o})}var Gr=/\\r\\n?/g,Kr=/\\u0000|\\uFFFD/g;function Zr(e){return(\"string\"==typeof e?e:\"\"+e).replace(Gr,\"\\n\").replace(Kr,\"\")}function Xr(e,t,n){if(t=Zr(t),Zr(e)!==t&&n)throw Error(i(425))}function Jr(){}var ea=null,ta=null;function na(e,t){return\"textarea\"===e||\"noscript\"===e||\"string\"==typeof t.children||\"number\"==typeof t.children||\"object\"==typeof t.dangerouslySetInnerHTML&&null!==t.dangerouslySetInnerHTML&&null!=t.dangerouslySetInnerHTML.__html}var ra=\"function\"==typeof setTimeout?setTimeout:void 0,aa=\"function\"==typeof clearTimeout?clearTimeout:void 0,ia=\"function\"==typeof Promise?Promise:void 0,oa=\"function\"==typeof queueMicrotask?queueMicrotask:void 0!==ia?function(e){return ia.resolve(null).then(e).catch(ua)}:ra;function ua(e){setTimeout((function(){throw e}))}function la(e,t){var n=t,r=0;do{var a=n.nextSibling;if(e.removeChild(n),a&&8===a.nodeType)if(\"/$\"===(n=a.data)){if(0===r)return e.removeChild(a),void Bt(t);r--}else\"$\"!==n&&\"$?\"!==n&&\"$!\"!==n||r++;n=a}while(n);Bt(t)}function sa(e){for(;null!=e;e=e.nextSibling){var t=e.nodeType;if(1===t||3===t)break;if(8===t){if(\"$\"===(t=e.data)||\"$!\"===t||\"$?\"===t)break;if(\"/$\"===t)return null}}return e}function ca(e){e=e.previousSibling;for(var t=0;e;){if(8===e.nodeType){var n=e.data;if(\"$\"===n||\"$!\"===n||\"$?\"===n){if(0===t)return e;t--}else\"/$\"===n&&t++}e=e.previousSibling}return null}var fa=Math.random().toString(36).slice(2),pa=\"__reactFiber$\"+fa,da=\"__reactProps$\"+fa,ha=\"__reactContainer$\"+fa,va=\"__reactEvents$\"+fa,ga=\"__reactListeners$\"+fa,ya=\"__reactHandles$\"+fa;function ma(e){var t=e[pa];if(t)return t;for(var n=e.parentNode;n;){if(t=n[ha]||n[pa]){if(n=t.alternate,null!==t.child||null!==n&&null!==n.child)for(e=ca(e);null!==e;){if(n=e[pa])return n;e=ca(e)}return t}n=(e=n).parentNode}return null}function ba(e){return!(e=e[pa]||e[ha])||5!==e.tag&&6!==e.tag&&13!==e.tag&&3!==e.tag?null:e}function _a(e){if(5===e.tag||6===e.tag)return e.stateNode;throw Error(i(33))}function wa(e){return e[da]||null}var xa=[],ka=-1;function Sa(e){return{current:e}}function Ea(e){0>ka||(e.current=xa[ka],xa[ka]=null,ka--)}function Ca(e,t){ka++,xa[ka]=e.current,e.current=t}var Ta={},Ma=Sa(Ta),Na=Sa(!1),Pa=Ta;function za(e,t){var n=e.type.contextTypes;if(!n)return Ta;var r=e.stateNode;if(r&&r.__reactInternalMemoizedUnmaskedChildContext===t)return r.__reactInternalMemoizedMaskedChildContext;var a,i={};for(a in n)i[a]=t[a];return r&&((e=e.stateNode).__reactInternalMemoizedUnmaskedChildContext=t,e.__reactInternalMemoizedMaskedChildContext=i),i}function La(e){return null!=e.childContextTypes}function Oa(){Ea(Na),Ea(Ma)}function Aa(e,t,n){if(Ma.current!==Ta)throw Error(i(168));Ca(Ma,t),Ca(Na,n)}function Fa(e,t,n){var r=e.stateNode;if(t=t.childContextTypes,\"function\"!=typeof r.getChildContext)return n;for(var a in r=r.getChildContext())if(!(a in t))throw Error(i(108,W(e)||\"Unknown\",a));return R({},n,r)}function Da(e){return e=(e=e.stateNode)&&e.__reactInternalMemoizedMergedChildContext||Ta,Pa=Ma.current,Ca(Ma,e),Ca(Na,Na.current),!0}function Ra(e,t,n){var r=e.stateNode;if(!r)throw Error(i(169));n?(e=Fa(e,t,Pa),r.__reactInternalMemoizedMergedChildContext=e,Ea(Na),Ea(Ma),Ca(Ma,e)):Ea(Na),Ca(Na,n)}var ja=null,Ua=!1,Ia=!1;function $a(e){null===ja?ja=[e]:ja.push(e)}function Ba(){if(!Ia&&null!==ja){Ia=!0;var e=0,t=bt;try{var n=ja;for(bt=1;e<n.length;e++){var r=n[e];do{r=r(!0)}while(null!==r)}ja=null,Ua=!1}catch(t){throw null!==ja&&(ja=ja.slice(e+1)),Qe(Je,Ba),t}finally{bt=t,Ia=!1}}return null}var Wa=[],Va=0,Ha=null,qa=0,Qa=[],Ya=0,Ga=null,Ka=1,Za=\"\";function Xa(e,t){Wa[Va++]=qa,Wa[Va++]=Ha,Ha=e,qa=t}function Ja(e,t,n){Qa[Ya++]=Ka,Qa[Ya++]=Za,Qa[Ya++]=Ga,Ga=e;var r=Ka;e=Za;var a=32-ot(r)-1;r&=~(1<<a),n+=1;var i=32-ot(t)+a;if(30<i){var o=a-a%5;i=(r&(1<<o)-1).toString(32),r>>=o,a-=o,Ka=1<<32-ot(t)+a|n<<a|r,Za=i+e}else Ka=1<<i|n<<a|r,Za=e}function ei(e){null!==e.return&&(Xa(e,1),Ja(e,1,0))}function ti(e){for(;e===Ha;)Ha=Wa[--Va],Wa[Va]=null,qa=Wa[--Va],Wa[Va]=null;for(;e===Ga;)Ga=Qa[--Ya],Qa[Ya]=null,Za=Qa[--Ya],Qa[Ya]=null,Ka=Qa[--Ya],Qa[Ya]=null}var ni=null,ri=null,ai=!1,ii=null;function oi(e,t){var n=Ls(5,null,null,0);n.elementType=\"DELETED\",n.stateNode=t,n.return=e,null===(t=e.deletions)?(e.deletions=[n],e.flags|=16):t.push(n)}function ui(e,t){switch(e.tag){case 5:var n=e.type;return null!==(t=1!==t.nodeType||n.toLowerCase()!==t.nodeName.toLowerCase()?null:t)&&(e.stateNode=t,ni=e,ri=sa(t.firstChild),!0);case 6:return null!==(t=\"\"===e.pendingProps||3!==t.nodeType?null:t)&&(e.stateNode=t,ni=e,ri=null,!0);case 13:return null!==(t=8!==t.nodeType?null:t)&&(n=null!==Ga?{id:Ka,overflow:Za}:null,e.memoizedState={dehydrated:t,treeContext:n,retryLane:1073741824},(n=Ls(18,null,null,0)).stateNode=t,n.return=e,e.child=n,ni=e,ri=null,!0);default:return!1}}function li(e){return 0!=(1&e.mode)&&0==(128&e.flags)}function si(e){if(ai){var t=ri;if(t){var n=t;if(!ui(e,t)){if(li(e))throw Error(i(418));t=sa(n.nextSibling);var r=ni;t&&ui(e,t)?oi(r,n):(e.flags=-4097&e.flags|2,ai=!1,ni=e)}}else{if(li(e))throw Error(i(418));e.flags=-4097&e.flags|2,ai=!1,ni=e}}}function ci(e){for(e=e.return;null!==e&&5!==e.tag&&3!==e.tag&&13!==e.tag;)e=e.return;ni=e}function fi(e){if(e!==ni)return!1;if(!ai)return ci(e),ai=!0,!1;var t;if((t=3!==e.tag)&&!(t=5!==e.tag)&&(t=\"head\"!==(t=e.type)&&\"body\"!==t&&!na(e.type,e.memoizedProps)),t&&(t=ri)){if(li(e))throw pi(),Error(i(418));for(;t;)oi(e,t),t=sa(t.nextSibling)}if(ci(e),13===e.tag){if(!(e=null!==(e=e.memoizedState)?e.dehydrated:null))throw Error(i(317));e:{for(e=e.nextSibling,t=0;e;){if(8===e.nodeType){var n=e.data;if(\"/$\"===n){if(0===t){ri=sa(e.nextSibling);break e}t--}else\"$\"!==n&&\"$!\"!==n&&\"$?\"!==n||t++}e=e.nextSibling}ri=null}}else ri=ni?sa(e.stateNode.nextSibling):null;return!0}function pi(){for(var e=ri;e;)e=sa(e.nextSibling)}function di(){ri=ni=null,ai=!1}function hi(e){null===ii?ii=[e]:ii.push(e)}var vi=_.ReactCurrentBatchConfig;function gi(e,t){if(e&&e.defaultProps){for(var n in t=R({},t),e=e.defaultProps)void 0===t[n]&&(t[n]=e[n]);return t}return t}var yi=Sa(null),mi=null,bi=null,_i=null;function wi(){_i=bi=mi=null}function xi(e){var t=yi.current;Ea(yi),e._currentValue=t}function ki(e,t,n){for(;null!==e;){var r=e.alternate;if((e.childLanes&t)!==t?(e.childLanes|=t,null!==r&&(r.childLanes|=t)):null!==r&&(r.childLanes&t)!==t&&(r.childLanes|=t),e===n)break;e=e.return}}function Si(e,t){mi=e,_i=bi=null,null!==(e=e.dependencies)&&null!==e.firstContext&&(0!=(e.lanes&t)&&(_u=!0),e.firstContext=null)}function Ei(e){var t=e._currentValue;if(_i!==e)if(e={context:e,memoizedValue:t,next:null},null===bi){if(null===mi)throw Error(i(308));bi=e,mi.dependencies={lanes:0,firstContext:e}}else bi=bi.next=e;return t}var Ci=null;function Ti(e){null===Ci?Ci=[e]:Ci.push(e)}function Mi(e,t,n,r){var a=t.interleaved;return null===a?(n.next=n,Ti(t)):(n.next=a.next,a.next=n),t.interleaved=n,Ni(e,r)}function Ni(e,t){e.lanes|=t;var n=e.alternate;for(null!==n&&(n.lanes|=t),n=e,e=e.return;null!==e;)e.childLanes|=t,null!==(n=e.alternate)&&(n.childLanes|=t),n=e,e=e.return;return 3===n.tag?n.stateNode:null}var Pi=!1;function zi(e){e.updateQueue={baseState:e.memoizedState,firstBaseUpdate:null,lastBaseUpdate:null,shared:{pending:null,interleaved:null,lanes:0},effects:null}}function Li(e,t){e=e.updateQueue,t.updateQueue===e&&(t.updateQueue={baseState:e.baseState,firstBaseUpdate:e.firstBaseUpdate,lastBaseUpdate:e.lastBaseUpdate,shared:e.shared,effects:e.effects})}function Oi(e,t){return{eventTime:e,lane:t,tag:0,payload:null,callback:null,next:null}}function Ai(e,t,n){var r=e.updateQueue;if(null===r)return null;if(r=r.shared,0!=(2&Nl)){var a=r.pending;return null===a?t.next=t:(t.next=a.next,a.next=t),r.pending=t,Ni(e,n)}return null===(a=r.interleaved)?(t.next=t,Ti(r)):(t.next=a.next,a.next=t),r.interleaved=t,Ni(e,n)}function Fi(e,t,n){if(null!==(t=t.updateQueue)&&(t=t.shared,0!=(4194240&n))){var r=t.lanes;n|=r&=e.pendingLanes,t.lanes=n,mt(e,n)}}function Di(e,t){var n=e.updateQueue,r=e.alternate;if(null!==r&&n===(r=r.updateQueue)){var a=null,i=null;if(null!==(n=n.firstBaseUpdate)){do{var o={eventTime:n.eventTime,lane:n.lane,tag:n.tag,payload:n.payload,callback:n.callback,next:null};null===i?a=i=o:i=i.next=o,n=n.next}while(null!==n);null===i?a=i=t:i=i.next=t}else a=i=t;return n={baseState:r.baseState,firstBaseUpdate:a,lastBaseUpdate:i,shared:r.shared,effects:r.effects},void(e.updateQueue=n)}null===(e=n.lastBaseUpdate)?n.firstBaseUpdate=t:e.next=t,n.lastBaseUpdate=t}function Ri(e,t,n,r){var a=e.updateQueue;Pi=!1;var i=a.firstBaseUpdate,o=a.lastBaseUpdate,u=a.shared.pending;if(null!==u){a.shared.pending=null;var l=u,s=l.next;l.next=null,null===o?i=s:o.next=s,o=l;var c=e.alternate;null!==c&&(u=(c=c.updateQueue).lastBaseUpdate)!==o&&(null===u?c.firstBaseUpdate=s:u.next=s,c.lastBaseUpdate=l)}if(null!==i){var f=a.baseState;for(o=0,c=s=l=null,u=i;;){var p=u.lane,d=u.eventTime;if((r&p)===p){null!==c&&(c=c.next={eventTime:d,lane:0,tag:u.tag,payload:u.payload,callback:u.callback,next:null});e:{var h=e,v=u;switch(p=t,d=n,v.tag){case 1:if(\"function\"==typeof(h=v.payload)){f=h.call(d,f,p);break e}f=h;break e;case 3:h.flags=-65537&h.flags|128;case 0:if(null==(p=\"function\"==typeof(h=v.payload)?h.call(d,f,p):h))break e;f=R({},f,p);break e;case 2:Pi=!0}}null!==u.callback&&0!==u.lane&&(e.flags|=64,null===(p=a.effects)?a.effects=[u]:p.push(u))}else d={eventTime:d,lane:p,tag:u.tag,payload:u.payload,callback:u.callback,next:null},null===c?(s=c=d,l=f):c=c.next=d,o|=p;if(null===(u=u.next)){if(null===(u=a.shared.pending))break;u=(p=u).next,p.next=null,a.lastBaseUpdate=p,a.shared.pending=null}}if(null===c&&(l=f),a.baseState=l,a.firstBaseUpdate=s,a.lastBaseUpdate=c,null!==(t=a.shared.interleaved)){a=t;do{o|=a.lane,a=a.next}while(a!==t)}else null===i&&(a.shared.lanes=0);Rl|=o,e.lanes=o,e.memoizedState=f}}function ji(e,t,n){if(e=t.effects,t.effects=null,null!==e)for(t=0;t<e.length;t++){var r=e[t],a=r.callback;if(null!==a){if(r.callback=null,r=n,\"function\"!=typeof a)throw Error(i(191,a));a.call(r)}}}var Ui=(new r.Component).refs;function Ii(e,t,n,r){n=null==(n=n(r,t=e.memoizedState))?t:R({},t,n),e.memoizedState=n,0===e.lanes&&(e.updateQueue.baseState=n)}var $i={isMounted:function(e){return!!(e=e._reactInternals)&&Be(e)===e},enqueueSetState:function(e,t,n){e=e._reactInternals;var r=ts(),a=ns(e),i=Oi(r,a);i.payload=t,null!=n&&(i.callback=n),null!==(t=Ai(e,i,a))&&(rs(t,e,a,r),Fi(t,e,a))},enqueueReplaceState:function(e,t,n){e=e._reactInternals;var r=ts(),a=ns(e),i=Oi(r,a);i.tag=1,i.payload=t,null!=n&&(i.callback=n),null!==(t=Ai(e,i,a))&&(rs(t,e,a,r),Fi(t,e,a))},enqueueForceUpdate:function(e,t){e=e._reactInternals;var n=ts(),r=ns(e),a=Oi(n,r);a.tag=2,null!=t&&(a.callback=t),null!==(t=Ai(e,a,r))&&(rs(t,e,r,n),Fi(t,e,r))}};function Bi(e,t,n,r,a,i,o){return\"function\"==typeof(e=e.stateNode).shouldComponentUpdate?e.shouldComponentUpdate(r,i,o):!(t.prototype&&t.prototype.isPureReactComponent&&lr(n,r)&&lr(a,i))}function Wi(e,t,n){var r=!1,a=Ta,i=t.contextType;return\"object\"==typeof i&&null!==i?i=Ei(i):(a=La(t)?Pa:Ma.current,i=(r=null!=(r=t.contextTypes))?za(e,a):Ta),t=new t(n,i),e.memoizedState=null!==t.state&&void 0!==t.state?t.state:null,t.updater=$i,e.stateNode=t,t._reactInternals=e,r&&((e=e.stateNode).__reactInternalMemoizedUnmaskedChildContext=a,e.__reactInternalMemoizedMaskedChildContext=i),t}function Vi(e,t,n,r){e=t.state,\"function\"==typeof t.componentWillReceiveProps&&t.componentWillReceiveProps(n,r),\"function\"==typeof t.UNSAFE_componentWillReceiveProps&&t.UNSAFE_componentWillReceiveProps(n,r),t.state!==e&&$i.enqueueReplaceState(t,t.state,null)}function Hi(e,t,n,r){var a=e.stateNode;a.props=n,a.state=e.memoizedState,a.refs=Ui,zi(e);var i=t.contextType;\"object\"==typeof i&&null!==i?a.context=Ei(i):(i=La(t)?Pa:Ma.current,a.context=za(e,i)),a.state=e.memoizedState,\"function\"==typeof(i=t.getDerivedStateFromProps)&&(Ii(e,t,i,n),a.state=e.memoizedState),\"function\"==typeof t.getDerivedStateFromProps||\"function\"==typeof a.getSnapshotBeforeUpdate||\"function\"!=typeof a.UNSAFE_componentWillMount&&\"function\"!=typeof a.componentWillMount||(t=a.state,\"function\"==typeof a.componentWillMount&&a.componentWillMount(),\"function\"==typeof a.UNSAFE_componentWillMount&&a.UNSAFE_componentWillMount(),t!==a.state&&$i.enqueueReplaceState(a,a.state,null),Ri(e,n,a,r),a.state=e.memoizedState),\"function\"==typeof a.componentDidMount&&(e.flags|=4194308)}function qi(e,t,n){if(null!==(e=n.ref)&&\"function\"!=typeof e&&\"object\"!=typeof e){if(n._owner){if(n=n._owner){if(1!==n.tag)throw Error(i(309));var r=n.stateNode}if(!r)throw Error(i(147,e));var a=r,o=\"\"+e;return null!==t&&null!==t.ref&&\"function\"==typeof t.ref&&t.ref._stringRef===o?t.ref:(t=function(e){var t=a.refs;t===Ui&&(t=a.refs={}),null===e?delete t[o]:t[o]=e},t._stringRef=o,t)}if(\"string\"!=typeof e)throw Error(i(284));if(!n._owner)throw Error(i(290,e))}return e}function Qi(e,t){throw e=Object.prototype.toString.call(t),Error(i(31,\"[object Object]\"===e?\"object with keys {\"+Object.keys(t).join(\", \")+\"}\":e))}function Yi(e){return(0,e._init)(e._payload)}function Gi(e){function t(t,n){if(e){var r=t.deletions;null===r?(t.deletions=[n],t.flags|=16):r.push(n)}}function n(n,r){if(!e)return null;for(;null!==r;)t(n,r),r=r.sibling;return null}function r(e,t){for(e=new Map;null!==t;)null!==t.key?e.set(t.key,t):e.set(t.index,t),t=t.sibling;return e}function a(e,t){return(e=As(e,t)).index=0,e.sibling=null,e}function o(t,n,r){return t.index=r,e?null!==(r=t.alternate)?(r=r.index)<n?(t.flags|=2,n):r:(t.flags|=2,n):(t.flags|=1048576,n)}function u(t){return e&&null===t.alternate&&(t.flags|=2),t}function l(e,t,n,r){return null===t||6!==t.tag?((t=js(n,e.mode,r)).return=e,t):((t=a(t,n)).return=e,t)}function s(e,t,n,r){var i=n.type;return i===k?f(e,t,n.props.children,r,n.key):null!==t&&(t.elementType===i||\"object\"==typeof i&&null!==i&&i.$$typeof===L&&Yi(i)===t.type)?((r=a(t,n.props)).ref=qi(e,t,n),r.return=e,r):((r=Fs(n.type,n.key,n.props,null,e.mode,r)).ref=qi(e,t,n),r.return=e,r)}function c(e,t,n,r){return null===t||4!==t.tag||t.stateNode.containerInfo!==n.containerInfo||t.stateNode.implementation!==n.implementation?((t=Us(n,e.mode,r)).return=e,t):((t=a(t,n.children||[])).return=e,t)}function f(e,t,n,r,i){return null===t||7!==t.tag?((t=Ds(n,e.mode,r,i)).return=e,t):((t=a(t,n)).return=e,t)}function p(e,t,n){if(\"string\"==typeof t&&\"\"!==t||\"number\"==typeof t)return(t=js(\"\"+t,e.mode,n)).return=e,t;if(\"object\"==typeof t&&null!==t){switch(t.$$typeof){case w:return(n=Fs(t.type,t.key,t.props,null,e.mode,n)).ref=qi(e,null,t),n.return=e,n;case x:return(t=Us(t,e.mode,n)).return=e,t;case L:return p(e,(0,t._init)(t._payload),n)}if(te(t)||F(t))return(t=Ds(t,e.mode,n,null)).return=e,t;Qi(e,t)}return null}function d(e,t,n,r){var a=null!==t?t.key:null;if(\"string\"==typeof n&&\"\"!==n||\"number\"==typeof n)return null!==a?null:l(e,t,\"\"+n,r);if(\"object\"==typeof n&&null!==n){switch(n.$$typeof){case w:return n.key===a?s(e,t,n,r):null;case x:return n.key===a?c(e,t,n,r):null;case L:return d(e,t,(a=n._init)(n._payload),r)}if(te(n)||F(n))return null!==a?null:f(e,t,n,r,null);Qi(e,n)}return null}function h(e,t,n,r,a){if(\"string\"==typeof r&&\"\"!==r||\"number\"==typeof r)return l(t,e=e.get(n)||null,\"\"+r,a);if(\"object\"==typeof r&&null!==r){switch(r.$$typeof){case w:return s(t,e=e.get(null===r.key?n:r.key)||null,r,a);case x:return c(t,e=e.get(null===r.key?n:r.key)||null,r,a);case L:return h(e,t,n,(0,r._init)(r._payload),a)}if(te(r)||F(r))return f(t,e=e.get(n)||null,r,a,null);Qi(t,r)}return null}function v(a,i,u,l){for(var s=null,c=null,f=i,v=i=0,g=null;null!==f&&v<u.length;v++){f.index>v?(g=f,f=null):g=f.sibling;var y=d(a,f,u[v],l);if(null===y){null===f&&(f=g);break}e&&f&&null===y.alternate&&t(a,f),i=o(y,i,v),null===c?s=y:c.sibling=y,c=y,f=g}if(v===u.length)return n(a,f),ai&&Xa(a,v),s;if(null===f){for(;v<u.length;v++)null!==(f=p(a,u[v],l))&&(i=o(f,i,v),null===c?s=f:c.sibling=f,c=f);return ai&&Xa(a,v),s}for(f=r(a,f);v<u.length;v++)null!==(g=h(f,a,v,u[v],l))&&(e&&null!==g.alternate&&f.delete(null===g.key?v:g.key),i=o(g,i,v),null===c?s=g:c.sibling=g,c=g);return e&&f.forEach((function(e){return t(a,e)})),ai&&Xa(a,v),s}function g(a,u,l,s){var c=F(l);if(\"function\"!=typeof c)throw Error(i(150));if(null==(l=c.call(l)))throw Error(i(151));for(var f=c=null,v=u,g=u=0,y=null,m=l.next();null!==v&&!m.done;g++,m=l.next()){v.index>g?(y=v,v=null):y=v.sibling;var b=d(a,v,m.value,s);if(null===b){null===v&&(v=y);break}e&&v&&null===b.alternate&&t(a,v),u=o(b,u,g),null===f?c=b:f.sibling=b,f=b,v=y}if(m.done)return n(a,v),ai&&Xa(a,g),c;if(null===v){for(;!m.done;g++,m=l.next())null!==(m=p(a,m.value,s))&&(u=o(m,u,g),null===f?c=m:f.sibling=m,f=m);return ai&&Xa(a,g),c}for(v=r(a,v);!m.done;g++,m=l.next())null!==(m=h(v,a,g,m.value,s))&&(e&&null!==m.alternate&&v.delete(null===m.key?g:m.key),u=o(m,u,g),null===f?c=m:f.sibling=m,f=m);return e&&v.forEach((function(e){return t(a,e)})),ai&&Xa(a,g),c}return function e(r,i,o,l){if(\"object\"==typeof o&&null!==o&&o.type===k&&null===o.key&&(o=o.props.children),\"object\"==typeof o&&null!==o){switch(o.$$typeof){case w:e:{for(var s=o.key,c=i;null!==c;){if(c.key===s){if((s=o.type)===k){if(7===c.tag){n(r,c.sibling),(i=a(c,o.props.children)).return=r,r=i;break e}}else if(c.elementType===s||\"object\"==typeof s&&null!==s&&s.$$typeof===L&&Yi(s)===c.type){n(r,c.sibling),(i=a(c,o.props)).ref=qi(r,c,o),i.return=r,r=i;break e}n(r,c);break}t(r,c),c=c.sibling}o.type===k?((i=Ds(o.props.children,r.mode,l,o.key)).return=r,r=i):((l=Fs(o.type,o.key,o.props,null,r.mode,l)).ref=qi(r,i,o),l.return=r,r=l)}return u(r);case x:e:{for(c=o.key;null!==i;){if(i.key===c){if(4===i.tag&&i.stateNode.containerInfo===o.containerInfo&&i.stateNode.implementation===o.implementation){n(r,i.sibling),(i=a(i,o.children||[])).return=r,r=i;break e}n(r,i);break}t(r,i),i=i.sibling}(i=Us(o,r.mode,l)).return=r,r=i}return u(r);case L:return e(r,i,(c=o._init)(o._payload),l)}if(te(o))return v(r,i,o,l);if(F(o))return g(r,i,o,l);Qi(r,o)}return\"string\"==typeof o&&\"\"!==o||\"number\"==typeof o?(o=\"\"+o,null!==i&&6===i.tag?(n(r,i.sibling),(i=a(i,o)).return=r,r=i):(n(r,i),(i=js(o,r.mode,l)).return=r,r=i),u(r)):n(r,i)}}var Ki=Gi(!0),Zi=Gi(!1),Xi={},Ji=Sa(Xi),eo=Sa(Xi),to=Sa(Xi);function no(e){if(e===Xi)throw Error(i(174));return e}function ro(e,t){switch(Ca(to,t),Ca(eo,e),Ca(Ji,Xi),e=t.nodeType){case 9:case 11:t=(t=t.documentElement)?t.namespaceURI:le(null,\"\");break;default:t=le(t=(e=8===e?t.parentNode:t).namespaceURI||null,e=e.tagName)}Ea(Ji),Ca(Ji,t)}function ao(){Ea(Ji),Ea(eo),Ea(to)}function io(e){no(to.current);var t=no(Ji.current),n=le(t,e.type);t!==n&&(Ca(eo,e),Ca(Ji,n))}function oo(e){eo.current===e&&(Ea(Ji),Ea(eo))}var uo=Sa(0);function lo(e){for(var t=e;null!==t;){if(13===t.tag){var n=t.memoizedState;if(null!==n&&(null===(n=n.dehydrated)||\"$?\"===n.data||\"$!\"===n.data))return t}else if(19===t.tag&&void 0!==t.memoizedProps.revealOrder){if(0!=(128&t.flags))return t}else if(null!==t.child){t.child.return=t,t=t.child;continue}if(t===e)break;for(;null===t.sibling;){if(null===t.return||t.return===e)return null;t=t.return}t.sibling.return=t.return,t=t.sibling}return null}var so=[];function co(){for(var e=0;e<so.length;e++)so[e]._workInProgressVersionPrimary=null;so.length=0}var fo=_.ReactCurrentDispatcher,po=_.ReactCurrentBatchConfig,ho=0,vo=null,go=null,yo=null,mo=!1,bo=!1,_o=0,wo=0;function xo(){throw Error(i(321))}function ko(e,t){if(null===t)return!1;for(var n=0;n<t.length&&n<e.length;n++)if(!ur(e[n],t[n]))return!1;return!0}function So(e,t,n,r,a,o){if(ho=o,vo=t,t.memoizedState=null,t.updateQueue=null,t.lanes=0,fo.current=null===e||null===e.memoizedState?uu:lu,e=n(r,a),bo){o=0;do{if(bo=!1,_o=0,25<=o)throw Error(i(301));o+=1,yo=go=null,t.updateQueue=null,fo.current=su,e=n(r,a)}while(bo)}if(fo.current=ou,t=null!==go&&null!==go.next,ho=0,yo=go=vo=null,mo=!1,t)throw Error(i(300));return e}function Eo(){var e=0!==_o;return _o=0,e}function Co(){var e={memoizedState:null,baseState:null,baseQueue:null,queue:null,next:null};return null===yo?vo.memoizedState=yo=e:yo=yo.next=e,yo}function To(){if(null===go){var e=vo.alternate;e=null!==e?e.memoizedState:null}else e=go.next;var t=null===yo?vo.memoizedState:yo.next;if(null!==t)yo=t,go=e;else{if(null===e)throw Error(i(310));e={memoizedState:(go=e).memoizedState,baseState:go.baseState,baseQueue:go.baseQueue,queue:go.queue,next:null},null===yo?vo.memoizedState=yo=e:yo=yo.next=e}return yo}function Mo(e,t){return\"function\"==typeof t?t(e):t}function No(e){var t=To(),n=t.queue;if(null===n)throw Error(i(311));n.lastRenderedReducer=e;var r=go,a=r.baseQueue,o=n.pending;if(null!==o){if(null!==a){var u=a.next;a.next=o.next,o.next=u}r.baseQueue=a=o,n.pending=null}if(null!==a){o=a.next,r=r.baseState;var l=u=null,s=null,c=o;do{var f=c.lane;if((ho&f)===f)null!==s&&(s=s.next={lane:0,action:c.action,hasEagerState:c.hasEagerState,eagerState:c.eagerState,next:null}),r=c.hasEagerState?c.eagerState:e(r,c.action);else{var p={lane:f,action:c.action,hasEagerState:c.hasEagerState,eagerState:c.eagerState,next:null};null===s?(l=s=p,u=r):s=s.next=p,vo.lanes|=f,Rl|=f}c=c.next}while(null!==c&&c!==o);null===s?u=r:s.next=l,ur(r,t.memoizedState)||(_u=!0),t.memoizedState=r,t.baseState=u,t.baseQueue=s,n.lastRenderedState=r}if(null!==(e=n.interleaved)){a=e;do{o=a.lane,vo.lanes|=o,Rl|=o,a=a.next}while(a!==e)}else null===a&&(n.lanes=0);return[t.memoizedState,n.dispatch]}function Po(e){var t=To(),n=t.queue;if(null===n)throw Error(i(311));n.lastRenderedReducer=e;var r=n.dispatch,a=n.pending,o=t.memoizedState;if(null!==a){n.pending=null;var u=a=a.next;do{o=e(o,u.action),u=u.next}while(u!==a);ur(o,t.memoizedState)||(_u=!0),t.memoizedState=o,null===t.baseQueue&&(t.baseState=o),n.lastRenderedState=o}return[o,r]}function zo(){}function Lo(e,t){var n=vo,r=To(),a=t(),o=!ur(r.memoizedState,a);if(o&&(r.memoizedState=a,_u=!0),r=r.queue,Vo(Fo.bind(null,n,r,e),[e]),r.getSnapshot!==t||o||null!==yo&&1&yo.memoizedState.tag){if(n.flags|=2048,Uo(9,Ao.bind(null,n,r,a,t),void 0,null),null===Pl)throw Error(i(349));0!=(30&ho)||Oo(n,t,a)}return a}function Oo(e,t,n){e.flags|=16384,e={getSnapshot:t,value:n},null===(t=vo.updateQueue)?(t={lastEffect:null,stores:null},vo.updateQueue=t,t.stores=[e]):null===(n=t.stores)?t.stores=[e]:n.push(e)}function Ao(e,t,n,r){t.value=n,t.getSnapshot=r,Do(t)&&Ro(e)}function Fo(e,t,n){return n((function(){Do(t)&&Ro(e)}))}function Do(e){var t=e.getSnapshot;e=e.value;try{var n=t();return!ur(e,n)}catch(e){return!0}}function Ro(e){var t=Ni(e,1);null!==t&&rs(t,e,1,-1)}function jo(e){var t=Co();return\"function\"==typeof e&&(e=e()),t.memoizedState=t.baseState=e,e={pending:null,interleaved:null,lanes:0,dispatch:null,lastRenderedReducer:Mo,lastRenderedState:e},t.queue=e,e=e.dispatch=nu.bind(null,vo,e),[t.memoizedState,e]}function Uo(e,t,n,r){return e={tag:e,create:t,destroy:n,deps:r,next:null},null===(t=vo.updateQueue)?(t={lastEffect:null,stores:null},vo.updateQueue=t,t.lastEffect=e.next=e):null===(n=t.lastEffect)?t.lastEffect=e.next=e:(r=n.next,n.next=e,e.next=r,t.lastEffect=e),e}function Io(){return To().memoizedState}function $o(e,t,n,r){var a=Co();vo.flags|=e,a.memoizedState=Uo(1|t,n,void 0,void 0===r?null:r)}function Bo(e,t,n,r){var a=To();r=void 0===r?null:r;var i=void 0;if(null!==go){var o=go.memoizedState;if(i=o.destroy,null!==r&&ko(r,o.deps))return void(a.memoizedState=Uo(t,n,i,r))}vo.flags|=e,a.memoizedState=Uo(1|t,n,i,r)}function Wo(e,t){return $o(8390656,8,e,t)}function Vo(e,t){return Bo(2048,8,e,t)}function Ho(e,t){return Bo(4,2,e,t)}function qo(e,t){return Bo(4,4,e,t)}function Qo(e,t){return\"function\"==typeof t?(e=e(),t(e),function(){t(null)}):null!=t?(e=e(),t.current=e,function(){t.current=null}):void 0}function Yo(e,t,n){return n=null!=n?n.concat([e]):null,Bo(4,4,Qo.bind(null,t,e),n)}function Go(){}function Ko(e,t){var n=To();t=void 0===t?null:t;var r=n.memoizedState;return null!==r&&null!==t&&ko(t,r[1])?r[0]:(n.memoizedState=[e,t],e)}function Zo(e,t){var n=To();t=void 0===t?null:t;var r=n.memoizedState;return null!==r&&null!==t&&ko(t,r[1])?r[0]:(e=e(),n.memoizedState=[e,t],e)}function Xo(e,t,n){return 0==(21&ho)?(e.baseState&&(e.baseState=!1,_u=!0),e.memoizedState=n):(ur(n,t)||(n=vt(),vo.lanes|=n,Rl|=n,e.baseState=!0),t)}function Jo(e,t){var n=bt;bt=0!==n&&4>n?n:4,e(!0);var r=po.transition;po.transition={};try{e(!1),t()}finally{bt=n,po.transition=r}}function eu(){return To().memoizedState}function tu(e,t,n){var r=ns(e);n={lane:r,action:n,hasEagerState:!1,eagerState:null,next:null},ru(e)?au(t,n):null!==(n=Mi(e,t,n,r))&&(rs(n,e,r,ts()),iu(n,t,r))}function nu(e,t,n){var r=ns(e),a={lane:r,action:n,hasEagerState:!1,eagerState:null,next:null};if(ru(e))au(t,a);else{var i=e.alternate;if(0===e.lanes&&(null===i||0===i.lanes)&&null!==(i=t.lastRenderedReducer))try{var o=t.lastRenderedState,u=i(o,n);if(a.hasEagerState=!0,a.eagerState=u,ur(u,o)){var l=t.interleaved;return null===l?(a.next=a,Ti(t)):(a.next=l.next,l.next=a),void(t.interleaved=a)}}catch(e){}null!==(n=Mi(e,t,a,r))&&(rs(n,e,r,a=ts()),iu(n,t,r))}}function ru(e){var t=e.alternate;return e===vo||null!==t&&t===vo}function au(e,t){bo=mo=!0;var n=e.pending;null===n?t.next=t:(t.next=n.next,n.next=t),e.pending=t}function iu(e,t,n){if(0!=(4194240&n)){var r=t.lanes;n|=r&=e.pendingLanes,t.lanes=n,mt(e,n)}}var ou={readContext:Ei,useCallback:xo,useContext:xo,useEffect:xo,useImperativeHandle:xo,useInsertionEffect:xo,useLayoutEffect:xo,useMemo:xo,useReducer:xo,useRef:xo,useState:xo,useDebugValue:xo,useDeferredValue:xo,useTransition:xo,useMutableSource:xo,useSyncExternalStore:xo,useId:xo,unstable_isNewReconciler:!1},uu={readContext:Ei,useCallback:function(e,t){return Co().memoizedState=[e,void 0===t?null:t],e},useContext:Ei,useEffect:Wo,useImperativeHandle:function(e,t,n){return n=null!=n?n.concat([e]):null,$o(4194308,4,Qo.bind(null,t,e),n)},useLayoutEffect:function(e,t){return $o(4194308,4,e,t)},useInsertionEffect:function(e,t){return $o(4,2,e,t)},useMemo:function(e,t){var n=Co();return t=void 0===t?null:t,e=e(),n.memoizedState=[e,t],e},useReducer:function(e,t,n){var r=Co();return t=void 0!==n?n(t):t,r.memoizedState=r.baseState=t,e={pending:null,interleaved:null,lanes:0,dispatch:null,lastRenderedReducer:e,lastRenderedState:t},r.queue=e,e=e.dispatch=tu.bind(null,vo,e),[r.memoizedState,e]},useRef:function(e){return e={current:e},Co().memoizedState=e},useState:jo,useDebugValue:Go,useDeferredValue:function(e){return Co().memoizedState=e},useTransition:function(){var e=jo(!1),t=e[0];return e=Jo.bind(null,e[1]),Co().memoizedState=e,[t,e]},useMutableSource:function(){},useSyncExternalStore:function(e,t,n){var r=vo,a=Co();if(ai){if(void 0===n)throw Error(i(407));n=n()}else{if(n=t(),null===Pl)throw Error(i(349));0!=(30&ho)||Oo(r,t,n)}a.memoizedState=n;var o={value:n,getSnapshot:t};return a.queue=o,Wo(Fo.bind(null,r,o,e),[e]),r.flags|=2048,Uo(9,Ao.bind(null,r,o,n,t),void 0,null),n},useId:function(){var e=Co(),t=Pl.identifierPrefix;if(ai){var n=Za;t=\":\"+t+\"R\"+(n=(Ka&~(1<<32-ot(Ka)-1)).toString(32)+n),0<(n=_o++)&&(t+=\"H\"+n.toString(32)),t+=\":\"}else t=\":\"+t+\"r\"+(n=wo++).toString(32)+\":\";return e.memoizedState=t},unstable_isNewReconciler:!1},lu={readContext:Ei,useCallback:Ko,useContext:Ei,useEffect:Vo,useImperativeHandle:Yo,useInsertionEffect:Ho,useLayoutEffect:qo,useMemo:Zo,useReducer:No,useRef:Io,useState:function(){return No(Mo)},useDebugValue:Go,useDeferredValue:function(e){return Xo(To(),go.memoizedState,e)},useTransition:function(){return[No(Mo)[0],To().memoizedState]},useMutableSource:zo,useSyncExternalStore:Lo,useId:eu,unstable_isNewReconciler:!1},su={readContext:Ei,useCallback:Ko,useContext:Ei,useEffect:Vo,useImperativeHandle:Yo,useInsertionEffect:Ho,useLayoutEffect:qo,useMemo:Zo,useReducer:Po,useRef:Io,useState:function(){return Po(Mo)},useDebugValue:Go,useDeferredValue:function(e){var t=To();return null===go?t.memoizedState=e:Xo(t,go.memoizedState,e)},useTransition:function(){return[Po(Mo)[0],To().memoizedState]},useMutableSource:zo,useSyncExternalStore:Lo,useId:eu,unstable_isNewReconciler:!1};function cu(e,t){try{var n=\"\",r=t;do{n+=$(r),r=r.return}while(r);var a=n}catch(e){a=\"\\nError generating stack: \"+e.message+\"\\n\"+e.stack}return{value:e,source:t,stack:a,digest:null}}function fu(e,t,n){return{value:e,source:null,stack:null!=n?n:null,digest:null!=t?t:null}}function pu(e,t){try{console.error(t.value)}catch(e){setTimeout((function(){throw e}))}}var du=\"function\"==typeof WeakMap?WeakMap:Map;function hu(e,t,n){(n=Oi(-1,n)).tag=3,n.payload={element:null};var r=t.value;return n.callback=function(){Hl||(Hl=!0,ql=r),pu(0,t)},n}function vu(e,t,n){(n=Oi(-1,n)).tag=3;var r=e.type.getDerivedStateFromError;if(\"function\"==typeof r){var a=t.value;n.payload=function(){return r(a)},n.callback=function(){pu(0,t)}}var i=e.stateNode;return null!==i&&\"function\"==typeof i.componentDidCatch&&(n.callback=function(){pu(0,t),\"function\"!=typeof r&&(null===Ql?Ql=new Set([this]):Ql.add(this));var e=t.stack;this.componentDidCatch(t.value,{componentStack:null!==e?e:\"\"})}),n}function gu(e,t,n){var r=e.pingCache;if(null===r){r=e.pingCache=new du;var a=new Set;r.set(t,a)}else void 0===(a=r.get(t))&&(a=new Set,r.set(t,a));a.has(n)||(a.add(n),e=Cs.bind(null,e,t,n),t.then(e,e))}function yu(e){do{var t;if((t=13===e.tag)&&(t=null===(t=e.memoizedState)||null!==t.dehydrated),t)return e;e=e.return}while(null!==e);return null}function mu(e,t,n,r,a){return 0==(1&e.mode)?(e===t?e.flags|=65536:(e.flags|=128,n.flags|=131072,n.flags&=-52805,1===n.tag&&(null===n.alternate?n.tag=17:((t=Oi(-1,1)).tag=2,Ai(n,t,1))),n.lanes|=1),e):(e.flags|=65536,e.lanes=a,e)}var bu=_.ReactCurrentOwner,_u=!1;function wu(e,t,n,r){t.child=null===e?Zi(t,null,n,r):Ki(t,e.child,n,r)}function xu(e,t,n,r,a){n=n.render;var i=t.ref;return Si(t,a),r=So(e,t,n,r,i,a),n=Eo(),null===e||_u?(ai&&n&&ei(t),t.flags|=1,wu(e,t,r,a),t.child):(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~a,Hu(e,t,a))}function ku(e,t,n,r,a){if(null===e){var i=n.type;return\"function\"!=typeof i||Os(i)||void 0!==i.defaultProps||null!==n.compare||void 0!==n.defaultProps?((e=Fs(n.type,null,r,t,t.mode,a)).ref=t.ref,e.return=t,t.child=e):(t.tag=15,t.type=i,Su(e,t,i,r,a))}if(i=e.child,0==(e.lanes&a)){var o=i.memoizedProps;if((n=null!==(n=n.compare)?n:lr)(o,r)&&e.ref===t.ref)return Hu(e,t,a)}return t.flags|=1,(e=As(i,r)).ref=t.ref,e.return=t,t.child=e}function Su(e,t,n,r,a){if(null!==e){var i=e.memoizedProps;if(lr(i,r)&&e.ref===t.ref){if(_u=!1,t.pendingProps=r=i,0==(e.lanes&a))return t.lanes=e.lanes,Hu(e,t,a);0!=(131072&e.flags)&&(_u=!0)}}return Tu(e,t,n,r,a)}function Eu(e,t,n){var r=t.pendingProps,a=r.children,i=null!==e?e.memoizedState:null;if(\"hidden\"===r.mode)if(0==(1&t.mode))t.memoizedState={baseLanes:0,cachePool:null,transitions:null},Ca(Al,Ol),Ol|=n;else{if(0==(1073741824&n))return e=null!==i?i.baseLanes|n:n,t.lanes=t.childLanes=1073741824,t.memoizedState={baseLanes:e,cachePool:null,transitions:null},t.updateQueue=null,Ca(Al,Ol),Ol|=e,null;t.memoizedState={baseLanes:0,cachePool:null,transitions:null},r=null!==i?i.baseLanes:n,Ca(Al,Ol),Ol|=r}else null!==i?(r=i.baseLanes|n,t.memoizedState=null):r=n,Ca(Al,Ol),Ol|=r;return wu(e,t,a,n),t.child}function Cu(e,t){var n=t.ref;(null===e&&null!==n||null!==e&&e.ref!==n)&&(t.flags|=512,t.flags|=2097152)}function Tu(e,t,n,r,a){var i=La(n)?Pa:Ma.current;return i=za(t,i),Si(t,a),n=So(e,t,n,r,i,a),r=Eo(),null===e||_u?(ai&&r&&ei(t),t.flags|=1,wu(e,t,n,a),t.child):(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~a,Hu(e,t,a))}function Mu(e,t,n,r,a){if(La(n)){var i=!0;Da(t)}else i=!1;if(Si(t,a),null===t.stateNode)Vu(e,t),Wi(t,n,r),Hi(t,n,r,a),r=!0;else if(null===e){var o=t.stateNode,u=t.memoizedProps;o.props=u;var l=o.context,s=n.contextType;s=\"object\"==typeof s&&null!==s?Ei(s):za(t,s=La(n)?Pa:Ma.current);var c=n.getDerivedStateFromProps,f=\"function\"==typeof c||\"function\"==typeof o.getSnapshotBeforeUpdate;f||\"function\"!=typeof o.UNSAFE_componentWillReceiveProps&&\"function\"!=typeof o.componentWillReceiveProps||(u!==r||l!==s)&&Vi(t,o,r,s),Pi=!1;var p=t.memoizedState;o.state=p,Ri(t,r,o,a),l=t.memoizedState,u!==r||p!==l||Na.current||Pi?(\"function\"==typeof c&&(Ii(t,n,c,r),l=t.memoizedState),(u=Pi||Bi(t,n,u,r,p,l,s))?(f||\"function\"!=typeof o.UNSAFE_componentWillMount&&\"function\"!=typeof o.componentWillMount||(\"function\"==typeof o.componentWillMount&&o.componentWillMount(),\"function\"==typeof o.UNSAFE_componentWillMount&&o.UNSAFE_componentWillMount()),\"function\"==typeof o.componentDidMount&&(t.flags|=4194308)):(\"function\"==typeof o.componentDidMount&&(t.flags|=4194308),t.memoizedProps=r,t.memoizedState=l),o.props=r,o.state=l,o.context=s,r=u):(\"function\"==typeof o.componentDidMount&&(t.flags|=4194308),r=!1)}else{o=t.stateNode,Li(e,t),u=t.memoizedProps,s=t.type===t.elementType?u:gi(t.type,u),o.props=s,f=t.pendingProps,p=o.context,l=\"object\"==typeof(l=n.contextType)&&null!==l?Ei(l):za(t,l=La(n)?Pa:Ma.current);var d=n.getDerivedStateFromProps;(c=\"function\"==typeof d||\"function\"==typeof o.getSnapshotBeforeUpdate)||\"function\"!=typeof o.UNSAFE_componentWillReceiveProps&&\"function\"!=typeof o.componentWillReceiveProps||(u!==f||p!==l)&&Vi(t,o,r,l),Pi=!1,p=t.memoizedState,o.state=p,Ri(t,r,o,a);var h=t.memoizedState;u!==f||p!==h||Na.current||Pi?(\"function\"==typeof d&&(Ii(t,n,d,r),h=t.memoizedState),(s=Pi||Bi(t,n,s,r,p,h,l)||!1)?(c||\"function\"!=typeof o.UNSAFE_componentWillUpdate&&\"function\"!=typeof o.componentWillUpdate||(\"function\"==typeof o.componentWillUpdate&&o.componentWillUpdate(r,h,l),\"function\"==typeof o.UNSAFE_componentWillUpdate&&o.UNSAFE_componentWillUpdate(r,h,l)),\"function\"==typeof o.componentDidUpdate&&(t.flags|=4),\"function\"==typeof o.getSnapshotBeforeUpdate&&(t.flags|=1024)):(\"function\"!=typeof o.componentDidUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=4),\"function\"!=typeof o.getSnapshotBeforeUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=1024),t.memoizedProps=r,t.memoizedState=h),o.props=r,o.state=h,o.context=l,r=s):(\"function\"!=typeof o.componentDidUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=4),\"function\"!=typeof o.getSnapshotBeforeUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=1024),r=!1)}return Nu(e,t,n,r,i,a)}function Nu(e,t,n,r,a,i){Cu(e,t);var o=0!=(128&t.flags);if(!r&&!o)return a&&Ra(t,n,!1),Hu(e,t,i);r=t.stateNode,bu.current=t;var u=o&&\"function\"!=typeof n.getDerivedStateFromError?null:r.render();return t.flags|=1,null!==e&&o?(t.child=Ki(t,e.child,null,i),t.child=Ki(t,null,u,i)):wu(e,t,u,i),t.memoizedState=r.state,a&&Ra(t,n,!0),t.child}function Pu(e){var t=e.stateNode;t.pendingContext?Aa(0,t.pendingContext,t.pendingContext!==t.context):t.context&&Aa(0,t.context,!1),ro(e,t.containerInfo)}function zu(e,t,n,r,a){return di(),hi(a),t.flags|=256,wu(e,t,n,r),t.child}var Lu,Ou,Au,Fu,Du={dehydrated:null,treeContext:null,retryLane:0};function Ru(e){return{baseLanes:e,cachePool:null,transitions:null}}function ju(e,t,n){var r,a=t.pendingProps,o=uo.current,u=!1,l=0!=(128&t.flags);if((r=l)||(r=(null===e||null!==e.memoizedState)&&0!=(2&o)),r?(u=!0,t.flags&=-129):null!==e&&null===e.memoizedState||(o|=1),Ca(uo,1&o),null===e)return si(t),null!==(e=t.memoizedState)&&null!==(e=e.dehydrated)?(0==(1&t.mode)?t.lanes=1:\"$!\"===e.data?t.lanes=8:t.lanes=1073741824,null):(l=a.children,e=a.fallback,u?(a=t.mode,u=t.child,l={mode:\"hidden\",children:l},0==(1&a)&&null!==u?(u.childLanes=0,u.pendingProps=l):u=Rs(l,a,0,null),e=Ds(e,a,n,null),u.return=t,e.return=t,u.sibling=e,t.child=u,t.child.memoizedState=Ru(n),t.memoizedState=Du,e):Uu(t,l));if(null!==(o=e.memoizedState)&&null!==(r=o.dehydrated))return function(e,t,n,r,a,o,u){if(n)return 256&t.flags?(t.flags&=-257,Iu(e,t,u,r=fu(Error(i(422))))):null!==t.memoizedState?(t.child=e.child,t.flags|=128,null):(o=r.fallback,a=t.mode,r=Rs({mode:\"visible\",children:r.children},a,0,null),(o=Ds(o,a,u,null)).flags|=2,r.return=t,o.return=t,r.sibling=o,t.child=r,0!=(1&t.mode)&&Ki(t,e.child,null,u),t.child.memoizedState=Ru(u),t.memoizedState=Du,o);if(0==(1&t.mode))return Iu(e,t,u,null);if(\"$!\"===a.data){if(r=a.nextSibling&&a.nextSibling.dataset)var l=r.dgst;return r=l,Iu(e,t,u,r=fu(o=Error(i(419)),r,void 0))}if(l=0!=(u&e.childLanes),_u||l){if(null!==(r=Pl)){switch(u&-u){case 4:a=2;break;case 16:a=8;break;case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:a=32;break;case 536870912:a=268435456;break;default:a=0}0!==(a=0!=(a&(r.suspendedLanes|u))?0:a)&&a!==o.retryLane&&(o.retryLane=a,Ni(e,a),rs(r,e,a,-1))}return gs(),Iu(e,t,u,r=fu(Error(i(421))))}return\"$?\"===a.data?(t.flags|=128,t.child=e.child,t=Ms.bind(null,e),a._reactRetry=t,null):(e=o.treeContext,ri=sa(a.nextSibling),ni=t,ai=!0,ii=null,null!==e&&(Qa[Ya++]=Ka,Qa[Ya++]=Za,Qa[Ya++]=Ga,Ka=e.id,Za=e.overflow,Ga=t),(t=Uu(t,r.children)).flags|=4096,t)}(e,t,l,a,r,o,n);if(u){u=a.fallback,l=t.mode,r=(o=e.child).sibling;var s={mode:\"hidden\",children:a.children};return 0==(1&l)&&t.child!==o?((a=t.child).childLanes=0,a.pendingProps=s,t.deletions=null):(a=As(o,s)).subtreeFlags=14680064&o.subtreeFlags,null!==r?u=As(r,u):(u=Ds(u,l,n,null)).flags|=2,u.return=t,a.return=t,a.sibling=u,t.child=a,a=u,u=t.child,l=null===(l=e.child.memoizedState)?Ru(n):{baseLanes:l.baseLanes|n,cachePool:null,transitions:l.transitions},u.memoizedState=l,u.childLanes=e.childLanes&~n,t.memoizedState=Du,a}return e=(u=e.child).sibling,a=As(u,{mode:\"visible\",children:a.children}),0==(1&t.mode)&&(a.lanes=n),a.return=t,a.sibling=null,null!==e&&(null===(n=t.deletions)?(t.deletions=[e],t.flags|=16):n.push(e)),t.child=a,t.memoizedState=null,a}function Uu(e,t){return(t=Rs({mode:\"visible\",children:t},e.mode,0,null)).return=e,e.child=t}function Iu(e,t,n,r){return null!==r&&hi(r),Ki(t,e.child,null,n),(e=Uu(t,t.pendingProps.children)).flags|=2,t.memoizedState=null,e}function $u(e,t,n){e.lanes|=t;var r=e.alternate;null!==r&&(r.lanes|=t),ki(e.return,t,n)}function Bu(e,t,n,r,a){var i=e.memoizedState;null===i?e.memoizedState={isBackwards:t,rendering:null,renderingStartTime:0,last:r,tail:n,tailMode:a}:(i.isBackwards=t,i.rendering=null,i.renderingStartTime=0,i.last=r,i.tail=n,i.tailMode=a)}function Wu(e,t,n){var r=t.pendingProps,a=r.revealOrder,i=r.tail;if(wu(e,t,r.children,n),0!=(2&(r=uo.current)))r=1&r|2,t.flags|=128;else{if(null!==e&&0!=(128&e.flags))e:for(e=t.child;null!==e;){if(13===e.tag)null!==e.memoizedState&&$u(e,n,t);else if(19===e.tag)$u(e,n,t);else if(null!==e.child){e.child.return=e,e=e.child;continue}if(e===t)break e;for(;null===e.sibling;){if(null===e.return||e.return===t)break e;e=e.return}e.sibling.return=e.return,e=e.sibling}r&=1}if(Ca(uo,r),0==(1&t.mode))t.memoizedState=null;else switch(a){case\"forwards\":for(n=t.child,a=null;null!==n;)null!==(e=n.alternate)&&null===lo(e)&&(a=n),n=n.sibling;null===(n=a)?(a=t.child,t.child=null):(a=n.sibling,n.sibling=null),Bu(t,!1,a,n,i);break;case\"backwards\":for(n=null,a=t.child,t.child=null;null!==a;){if(null!==(e=a.alternate)&&null===lo(e)){t.child=a;break}e=a.sibling,a.sibling=n,n=a,a=e}Bu(t,!0,n,null,i);break;case\"together\":Bu(t,!1,null,null,void 0);break;default:t.memoizedState=null}return t.child}function Vu(e,t){0==(1&t.mode)&&null!==e&&(e.alternate=null,t.alternate=null,t.flags|=2)}function Hu(e,t,n){if(null!==e&&(t.dependencies=e.dependencies),Rl|=t.lanes,0==(n&t.childLanes))return null;if(null!==e&&t.child!==e.child)throw Error(i(153));if(null!==t.child){for(n=As(e=t.child,e.pendingProps),t.child=n,n.return=t;null!==e.sibling;)e=e.sibling,(n=n.sibling=As(e,e.pendingProps)).return=t;n.sibling=null}return t.child}function qu(e,t){if(!ai)switch(e.tailMode){case\"hidden\":t=e.tail;for(var n=null;null!==t;)null!==t.alternate&&(n=t),t=t.sibling;null===n?e.tail=null:n.sibling=null;break;case\"collapsed\":n=e.tail;for(var r=null;null!==n;)null!==n.alternate&&(r=n),n=n.sibling;null===r?t||null===e.tail?e.tail=null:e.tail.sibling=null:r.sibling=null}}function Qu(e){var t=null!==e.alternate&&e.alternate.child===e.child,n=0,r=0;if(t)for(var a=e.child;null!==a;)n|=a.lanes|a.childLanes,r|=14680064&a.subtreeFlags,r|=14680064&a.flags,a.return=e,a=a.sibling;else for(a=e.child;null!==a;)n|=a.lanes|a.childLanes,r|=a.subtreeFlags,r|=a.flags,a.return=e,a=a.sibling;return e.subtreeFlags|=r,e.childLanes=n,t}function Yu(e,t,n){var r=t.pendingProps;switch(ti(t),t.tag){case 2:case 16:case 15:case 0:case 11:case 7:case 8:case 12:case 9:case 14:return Qu(t),null;case 1:case 17:return La(t.type)&&Oa(),Qu(t),null;case 3:return r=t.stateNode,ao(),Ea(Na),Ea(Ma),co(),r.pendingContext&&(r.context=r.pendingContext,r.pendingContext=null),null!==e&&null!==e.child||(fi(t)?t.flags|=4:null===e||e.memoizedState.isDehydrated&&0==(256&t.flags)||(t.flags|=1024,null!==ii&&(us(ii),ii=null))),Ou(e,t),Qu(t),null;case 5:oo(t);var a=no(to.current);if(n=t.type,null!==e&&null!=t.stateNode)Au(e,t,n,r,a),e.ref!==t.ref&&(t.flags|=512,t.flags|=2097152);else{if(!r){if(null===t.stateNode)throw Error(i(166));return Qu(t),null}if(e=no(Ji.current),fi(t)){r=t.stateNode,n=t.type;var o=t.memoizedProps;switch(r[pa]=t,r[da]=o,e=0!=(1&t.mode),n){case\"dialog\":Ur(\"cancel\",r),Ur(\"close\",r);break;case\"iframe\":case\"object\":case\"embed\":Ur(\"load\",r);break;case\"video\":case\"audio\":for(a=0;a<Fr.length;a++)Ur(Fr[a],r);break;case\"source\":Ur(\"error\",r);break;case\"img\":case\"image\":case\"link\":Ur(\"error\",r),Ur(\"load\",r);break;case\"details\":Ur(\"toggle\",r);break;case\"input\":K(r,o),Ur(\"invalid\",r);break;case\"select\":r._wrapperState={wasMultiple:!!o.multiple},Ur(\"invalid\",r);break;case\"textarea\":ae(r,o),Ur(\"invalid\",r)}for(var l in me(n,o),a=null,o)if(o.hasOwnProperty(l)){var s=o[l];\"children\"===l?\"string\"==typeof s?r.textContent!==s&&(!0!==o.suppressHydrationWarning&&Xr(r.textContent,s,e),a=[\"children\",s]):\"number\"==typeof s&&r.textContent!==\"\"+s&&(!0!==o.suppressHydrationWarning&&Xr(r.textContent,s,e),a=[\"children\",\"\"+s]):u.hasOwnProperty(l)&&null!=s&&\"onScroll\"===l&&Ur(\"scroll\",r)}switch(n){case\"input\":q(r),J(r,o,!0);break;case\"textarea\":q(r),oe(r);break;case\"select\":case\"option\":break;default:\"function\"==typeof o.onClick&&(r.onclick=Jr)}r=a,t.updateQueue=r,null!==r&&(t.flags|=4)}else{l=9===a.nodeType?a:a.ownerDocument,\"http://www.w3.org/1999/xhtml\"===e&&(e=ue(n)),\"http://www.w3.org/1999/xhtml\"===e?\"script\"===n?((e=l.createElement(\"div\")).innerHTML=\"<script><\\/script>\",e=e.removeChild(e.firstChild)):\"string\"==typeof r.is?e=l.createElement(n,{is:r.is}):(e=l.createElement(n),\"select\"===n&&(l=e,r.multiple?l.multiple=!0:r.size&&(l.size=r.size))):e=l.createElementNS(e,n),e[pa]=t,e[da]=r,Lu(e,t,!1,!1),t.stateNode=e;e:{switch(l=be(n,r),n){case\"dialog\":Ur(\"cancel\",e),Ur(\"close\",e),a=r;break;case\"iframe\":case\"object\":case\"embed\":Ur(\"load\",e),a=r;break;case\"video\":case\"audio\":for(a=0;a<Fr.length;a++)Ur(Fr[a],e);a=r;break;case\"source\":Ur(\"error\",e),a=r;break;case\"img\":case\"image\":case\"link\":Ur(\"error\",e),Ur(\"load\",e),a=r;break;case\"details\":Ur(\"toggle\",e),a=r;break;case\"input\":K(e,r),a=G(e,r),Ur(\"invalid\",e);break;case\"option\":default:a=r;break;case\"select\":e._wrapperState={wasMultiple:!!r.multiple},a=R({},r,{value:void 0}),Ur(\"invalid\",e);break;case\"textarea\":ae(e,r),a=re(e,r),Ur(\"invalid\",e)}for(o in me(n,a),s=a)if(s.hasOwnProperty(o)){var c=s[o];\"style\"===o?ge(e,c):\"dangerouslySetInnerHTML\"===o?null!=(c=c?c.__html:void 0)&&fe(e,c):\"children\"===o?\"string\"==typeof c?(\"textarea\"!==n||\"\"!==c)&&pe(e,c):\"number\"==typeof c&&pe(e,\"\"+c):\"suppressContentEditableWarning\"!==o&&\"suppressHydrationWarning\"!==o&&\"autoFocus\"!==o&&(u.hasOwnProperty(o)?null!=c&&\"onScroll\"===o&&Ur(\"scroll\",e):null!=c&&b(e,o,c,l))}switch(n){case\"input\":q(e),J(e,r,!1);break;case\"textarea\":q(e),oe(e);break;case\"option\":null!=r.value&&e.setAttribute(\"value\",\"\"+V(r.value));break;case\"select\":e.multiple=!!r.multiple,null!=(o=r.value)?ne(e,!!r.multiple,o,!1):null!=r.defaultValue&&ne(e,!!r.multiple,r.defaultValue,!0);break;default:\"function\"==typeof a.onClick&&(e.onclick=Jr)}switch(n){case\"button\":case\"input\":case\"select\":case\"textarea\":r=!!r.autoFocus;break e;case\"img\":r=!0;break e;default:r=!1}}r&&(t.flags|=4)}null!==t.ref&&(t.flags|=512,t.flags|=2097152)}return Qu(t),null;case 6:if(e&&null!=t.stateNode)Fu(e,t,e.memoizedProps,r);else{if(\"string\"!=typeof r&&null===t.stateNode)throw Error(i(166));if(n=no(to.current),no(Ji.current),fi(t)){if(r=t.stateNode,n=t.memoizedProps,r[pa]=t,(o=r.nodeValue!==n)&&null!==(e=ni))switch(e.tag){case 3:Xr(r.nodeValue,n,0!=(1&e.mode));break;case 5:!0!==e.memoizedProps.suppressHydrationWarning&&Xr(r.nodeValue,n,0!=(1&e.mode))}o&&(t.flags|=4)}else(r=(9===n.nodeType?n:n.ownerDocument).createTextNode(r))[pa]=t,t.stateNode=r}return Qu(t),null;case 13:if(Ea(uo),r=t.memoizedState,null===e||null!==e.memoizedState&&null!==e.memoizedState.dehydrated){if(ai&&null!==ri&&0!=(1&t.mode)&&0==(128&t.flags))pi(),di(),t.flags|=98560,o=!1;else if(o=fi(t),null!==r&&null!==r.dehydrated){if(null===e){if(!o)throw Error(i(318));if(!(o=null!==(o=t.memoizedState)?o.dehydrated:null))throw Error(i(317));o[pa]=t}else di(),0==(128&t.flags)&&(t.memoizedState=null),t.flags|=4;Qu(t),o=!1}else null!==ii&&(us(ii),ii=null),o=!0;if(!o)return 65536&t.flags?t:null}return 0!=(128&t.flags)?(t.lanes=n,t):((r=null!==r)!=(null!==e&&null!==e.memoizedState)&&r&&(t.child.flags|=8192,0!=(1&t.mode)&&(null===e||0!=(1&uo.current)?0===Fl&&(Fl=3):gs())),null!==t.updateQueue&&(t.flags|=4),Qu(t),null);case 4:return ao(),Ou(e,t),null===e&&Br(t.stateNode.containerInfo),Qu(t),null;case 10:return xi(t.type._context),Qu(t),null;case 19:if(Ea(uo),null===(o=t.memoizedState))return Qu(t),null;if(r=0!=(128&t.flags),null===(l=o.rendering))if(r)qu(o,!1);else{if(0!==Fl||null!==e&&0!=(128&e.flags))for(e=t.child;null!==e;){if(null!==(l=lo(e))){for(t.flags|=128,qu(o,!1),null!==(r=l.updateQueue)&&(t.updateQueue=r,t.flags|=4),t.subtreeFlags=0,r=n,n=t.child;null!==n;)e=r,(o=n).flags&=14680066,null===(l=o.alternate)?(o.childLanes=0,o.lanes=e,o.child=null,o.subtreeFlags=0,o.memoizedProps=null,o.memoizedState=null,o.updateQueue=null,o.dependencies=null,o.stateNode=null):(o.childLanes=l.childLanes,o.lanes=l.lanes,o.child=l.child,o.subtreeFlags=0,o.deletions=null,o.memoizedProps=l.memoizedProps,o.memoizedState=l.memoizedState,o.updateQueue=l.updateQueue,o.type=l.type,e=l.dependencies,o.dependencies=null===e?null:{lanes:e.lanes,firstContext:e.firstContext}),n=n.sibling;return Ca(uo,1&uo.current|2),t.child}e=e.sibling}null!==o.tail&&Ze()>Wl&&(t.flags|=128,r=!0,qu(o,!1),t.lanes=4194304)}else{if(!r)if(null!==(e=lo(l))){if(t.flags|=128,r=!0,null!==(n=e.updateQueue)&&(t.updateQueue=n,t.flags|=4),qu(o,!0),null===o.tail&&\"hidden\"===o.tailMode&&!l.alternate&&!ai)return Qu(t),null}else 2*Ze()-o.renderingStartTime>Wl&&1073741824!==n&&(t.flags|=128,r=!0,qu(o,!1),t.lanes=4194304);o.isBackwards?(l.sibling=t.child,t.child=l):(null!==(n=o.last)?n.sibling=l:t.child=l,o.last=l)}return null!==o.tail?(t=o.tail,o.rendering=t,o.tail=t.sibling,o.renderingStartTime=Ze(),t.sibling=null,n=uo.current,Ca(uo,r?1&n|2:1&n),t):(Qu(t),null);case 22:case 23:return ps(),r=null!==t.memoizedState,null!==e&&null!==e.memoizedState!==r&&(t.flags|=8192),r&&0!=(1&t.mode)?0!=(1073741824&Ol)&&(Qu(t),6&t.subtreeFlags&&(t.flags|=8192)):Qu(t),null;case 24:case 25:return null}throw Error(i(156,t.tag))}function Gu(e,t){switch(ti(t),t.tag){case 1:return La(t.type)&&Oa(),65536&(e=t.flags)?(t.flags=-65537&e|128,t):null;case 3:return ao(),Ea(Na),Ea(Ma),co(),0!=(65536&(e=t.flags))&&0==(128&e)?(t.flags=-65537&e|128,t):null;case 5:return oo(t),null;case 13:if(Ea(uo),null!==(e=t.memoizedState)&&null!==e.dehydrated){if(null===t.alternate)throw Error(i(340));di()}return 65536&(e=t.flags)?(t.flags=-65537&e|128,t):null;case 19:return Ea(uo),null;case 4:return ao(),null;case 10:return xi(t.type._context),null;case 22:case 23:return ps(),null;default:return null}}Lu=function(e,t){for(var n=t.child;null!==n;){if(5===n.tag||6===n.tag)e.appendChild(n.stateNode);else if(4!==n.tag&&null!==n.child){n.child.return=n,n=n.child;continue}if(n===t)break;for(;null===n.sibling;){if(null===n.return||n.return===t)return;n=n.return}n.sibling.return=n.return,n=n.sibling}},Ou=function(){},Au=function(e,t,n,r){var a=e.memoizedProps;if(a!==r){e=t.stateNode,no(Ji.current);var i,o=null;switch(n){case\"input\":a=G(e,a),r=G(e,r),o=[];break;case\"select\":a=R({},a,{value:void 0}),r=R({},r,{value:void 0}),o=[];break;case\"textarea\":a=re(e,a),r=re(e,r),o=[];break;default:\"function\"!=typeof a.onClick&&\"function\"==typeof r.onClick&&(e.onclick=Jr)}for(c in me(n,r),n=null,a)if(!r.hasOwnProperty(c)&&a.hasOwnProperty(c)&&null!=a[c])if(\"style\"===c){var l=a[c];for(i in l)l.hasOwnProperty(i)&&(n||(n={}),n[i]=\"\")}else\"dangerouslySetInnerHTML\"!==c&&\"children\"!==c&&\"suppressContentEditableWarning\"!==c&&\"suppressHydrationWarning\"!==c&&\"autoFocus\"!==c&&(u.hasOwnProperty(c)?o||(o=[]):(o=o||[]).push(c,null));for(c in r){var s=r[c];if(l=null!=a?a[c]:void 0,r.hasOwnProperty(c)&&s!==l&&(null!=s||null!=l))if(\"style\"===c)if(l){for(i in l)!l.hasOwnProperty(i)||s&&s.hasOwnProperty(i)||(n||(n={}),n[i]=\"\");for(i in s)s.hasOwnProperty(i)&&l[i]!==s[i]&&(n||(n={}),n[i]=s[i])}else n||(o||(o=[]),o.push(c,n)),n=s;else\"dangerouslySetInnerHTML\"===c?(s=s?s.__html:void 0,l=l?l.__html:void 0,null!=s&&l!==s&&(o=o||[]).push(c,s)):\"children\"===c?\"string\"!=typeof s&&\"number\"!=typeof s||(o=o||[]).push(c,\"\"+s):\"suppressContentEditableWarning\"!==c&&\"suppressHydrationWarning\"!==c&&(u.hasOwnProperty(c)?(null!=s&&\"onScroll\"===c&&Ur(\"scroll\",e),o||l===s||(o=[])):(o=o||[]).push(c,s))}n&&(o=o||[]).push(\"style\",n);var c=o;(t.updateQueue=c)&&(t.flags|=4)}},Fu=function(e,t,n,r){n!==r&&(t.flags|=4)};var Ku=!1,Zu=!1,Xu=\"function\"==typeof WeakSet?WeakSet:Set,Ju=null;function el(e,t){var n=e.ref;if(null!==n)if(\"function\"==typeof n)try{n(null)}catch(n){Es(e,t,n)}else n.current=null}function tl(e,t,n){try{n()}catch(n){Es(e,t,n)}}var nl=!1;function rl(e,t,n){var r=t.updateQueue;if(null!==(r=null!==r?r.lastEffect:null)){var a=r=r.next;do{if((a.tag&e)===e){var i=a.destroy;a.destroy=void 0,void 0!==i&&tl(t,n,i)}a=a.next}while(a!==r)}}function al(e,t){if(null!==(t=null!==(t=t.updateQueue)?t.lastEffect:null)){var n=t=t.next;do{if((n.tag&e)===e){var r=n.create;n.destroy=r()}n=n.next}while(n!==t)}}function il(e){var t=e.ref;if(null!==t){var n=e.stateNode;e.tag,e=n,\"function\"==typeof t?t(e):t.current=e}}function ol(e){var t=e.alternate;null!==t&&(e.alternate=null,ol(t)),e.child=null,e.deletions=null,e.sibling=null,5===e.tag&&null!==(t=e.stateNode)&&(delete t[pa],delete t[da],delete t[va],delete t[ga],delete t[ya]),e.stateNode=null,e.return=null,e.dependencies=null,e.memoizedProps=null,e.memoizedState=null,e.pendingProps=null,e.stateNode=null,e.updateQueue=null}function ul(e){return 5===e.tag||3===e.tag||4===e.tag}function ll(e){e:for(;;){for(;null===e.sibling;){if(null===e.return||ul(e.return))return null;e=e.return}for(e.sibling.return=e.return,e=e.sibling;5!==e.tag&&6!==e.tag&&18!==e.tag;){if(2&e.flags)continue e;if(null===e.child||4===e.tag)continue e;e.child.return=e,e=e.child}if(!(2&e.flags))return e.stateNode}}function sl(e,t,n){var r=e.tag;if(5===r||6===r)e=e.stateNode,t?8===n.nodeType?n.parentNode.insertBefore(e,t):n.insertBefore(e,t):(8===n.nodeType?(t=n.parentNode).insertBefore(e,n):(t=n).appendChild(e),null!=(n=n._reactRootContainer)||null!==t.onclick||(t.onclick=Jr));else if(4!==r&&null!==(e=e.child))for(sl(e,t,n),e=e.sibling;null!==e;)sl(e,t,n),e=e.sibling}function cl(e,t,n){var r=e.tag;if(5===r||6===r)e=e.stateNode,t?n.insertBefore(e,t):n.appendChild(e);else if(4!==r&&null!==(e=e.child))for(cl(e,t,n),e=e.sibling;null!==e;)cl(e,t,n),e=e.sibling}var fl=null,pl=!1;function dl(e,t,n){for(n=n.child;null!==n;)hl(e,t,n),n=n.sibling}function hl(e,t,n){if(it&&\"function\"==typeof it.onCommitFiberUnmount)try{it.onCommitFiberUnmount(at,n)}catch(e){}switch(n.tag){case 5:Zu||el(n,t);case 6:var r=fl,a=pl;fl=null,dl(e,t,n),pl=a,null!==(fl=r)&&(pl?(e=fl,n=n.stateNode,8===e.nodeType?e.parentNode.removeChild(n):e.removeChild(n)):fl.removeChild(n.stateNode));break;case 18:null!==fl&&(pl?(e=fl,n=n.stateNode,8===e.nodeType?la(e.parentNode,n):1===e.nodeType&&la(e,n),Bt(e)):la(fl,n.stateNode));break;case 4:r=fl,a=pl,fl=n.stateNode.containerInfo,pl=!0,dl(e,t,n),fl=r,pl=a;break;case 0:case 11:case 14:case 15:if(!Zu&&null!==(r=n.updateQueue)&&null!==(r=r.lastEffect)){a=r=r.next;do{var i=a,o=i.destroy;i=i.tag,void 0!==o&&(0!=(2&i)||0!=(4&i))&&tl(n,t,o),a=a.next}while(a!==r)}dl(e,t,n);break;case 1:if(!Zu&&(el(n,t),\"function\"==typeof(r=n.stateNode).componentWillUnmount))try{r.props=n.memoizedProps,r.state=n.memoizedState,r.componentWillUnmount()}catch(e){Es(n,t,e)}dl(e,t,n);break;case 21:dl(e,t,n);break;case 22:1&n.mode?(Zu=(r=Zu)||null!==n.memoizedState,dl(e,t,n),Zu=r):dl(e,t,n);break;default:dl(e,t,n)}}function vl(e){var t=e.updateQueue;if(null!==t){e.updateQueue=null;var n=e.stateNode;null===n&&(n=e.stateNode=new Xu),t.forEach((function(t){var r=Ns.bind(null,e,t);n.has(t)||(n.add(t),t.then(r,r))}))}}function gl(e,t){var n=t.deletions;if(null!==n)for(var r=0;r<n.length;r++){var a=n[r];try{var o=e,u=t,l=u;e:for(;null!==l;){switch(l.tag){case 5:fl=l.stateNode,pl=!1;break e;case 3:case 4:fl=l.stateNode.containerInfo,pl=!0;break e}l=l.return}if(null===fl)throw Error(i(160));hl(o,u,a),fl=null,pl=!1;var s=a.alternate;null!==s&&(s.return=null),a.return=null}catch(e){Es(a,t,e)}}if(12854&t.subtreeFlags)for(t=t.child;null!==t;)yl(t,e),t=t.sibling}function yl(e,t){var n=e.alternate,r=e.flags;switch(e.tag){case 0:case 11:case 14:case 15:if(gl(t,e),ml(e),4&r){try{rl(3,e,e.return),al(3,e)}catch(t){Es(e,e.return,t)}try{rl(5,e,e.return)}catch(t){Es(e,e.return,t)}}break;case 1:gl(t,e),ml(e),512&r&&null!==n&&el(n,n.return);break;case 5:if(gl(t,e),ml(e),512&r&&null!==n&&el(n,n.return),32&e.flags){var a=e.stateNode;try{pe(a,\"\")}catch(t){Es(e,e.return,t)}}if(4&r&&null!=(a=e.stateNode)){var o=e.memoizedProps,u=null!==n?n.memoizedProps:o,l=e.type,s=e.updateQueue;if(e.updateQueue=null,null!==s)try{\"input\"===l&&\"radio\"===o.type&&null!=o.name&&Z(a,o),be(l,u);var c=be(l,o);for(u=0;u<s.length;u+=2){var f=s[u],p=s[u+1];\"style\"===f?ge(a,p):\"dangerouslySetInnerHTML\"===f?fe(a,p):\"children\"===f?pe(a,p):b(a,f,p,c)}switch(l){case\"input\":X(a,o);break;case\"textarea\":ie(a,o);break;case\"select\":var d=a._wrapperState.wasMultiple;a._wrapperState.wasMultiple=!!o.multiple;var h=o.value;null!=h?ne(a,!!o.multiple,h,!1):d!==!!o.multiple&&(null!=o.defaultValue?ne(a,!!o.multiple,o.defaultValue,!0):ne(a,!!o.multiple,o.multiple?[]:\"\",!1))}a[da]=o}catch(t){Es(e,e.return,t)}}break;case 6:if(gl(t,e),ml(e),4&r){if(null===e.stateNode)throw Error(i(162));a=e.stateNode,o=e.memoizedProps;try{a.nodeValue=o}catch(t){Es(e,e.return,t)}}break;case 3:if(gl(t,e),ml(e),4&r&&null!==n&&n.memoizedState.isDehydrated)try{Bt(t.containerInfo)}catch(t){Es(e,e.return,t)}break;case 4:default:gl(t,e),ml(e);break;case 13:gl(t,e),ml(e),8192&(a=e.child).flags&&(o=null!==a.memoizedState,a.stateNode.isHidden=o,!o||null!==a.alternate&&null!==a.alternate.memoizedState||(Bl=Ze())),4&r&&vl(e);break;case 22:if(f=null!==n&&null!==n.memoizedState,1&e.mode?(Zu=(c=Zu)||f,gl(t,e),Zu=c):gl(t,e),ml(e),8192&r){if(c=null!==e.memoizedState,(e.stateNode.isHidden=c)&&!f&&0!=(1&e.mode))for(Ju=e,f=e.child;null!==f;){for(p=Ju=f;null!==Ju;){switch(h=(d=Ju).child,d.tag){case 0:case 11:case 14:case 15:rl(4,d,d.return);break;case 1:el(d,d.return);var v=d.stateNode;if(\"function\"==typeof v.componentWillUnmount){r=d,n=d.return;try{t=r,v.props=t.memoizedProps,v.state=t.memoizedState,v.componentWillUnmount()}catch(e){Es(r,n,e)}}break;case 5:el(d,d.return);break;case 22:if(null!==d.memoizedState){xl(p);continue}}null!==h?(h.return=d,Ju=h):xl(p)}f=f.sibling}e:for(f=null,p=e;;){if(5===p.tag){if(null===f){f=p;try{a=p.stateNode,c?\"function\"==typeof(o=a.style).setProperty?o.setProperty(\"display\",\"none\",\"important\"):o.display=\"none\":(l=p.stateNode,u=null!=(s=p.memoizedProps.style)&&s.hasOwnProperty(\"display\")?s.display:null,l.style.display=ve(\"display\",u))}catch(t){Es(e,e.return,t)}}}else if(6===p.tag){if(null===f)try{p.stateNode.nodeValue=c?\"\":p.memoizedProps}catch(t){Es(e,e.return,t)}}else if((22!==p.tag&&23!==p.tag||null===p.memoizedState||p===e)&&null!==p.child){p.child.return=p,p=p.child;continue}if(p===e)break e;for(;null===p.sibling;){if(null===p.return||p.return===e)break e;f===p&&(f=null),p=p.return}f===p&&(f=null),p.sibling.return=p.return,p=p.sibling}}break;case 19:gl(t,e),ml(e),4&r&&vl(e);case 21:}}function ml(e){var t=e.flags;if(2&t){try{e:{for(var n=e.return;null!==n;){if(ul(n)){var r=n;break e}n=n.return}throw Error(i(160))}switch(r.tag){case 5:var a=r.stateNode;32&r.flags&&(pe(a,\"\"),r.flags&=-33),cl(e,ll(e),a);break;case 3:case 4:var o=r.stateNode.containerInfo;sl(e,ll(e),o);break;default:throw Error(i(161))}}catch(t){Es(e,e.return,t)}e.flags&=-3}4096&t&&(e.flags&=-4097)}function bl(e,t,n){Ju=e,_l(e,t,n)}function _l(e,t,n){for(var r=0!=(1&e.mode);null!==Ju;){var a=Ju,i=a.child;if(22===a.tag&&r){var o=null!==a.memoizedState||Ku;if(!o){var u=a.alternate,l=null!==u&&null!==u.memoizedState||Zu;u=Ku;var s=Zu;if(Ku=o,(Zu=l)&&!s)for(Ju=a;null!==Ju;)l=(o=Ju).child,22===o.tag&&null!==o.memoizedState?kl(a):null!==l?(l.return=o,Ju=l):kl(a);for(;null!==i;)Ju=i,_l(i,t,n),i=i.sibling;Ju=a,Ku=u,Zu=s}wl(e)}else 0!=(8772&a.subtreeFlags)&&null!==i?(i.return=a,Ju=i):wl(e)}}function wl(e){for(;null!==Ju;){var t=Ju;if(0!=(8772&t.flags)){var n=t.alternate;try{if(0!=(8772&t.flags))switch(t.tag){case 0:case 11:case 15:Zu||al(5,t);break;case 1:var r=t.stateNode;if(4&t.flags&&!Zu)if(null===n)r.componentDidMount();else{var a=t.elementType===t.type?n.memoizedProps:gi(t.type,n.memoizedProps);r.componentDidUpdate(a,n.memoizedState,r.__reactInternalSnapshotBeforeUpdate)}var o=t.updateQueue;null!==o&&ji(t,o,r);break;case 3:var u=t.updateQueue;if(null!==u){if(n=null,null!==t.child)switch(t.child.tag){case 5:case 1:n=t.child.stateNode}ji(t,u,n)}break;case 5:var l=t.stateNode;if(null===n&&4&t.flags){n=l;var s=t.memoizedProps;switch(t.type){case\"button\":case\"input\":case\"select\":case\"textarea\":s.autoFocus&&n.focus();break;case\"img\":s.src&&(n.src=s.src)}}break;case 6:case 4:case 12:case 19:case 17:case 21:case 22:case 23:case 25:break;case 13:if(null===t.memoizedState){var c=t.alternate;if(null!==c){var f=c.memoizedState;if(null!==f){var p=f.dehydrated;null!==p&&Bt(p)}}}break;default:throw Error(i(163))}Zu||512&t.flags&&il(t)}catch(e){Es(t,t.return,e)}}if(t===e){Ju=null;break}if(null!==(n=t.sibling)){n.return=t.return,Ju=n;break}Ju=t.return}}function xl(e){for(;null!==Ju;){var t=Ju;if(t===e){Ju=null;break}var n=t.sibling;if(null!==n){n.return=t.return,Ju=n;break}Ju=t.return}}function kl(e){for(;null!==Ju;){var t=Ju;try{switch(t.tag){case 0:case 11:case 15:var n=t.return;try{al(4,t)}catch(e){Es(t,n,e)}break;case 1:var r=t.stateNode;if(\"function\"==typeof r.componentDidMount){var a=t.return;try{r.componentDidMount()}catch(e){Es(t,a,e)}}var i=t.return;try{il(t)}catch(e){Es(t,i,e)}break;case 5:var o=t.return;try{il(t)}catch(e){Es(t,o,e)}}}catch(e){Es(t,t.return,e)}if(t===e){Ju=null;break}var u=t.sibling;if(null!==u){u.return=t.return,Ju=u;break}Ju=t.return}}var Sl,El=Math.ceil,Cl=_.ReactCurrentDispatcher,Tl=_.ReactCurrentOwner,Ml=_.ReactCurrentBatchConfig,Nl=0,Pl=null,zl=null,Ll=0,Ol=0,Al=Sa(0),Fl=0,Dl=null,Rl=0,jl=0,Ul=0,Il=null,$l=null,Bl=0,Wl=1/0,Vl=null,Hl=!1,ql=null,Ql=null,Yl=!1,Gl=null,Kl=0,Zl=0,Xl=null,Jl=-1,es=0;function ts(){return 0!=(6&Nl)?Ze():-1!==Jl?Jl:Jl=Ze()}function ns(e){return 0==(1&e.mode)?1:0!=(2&Nl)&&0!==Ll?Ll&-Ll:null!==vi.transition?(0===es&&(es=vt()),es):0!==(e=bt)?e:e=void 0===(e=window.event)?16:Kt(e.type)}function rs(e,t,n,r){if(50<Zl)throw Zl=0,Xl=null,Error(i(185));yt(e,n,r),0!=(2&Nl)&&e===Pl||(e===Pl&&(0==(2&Nl)&&(jl|=n),4===Fl&&ls(e,Ll)),as(e,r),1===n&&0===Nl&&0==(1&t.mode)&&(Wl=Ze()+500,Ua&&Ba()))}function as(e,t){var n=e.callbackNode;!function(e,t){for(var n=e.suspendedLanes,r=e.pingedLanes,a=e.expirationTimes,i=e.pendingLanes;0<i;){var o=31-ot(i),u=1<<o,l=a[o];-1===l?0!=(u&n)&&0==(u&r)||(a[o]=dt(u,t)):l<=t&&(e.expiredLanes|=u),i&=~u}}(e,t);var r=pt(e,e===Pl?Ll:0);if(0===r)null!==n&&Ye(n),e.callbackNode=null,e.callbackPriority=0;else if(t=r&-r,e.callbackPriority!==t){if(null!=n&&Ye(n),1===t)0===e.tag?function(e){Ua=!0,$a(e)}(ss.bind(null,e)):$a(ss.bind(null,e)),oa((function(){0==(6&Nl)&&Ba()})),n=null;else{switch(_t(r)){case 1:n=Je;break;case 4:n=et;break;case 16:default:n=tt;break;case 536870912:n=rt}n=Ps(n,is.bind(null,e))}e.callbackPriority=t,e.callbackNode=n}}function is(e,t){if(Jl=-1,es=0,0!=(6&Nl))throw Error(i(327));var n=e.callbackNode;if(ks()&&e.callbackNode!==n)return null;var r=pt(e,e===Pl?Ll:0);if(0===r)return null;if(0!=(30&r)||0!=(r&e.expiredLanes)||t)t=ys(e,r);else{t=r;var a=Nl;Nl|=2;var o=vs();for(Pl===e&&Ll===t||(Vl=null,Wl=Ze()+500,ds(e,t));;)try{bs();break}catch(t){hs(e,t)}wi(),Cl.current=o,Nl=a,null!==zl?t=0:(Pl=null,Ll=0,t=Fl)}if(0!==t){if(2===t&&0!==(a=ht(e))&&(r=a,t=os(e,a)),1===t)throw n=Dl,ds(e,0),ls(e,r),as(e,Ze()),n;if(6===t)ls(e,r);else{if(a=e.current.alternate,0==(30&r)&&!function(e){for(var t=e;;){if(16384&t.flags){var n=t.updateQueue;if(null!==n&&null!==(n=n.stores))for(var r=0;r<n.length;r++){var a=n[r],i=a.getSnapshot;a=a.value;try{if(!ur(i(),a))return!1}catch(e){return!1}}}if(n=t.child,16384&t.subtreeFlags&&null!==n)n.return=t,t=n;else{if(t===e)break;for(;null===t.sibling;){if(null===t.return||t.return===e)return!0;t=t.return}t.sibling.return=t.return,t=t.sibling}}return!0}(a)&&(2===(t=ys(e,r))&&0!==(o=ht(e))&&(r=o,t=os(e,o)),1===t))throw n=Dl,ds(e,0),ls(e,r),as(e,Ze()),n;switch(e.finishedWork=a,e.finishedLanes=r,t){case 0:case 1:throw Error(i(345));case 2:case 5:xs(e,$l,Vl);break;case 3:if(ls(e,r),(130023424&r)===r&&10<(t=Bl+500-Ze())){if(0!==pt(e,0))break;if(((a=e.suspendedLanes)&r)!==r){ts(),e.pingedLanes|=e.suspendedLanes&a;break}e.timeoutHandle=ra(xs.bind(null,e,$l,Vl),t);break}xs(e,$l,Vl);break;case 4:if(ls(e,r),(4194240&r)===r)break;for(t=e.eventTimes,a=-1;0<r;){var u=31-ot(r);o=1<<u,(u=t[u])>a&&(a=u),r&=~o}if(r=a,10<(r=(120>(r=Ze()-r)?120:480>r?480:1080>r?1080:1920>r?1920:3e3>r?3e3:4320>r?4320:1960*El(r/1960))-r)){e.timeoutHandle=ra(xs.bind(null,e,$l,Vl),r);break}xs(e,$l,Vl);break;default:throw Error(i(329))}}}return as(e,Ze()),e.callbackNode===n?is.bind(null,e):null}function os(e,t){var n=Il;return e.current.memoizedState.isDehydrated&&(ds(e,t).flags|=256),2!==(e=ys(e,t))&&(t=$l,$l=n,null!==t&&us(t)),e}function us(e){null===$l?$l=e:$l.push.apply($l,e)}function ls(e,t){for(t&=~Ul,t&=~jl,e.suspendedLanes|=t,e.pingedLanes&=~t,e=e.expirationTimes;0<t;){var n=31-ot(t),r=1<<n;e[n]=-1,t&=~r}}function ss(e){if(0!=(6&Nl))throw Error(i(327));ks();var t=pt(e,0);if(0==(1&t))return as(e,Ze()),null;var n=ys(e,t);if(0!==e.tag&&2===n){var r=ht(e);0!==r&&(t=r,n=os(e,r))}if(1===n)throw n=Dl,ds(e,0),ls(e,t),as(e,Ze()),n;if(6===n)throw Error(i(345));return e.finishedWork=e.current.alternate,e.finishedLanes=t,xs(e,$l,Vl),as(e,Ze()),null}function cs(e,t){var n=Nl;Nl|=1;try{return e(t)}finally{0===(Nl=n)&&(Wl=Ze()+500,Ua&&Ba())}}function fs(e){null!==Gl&&0===Gl.tag&&0==(6&Nl)&&ks();var t=Nl;Nl|=1;var n=Ml.transition,r=bt;try{if(Ml.transition=null,bt=1,e)return e()}finally{bt=r,Ml.transition=n,0==(6&(Nl=t))&&Ba()}}function ps(){Ol=Al.current,Ea(Al)}function ds(e,t){e.finishedWork=null,e.finishedLanes=0;var n=e.timeoutHandle;if(-1!==n&&(e.timeoutHandle=-1,aa(n)),null!==zl)for(n=zl.return;null!==n;){var r=n;switch(ti(r),r.tag){case 1:null!=(r=r.type.childContextTypes)&&Oa();break;case 3:ao(),Ea(Na),Ea(Ma),co();break;case 5:oo(r);break;case 4:ao();break;case 13:case 19:Ea(uo);break;case 10:xi(r.type._context);break;case 22:case 23:ps()}n=n.return}if(Pl=e,zl=e=As(e.current,null),Ll=Ol=t,Fl=0,Dl=null,Ul=jl=Rl=0,$l=Il=null,null!==Ci){for(t=0;t<Ci.length;t++)if(null!==(r=(n=Ci[t]).interleaved)){n.interleaved=null;var a=r.next,i=n.pending;if(null!==i){var o=i.next;i.next=a,r.next=o}n.pending=r}Ci=null}return e}function hs(e,t){for(;;){var n=zl;try{if(wi(),fo.current=ou,mo){for(var r=vo.memoizedState;null!==r;){var a=r.queue;null!==a&&(a.pending=null),r=r.next}mo=!1}if(ho=0,yo=go=vo=null,bo=!1,_o=0,Tl.current=null,null===n||null===n.return){Fl=1,Dl=t,zl=null;break}e:{var o=e,u=n.return,l=n,s=t;if(t=Ll,l.flags|=32768,null!==s&&\"object\"==typeof s&&\"function\"==typeof s.then){var c=s,f=l,p=f.tag;if(0==(1&f.mode)&&(0===p||11===p||15===p)){var d=f.alternate;d?(f.updateQueue=d.updateQueue,f.memoizedState=d.memoizedState,f.lanes=d.lanes):(f.updateQueue=null,f.memoizedState=null)}var h=yu(u);if(null!==h){h.flags&=-257,mu(h,u,l,0,t),1&h.mode&&gu(o,c,t),s=c;var v=(t=h).updateQueue;if(null===v){var g=new Set;g.add(s),t.updateQueue=g}else v.add(s);break e}if(0==(1&t)){gu(o,c,t),gs();break e}s=Error(i(426))}else if(ai&&1&l.mode){var y=yu(u);if(null!==y){0==(65536&y.flags)&&(y.flags|=256),mu(y,u,l,0,t),hi(cu(s,l));break e}}o=s=cu(s,l),4!==Fl&&(Fl=2),null===Il?Il=[o]:Il.push(o),o=u;do{switch(o.tag){case 3:o.flags|=65536,t&=-t,o.lanes|=t,Di(o,hu(0,s,t));break e;case 1:l=s;var m=o.type,b=o.stateNode;if(0==(128&o.flags)&&(\"function\"==typeof m.getDerivedStateFromError||null!==b&&\"function\"==typeof b.componentDidCatch&&(null===Ql||!Ql.has(b)))){o.flags|=65536,t&=-t,o.lanes|=t,Di(o,vu(o,l,t));break e}}o=o.return}while(null!==o)}ws(n)}catch(e){t=e,zl===n&&null!==n&&(zl=n=n.return);continue}break}}function vs(){var e=Cl.current;return Cl.current=ou,null===e?ou:e}function gs(){0!==Fl&&3!==Fl&&2!==Fl||(Fl=4),null===Pl||0==(268435455&Rl)&&0==(268435455&jl)||ls(Pl,Ll)}function ys(e,t){var n=Nl;Nl|=2;var r=vs();for(Pl===e&&Ll===t||(Vl=null,ds(e,t));;)try{ms();break}catch(t){hs(e,t)}if(wi(),Nl=n,Cl.current=r,null!==zl)throw Error(i(261));return Pl=null,Ll=0,Fl}function ms(){for(;null!==zl;)_s(zl)}function bs(){for(;null!==zl&&!Ge();)_s(zl)}function _s(e){var t=Sl(e.alternate,e,Ol);e.memoizedProps=e.pendingProps,null===t?ws(e):zl=t,Tl.current=null}function ws(e){var t=e;do{var n=t.alternate;if(e=t.return,0==(32768&t.flags)){if(null!==(n=Yu(n,t,Ol)))return void(zl=n)}else{if(null!==(n=Gu(n,t)))return n.flags&=32767,void(zl=n);if(null===e)return Fl=6,void(zl=null);e.flags|=32768,e.subtreeFlags=0,e.deletions=null}if(null!==(t=t.sibling))return void(zl=t);zl=t=e}while(null!==t);0===Fl&&(Fl=5)}function xs(e,t,n){var r=bt,a=Ml.transition;try{Ml.transition=null,bt=1,function(e,t,n,r){do{ks()}while(null!==Gl);if(0!=(6&Nl))throw Error(i(327));n=e.finishedWork;var a=e.finishedLanes;if(null===n)return null;if(e.finishedWork=null,e.finishedLanes=0,n===e.current)throw Error(i(177));e.callbackNode=null,e.callbackPriority=0;var o=n.lanes|n.childLanes;if(function(e,t){var n=e.pendingLanes&~t;e.pendingLanes=t,e.suspendedLanes=0,e.pingedLanes=0,e.expiredLanes&=t,e.mutableReadLanes&=t,e.entangledLanes&=t,t=e.entanglements;var r=e.eventTimes;for(e=e.expirationTimes;0<n;){var a=31-ot(n),i=1<<a;t[a]=0,r[a]=-1,e[a]=-1,n&=~i}}(e,o),e===Pl&&(zl=Pl=null,Ll=0),0==(2064&n.subtreeFlags)&&0==(2064&n.flags)||Yl||(Yl=!0,Ps(tt,(function(){return ks(),null}))),o=0!=(15990&n.flags),0!=(15990&n.subtreeFlags)||o){o=Ml.transition,Ml.transition=null;var u=bt;bt=1;var l=Nl;Nl|=4,Tl.current=null,function(e,t){if(ea=Vt,dr(e=pr())){if(\"selectionStart\"in e)var n={start:e.selectionStart,end:e.selectionEnd};else e:{var r=(n=(n=e.ownerDocument)&&n.defaultView||window).getSelection&&n.getSelection();if(r&&0!==r.rangeCount){n=r.anchorNode;var a=r.anchorOffset,o=r.focusNode;r=r.focusOffset;try{n.nodeType,o.nodeType}catch(e){n=null;break e}var u=0,l=-1,s=-1,c=0,f=0,p=e,d=null;t:for(;;){for(var h;p!==n||0!==a&&3!==p.nodeType||(l=u+a),p!==o||0!==r&&3!==p.nodeType||(s=u+r),3===p.nodeType&&(u+=p.nodeValue.length),null!==(h=p.firstChild);)d=p,p=h;for(;;){if(p===e)break t;if(d===n&&++c===a&&(l=u),d===o&&++f===r&&(s=u),null!==(h=p.nextSibling))break;d=(p=d).parentNode}p=h}n=-1===l||-1===s?null:{start:l,end:s}}else n=null}n=n||{start:0,end:0}}else n=null;for(ta={focusedElem:e,selectionRange:n},Vt=!1,Ju=t;null!==Ju;)if(e=(t=Ju).child,0!=(1028&t.subtreeFlags)&&null!==e)e.return=t,Ju=e;else for(;null!==Ju;){t=Ju;try{var v=t.alternate;if(0!=(1024&t.flags))switch(t.tag){case 0:case 11:case 15:case 5:case 6:case 4:case 17:break;case 1:if(null!==v){var g=v.memoizedProps,y=v.memoizedState,m=t.stateNode,b=m.getSnapshotBeforeUpdate(t.elementType===t.type?g:gi(t.type,g),y);m.__reactInternalSnapshotBeforeUpdate=b}break;case 3:var _=t.stateNode.containerInfo;1===_.nodeType?_.textContent=\"\":9===_.nodeType&&_.documentElement&&_.removeChild(_.documentElement);break;default:throw Error(i(163))}}catch(e){Es(t,t.return,e)}if(null!==(e=t.sibling)){e.return=t.return,Ju=e;break}Ju=t.return}v=nl,nl=!1}(e,n),yl(n,e),hr(ta),Vt=!!ea,ta=ea=null,e.current=n,bl(n,e,a),Ke(),Nl=l,bt=u,Ml.transition=o}else e.current=n;if(Yl&&(Yl=!1,Gl=e,Kl=a),0===(o=e.pendingLanes)&&(Ql=null),function(e){if(it&&\"function\"==typeof it.onCommitFiberRoot)try{it.onCommitFiberRoot(at,e,void 0,128==(128&e.current.flags))}catch(e){}}(n.stateNode),as(e,Ze()),null!==t)for(r=e.onRecoverableError,n=0;n<t.length;n++)r((a=t[n]).value,{componentStack:a.stack,digest:a.digest});if(Hl)throw Hl=!1,e=ql,ql=null,e;0!=(1&Kl)&&0!==e.tag&&ks(),0!=(1&(o=e.pendingLanes))?e===Xl?Zl++:(Zl=0,Xl=e):Zl=0,Ba()}(e,t,n,r)}finally{Ml.transition=a,bt=r}return null}function ks(){if(null!==Gl){var e=_t(Kl),t=Ml.transition,n=bt;try{if(Ml.transition=null,bt=16>e?16:e,null===Gl)var r=!1;else{if(e=Gl,Gl=null,Kl=0,0!=(6&Nl))throw Error(i(331));var a=Nl;for(Nl|=4,Ju=e.current;null!==Ju;){var o=Ju,u=o.child;if(0!=(16&Ju.flags)){var l=o.deletions;if(null!==l){for(var s=0;s<l.length;s++){var c=l[s];for(Ju=c;null!==Ju;){var f=Ju;switch(f.tag){case 0:case 11:case 15:rl(8,f,o)}var p=f.child;if(null!==p)p.return=f,Ju=p;else for(;null!==Ju;){var d=(f=Ju).sibling,h=f.return;if(ol(f),f===c){Ju=null;break}if(null!==d){d.return=h,Ju=d;break}Ju=h}}}var v=o.alternate;if(null!==v){var g=v.child;if(null!==g){v.child=null;do{var y=g.sibling;g.sibling=null,g=y}while(null!==g)}}Ju=o}}if(0!=(2064&o.subtreeFlags)&&null!==u)u.return=o,Ju=u;else e:for(;null!==Ju;){if(0!=(2048&(o=Ju).flags))switch(o.tag){case 0:case 11:case 15:rl(9,o,o.return)}var m=o.sibling;if(null!==m){m.return=o.return,Ju=m;break e}Ju=o.return}}var b=e.current;for(Ju=b;null!==Ju;){var _=(u=Ju).child;if(0!=(2064&u.subtreeFlags)&&null!==_)_.return=u,Ju=_;else e:for(u=b;null!==Ju;){if(0!=(2048&(l=Ju).flags))try{switch(l.tag){case 0:case 11:case 15:al(9,l)}}catch(e){Es(l,l.return,e)}if(l===u){Ju=null;break e}var w=l.sibling;if(null!==w){w.return=l.return,Ju=w;break e}Ju=l.return}}if(Nl=a,Ba(),it&&\"function\"==typeof it.onPostCommitFiberRoot)try{it.onPostCommitFiberRoot(at,e)}catch(e){}r=!0}return r}finally{bt=n,Ml.transition=t}}return!1}function Ss(e,t,n){e=Ai(e,t=hu(0,t=cu(n,t),1),1),t=ts(),null!==e&&(yt(e,1,t),as(e,t))}function Es(e,t,n){if(3===e.tag)Ss(e,e,n);else for(;null!==t;){if(3===t.tag){Ss(t,e,n);break}if(1===t.tag){var r=t.stateNode;if(\"function\"==typeof t.type.getDerivedStateFromError||\"function\"==typeof r.componentDidCatch&&(null===Ql||!Ql.has(r))){t=Ai(t,e=vu(t,e=cu(n,e),1),1),e=ts(),null!==t&&(yt(t,1,e),as(t,e));break}}t=t.return}}function Cs(e,t,n){var r=e.pingCache;null!==r&&r.delete(t),t=ts(),e.pingedLanes|=e.suspendedLanes&n,Pl===e&&(Ll&n)===n&&(4===Fl||3===Fl&&(130023424&Ll)===Ll&&500>Ze()-Bl?ds(e,0):Ul|=n),as(e,t)}function Ts(e,t){0===t&&(0==(1&e.mode)?t=1:(t=ct,0==(130023424&(ct<<=1))&&(ct=4194304)));var n=ts();null!==(e=Ni(e,t))&&(yt(e,t,n),as(e,n))}function Ms(e){var t=e.memoizedState,n=0;null!==t&&(n=t.retryLane),Ts(e,n)}function Ns(e,t){var n=0;switch(e.tag){case 13:var r=e.stateNode,a=e.memoizedState;null!==a&&(n=a.retryLane);break;case 19:r=e.stateNode;break;default:throw Error(i(314))}null!==r&&r.delete(t),Ts(e,n)}function Ps(e,t){return Qe(e,t)}function zs(e,t,n,r){this.tag=e,this.key=n,this.sibling=this.child=this.return=this.stateNode=this.type=this.elementType=null,this.index=0,this.ref=null,this.pendingProps=t,this.dependencies=this.memoizedState=this.updateQueue=this.memoizedProps=null,this.mode=r,this.subtreeFlags=this.flags=0,this.deletions=null,this.childLanes=this.lanes=0,this.alternate=null}function Ls(e,t,n,r){return new zs(e,t,n,r)}function Os(e){return!(!(e=e.prototype)||!e.isReactComponent)}function As(e,t){var n=e.alternate;return null===n?((n=Ls(e.tag,t,e.key,e.mode)).elementType=e.elementType,n.type=e.type,n.stateNode=e.stateNode,n.alternate=e,e.alternate=n):(n.pendingProps=t,n.type=e.type,n.flags=0,n.subtreeFlags=0,n.deletions=null),n.flags=14680064&e.flags,n.childLanes=e.childLanes,n.lanes=e.lanes,n.child=e.child,n.memoizedProps=e.memoizedProps,n.memoizedState=e.memoizedState,n.updateQueue=e.updateQueue,t=e.dependencies,n.dependencies=null===t?null:{lanes:t.lanes,firstContext:t.firstContext},n.sibling=e.sibling,n.index=e.index,n.ref=e.ref,n}function Fs(e,t,n,r,a,o){var u=2;if(r=e,\"function\"==typeof e)Os(e)&&(u=1);else if(\"string\"==typeof e)u=5;else e:switch(e){case k:return Ds(n.children,a,o,t);case S:u=8,a|=8;break;case E:return(e=Ls(12,n,t,2|a)).elementType=E,e.lanes=o,e;case N:return(e=Ls(13,n,t,a)).elementType=N,e.lanes=o,e;case P:return(e=Ls(19,n,t,a)).elementType=P,e.lanes=o,e;case O:return Rs(n,a,o,t);default:if(\"object\"==typeof e&&null!==e)switch(e.$$typeof){case C:u=10;break e;case T:u=9;break e;case M:u=11;break e;case z:u=14;break e;case L:u=16,r=null;break e}throw Error(i(130,null==e?e:typeof e,\"\"))}return(t=Ls(u,n,t,a)).elementType=e,t.type=r,t.lanes=o,t}function Ds(e,t,n,r){return(e=Ls(7,e,r,t)).lanes=n,e}function Rs(e,t,n,r){return(e=Ls(22,e,r,t)).elementType=O,e.lanes=n,e.stateNode={isHidden:!1},e}function js(e,t,n){return(e=Ls(6,e,null,t)).lanes=n,e}function Us(e,t,n){return(t=Ls(4,null!==e.children?e.children:[],e.key,t)).lanes=n,t.stateNode={containerInfo:e.containerInfo,pendingChildren:null,implementation:e.implementation},t}function Is(e,t,n,r,a){this.tag=t,this.containerInfo=e,this.finishedWork=this.pingCache=this.current=this.pendingChildren=null,this.timeoutHandle=-1,this.callbackNode=this.pendingContext=this.context=null,this.callbackPriority=0,this.eventTimes=gt(0),this.expirationTimes=gt(-1),this.entangledLanes=this.finishedLanes=this.mutableReadLanes=this.expiredLanes=this.pingedLanes=this.suspendedLanes=this.pendingLanes=0,this.entanglements=gt(0),this.identifierPrefix=r,this.onRecoverableError=a,this.mutableSourceEagerHydrationData=null}function $s(e,t,n,r,a,i,o,u,l){return e=new Is(e,t,n,u,l),1===t?(t=1,!0===i&&(t|=8)):t=0,i=Ls(3,null,null,t),e.current=i,i.stateNode=e,i.memoizedState={element:r,isDehydrated:n,cache:null,transitions:null,pendingSuspenseBoundaries:null},zi(i),e}function Bs(e){if(!e)return Ta;e:{if(Be(e=e._reactInternals)!==e||1!==e.tag)throw Error(i(170));var t=e;do{switch(t.tag){case 3:t=t.stateNode.context;break e;case 1:if(La(t.type)){t=t.stateNode.__reactInternalMemoizedMergedChildContext;break e}}t=t.return}while(null!==t);throw Error(i(171))}if(1===e.tag){var n=e.type;if(La(n))return Fa(e,n,t)}return t}function Ws(e,t,n,r,a,i,o,u,l){return(e=$s(n,r,!0,e,0,i,0,u,l)).context=Bs(null),n=e.current,(i=Oi(r=ts(),a=ns(n))).callback=null!=t?t:null,Ai(n,i,a),e.current.lanes=a,yt(e,a,r),as(e,r),e}function Vs(e,t,n,r){var a=t.current,i=ts(),o=ns(a);return n=Bs(n),null===t.context?t.context=n:t.pendingContext=n,(t=Oi(i,o)).payload={element:e},null!==(r=void 0===r?null:r)&&(t.callback=r),null!==(e=Ai(a,t,o))&&(rs(e,a,o,i),Fi(e,a,o)),o}function Hs(e){return(e=e.current).child?(e.child.tag,e.child.stateNode):null}function qs(e,t){if(null!==(e=e.memoizedState)&&null!==e.dehydrated){var n=e.retryLane;e.retryLane=0!==n&&n<t?n:t}}function Qs(e,t){qs(e,t),(e=e.alternate)&&qs(e,t)}Sl=function(e,t,n){if(null!==e)if(e.memoizedProps!==t.pendingProps||Na.current)_u=!0;else{if(0==(e.lanes&n)&&0==(128&t.flags))return _u=!1,function(e,t,n){switch(t.tag){case 3:Pu(t),di();break;case 5:io(t);break;case 1:La(t.type)&&Da(t);break;case 4:ro(t,t.stateNode.containerInfo);break;case 10:var r=t.type._context,a=t.memoizedProps.value;Ca(yi,r._currentValue),r._currentValue=a;break;case 13:if(null!==(r=t.memoizedState))return null!==r.dehydrated?(Ca(uo,1&uo.current),t.flags|=128,null):0!=(n&t.child.childLanes)?ju(e,t,n):(Ca(uo,1&uo.current),null!==(e=Hu(e,t,n))?e.sibling:null);Ca(uo,1&uo.current);break;case 19:if(r=0!=(n&t.childLanes),0!=(128&e.flags)){if(r)return Wu(e,t,n);t.flags|=128}if(null!==(a=t.memoizedState)&&(a.rendering=null,a.tail=null,a.lastEffect=null),Ca(uo,uo.current),r)break;return null;case 22:case 23:return t.lanes=0,Eu(e,t,n)}return Hu(e,t,n)}(e,t,n);_u=0!=(131072&e.flags)}else _u=!1,ai&&0!=(1048576&t.flags)&&Ja(t,qa,t.index);switch(t.lanes=0,t.tag){case 2:var r=t.type;Vu(e,t),e=t.pendingProps;var a=za(t,Ma.current);Si(t,n),a=So(null,t,r,e,a,n);var o=Eo();return t.flags|=1,\"object\"==typeof a&&null!==a&&\"function\"==typeof a.render&&void 0===a.$$typeof?(t.tag=1,t.memoizedState=null,t.updateQueue=null,La(r)?(o=!0,Da(t)):o=!1,t.memoizedState=null!==a.state&&void 0!==a.state?a.state:null,zi(t),a.updater=$i,t.stateNode=a,a._reactInternals=t,Hi(t,r,e,n),t=Nu(null,t,r,!0,o,n)):(t.tag=0,ai&&o&&ei(t),wu(null,t,a,n),t=t.child),t;case 16:r=t.elementType;e:{switch(Vu(e,t),e=t.pendingProps,r=(a=r._init)(r._payload),t.type=r,a=t.tag=function(e){if(\"function\"==typeof e)return Os(e)?1:0;if(null!=e){if((e=e.$$typeof)===M)return 11;if(e===z)return 14}return 2}(r),e=gi(r,e),a){case 0:t=Tu(null,t,r,e,n);break e;case 1:t=Mu(null,t,r,e,n);break e;case 11:t=xu(null,t,r,e,n);break e;case 14:t=ku(null,t,r,gi(r.type,e),n);break e}throw Error(i(306,r,\"\"))}return t;case 0:return r=t.type,a=t.pendingProps,Tu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 1:return r=t.type,a=t.pendingProps,Mu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 3:e:{if(Pu(t),null===e)throw Error(i(387));r=t.pendingProps,a=(o=t.memoizedState).element,Li(e,t),Ri(t,r,null,n);var u=t.memoizedState;if(r=u.element,o.isDehydrated){if(o={element:r,isDehydrated:!1,cache:u.cache,pendingSuspenseBoundaries:u.pendingSuspenseBoundaries,transitions:u.transitions},t.updateQueue.baseState=o,t.memoizedState=o,256&t.flags){t=zu(e,t,r,n,a=cu(Error(i(423)),t));break e}if(r!==a){t=zu(e,t,r,n,a=cu(Error(i(424)),t));break e}for(ri=sa(t.stateNode.containerInfo.firstChild),ni=t,ai=!0,ii=null,n=Zi(t,null,r,n),t.child=n;n;)n.flags=-3&n.flags|4096,n=n.sibling}else{if(di(),r===a){t=Hu(e,t,n);break e}wu(e,t,r,n)}t=t.child}return t;case 5:return io(t),null===e&&si(t),r=t.type,a=t.pendingProps,o=null!==e?e.memoizedProps:null,u=a.children,na(r,a)?u=null:null!==o&&na(r,o)&&(t.flags|=32),Cu(e,t),wu(e,t,u,n),t.child;case 6:return null===e&&si(t),null;case 13:return ju(e,t,n);case 4:return ro(t,t.stateNode.containerInfo),r=t.pendingProps,null===e?t.child=Ki(t,null,r,n):wu(e,t,r,n),t.child;case 11:return r=t.type,a=t.pendingProps,xu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 7:return wu(e,t,t.pendingProps,n),t.child;case 8:case 12:return wu(e,t,t.pendingProps.children,n),t.child;case 10:e:{if(r=t.type._context,a=t.pendingProps,o=t.memoizedProps,u=a.value,Ca(yi,r._currentValue),r._currentValue=u,null!==o)if(ur(o.value,u)){if(o.children===a.children&&!Na.current){t=Hu(e,t,n);break e}}else for(null!==(o=t.child)&&(o.return=t);null!==o;){var l=o.dependencies;if(null!==l){u=o.child;for(var s=l.firstContext;null!==s;){if(s.context===r){if(1===o.tag){(s=Oi(-1,n&-n)).tag=2;var c=o.updateQueue;if(null!==c){var f=(c=c.shared).pending;null===f?s.next=s:(s.next=f.next,f.next=s),c.pending=s}}o.lanes|=n,null!==(s=o.alternate)&&(s.lanes|=n),ki(o.return,n,t),l.lanes|=n;break}s=s.next}}else if(10===o.tag)u=o.type===t.type?null:o.child;else if(18===o.tag){if(null===(u=o.return))throw Error(i(341));u.lanes|=n,null!==(l=u.alternate)&&(l.lanes|=n),ki(u,n,t),u=o.sibling}else u=o.child;if(null!==u)u.return=o;else for(u=o;null!==u;){if(u===t){u=null;break}if(null!==(o=u.sibling)){o.return=u.return,u=o;break}u=u.return}o=u}wu(e,t,a.children,n),t=t.child}return t;case 9:return a=t.type,r=t.pendingProps.children,Si(t,n),r=r(a=Ei(a)),t.flags|=1,wu(e,t,r,n),t.child;case 14:return a=gi(r=t.type,t.pendingProps),ku(e,t,r,a=gi(r.type,a),n);case 15:return Su(e,t,t.type,t.pendingProps,n);case 17:return r=t.type,a=t.pendingProps,a=t.elementType===r?a:gi(r,a),Vu(e,t),t.tag=1,La(r)?(e=!0,Da(t)):e=!1,Si(t,n),Wi(t,r,a),Hi(t,r,a,n),Nu(null,t,r,!0,e,n);case 19:return Wu(e,t,n);case 22:return Eu(e,t,n)}throw Error(i(156,t.tag))};var Ys=\"function\"==typeof reportError?reportError:function(e){console.error(e)};function Gs(e){this._internalRoot=e}function Ks(e){this._internalRoot=e}function Zs(e){return!(!e||1!==e.nodeType&&9!==e.nodeType&&11!==e.nodeType)}function Xs(e){return!(!e||1!==e.nodeType&&9!==e.nodeType&&11!==e.nodeType&&(8!==e.nodeType||\" react-mount-point-unstable \"!==e.nodeValue))}function Js(){}function ec(e,t,n,r,a){var i=n._reactRootContainer;if(i){var o=i;if(\"function\"==typeof a){var u=a;a=function(){var e=Hs(o);u.call(e)}}Vs(t,o,e,a)}else o=function(e,t,n,r,a){if(a){if(\"function\"==typeof r){var i=r;r=function(){var e=Hs(o);i.call(e)}}var o=Ws(t,r,e,0,null,!1,0,\"\",Js);return e._reactRootContainer=o,e[ha]=o.current,Br(8===e.nodeType?e.parentNode:e),fs(),o}for(;a=e.lastChild;)e.removeChild(a);if(\"function\"==typeof r){var u=r;r=function(){var e=Hs(l);u.call(e)}}var l=$s(e,0,!1,null,0,!1,0,\"\",Js);return e._reactRootContainer=l,e[ha]=l.current,Br(8===e.nodeType?e.parentNode:e),fs((function(){Vs(t,l,n,r)})),l}(n,t,e,a,r);return Hs(o)}Ks.prototype.render=Gs.prototype.render=function(e){var t=this._internalRoot;if(null===t)throw Error(i(409));Vs(e,t,null,null)},Ks.prototype.unmount=Gs.prototype.unmount=function(){var e=this._internalRoot;if(null!==e){this._internalRoot=null;var t=e.containerInfo;fs((function(){Vs(null,e,null,null)})),t[ha]=null}},Ks.prototype.unstable_scheduleHydration=function(e){if(e){var t=St();e={blockedOn:null,target:e,priority:t};for(var n=0;n<Ot.length&&0!==t&&t<Ot[n].priority;n++);Ot.splice(n,0,e),0===n&&Rt(e)}},wt=function(e){switch(e.tag){case 3:var t=e.stateNode;if(t.current.memoizedState.isDehydrated){var n=ft(t.pendingLanes);0!==n&&(mt(t,1|n),as(t,Ze()),0==(6&Nl)&&(Wl=Ze()+500,Ba()))}break;case 13:fs((function(){var t=Ni(e,1);if(null!==t){var n=ts();rs(t,e,1,n)}})),Qs(e,1)}},xt=function(e){if(13===e.tag){var t=Ni(e,134217728);null!==t&&rs(t,e,134217728,ts()),Qs(e,134217728)}},kt=function(e){if(13===e.tag){var t=ns(e),n=Ni(e,t);null!==n&&rs(n,e,t,ts()),Qs(e,t)}},St=function(){return bt},Et=function(e,t){var n=bt;try{return bt=e,t()}finally{bt=n}},xe=function(e,t,n){switch(t){case\"input\":if(X(e,n),t=n.name,\"radio\"===n.type&&null!=t){for(n=e;n.parentNode;)n=n.parentNode;for(n=n.querySelectorAll(\"input[name=\"+JSON.stringify(\"\"+t)+'][type=\"radio\"]'),t=0;t<n.length;t++){var r=n[t];if(r!==e&&r.form===e.form){var a=wa(r);if(!a)throw Error(i(90));Q(r),X(r,a)}}}break;case\"textarea\":ie(e,n);break;case\"select\":null!=(t=n.value)&&ne(e,!!n.multiple,t,!1)}},Me=cs,Ne=fs;var tc={usingClientEntryPoint:!1,Events:[ba,_a,wa,Ce,Te,cs]},nc={findFiberByHostInstance:ma,bundleType:0,version:\"18.2.0\",rendererPackageName:\"react-dom\"},rc={bundleType:nc.bundleType,version:nc.version,rendererPackageName:nc.rendererPackageName,rendererConfig:nc.rendererConfig,overrideHookState:null,overrideHookStateDeletePath:null,overrideHookStateRenamePath:null,overrideProps:null,overridePropsDeletePath:null,overridePropsRenamePath:null,setErrorHandler:null,setSuspenseHandler:null,scheduleUpdate:null,currentDispatcherRef:_.ReactCurrentDispatcher,findHostInstanceByFiber:function(e){return null===(e=He(e))?null:e.stateNode},findFiberByHostInstance:nc.findFiberByHostInstance||function(){return null},findHostInstancesForRefresh:null,scheduleRefresh:null,scheduleRoot:null,setRefreshHandler:null,getCurrentFiber:null,reconcilerVersion:\"18.2.0-next-9e3b772b8-20220608\"};if(\"undefined\"!=typeof __REACT_DEVTOOLS_GLOBAL_HOOK__){var ac=__REACT_DEVTOOLS_GLOBAL_HOOK__;if(!ac.isDisabled&&ac.supportsFiber)try{at=ac.inject(rc),it=ac}catch(ce){}}t.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED=tc,t.createPortal=function(e,t){var n=2<arguments.length&&void 0!==arguments[2]?arguments[2]:null;if(!Zs(t))throw Error(i(200));return function(e,t,n){var r=3<arguments.length&&void 0!==arguments[3]?arguments[3]:null;return{$$typeof:x,key:null==r?null:\"\"+r,children:e,containerInfo:t,implementation:n}}(e,t,null,n)},t.createRoot=function(e,t){if(!Zs(e))throw Error(i(299));var n=!1,r=\"\",a=Ys;return null!=t&&(!0===t.unstable_strictMode&&(n=!0),void 0!==t.identifierPrefix&&(r=t.identifierPrefix),void 0!==t.onRecoverableError&&(a=t.onRecoverableError)),t=$s(e,1,!1,null,0,n,0,r,a),e[ha]=t.current,Br(8===e.nodeType?e.parentNode:e),new Gs(t)},t.findDOMNode=function(e){if(null==e)return null;if(1===e.nodeType)return e;var t=e._reactInternals;if(void 0===t){if(\"function\"==typeof e.render)throw Error(i(188));throw e=Object.keys(e).join(\",\"),Error(i(268,e))}return null===(e=He(t))?null:e.stateNode},t.flushSync=function(e){return fs(e)},t.hydrate=function(e,t,n){if(!Xs(t))throw Error(i(200));return ec(null,e,t,!0,n)},t.hydrateRoot=function(e,t,n){if(!Zs(e))throw Error(i(405));var r=null!=n&&n.hydratedSources||null,a=!1,o=\"\",u=Ys;if(null!=n&&(!0===n.unstable_strictMode&&(a=!0),void 0!==n.identifierPrefix&&(o=n.identifierPrefix),void 0!==n.onRecoverableError&&(u=n.onRecoverableError)),t=Ws(t,null,e,1,null!=n?n:null,a,0,o,u),e[ha]=t.current,Br(e),r)for(e=0;e<r.length;e++)a=(a=(n=r[e])._getVersion)(n._source),null==t.mutableSourceEagerHydrationData?t.mutableSourceEagerHydrationData=[n,a]:t.mutableSourceEagerHydrationData.push(n,a);return new Ks(t)},t.render=function(e,t,n){if(!Xs(t))throw Error(i(200));return ec(null,e,t,!1,n)},t.unmountComponentAtNode=function(e){if(!Xs(e))throw Error(i(40));return!!e._reactRootContainer&&(fs((function(){ec(null,null,e,!1,(function(){e._reactRootContainer=null,e[ha]=null}))})),!0)},t.unstable_batchedUpdates=cs,t.unstable_renderSubtreeIntoContainer=function(e,t,n,r){if(!Xs(n))throw Error(i(200));if(null==e||void 0===e._reactInternals)throw Error(i(38));return ec(e,t,n,!1,r)},t.version=\"18.2.0-next-9e3b772b8-20220608\"},935:(e,t,n)=>{\"use strict\";!function e(){if(\"undefined\"!=typeof __REACT_DEVTOOLS_GLOBAL_HOOK__&&\"function\"==typeof __REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE)try{__REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE(e)}catch(e){console.error(e)}}(),e.exports=n(448)},408:(e,t)=>{\"use strict\";var n=Symbol.for(\"react.element\"),r=Symbol.for(\"react.portal\"),a=Symbol.for(\"react.fragment\"),i=Symbol.for(\"react.strict_mode\"),o=Symbol.for(\"react.profiler\"),u=Symbol.for(\"react.provider\"),l=Symbol.for(\"react.context\"),s=Symbol.for(\"react.forward_ref\"),c=Symbol.for(\"react.suspense\"),f=Symbol.for(\"react.memo\"),p=Symbol.for(\"react.lazy\"),d=Symbol.iterator,h={isMounted:function(){return!1},enqueueForceUpdate:function(){},enqueueReplaceState:function(){},enqueueSetState:function(){}},v=Object.assign,g={};function y(e,t,n){this.props=e,this.context=t,this.refs=g,this.updater=n||h}function m(){}function b(e,t,n){this.props=e,this.context=t,this.refs=g,this.updater=n||h}y.prototype.isReactComponent={},y.prototype.setState=function(e,t){if(\"object\"!=typeof e&&\"function\"!=typeof e&&null!=e)throw Error(\"setState(...): takes an object of state variables to update or a function which returns an object of state variables.\");this.updater.enqueueSetState(this,e,t,\"setState\")},y.prototype.forceUpdate=function(e){this.updater.enqueueForceUpdate(this,e,\"forceUpdate\")},m.prototype=y.prototype;var _=b.prototype=new m;_.constructor=b,v(_,y.prototype),_.isPureReactComponent=!0;var w=Array.isArray,x=Object.prototype.hasOwnProperty,k={current:null},S={key:!0,ref:!0,__self:!0,__source:!0};function E(e,t,r){var a,i={},o=null,u=null;if(null!=t)for(a in void 0!==t.ref&&(u=t.ref),void 0!==t.key&&(o=\"\"+t.key),t)x.call(t,a)&&!S.hasOwnProperty(a)&&(i[a]=t[a]);var l=arguments.length-2;if(1===l)i.children=r;else if(1<l){for(var s=Array(l),c=0;c<l;c++)s[c]=arguments[c+2];i.children=s}if(e&&e.defaultProps)for(a in l=e.defaultProps)void 0===i[a]&&(i[a]=l[a]);return{$$typeof:n,type:e,key:o,ref:u,props:i,_owner:k.current}}function C(e){return\"object\"==typeof e&&null!==e&&e.$$typeof===n}var T=/\\/+/g;function M(e,t){return\"object\"==typeof e&&null!==e&&null!=e.key?function(e){var t={\"=\":\"=0\",\":\":\"=2\"};return\"$\"+e.replace(/[=:]/g,(function(e){return t[e]}))}(\"\"+e.key):t.toString(36)}function N(e,t,a,i,o){var u=typeof e;\"undefined\"!==u&&\"boolean\"!==u||(e=null);var l=!1;if(null===e)l=!0;else switch(u){case\"string\":case\"number\":l=!0;break;case\"object\":switch(e.$$typeof){case n:case r:l=!0}}if(l)return o=o(l=e),e=\"\"===i?\".\"+M(l,0):i,w(o)?(a=\"\",null!=e&&(a=e.replace(T,\"$&/\")+\"/\"),N(o,t,a,\"\",(function(e){return e}))):null!=o&&(C(o)&&(o=function(e,t){return{$$typeof:n,type:e.type,key:t,ref:e.ref,props:e.props,_owner:e._owner}}(o,a+(!o.key||l&&l.key===o.key?\"\":(\"\"+o.key).replace(T,\"$&/\")+\"/\")+e)),t.push(o)),1;if(l=0,i=\"\"===i?\".\":i+\":\",w(e))for(var s=0;s<e.length;s++){var c=i+M(u=e[s],s);l+=N(u,t,a,c,o)}else if(c=function(e){return null===e||\"object\"!=typeof e?null:\"function\"==typeof(e=d&&e[d]||e[\"@@iterator\"])?e:null}(e),\"function\"==typeof c)for(e=c.call(e),s=0;!(u=e.next()).done;)l+=N(u=u.value,t,a,c=i+M(u,s++),o);else if(\"object\"===u)throw t=String(e),Error(\"Objects are not valid as a React child (found: \"+(\"[object Object]\"===t?\"object with keys {\"+Object.keys(e).join(\", \")+\"}\":t)+\"). If you meant to render a collection of children, use an array instead.\");return l}function P(e,t,n){if(null==e)return e;var r=[],a=0;return N(e,r,\"\",\"\",(function(e){return t.call(n,e,a++)})),r}function z(e){if(-1===e._status){var t=e._result;(t=t()).then((function(t){0!==e._status&&-1!==e._status||(e._status=1,e._result=t)}),(function(t){0!==e._status&&-1!==e._status||(e._status=2,e._result=t)})),-1===e._status&&(e._status=0,e._result=t)}if(1===e._status)return e._result.default;throw e._result}var L={current:null},O={transition:null},A={ReactCurrentDispatcher:L,ReactCurrentBatchConfig:O,ReactCurrentOwner:k};t.Children={map:P,forEach:function(e,t,n){P(e,(function(){t.apply(this,arguments)}),n)},count:function(e){var t=0;return P(e,(function(){t++})),t},toArray:function(e){return P(e,(function(e){return e}))||[]},only:function(e){if(!C(e))throw Error(\"React.Children.only expected to receive a single React element child.\");return e}},t.Component=y,t.Fragment=a,t.Profiler=o,t.PureComponent=b,t.StrictMode=i,t.Suspense=c,t.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED=A,t.cloneElement=function(e,t,r){if(null==e)throw Error(\"React.cloneElement(...): The argument must be a React element, but you passed \"+e+\".\");var a=v({},e.props),i=e.key,o=e.ref,u=e._owner;if(null!=t){if(void 0!==t.ref&&(o=t.ref,u=k.current),void 0!==t.key&&(i=\"\"+t.key),e.type&&e.type.defaultProps)var l=e.type.defaultProps;for(s in t)x.call(t,s)&&!S.hasOwnProperty(s)&&(a[s]=void 0===t[s]&&void 0!==l?l[s]:t[s])}var s=arguments.length-2;if(1===s)a.children=r;else if(1<s){l=Array(s);for(var c=0;c<s;c++)l[c]=arguments[c+2];a.children=l}return{$$typeof:n,type:e.type,key:i,ref:o,props:a,_owner:u}},t.createContext=function(e){return(e={$$typeof:l,_currentValue:e,_currentValue2:e,_threadCount:0,Provider:null,Consumer:null,_defaultValue:null,_globalName:null}).Provider={$$typeof:u,_context:e},e.Consumer=e},t.createElement=E,t.createFactory=function(e){var t=E.bind(null,e);return t.type=e,t},t.createRef=function(){return{current:null}},t.forwardRef=function(e){return{$$typeof:s,render:e}},t.isValidElement=C,t.lazy=function(e){return{$$typeof:p,_payload:{_status:-1,_result:e},_init:z}},t.memo=function(e,t){return{$$typeof:f,type:e,compare:void 0===t?null:t}},t.startTransition=function(e){var t=O.transition;O.transition={};try{e()}finally{O.transition=t}},t.unstable_act=function(){throw Error(\"act(...) is not supported in production builds of React.\")},t.useCallback=function(e,t){return L.current.useCallback(e,t)},t.useContext=function(e){return L.current.useContext(e)},t.useDebugValue=function(){},t.useDeferredValue=function(e){return L.current.useDeferredValue(e)},t.useEffect=function(e,t){return L.current.useEffect(e,t)},t.useId=function(){return L.current.useId()},t.useImperativeHandle=function(e,t,n){return L.current.useImperativeHandle(e,t,n)},t.useInsertionEffect=function(e,t){return L.current.useInsertionEffect(e,t)},t.useLayoutEffect=function(e,t){return L.current.useLayoutEffect(e,t)},t.useMemo=function(e,t){return L.current.useMemo(e,t)},t.useReducer=function(e,t,n){return L.current.useReducer(e,t,n)},t.useRef=function(e){return L.current.useRef(e)},t.useState=function(e){return L.current.useState(e)},t.useSyncExternalStore=function(e,t,n){return L.current.useSyncExternalStore(e,t,n)},t.useTransition=function(){return L.current.useTransition()},t.version=\"18.2.0\"},294:(e,t,n)=>{\"use strict\";e.exports=n(408)},53:(e,t)=>{\"use strict\";function n(e,t){var n=e.length;e.push(t);e:for(;0<n;){var r=n-1>>>1,a=e[r];if(!(0<i(a,t)))break e;e[r]=t,e[n]=a,n=r}}function r(e){return 0===e.length?null:e[0]}function a(e){if(0===e.length)return null;var t=e[0],n=e.pop();if(n!==t){e[0]=n;e:for(var r=0,a=e.length,o=a>>>1;r<o;){var u=2*(r+1)-1,l=e[u],s=u+1,c=e[s];if(0>i(l,n))s<a&&0>i(c,l)?(e[r]=c,e[s]=n,r=s):(e[r]=l,e[u]=n,r=u);else{if(!(s<a&&0>i(c,n)))break e;e[r]=c,e[s]=n,r=s}}}return t}function i(e,t){var n=e.sortIndex-t.sortIndex;return 0!==n?n:e.id-t.id}if(\"object\"==typeof performance&&\"function\"==typeof performance.now){var o=performance;t.unstable_now=function(){return o.now()}}else{var u=Date,l=u.now();t.unstable_now=function(){return u.now()-l}}var s=[],c=[],f=1,p=null,d=3,h=!1,v=!1,g=!1,y=\"function\"==typeof setTimeout?setTimeout:null,m=\"function\"==typeof clearTimeout?clearTimeout:null,b=\"undefined\"!=typeof setImmediate?setImmediate:null;function _(e){for(var t=r(c);null!==t;){if(null===t.callback)a(c);else{if(!(t.startTime<=e))break;a(c),t.sortIndex=t.expirationTime,n(s,t)}t=r(c)}}function w(e){if(g=!1,_(e),!v)if(null!==r(s))v=!0,O(x);else{var t=r(c);null!==t&&A(w,t.startTime-e)}}function x(e,n){v=!1,g&&(g=!1,m(C),C=-1),h=!0;var i=d;try{for(_(n),p=r(s);null!==p&&(!(p.expirationTime>n)||e&&!N());){var o=p.callback;if(\"function\"==typeof o){p.callback=null,d=p.priorityLevel;var u=o(p.expirationTime<=n);n=t.unstable_now(),\"function\"==typeof u?p.callback=u:p===r(s)&&a(s),_(n)}else a(s);p=r(s)}if(null!==p)var l=!0;else{var f=r(c);null!==f&&A(w,f.startTime-n),l=!1}return l}finally{p=null,d=i,h=!1}}\"undefined\"!=typeof navigator&&void 0!==navigator.scheduling&&void 0!==navigator.scheduling.isInputPending&&navigator.scheduling.isInputPending.bind(navigator.scheduling);var k,S=!1,E=null,C=-1,T=5,M=-1;function N(){return!(t.unstable_now()-M<T)}function P(){if(null!==E){var e=t.unstable_now();M=e;var n=!0;try{n=E(!0,e)}finally{n?k():(S=!1,E=null)}}else S=!1}if(\"function\"==typeof b)k=function(){b(P)};else if(\"undefined\"!=typeof MessageChannel){var z=new MessageChannel,L=z.port2;z.port1.onmessage=P,k=function(){L.postMessage(null)}}else k=function(){y(P,0)};function O(e){E=e,S||(S=!0,k())}function A(e,n){C=y((function(){e(t.unstable_now())}),n)}t.unstable_IdlePriority=5,t.unstable_ImmediatePriority=1,t.unstable_LowPriority=4,t.unstable_NormalPriority=3,t.unstable_Profiling=null,t.unstable_UserBlockingPriority=2,t.unstable_cancelCallback=function(e){e.callback=null},t.unstable_continueExecution=function(){v||h||(v=!0,O(x))},t.unstable_forceFrameRate=function(e){0>e||125<e?console.error(\"forceFrameRate takes a positive int between 0 and 125, forcing frame rates higher than 125 fps is not supported\"):T=0<e?Math.floor(1e3/e):5},t.unstable_getCurrentPriorityLevel=function(){return d},t.unstable_getFirstCallbackNode=function(){return r(s)},t.unstable_next=function(e){switch(d){case 1:case 2:case 3:var t=3;break;default:t=d}var n=d;d=t;try{return e()}finally{d=n}},t.unstable_pauseExecution=function(){},t.unstable_requestPaint=function(){},t.unstable_runWithPriority=function(e,t){switch(e){case 1:case 2:case 3:case 4:case 5:break;default:e=3}var n=d;d=e;try{return t()}finally{d=n}},t.unstable_scheduleCallback=function(e,a,i){var o=t.unstable_now();switch(i=\"object\"==typeof i&&null!==i&&\"number\"==typeof(i=i.delay)&&0<i?o+i:o,e){case 1:var u=-1;break;case 2:u=250;break;case 5:u=1073741823;break;case 4:u=1e4;break;default:u=5e3}return e={id:f++,callback:a,priorityLevel:e,startTime:i,expirationTime:u=i+u,sortIndex:-1},i>o?(e.sortIndex=i,n(c,e),null===r(s)&&e===r(c)&&(g?(m(C),C=-1):g=!0,A(w,i-o))):(e.sortIndex=u,n(s,e),v||h||(v=!0,O(x))),e},t.unstable_shouldYield=N,t.unstable_wrapCallback=function(e){var t=d;return function(){var n=d;d=t;try{return e.apply(this,arguments)}finally{d=n}}}},840:(e,t,n)=>{\"use strict\";e.exports=n(53)}},t={};function n(r){var a=t[r];if(void 0!==a)return a.exports;var i=t[r]={id:r,loaded:!1,exports:{}};return e[r].call(i.exports,i,i.exports,n),i.loaded=!0,i.exports}n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(e){if(\"object\"==typeof window)return window}}(),n.nmd=e=>(e.paths=[],e.children||(e.children=[]),e),(()=>{\"use strict\";var e=n(294),t=n(935);const r=Math.sqrt(50),a=Math.sqrt(10),i=Math.sqrt(2);function o(e,t,n){const u=(t-e)/Math.max(0,n),l=Math.floor(Math.log10(u)),s=u/Math.pow(10,l),c=s>=r?10:s>=a?5:s>=i?2:1;let f,p,d;return l<0?(d=Math.pow(10,-l)/c,f=Math.round(e*d),p=Math.round(t*d),f/d<e&&++f,p/d>t&&--p,d=-d):(d=Math.pow(10,l)*c,f=Math.round(e/d),p=Math.round(t/d),f*d<e&&++f,p*d>t&&--p),p<f&&.5<=n&&n<2?o(e,t,2*n):[f,p,d]}function u(e,t,n){return o(e=+e,t=+t,n=+n)[2]}function l(e,t,n){n=+n;const r=(t=+t)<(e=+e),a=r?u(t,e,n):u(e,t,n);return(r?-1:1)*(a<0?1/-a:a)}function s(e,t){return null==e||null==t?NaN:e<t?-1:e>t?1:e>=t?0:NaN}function c(e,t){return null==e||null==t?NaN:t<e?-1:t>e?1:t>=e?0:NaN}function f(e){let t,n,r;function a(e,r,a=0,i=e.length){if(a<i){if(0!==t(r,r))return i;do{const t=a+i>>>1;n(e[t],r)<0?a=t+1:i=t}while(a<i)}return a}return 2!==e.length?(t=s,n=(t,n)=>s(e(t),n),r=(t,n)=>e(t)-n):(t=e===s||e===c?e:p,n=e,r=e),{left:a,center:function(e,t,n=0,i=e.length){const o=a(e,t,n,i-1);return o>n&&r(e[o-1],t)>-r(e[o],t)?o-1:o},right:function(e,r,a=0,i=e.length){if(a<i){if(0!==t(r,r))return i;do{const t=a+i>>>1;n(e[t],r)<=0?a=t+1:i=t}while(a<i)}return a}}}function p(){return 0}const d=f(s),h=d.right,v=(d.left,f((function(e){return null===e?NaN:+e})).center,h);function g(e,t,n){e.prototype=t.prototype=n,n.constructor=e}function y(e,t){var n=Object.create(e.prototype);for(var r in t)n[r]=t[r];return n}function m(){}var b=.7,_=1/b,w=\"\\\\s*([+-]?\\\\d+)\\\\s*\",x=\"\\\\s*([+-]?(?:\\\\d*\\\\.)?\\\\d+(?:[eE][+-]?\\\\d+)?)\\\\s*\",k=\"\\\\s*([+-]?(?:\\\\d*\\\\.)?\\\\d+(?:[eE][+-]?\\\\d+)?)%\\\\s*\",S=/^#([0-9a-f]{3,8})$/,E=new RegExp(`^rgb\\\\(${w},${w},${w}\\\\)$`),C=new RegExp(`^rgb\\\\(${k},${k},${k}\\\\)$`),T=new RegExp(`^rgba\\\\(${w},${w},${w},${x}\\\\)$`),M=new RegExp(`^rgba\\\\(${k},${k},${k},${x}\\\\)$`),N=new RegExp(`^hsl\\\\(${x},${k},${k}\\\\)$`),P=new RegExp(`^hsla\\\\(${x},${k},${k},${x}\\\\)$`),z={aliceblue:15792383,antiquewhite:16444375,aqua:65535,aquamarine:8388564,azure:15794175,beige:16119260,bisque:16770244,black:0,blanchedalmond:16772045,blue:255,blueviolet:9055202,brown:10824234,burlywood:14596231,cadetblue:6266528,chartreuse:8388352,chocolate:13789470,coral:16744272,cornflowerblue:6591981,cornsilk:16775388,crimson:14423100,cyan:65535,darkblue:139,darkcyan:35723,darkgoldenrod:12092939,darkgray:11119017,darkgreen:25600,darkgrey:11119017,darkkhaki:12433259,darkmagenta:9109643,darkolivegreen:5597999,darkorange:16747520,darkorchid:10040012,darkred:9109504,darksalmon:15308410,darkseagreen:9419919,darkslateblue:4734347,darkslategray:3100495,darkslategrey:3100495,darkturquoise:52945,darkviolet:9699539,deeppink:16716947,deepskyblue:49151,dimgray:6908265,dimgrey:6908265,dodgerblue:2003199,firebrick:11674146,floralwhite:16775920,forestgreen:2263842,fuchsia:16711935,gainsboro:14474460,ghostwhite:16316671,gold:16766720,goldenrod:14329120,gray:8421504,green:32768,greenyellow:11403055,grey:8421504,honeydew:15794160,hotpink:16738740,indianred:13458524,indigo:4915330,ivory:16777200,khaki:15787660,lavender:15132410,lavenderblush:16773365,lawngreen:8190976,lemonchiffon:16775885,lightblue:11393254,lightcoral:15761536,lightcyan:14745599,lightgoldenrodyellow:16448210,lightgray:13882323,lightgreen:9498256,lightgrey:13882323,lightpink:16758465,lightsalmon:16752762,lightseagreen:2142890,lightskyblue:8900346,lightslategray:7833753,lightslategrey:7833753,lightsteelblue:11584734,lightyellow:16777184,lime:65280,limegreen:3329330,linen:16445670,magenta:16711935,maroon:8388608,mediumaquamarine:6737322,mediumblue:205,mediumorchid:12211667,mediumpurple:9662683,mediumseagreen:3978097,mediumslateblue:8087790,mediumspringgreen:64154,mediumturquoise:4772300,mediumvioletred:13047173,midnightblue:1644912,mintcream:16121850,mistyrose:16770273,moccasin:16770229,navajowhite:16768685,navy:128,oldlace:16643558,olive:8421376,olivedrab:7048739,orange:16753920,orangered:16729344,orchid:14315734,palegoldenrod:15657130,palegreen:10025880,paleturquoise:11529966,palevioletred:14381203,papayawhip:16773077,peachpuff:16767673,peru:13468991,pink:16761035,plum:14524637,powderblue:11591910,purple:8388736,rebeccapurple:6697881,red:16711680,rosybrown:12357519,royalblue:4286945,saddlebrown:9127187,salmon:16416882,sandybrown:16032864,seagreen:3050327,seashell:16774638,sienna:10506797,silver:12632256,skyblue:8900331,slateblue:6970061,slategray:7372944,slategrey:7372944,snow:16775930,springgreen:65407,steelblue:4620980,tan:13808780,teal:32896,thistle:14204888,tomato:16737095,turquoise:4251856,violet:15631086,wheat:16113331,white:16777215,whitesmoke:16119285,yellow:16776960,yellowgreen:10145074};function L(){return this.rgb().formatHex()}function O(){return this.rgb().formatRgb()}function A(e){var t,n;return e=(e+\"\").trim().toLowerCase(),(t=S.exec(e))?(n=t[1].length,t=parseInt(t[1],16),6===n?F(t):3===n?new j(t>>8&15|t>>4&240,t>>4&15|240&t,(15&t)<<4|15&t,1):8===n?D(t>>24&255,t>>16&255,t>>8&255,(255&t)/255):4===n?D(t>>12&15|t>>8&240,t>>8&15|t>>4&240,t>>4&15|240&t,((15&t)<<4|15&t)/255):null):(t=E.exec(e))?new j(t[1],t[2],t[3],1):(t=C.exec(e))?new j(255*t[1]/100,255*t[2]/100,255*t[3]/100,1):(t=T.exec(e))?D(t[1],t[2],t[3],t[4]):(t=M.exec(e))?D(255*t[1]/100,255*t[2]/100,255*t[3]/100,t[4]):(t=N.exec(e))?V(t[1],t[2]/100,t[3]/100,1):(t=P.exec(e))?V(t[1],t[2]/100,t[3]/100,t[4]):z.hasOwnProperty(e)?F(z[e]):\"transparent\"===e?new j(NaN,NaN,NaN,0):null}function F(e){return new j(e>>16&255,e>>8&255,255&e,1)}function D(e,t,n,r){return r<=0&&(e=t=n=NaN),new j(e,t,n,r)}function R(e,t,n,r){return 1===arguments.length?((a=e)instanceof m||(a=A(a)),a?new j((a=a.rgb()).r,a.g,a.b,a.opacity):new j):new j(e,t,n,null==r?1:r);var a}function j(e,t,n,r){this.r=+e,this.g=+t,this.b=+n,this.opacity=+r}function U(){return`#${W(this.r)}${W(this.g)}${W(this.b)}`}function I(){const e=$(this.opacity);return`${1===e?\"rgb(\":\"rgba(\"}${B(this.r)}, ${B(this.g)}, ${B(this.b)}${1===e?\")\":`, ${e})`}`}function $(e){return isNaN(e)?1:Math.max(0,Math.min(1,e))}function B(e){return Math.max(0,Math.min(255,Math.round(e)||0))}function W(e){return((e=B(e))<16?\"0\":\"\")+e.toString(16)}function V(e,t,n,r){return r<=0?e=t=n=NaN:n<=0||n>=1?e=t=NaN:t<=0&&(e=NaN),new Q(e,t,n,r)}function H(e){if(e instanceof Q)return new Q(e.h,e.s,e.l,e.opacity);if(e instanceof m||(e=A(e)),!e)return new Q;if(e instanceof Q)return e;var t=(e=e.rgb()).r/255,n=e.g/255,r=e.b/255,a=Math.min(t,n,r),i=Math.max(t,n,r),o=NaN,u=i-a,l=(i+a)/2;return u?(o=t===i?(n-r)/u+6*(n<r):n===i?(r-t)/u+2:(t-n)/u+4,u/=l<.5?i+a:2-i-a,o*=60):u=l>0&&l<1?0:o,new Q(o,u,l,e.opacity)}function q(e,t,n,r){return 1===arguments.length?H(e):new Q(e,t,n,null==r?1:r)}function Q(e,t,n,r){this.h=+e,this.s=+t,this.l=+n,this.opacity=+r}function Y(e){return(e=(e||0)%360)<0?e+360:e}function G(e){return Math.max(0,Math.min(1,e||0))}function K(e,t,n){return 255*(e<60?t+(n-t)*e/60:e<180?n:e<240?t+(n-t)*(240-e)/60:t)}function Z(e,t,n,r,a){var i=e*e,o=i*e;return((1-3*e+3*i-o)*t+(4-6*i+3*o)*n+(1+3*e+3*i-3*o)*r+o*a)/6}g(m,A,{copy(e){return Object.assign(new this.constructor,this,e)},displayable(){return this.rgb().displayable()},hex:L,formatHex:L,formatHex8:function(){return this.rgb().formatHex8()},formatHsl:function(){return H(this).formatHsl()},formatRgb:O,toString:O}),g(j,R,y(m,{brighter(e){return e=null==e?_:Math.pow(_,e),new j(this.r*e,this.g*e,this.b*e,this.opacity)},darker(e){return e=null==e?b:Math.pow(b,e),new j(this.r*e,this.g*e,this.b*e,this.opacity)},rgb(){return this},clamp(){return new j(B(this.r),B(this.g),B(this.b),$(this.opacity))},displayable(){return-.5<=this.r&&this.r<255.5&&-.5<=this.g&&this.g<255.5&&-.5<=this.b&&this.b<255.5&&0<=this.opacity&&this.opacity<=1},hex:U,formatHex:U,formatHex8:function(){return`#${W(this.r)}${W(this.g)}${W(this.b)}${W(255*(isNaN(this.opacity)?1:this.opacity))}`},formatRgb:I,toString:I})),g(Q,q,y(m,{brighter(e){return e=null==e?_:Math.pow(_,e),new Q(this.h,this.s,this.l*e,this.opacity)},darker(e){return e=null==e?b:Math.pow(b,e),new Q(this.h,this.s,this.l*e,this.opacity)},rgb(){var e=this.h%360+360*(this.h<0),t=isNaN(e)||isNaN(this.s)?0:this.s,n=this.l,r=n+(n<.5?n:1-n)*t,a=2*n-r;return new j(K(e>=240?e-240:e+120,a,r),K(e,a,r),K(e<120?e+240:e-120,a,r),this.opacity)},clamp(){return new Q(Y(this.h),G(this.s),G(this.l),$(this.opacity))},displayable(){return(0<=this.s&&this.s<=1||isNaN(this.s))&&0<=this.l&&this.l<=1&&0<=this.opacity&&this.opacity<=1},formatHsl(){const e=$(this.opacity);return`${1===e?\"hsl(\":\"hsla(\"}${Y(this.h)}, ${100*G(this.s)}%, ${100*G(this.l)}%${1===e?\")\":`, ${e})`}`}}));const X=e=>()=>e;function J(e,t){var n=t-e;return n?function(e,t){return function(n){return e+n*t}}(e,n):X(isNaN(e)?t:e)}const ee=function e(t){var n=function(e){return 1==(e=+e)?J:function(t,n){return n-t?function(e,t,n){return e=Math.pow(e,n),t=Math.pow(t,n)-e,n=1/n,function(r){return Math.pow(e+r*t,n)}}(t,n,e):X(isNaN(t)?n:t)}}(t);function r(e,t){var r=n((e=R(e)).r,(t=R(t)).r),a=n(e.g,t.g),i=n(e.b,t.b),o=J(e.opacity,t.opacity);return function(t){return e.r=r(t),e.g=a(t),e.b=i(t),e.opacity=o(t),e+\"\"}}return r.gamma=e,r}(1);function te(e){return function(t){var n,r,a=t.length,i=new Array(a),o=new Array(a),u=new Array(a);for(n=0;n<a;++n)r=R(t[n]),i[n]=r.r||0,o[n]=r.g||0,u[n]=r.b||0;return i=e(i),o=e(o),u=e(u),r.opacity=1,function(e){return r.r=i(e),r.g=o(e),r.b=u(e),r+\"\"}}}function ne(e,t){var n,r=t?t.length:0,a=e?Math.min(r,e.length):0,i=new Array(a),o=new Array(r);for(n=0;n<a;++n)i[n]=ce(e[n],t[n]);for(;n<r;++n)o[n]=t[n];return function(e){for(n=0;n<a;++n)o[n]=i[n](e);return o}}function re(e,t){var n=new Date;return e=+e,t=+t,function(r){return n.setTime(e*(1-r)+t*r),n}}function ae(e,t){return e=+e,t=+t,function(n){return e*(1-n)+t*n}}function ie(e,t){var n,r={},a={};for(n in null!==e&&\"object\"==typeof e||(e={}),null!==t&&\"object\"==typeof t||(t={}),t)n in e?r[n]=ce(e[n],t[n]):a[n]=t[n];return function(e){for(n in r)a[n]=r[n](e);return a}}te((function(e){var t=e.length-1;return function(n){var r=n<=0?n=0:n>=1?(n=1,t-1):Math.floor(n*t),a=e[r],i=e[r+1],o=r>0?e[r-1]:2*a-i,u=r<t-1?e[r+2]:2*i-a;return Z((n-r/t)*t,o,a,i,u)}})),te((function(e){var t=e.length;return function(n){var r=Math.floor(((n%=1)<0?++n:n)*t),a=e[(r+t-1)%t],i=e[r%t],o=e[(r+1)%t],u=e[(r+2)%t];return Z((n-r/t)*t,a,i,o,u)}}));var oe=/[-+]?(?:\\d+\\.?\\d*|\\.?\\d+)(?:[eE][-+]?\\d+)?/g,ue=new RegExp(oe.source,\"g\");function le(e,t){var n,r,a,i=oe.lastIndex=ue.lastIndex=0,o=-1,u=[],l=[];for(e+=\"\",t+=\"\";(n=oe.exec(e))&&(r=ue.exec(t));)(a=r.index)>i&&(a=t.slice(i,a),u[o]?u[o]+=a:u[++o]=a),(n=n[0])===(r=r[0])?u[o]?u[o]+=r:u[++o]=r:(u[++o]=null,l.push({i:o,x:ae(n,r)})),i=ue.lastIndex;return i<t.length&&(a=t.slice(i),u[o]?u[o]+=a:u[++o]=a),u.length<2?l[0]?function(e){return function(t){return e(t)+\"\"}}(l[0].x):function(e){return function(){return e}}(t):(t=l.length,function(e){for(var n,r=0;r<t;++r)u[(n=l[r]).i]=n.x(e);return u.join(\"\")})}function se(e,t){t||(t=[]);var n,r=e?Math.min(t.length,e.length):0,a=t.slice();return function(i){for(n=0;n<r;++n)a[n]=e[n]*(1-i)+t[n]*i;return a}}function ce(e,t){var n,r,a=typeof t;return null==t||\"boolean\"===a?X(t):(\"number\"===a?ae:\"string\"===a?(n=A(t))?(t=n,ee):le:t instanceof A?ee:t instanceof Date?re:(r=t,!ArrayBuffer.isView(r)||r instanceof DataView?Array.isArray(t)?ne:\"function\"!=typeof t.valueOf&&\"function\"!=typeof t.toString||isNaN(t)?ie:ae:se))(e,t)}function fe(e,t){return e=+e,t=+t,function(n){return Math.round(e*(1-n)+t*n)}}function pe(e){return+e}var de=[0,1];function he(e){return e}function ve(e,t){return(t-=e=+e)?function(n){return(n-e)/t}:(n=isNaN(t)?NaN:.5,function(){return n});var n}function ge(e,t,n){var r=e[0],a=e[1],i=t[0],o=t[1];return a<r?(r=ve(a,r),i=n(o,i)):(r=ve(r,a),i=n(i,o)),function(e){return i(r(e))}}function ye(e,t,n){var r=Math.min(e.length,t.length)-1,a=new Array(r),i=new Array(r),o=-1;for(e[r]<e[0]&&(e=e.slice().reverse(),t=t.slice().reverse());++o<r;)a[o]=ve(e[o],e[o+1]),i[o]=n(t[o],t[o+1]);return function(t){var n=v(e,t,1,r)-1;return i[n](a[n](t))}}function me(e,t){return t.domain(e.domain()).range(e.range()).interpolate(e.interpolate()).clamp(e.clamp()).unknown(e.unknown())}function be(){return function(){var e,t,n,r,a,i,o=de,u=de,l=ce,s=he;function c(){var e,t,n,l=Math.min(o.length,u.length);return s!==he&&(e=o[0],t=o[l-1],e>t&&(n=e,e=t,t=n),s=function(n){return Math.max(e,Math.min(t,n))}),r=l>2?ye:ge,a=i=null,f}function f(t){return null==t||isNaN(t=+t)?n:(a||(a=r(o.map(e),u,l)))(e(s(t)))}return f.invert=function(n){return s(t((i||(i=r(u,o.map(e),ae)))(n)))},f.domain=function(e){return arguments.length?(o=Array.from(e,pe),c()):o.slice()},f.range=function(e){return arguments.length?(u=Array.from(e),c()):u.slice()},f.rangeRound=function(e){return u=Array.from(e),l=fe,c()},f.clamp=function(e){return arguments.length?(s=!!e||he,c()):s!==he},f.interpolate=function(e){return arguments.length?(l=e,c()):l},f.unknown=function(e){return arguments.length?(n=e,f):n},function(n,r){return e=n,t=r,c()}}()(he,he)}function _e(e,t){switch(arguments.length){case 0:break;case 1:this.range(e);break;default:this.range(t).domain(e)}return this}var we,xe=/^(?:(.)?([<>=^]))?([+\\-( ])?([$#])?(0)?(\\d+)?(,)?(\\.\\d+)?(~)?([a-z%])?$/i;function ke(e){if(!(t=xe.exec(e)))throw new Error(\"invalid format: \"+e);var t;return new Se({fill:t[1],align:t[2],sign:t[3],symbol:t[4],zero:t[5],width:t[6],comma:t[7],precision:t[8]&&t[8].slice(1),trim:t[9],type:t[10]})}function Se(e){this.fill=void 0===e.fill?\" \":e.fill+\"\",this.align=void 0===e.align?\">\":e.align+\"\",this.sign=void 0===e.sign?\"-\":e.sign+\"\",this.symbol=void 0===e.symbol?\"\":e.symbol+\"\",this.zero=!!e.zero,this.width=void 0===e.width?void 0:+e.width,this.comma=!!e.comma,this.precision=void 0===e.precision?void 0:+e.precision,this.trim=!!e.trim,this.type=void 0===e.type?\"\":e.type+\"\"}function Ee(e,t){if((n=(e=t?e.toExponential(t-1):e.toExponential()).indexOf(\"e\"))<0)return null;var n,r=e.slice(0,n);return[r.length>1?r[0]+r.slice(2):r,+e.slice(n+1)]}function Ce(e){return(e=Ee(Math.abs(e)))?e[1]:NaN}function Te(e,t){var n=Ee(e,t);if(!n)return e+\"\";var r=n[0],a=n[1];return a<0?\"0.\"+new Array(-a).join(\"0\")+r:r.length>a+1?r.slice(0,a+1)+\".\"+r.slice(a+1):r+new Array(a-r.length+2).join(\"0\")}ke.prototype=Se.prototype,Se.prototype.toString=function(){return this.fill+this.align+this.sign+this.symbol+(this.zero?\"0\":\"\")+(void 0===this.width?\"\":Math.max(1,0|this.width))+(this.comma?\",\":\"\")+(void 0===this.precision?\"\":\".\"+Math.max(0,0|this.precision))+(this.trim?\"~\":\"\")+this.type};const Me={\"%\":(e,t)=>(100*e).toFixed(t),b:e=>Math.round(e).toString(2),c:e=>e+\"\",d:function(e){return Math.abs(e=Math.round(e))>=1e21?e.toLocaleString(\"en\").replace(/,/g,\"\"):e.toString(10)},e:(e,t)=>e.toExponential(t),f:(e,t)=>e.toFixed(t),g:(e,t)=>e.toPrecision(t),o:e=>Math.round(e).toString(8),p:(e,t)=>Te(100*e,t),r:Te,s:function(e,t){var n=Ee(e,t);if(!n)return e+\"\";var r=n[0],a=n[1],i=a-(we=3*Math.max(-8,Math.min(8,Math.floor(a/3))))+1,o=r.length;return i===o?r:i>o?r+new Array(i-o+1).join(\"0\"):i>0?r.slice(0,i)+\".\"+r.slice(i):\"0.\"+new Array(1-i).join(\"0\")+Ee(e,Math.max(0,t+i-1))[0]},X:e=>Math.round(e).toString(16).toUpperCase(),x:e=>Math.round(e).toString(16)};function Ne(e){return e}var Pe,ze,Le,Oe=Array.prototype.map,Ae=[\"y\",\"z\",\"a\",\"f\",\"p\",\"n\",\"µ\",\"m\",\"\",\"k\",\"M\",\"G\",\"T\",\"P\",\"E\",\"Z\",\"Y\"];function Fe(e){var t=e.domain;return e.ticks=function(e){var n=t();return function(e,t,n){if(!((n=+n)>0))return[];if((e=+e)==(t=+t))return[e];const r=t<e,[a,i,u]=r?o(t,e,n):o(e,t,n);if(!(i>=a))return[];const l=i-a+1,s=new Array(l);if(r)if(u<0)for(let e=0;e<l;++e)s[e]=(i-e)/-u;else for(let e=0;e<l;++e)s[e]=(i-e)*u;else if(u<0)for(let e=0;e<l;++e)s[e]=(a+e)/-u;else for(let e=0;e<l;++e)s[e]=(a+e)*u;return s}(n[0],n[n.length-1],null==e?10:e)},e.tickFormat=function(e,n){var r=t();return function(e,t,n,r){var a,i=l(e,t,n);switch((r=ke(null==r?\",f\":r)).type){case\"s\":var o=Math.max(Math.abs(e),Math.abs(t));return null!=r.precision||isNaN(a=function(e,t){return Math.max(0,3*Math.max(-8,Math.min(8,Math.floor(Ce(t)/3)))-Ce(Math.abs(e)))}(i,o))||(r.precision=a),Le(r,o);case\"\":case\"e\":case\"g\":case\"p\":case\"r\":null!=r.precision||isNaN(a=function(e,t){return e=Math.abs(e),t=Math.abs(t)-e,Math.max(0,Ce(t)-Ce(e))+1}(i,Math.max(Math.abs(e),Math.abs(t))))||(r.precision=a-(\"e\"===r.type));break;case\"f\":case\"%\":null!=r.precision||isNaN(a=function(e){return Math.max(0,-Ce(Math.abs(e)))}(i))||(r.precision=a-2*(\"%\"===r.type))}return ze(r)}(r[0],r[r.length-1],null==e?10:e,n)},e.nice=function(n){null==n&&(n=10);var r,a,i=t(),o=0,l=i.length-1,s=i[o],c=i[l],f=10;for(c<s&&(a=s,s=c,c=a,a=o,o=l,l=a);f-- >0;){if((a=u(s,c,n))===r)return i[o]=s,i[l]=c,t(i);if(a>0)s=Math.floor(s/a)*a,c=Math.ceil(c/a)*a;else{if(!(a<0))break;s=Math.ceil(s*a)/a,c=Math.floor(c*a)/a}r=a}return e},e}function De(){var e=be();return e.copy=function(){return me(e,De())},_e.apply(e,arguments),Fe(e)}Pe=function(e){var t,n,r=void 0===e.grouping||void 0===e.thousands?Ne:(t=Oe.call(e.grouping,Number),n=e.thousands+\"\",function(e,r){for(var a=e.length,i=[],o=0,u=t[0],l=0;a>0&&u>0&&(l+u+1>r&&(u=Math.max(1,r-l)),i.push(e.substring(a-=u,a+u)),!((l+=u+1)>r));)u=t[o=(o+1)%t.length];return i.reverse().join(n)}),a=void 0===e.currency?\"\":e.currency[0]+\"\",i=void 0===e.currency?\"\":e.currency[1]+\"\",o=void 0===e.decimal?\".\":e.decimal+\"\",u=void 0===e.numerals?Ne:function(e){return function(t){return t.replace(/[0-9]/g,(function(t){return e[+t]}))}}(Oe.call(e.numerals,String)),l=void 0===e.percent?\"%\":e.percent+\"\",s=void 0===e.minus?\"−\":e.minus+\"\",c=void 0===e.nan?\"NaN\":e.nan+\"\";function f(e){var t=(e=ke(e)).fill,n=e.align,f=e.sign,p=e.symbol,d=e.zero,h=e.width,v=e.comma,g=e.precision,y=e.trim,m=e.type;\"n\"===m?(v=!0,m=\"g\"):Me[m]||(void 0===g&&(g=12),y=!0,m=\"g\"),(d||\"0\"===t&&\"=\"===n)&&(d=!0,t=\"0\",n=\"=\");var b=\"$\"===p?a:\"#\"===p&&/[boxX]/.test(m)?\"0\"+m.toLowerCase():\"\",_=\"$\"===p?i:/[%p]/.test(m)?l:\"\",w=Me[m],x=/[defgprs%]/.test(m);function k(e){var a,i,l,p=b,k=_;if(\"c\"===m)k=w(e)+k,e=\"\";else{var S=(e=+e)<0||1/e<0;if(e=isNaN(e)?c:w(Math.abs(e),g),y&&(e=function(e){e:for(var t,n=e.length,r=1,a=-1;r<n;++r)switch(e[r]){case\".\":a=t=r;break;case\"0\":0===a&&(a=r),t=r;break;default:if(!+e[r])break e;a>0&&(a=0)}return a>0?e.slice(0,a)+e.slice(t+1):e}(e)),S&&0==+e&&\"+\"!==f&&(S=!1),p=(S?\"(\"===f?f:s:\"-\"===f||\"(\"===f?\"\":f)+p,k=(\"s\"===m?Ae[8+we/3]:\"\")+k+(S&&\"(\"===f?\")\":\"\"),x)for(a=-1,i=e.length;++a<i;)if(48>(l=e.charCodeAt(a))||l>57){k=(46===l?o+e.slice(a+1):e.slice(a))+k,e=e.slice(0,a);break}}v&&!d&&(e=r(e,1/0));var E=p.length+e.length+k.length,C=E<h?new Array(h-E+1).join(t):\"\";switch(v&&d&&(e=r(C+e,C.length?h-k.length:1/0),C=\"\"),n){case\"<\":e=p+e+k+C;break;case\"=\":e=p+C+e+k;break;case\"^\":e=C.slice(0,E=C.length>>1)+p+e+k+C.slice(E);break;default:e=C+p+e+k}return u(e)}return g=void 0===g?6:/[gprs]/.test(m)?Math.max(1,Math.min(21,g)):Math.max(0,Math.min(20,g)),k.toString=function(){return e+\"\"},k}return{format:f,formatPrefix:function(e,t){var n=f(((e=ke(e)).type=\"f\",e)),r=3*Math.max(-8,Math.min(8,Math.floor(Ce(t)/3))),a=Math.pow(10,-r),i=Ae[8+r/3];return function(e){return n(a*e)+i}}}}({thousands:\",\",grouping:[3],currency:[\"$\",\"\"]}),ze=Pe.format,Le=Pe.formatPrefix;var Re=n(486);const je={colors:{RdBu:[\"rgb(255, 13, 87)\",\"rgb(30, 136, 229)\"],GnPR:[\"rgb(24, 196, 93)\",\"rgb(124, 82, 255)\"],CyPU:[\"#0099C6\",\"#990099\"],PkYg:[\"#DD4477\",\"#66AA00\"],DrDb:[\"#B82E2E\",\"#316395\"],LpLb:[\"#994499\",\"#22AA99\"],YlDp:[\"#AAAA11\",\"#6633CC\"],OrId:[\"#E67300\",\"#3E0099\"]},gray:\"#777\"};function Ue(e){return Ue=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Ue(e)}function Ie(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Ue(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Ue(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Ue(a)?a:String(a)),r)}var a}function $e(e,t){return $e=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},$e(e,t)}function Be(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function We(e){return We=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},We(e)}var Ve=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&$e(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=We(a);if(i){var n=We(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Ue(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return Be(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),(e=o.call(this)).width=100,window.lastSimpleListInstance=Be(e),e.effectFormat=ze(\".2\"),e}return n=u,(r=[{key:\"render\",value:function(){var t=this,n=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?n=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),n=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(n=this.props.plot_cmap),console.log(this.props.features,this.props.features),this.scale=De().domain([0,(0,Re.max)((0,Re.map)(this.props.features,(function(e){return Math.abs(e.effect)})))]).range([0,this.width]);var r=(0,Re.reverse)((0,Re.sortBy)(Object.keys(this.props.features),(function(e){return Math.abs(t.props.features[e].effect)}))).map((function(r){var a,i,o=t.props.features[r],u=t.props.featureNames[r],l={width:t.scale(Math.abs(o.effect)),height:\"20px\",background:o.effect<0?n[0]:n[1],display:\"inline-block\"},s={lineHeight:\"20px\",display:\"inline-block\",width:t.width+40,verticalAlign:\"top\",marginRight:\"5px\",textAlign:\"right\"},c={lineHeight:\"20px\",display:\"inline-block\",width:t.width+40,verticalAlign:\"top\",marginLeft:\"5px\"};return o.effect<0?(i=e.createElement(\"span\",{style:c},u),s.width=40+t.width-t.scale(Math.abs(o.effect)),s.textAlign=\"right\",s.color=\"#999\",s.fontSize=\"13px\",a=e.createElement(\"span\",{style:s},t.effectFormat(o.effect))):(s.textAlign=\"right\",a=e.createElement(\"span\",{style:s},u),c.width=40,c.textAlign=\"left\",c.color=\"#999\",c.fontSize=\"13px\",i=e.createElement(\"span\",{style:c},t.effectFormat(o.effect))),e.createElement(\"div\",{key:r,style:{marginTop:\"2px\"}},a,e.createElement(\"div\",{style:l}),i)}));return e.createElement(\"span\",null,r)}}])&&Ie(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);Ve.defaultProps={plot_cmap:\"RdBu\"};const He=Ve;function qe(){}function Qe(e){return null==e?qe:function(){return this.querySelector(e)}}function Ye(){return[]}function Ge(e){return function(t){return t.matches(e)}}var Ke=Array.prototype.find;function Ze(){return this.firstElementChild}var Xe=Array.prototype.filter;function Je(){return Array.from(this.children)}function et(e){return new Array(e.length)}function tt(e,t){this.ownerDocument=e.ownerDocument,this.namespaceURI=e.namespaceURI,this._next=null,this._parent=e,this.__data__=t}function nt(e,t,n,r,a,i){for(var o,u=0,l=t.length,s=i.length;u<s;++u)(o=t[u])?(o.__data__=i[u],r[u]=o):n[u]=new tt(e,i[u]);for(;u<l;++u)(o=t[u])&&(a[u]=o)}function rt(e,t,n,r,a,i,o){var u,l,s,c=new Map,f=t.length,p=i.length,d=new Array(f);for(u=0;u<f;++u)(l=t[u])&&(d[u]=s=o.call(l,l.__data__,u,t)+\"\",c.has(s)?a[u]=l:c.set(s,l));for(u=0;u<p;++u)s=o.call(e,i[u],u,i)+\"\",(l=c.get(s))?(r[u]=l,l.__data__=i[u],c.delete(s)):n[u]=new tt(e,i[u]);for(u=0;u<f;++u)(l=t[u])&&c.get(d[u])===l&&(a[u]=l)}function at(e){return e.__data__}function it(e){return\"object\"==typeof e&&\"length\"in e?e:Array.from(e)}function ot(e,t){return e<t?-1:e>t?1:e>=t?0:NaN}tt.prototype={constructor:tt,appendChild:function(e){return this._parent.insertBefore(e,this._next)},insertBefore:function(e,t){return this._parent.insertBefore(e,t)},querySelector:function(e){return this._parent.querySelector(e)},querySelectorAll:function(e){return this._parent.querySelectorAll(e)}};var ut=\"http://www.w3.org/1999/xhtml\";const lt={svg:\"http://www.w3.org/2000/svg\",xhtml:ut,xlink:\"http://www.w3.org/1999/xlink\",xml:\"http://www.w3.org/XML/1998/namespace\",xmlns:\"http://www.w3.org/2000/xmlns/\"};function st(e){var t=e+=\"\",n=t.indexOf(\":\");return n>=0&&\"xmlns\"!==(t=e.slice(0,n))&&(e=e.slice(n+1)),lt.hasOwnProperty(t)?{space:lt[t],local:e}:e}function ct(e){return function(){this.removeAttribute(e)}}function ft(e){return function(){this.removeAttributeNS(e.space,e.local)}}function pt(e,t){return function(){this.setAttribute(e,t)}}function dt(e,t){return function(){this.setAttributeNS(e.space,e.local,t)}}function ht(e,t){return function(){var n=t.apply(this,arguments);null==n?this.removeAttribute(e):this.setAttribute(e,n)}}function vt(e,t){return function(){var n=t.apply(this,arguments);null==n?this.removeAttributeNS(e.space,e.local):this.setAttributeNS(e.space,e.local,n)}}function gt(e){return e.ownerDocument&&e.ownerDocument.defaultView||e.document&&e||e.defaultView}function yt(e){return function(){this.style.removeProperty(e)}}function mt(e,t,n){return function(){this.style.setProperty(e,t,n)}}function bt(e,t,n){return function(){var r=t.apply(this,arguments);null==r?this.style.removeProperty(e):this.style.setProperty(e,r,n)}}function _t(e){return function(){delete this[e]}}function wt(e,t){return function(){this[e]=t}}function xt(e,t){return function(){var n=t.apply(this,arguments);null==n?delete this[e]:this[e]=n}}function kt(e){return e.trim().split(/^|\\s+/)}function St(e){return e.classList||new Et(e)}function Et(e){this._node=e,this._names=kt(e.getAttribute(\"class\")||\"\")}function Ct(e,t){for(var n=St(e),r=-1,a=t.length;++r<a;)n.add(t[r])}function Tt(e,t){for(var n=St(e),r=-1,a=t.length;++r<a;)n.remove(t[r])}function Mt(e){return function(){Ct(this,e)}}function Nt(e){return function(){Tt(this,e)}}function Pt(e,t){return function(){(t.apply(this,arguments)?Ct:Tt)(this,e)}}function zt(){this.textContent=\"\"}function Lt(e){return function(){this.textContent=e}}function Ot(e){return function(){var t=e.apply(this,arguments);this.textContent=null==t?\"\":t}}function At(){this.innerHTML=\"\"}function Ft(e){return function(){this.innerHTML=e}}function Dt(e){return function(){var t=e.apply(this,arguments);this.innerHTML=null==t?\"\":t}}function Rt(){this.nextSibling&&this.parentNode.appendChild(this)}function jt(){this.previousSibling&&this.parentNode.insertBefore(this,this.parentNode.firstChild)}function Ut(e){return function(){var t=this.ownerDocument,n=this.namespaceURI;return n===ut&&t.documentElement.namespaceURI===ut?t.createElement(e):t.createElementNS(n,e)}}function It(e){return function(){return this.ownerDocument.createElementNS(e.space,e.local)}}function $t(e){var t=st(e);return(t.local?It:Ut)(t)}function Bt(){return null}function Wt(){var e=this.parentNode;e&&e.removeChild(this)}function Vt(){var e=this.cloneNode(!1),t=this.parentNode;return t?t.insertBefore(e,this.nextSibling):e}function Ht(){var e=this.cloneNode(!0),t=this.parentNode;return t?t.insertBefore(e,this.nextSibling):e}function qt(e){return function(){var t=this.__on;if(t){for(var n,r=0,a=-1,i=t.length;r<i;++r)n=t[r],e.type&&n.type!==e.type||n.name!==e.name?t[++a]=n:this.removeEventListener(n.type,n.listener,n.options);++a?t.length=a:delete this.__on}}}function Qt(e,t,n){return function(){var r,a=this.__on,i=function(e){return function(t){e.call(this,t,this.__data__)}}(t);if(a)for(var o=0,u=a.length;o<u;++o)if((r=a[o]).type===e.type&&r.name===e.name)return this.removeEventListener(r.type,r.listener,r.options),this.addEventListener(r.type,r.listener=i,r.options=n),void(r.value=t);this.addEventListener(e.type,i,n),r={type:e.type,name:e.name,value:t,listener:i,options:n},a?a.push(r):this.__on=[r]}}function Yt(e,t,n){var r=gt(e),a=r.CustomEvent;\"function\"==typeof a?a=new a(t,n):(a=r.document.createEvent(\"Event\"),n?(a.initEvent(t,n.bubbles,n.cancelable),a.detail=n.detail):a.initEvent(t,!1,!1)),e.dispatchEvent(a)}function Gt(e,t){return function(){return Yt(this,e,t)}}function Kt(e,t){return function(){return Yt(this,e,t.apply(this,arguments))}}Et.prototype={add:function(e){this._names.indexOf(e)<0&&(this._names.push(e),this._node.setAttribute(\"class\",this._names.join(\" \")))},remove:function(e){var t=this._names.indexOf(e);t>=0&&(this._names.splice(t,1),this._node.setAttribute(\"class\",this._names.join(\" \")))},contains:function(e){return this._names.indexOf(e)>=0}};var Zt=[null];function Xt(e,t){this._groups=e,this._parents=t}function Jt(e){return\"string\"==typeof e?new Xt([[document.querySelector(e)]],[document.documentElement]):new Xt([[e]],Zt)}function en(e){return e}Xt.prototype=function(){return new Xt([[document.documentElement]],Zt)}.prototype={constructor:Xt,select:function(e){\"function\"!=typeof e&&(e=Qe(e));for(var t=this._groups,n=t.length,r=new Array(n),a=0;a<n;++a)for(var i,o,u=t[a],l=u.length,s=r[a]=new Array(l),c=0;c<l;++c)(i=u[c])&&(o=e.call(i,i.__data__,c,u))&&(\"__data__\"in i&&(o.__data__=i.__data__),s[c]=o);return new Xt(r,this._parents)},selectAll:function(e){e=\"function\"==typeof e?function(e){return function(){return null==(t=e.apply(this,arguments))?[]:Array.isArray(t)?t:Array.from(t);var t}}(e):function(e){return null==e?Ye:function(){return this.querySelectorAll(e)}}(e);for(var t=this._groups,n=t.length,r=[],a=[],i=0;i<n;++i)for(var o,u=t[i],l=u.length,s=0;s<l;++s)(o=u[s])&&(r.push(e.call(o,o.__data__,s,u)),a.push(o));return new Xt(r,a)},selectChild:function(e){return this.select(null==e?Ze:function(e){return function(){return Ke.call(this.children,e)}}(\"function\"==typeof e?e:Ge(e)))},selectChildren:function(e){return this.selectAll(null==e?Je:function(e){return function(){return Xe.call(this.children,e)}}(\"function\"==typeof e?e:Ge(e)))},filter:function(e){\"function\"!=typeof e&&(e=function(e){return function(){return this.matches(e)}}(e));for(var t=this._groups,n=t.length,r=new Array(n),a=0;a<n;++a)for(var i,o=t[a],u=o.length,l=r[a]=[],s=0;s<u;++s)(i=o[s])&&e.call(i,i.__data__,s,o)&&l.push(i);return new Xt(r,this._parents)},data:function(e,t){if(!arguments.length)return Array.from(this,at);var n,r=t?rt:nt,a=this._parents,i=this._groups;\"function\"!=typeof e&&(n=e,e=function(){return n});for(var o=i.length,u=new Array(o),l=new Array(o),s=new Array(o),c=0;c<o;++c){var f=a[c],p=i[c],d=p.length,h=it(e.call(f,f&&f.__data__,c,a)),v=h.length,g=l[c]=new Array(v),y=u[c]=new Array(v);r(f,p,g,y,s[c]=new Array(d),h,t);for(var m,b,_=0,w=0;_<v;++_)if(m=g[_]){for(_>=w&&(w=_+1);!(b=y[w])&&++w<v;);m._next=b||null}}return(u=new Xt(u,a))._enter=l,u._exit=s,u},enter:function(){return new Xt(this._enter||this._groups.map(et),this._parents)},exit:function(){return new Xt(this._exit||this._groups.map(et),this._parents)},join:function(e,t,n){var r=this.enter(),a=this,i=this.exit();return\"function\"==typeof e?(r=e(r))&&(r=r.selection()):r=r.append(e+\"\"),null!=t&&(a=t(a))&&(a=a.selection()),null==n?i.remove():n(i),r&&a?r.merge(a).order():a},merge:function(e){for(var t=e.selection?e.selection():e,n=this._groups,r=t._groups,a=n.length,i=r.length,o=Math.min(a,i),u=new Array(a),l=0;l<o;++l)for(var s,c=n[l],f=r[l],p=c.length,d=u[l]=new Array(p),h=0;h<p;++h)(s=c[h]||f[h])&&(d[h]=s);for(;l<a;++l)u[l]=n[l];return new Xt(u,this._parents)},selection:function(){return this},order:function(){for(var e=this._groups,t=-1,n=e.length;++t<n;)for(var r,a=e[t],i=a.length-1,o=a[i];--i>=0;)(r=a[i])&&(o&&4^r.compareDocumentPosition(o)&&o.parentNode.insertBefore(r,o),o=r);return this},sort:function(e){function t(t,n){return t&&n?e(t.__data__,n.__data__):!t-!n}e||(e=ot);for(var n=this._groups,r=n.length,a=new Array(r),i=0;i<r;++i){for(var o,u=n[i],l=u.length,s=a[i]=new Array(l),c=0;c<l;++c)(o=u[c])&&(s[c]=o);s.sort(t)}return new Xt(a,this._parents).order()},call:function(){var e=arguments[0];return arguments[0]=this,e.apply(null,arguments),this},nodes:function(){return Array.from(this)},node:function(){for(var e=this._groups,t=0,n=e.length;t<n;++t)for(var r=e[t],a=0,i=r.length;a<i;++a){var o=r[a];if(o)return o}return null},size:function(){let e=0;for(const t of this)++e;return e},empty:function(){return!this.node()},each:function(e){for(var t=this._groups,n=0,r=t.length;n<r;++n)for(var a,i=t[n],o=0,u=i.length;o<u;++o)(a=i[o])&&e.call(a,a.__data__,o,i);return this},attr:function(e,t){var n=st(e);if(arguments.length<2){var r=this.node();return n.local?r.getAttributeNS(n.space,n.local):r.getAttribute(n)}return this.each((null==t?n.local?ft:ct:\"function\"==typeof t?n.local?vt:ht:n.local?dt:pt)(n,t))},style:function(e,t,n){return arguments.length>1?this.each((null==t?yt:\"function\"==typeof t?bt:mt)(e,t,null==n?\"\":n)):function(e,t){return e.style.getPropertyValue(t)||gt(e).getComputedStyle(e,null).getPropertyValue(t)}(this.node(),e)},property:function(e,t){return arguments.length>1?this.each((null==t?_t:\"function\"==typeof t?xt:wt)(e,t)):this.node()[e]},classed:function(e,t){var n=kt(e+\"\");if(arguments.length<2){for(var r=St(this.node()),a=-1,i=n.length;++a<i;)if(!r.contains(n[a]))return!1;return!0}return this.each((\"function\"==typeof t?Pt:t?Mt:Nt)(n,t))},text:function(e){return arguments.length?this.each(null==e?zt:(\"function\"==typeof e?Ot:Lt)(e)):this.node().textContent},html:function(e){return arguments.length?this.each(null==e?At:(\"function\"==typeof e?Dt:Ft)(e)):this.node().innerHTML},raise:function(){return this.each(Rt)},lower:function(){return this.each(jt)},append:function(e){var t=\"function\"==typeof e?e:$t(e);return this.select((function(){return this.appendChild(t.apply(this,arguments))}))},insert:function(e,t){var n=\"function\"==typeof e?e:$t(e),r=null==t?Bt:\"function\"==typeof t?t:Qe(t);return this.select((function(){return this.insertBefore(n.apply(this,arguments),r.apply(this,arguments)||null)}))},remove:function(){return this.each(Wt)},clone:function(e){return this.select(e?Ht:Vt)},datum:function(e){return arguments.length?this.property(\"__data__\",e):this.node().__data__},on:function(e,t,n){var r,a,i=function(e){return e.trim().split(/^|\\s+/).map((function(e){var t=\"\",n=e.indexOf(\".\");return n>=0&&(t=e.slice(n+1),e=e.slice(0,n)),{type:e,name:t}}))}(e+\"\"),o=i.length;if(!(arguments.length<2)){for(u=t?Qt:qt,r=0;r<o;++r)this.each(u(i[r],t,n));return this}var u=this.node().__on;if(u)for(var l,s=0,c=u.length;s<c;++s)for(r=0,l=u[s];r<o;++r)if((a=i[r]).type===l.type&&a.name===l.name)return l.value},dispatch:function(e,t){return this.each((\"function\"==typeof t?Kt:Gt)(e,t))},[Symbol.iterator]:function*(){for(var e=this._groups,t=0,n=e.length;t<n;++t)for(var r,a=e[t],i=0,o=a.length;i<o;++i)(r=a[i])&&(yield r)}};var tn=1,nn=2,rn=3,an=4,on=1e-6;function un(e){return\"translate(\"+e+\",0)\"}function ln(e){return\"translate(0,\"+e+\")\"}function sn(e){return t=>+e(t)}function cn(e,t){return t=Math.max(0,e.bandwidth()-2*t)/2,e.round()&&(t=Math.round(t)),n=>+e(n)+t}function fn(){return!this.__axis}function pn(e,t){var n=[],r=null,a=null,i=6,o=6,u=3,l=\"undefined\"!=typeof window&&window.devicePixelRatio>1?0:.5,s=e===tn||e===an?-1:1,c=e===an||e===nn?\"x\":\"y\",f=e===tn||e===rn?un:ln;function p(p){var d=null==r?t.ticks?t.ticks.apply(t,n):t.domain():r,h=null==a?t.tickFormat?t.tickFormat.apply(t,n):en:a,v=Math.max(i,0)+u,g=t.range(),y=+g[0]+l,m=+g[g.length-1]+l,b=(t.bandwidth?cn:sn)(t.copy(),l),_=p.selection?p.selection():p,w=_.selectAll(\".domain\").data([null]),x=_.selectAll(\".tick\").data(d,t).order(),k=x.exit(),S=x.enter().append(\"g\").attr(\"class\",\"tick\"),E=x.select(\"line\"),C=x.select(\"text\");w=w.merge(w.enter().insert(\"path\",\".tick\").attr(\"class\",\"domain\").attr(\"stroke\",\"currentColor\")),x=x.merge(S),E=E.merge(S.append(\"line\").attr(\"stroke\",\"currentColor\").attr(c+\"2\",s*i)),C=C.merge(S.append(\"text\").attr(\"fill\",\"currentColor\").attr(c,s*v).attr(\"dy\",e===tn?\"0em\":e===rn?\"0.71em\":\"0.32em\")),p!==_&&(w=w.transition(p),x=x.transition(p),E=E.transition(p),C=C.transition(p),k=k.transition(p).attr(\"opacity\",on).attr(\"transform\",(function(e){return isFinite(e=b(e))?f(e+l):this.getAttribute(\"transform\")})),S.attr(\"opacity\",on).attr(\"transform\",(function(e){var t=this.parentNode.__axis;return f((t&&isFinite(t=t(e))?t:b(e))+l)}))),k.remove(),w.attr(\"d\",e===an||e===nn?o?\"M\"+s*o+\",\"+y+\"H\"+l+\"V\"+m+\"H\"+s*o:\"M\"+l+\",\"+y+\"V\"+m:o?\"M\"+y+\",\"+s*o+\"V\"+l+\"H\"+m+\"V\"+s*o:\"M\"+y+\",\"+l+\"H\"+m),x.attr(\"opacity\",1).attr(\"transform\",(function(e){return f(b(e)+l)})),E.attr(c+\"2\",s*i),C.attr(c,s*v).text(h),_.filter(fn).attr(\"fill\",\"none\").attr(\"font-size\",10).attr(\"font-family\",\"sans-serif\").attr(\"text-anchor\",e===nn?\"start\":e===an?\"end\":\"middle\"),_.each((function(){this.__axis=b}))}return p.scale=function(e){return arguments.length?(t=e,p):t},p.ticks=function(){return n=Array.from(arguments),p},p.tickArguments=function(e){return arguments.length?(n=null==e?[]:Array.from(e),p):n.slice()},p.tickValues=function(e){return arguments.length?(r=null==e?null:Array.from(e),p):r&&r.slice()},p.tickFormat=function(e){return arguments.length?(a=e,p):a},p.tickSize=function(e){return arguments.length?(i=o=+e,p):i},p.tickSizeInner=function(e){return arguments.length?(i=+e,p):i},p.tickSizeOuter=function(e){return arguments.length?(o=+e,p):o},p.tickPadding=function(e){return arguments.length?(u=+e,p):u},p.offset=function(e){return arguments.length?(l=+e,p):l},p}function dn(e){return pn(rn,e)}function hn(e){return function(){return e}}function vn(e){this._context=e}function gn(e){return new vn(e)}Array.prototype.slice,vn.prototype={areaStart:function(){this._line=0},areaEnd:function(){this._line=NaN},lineStart:function(){this._point=0},lineEnd:function(){(this._line||0!==this._line&&1===this._point)&&this._context.closePath(),this._line=1-this._line},point:function(e,t){switch(e=+e,t=+t,this._point){case 0:this._point=1,this._line?this._context.lineTo(e,t):this._context.moveTo(e,t);break;case 1:this._point=2;default:this._context.lineTo(e,t)}}};const yn=Math.PI,mn=2*yn,bn=1e-6,_n=mn-bn;function wn(e){this._+=e[0];for(let t=1,n=e.length;t<n;++t)this._+=arguments[t]+e[t]}class xn{constructor(e){this._x0=this._y0=this._x1=this._y1=null,this._=\"\",this._append=null==e?wn:function(e){let t=Math.floor(e);if(!(t>=0))throw new Error(`invalid digits: ${e}`);if(t>15)return wn;const n=10**t;return function(e){this._+=e[0];for(let t=1,r=e.length;t<r;++t)this._+=Math.round(arguments[t]*n)/n+e[t]}}(e)}moveTo(e,t){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+t}`}closePath(){null!==this._x1&&(this._x1=this._x0,this._y1=this._y0,this._append`Z`)}lineTo(e,t){this._append`L${this._x1=+e},${this._y1=+t}`}quadraticCurveTo(e,t,n,r){this._append`Q${+e},${+t},${this._x1=+n},${this._y1=+r}`}bezierCurveTo(e,t,n,r,a,i){this._append`C${+e},${+t},${+n},${+r},${this._x1=+a},${this._y1=+i}`}arcTo(e,t,n,r,a){if(e=+e,t=+t,n=+n,r=+r,(a=+a)<0)throw new Error(`negative radius: ${a}`);let i=this._x1,o=this._y1,u=n-e,l=r-t,s=i-e,c=o-t,f=s*s+c*c;if(null===this._x1)this._append`M${this._x1=e},${this._y1=t}`;else if(f>bn)if(Math.abs(c*u-l*s)>bn&&a){let p=n-i,d=r-o,h=u*u+l*l,v=p*p+d*d,g=Math.sqrt(h),y=Math.sqrt(f),m=a*Math.tan((yn-Math.acos((h+f-v)/(2*g*y)))/2),b=m/y,_=m/g;Math.abs(b-1)>bn&&this._append`L${e+b*s},${t+b*c}`,this._append`A${a},${a},0,0,${+(c*p>s*d)},${this._x1=e+_*u},${this._y1=t+_*l}`}else this._append`L${this._x1=e},${this._y1=t}`}arc(e,t,n,r,a,i){if(e=+e,t=+t,i=!!i,(n=+n)<0)throw new Error(`negative radius: ${n}`);let o=n*Math.cos(r),u=n*Math.sin(r),l=e+o,s=t+u,c=1^i,f=i?r-a:a-r;null===this._x1?this._append`M${l},${s}`:(Math.abs(this._x1-l)>bn||Math.abs(this._y1-s)>bn)&&this._append`L${l},${s}`,n&&(f<0&&(f=f%mn+mn),f>_n?this._append`A${n},${n},0,1,${c},${e-o},${t-u}A${n},${n},0,1,${c},${this._x1=l},${this._y1=s}`:f>bn&&this._append`A${n},${n},0,${+(f>=yn)},${c},${this._x1=e+n*Math.cos(a)},${this._y1=t+n*Math.sin(a)}`)}rect(e,t,n,r){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+t}h${n=+n}v${+r}h${-n}Z`}toString(){return this._}}function kn(e){return e[0]}function Sn(e){return e[1]}function En(e,t){var n=hn(!0),r=null,a=gn,i=null,o=function(e){let t=3;return e.digits=function(n){if(!arguments.length)return t;if(null==n)t=null;else{const e=Math.floor(n);if(!(e>=0))throw new RangeError(`invalid digits: ${n}`);t=e}return e},()=>new xn(t)}(u);function u(u){var l,s,c,f=(u=function(e){return\"object\"==typeof e&&\"length\"in e?e:Array.from(e)}(u)).length,p=!1;for(null==r&&(i=a(c=o())),l=0;l<=f;++l)!(l<f&&n(s=u[l],l,u))===p&&((p=!p)?i.lineStart():i.lineEnd()),p&&i.point(+e(s,l,u),+t(s,l,u));if(c)return i=null,c+\"\"||null}return e=\"function\"==typeof e?e:void 0===e?kn:hn(e),t=\"function\"==typeof t?t:void 0===t?Sn:hn(t),u.x=function(t){return arguments.length?(e=\"function\"==typeof t?t:hn(+t),u):e},u.y=function(e){return arguments.length?(t=\"function\"==typeof e?e:hn(+e),u):t},u.defined=function(e){return arguments.length?(n=\"function\"==typeof e?e:hn(!!e),u):n},u.curve=function(e){return arguments.length?(a=e,null!=r&&(i=a(r)),u):a},u.context=function(e){return arguments.length?(null==e?r=i=null:i=a(r=e),u):r},u}function Cn(e){return Cn=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Cn(e)}function Tn(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Cn(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Cn(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Cn(a)?a:String(a)),r)}var a}function Mn(e,t){return Mn=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},Mn(e,t)}function Nn(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function Pn(e){return Pn=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},Pn(e)}var zn=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&Mn(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=Pn(a);if(i){var n=Pn(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Cn(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return Nn(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),e=o.call(this),window.lastAdditiveForceVisualizer=Nn(e),e.effectFormat=ze(\".2\"),e.redraw=(0,Re.debounce)((function(){return e.draw()}),200),e}return n=u,(r=[{key:\"componentDidMount\",value:function(){var e=this;this.mainGroup=this.svg.append(\"g\"),this.axisElement=this.mainGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-axis\"),this.onTopGroup=this.svg.append(\"g\"),this.baseValueTitle=this.svg.append(\"text\"),this.joinPointLine=this.svg.append(\"line\"),this.joinPointLabelOutline=this.svg.append(\"text\"),this.joinPointLabel=this.svg.append(\"text\"),this.joinPointTitleLeft=this.svg.append(\"text\"),this.joinPointTitleLeftArrow=this.svg.append(\"text\"),this.joinPointTitle=this.svg.append(\"text\"),this.joinPointTitleRightArrow=this.svg.append(\"text\"),this.joinPointTitleRight=this.svg.append(\"text\"),this.hoverLabelBacking=this.svg.append(\"text\").attr(\"x\",10).attr(\"y\",20).attr(\"text-anchor\",\"middle\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").text(\"\").on(\"mouseover\",(function(){e.hoverLabel.attr(\"opacity\",1),e.hoverLabelBacking.attr(\"opacity\",1)})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)})),this.hoverLabel=this.svg.append(\"text\").attr(\"x\",10).attr(\"y\",20).attr(\"text-anchor\",\"middle\").attr(\"font-size\",12).attr(\"fill\",\"#0f0\").text(\"\").on(\"mouseover\",(function(){e.hoverLabel.attr(\"opacity\",1),e.hoverLabelBacking.attr(\"opacity\",1)})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)}));var t=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?t=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),t=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(t=this.props.plot_cmap),this.colors=t.map((function(e){return q(e)})),this.brighterColors=[1.45,1.6].map((function(t,n){return e.colors[n].brighter(t)})),this.colors.map((function(t,n){var r=e.svg.append(\"linearGradient\").attr(\"id\",\"linear-grad-\"+n).attr(\"x1\",\"0%\").attr(\"y1\",\"0%\").attr(\"x2\",\"0%\").attr(\"y2\",\"100%\");r.append(\"stop\").attr(\"offset\",\"0%\").attr(\"stop-color\",t).attr(\"stop-opacity\",.6),r.append(\"stop\").attr(\"offset\",\"100%\").attr(\"stop-color\",t).attr(\"stop-opacity\",0);var a=e.svg.append(\"linearGradient\").attr(\"id\",\"linear-backgrad-\"+n).attr(\"x1\",\"0%\").attr(\"y1\",\"0%\").attr(\"x2\",\"0%\").attr(\"y2\",\"100%\");a.append(\"stop\").attr(\"offset\",\"0%\").attr(\"stop-color\",t).attr(\"stop-opacity\",.5),a.append(\"stop\").attr(\"offset\",\"100%\").attr(\"stop-color\",t).attr(\"stop-opacity\",0)})),this.tickFormat=ze(\",.4\"),this.scaleCentered=De(),this.axis=dn().scale(this.scaleCentered).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.tickFormat(e.invLinkFunction(t))})).tickPadding(-18),window.addEventListener(\"resize\",this.redraw),window.setTimeout(this.redraw,50)}},{key:\"componentDidUpdate\",value:function(){this.draw()}},{key:\"draw\",value:function(){var e=this;(0,Re.each)(this.props.featureNames,(function(t,n){e.props.features[n]&&(e.props.features[n].name=t)})),\"identity\"===this.props.link?this.invLinkFunction=function(t){return e.props.baseValue+t}:\"logit\"===this.props.link?this.invLinkFunction=function(t){return 1/(1+Math.exp(-(e.props.baseValue+t)))}:console.log(\"ERROR: Unrecognized link function: \",this.props.link);var t=this.svg.node().parentNode.offsetWidth;if(0==t)return setTimeout((function(){return e.draw(e.props)}),500);this.svg.style(\"height\",\"150px\"),this.svg.style(\"width\",t+\"px\");var n=(0,Re.sortBy)(this.props.features,(function(e){return-1/(e.effect+1e-10)})),r=(0,Re.sum)((0,Re.map)(n,(function(e){return Math.abs(e.effect)}))),a=(0,Re.sum)((0,Re.map)((0,Re.filter)(n,(function(e){return e.effect>0})),(function(e){return e.effect})))||0,i=(0,Re.sum)((0,Re.map)((0,Re.filter)(n,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0;this.domainSize=3*Math.max(a,i);var o=De().domain([0,this.domainSize]).range([0,t]),u=t/2-o(i);this.scaleCentered.domain([-this.domainSize/2,this.domainSize/2]).range([0,t]).clamp(!0),this.axisElement.attr(\"transform\",\"translate(0,50)\").call(this.axis);var l,s,c,f=0;for(l=0;l<n.length;++l)n[l].x=f,n[l].effect<0&&void 0===s&&(s=f,c=l),f+=Math.abs(n[l].effect);void 0===s&&(s=f,c=l);var p=En().x((function(e){return e[0]})).y((function(e){return e[1]})),d=function(t){return void 0!==t.value&&null!==t.value&&\"\"!==t.value?t.name+\" = \"+(isNaN(t.value)?t.value:e.tickFormat(t.value)):t.name};n=this.props.hideBars?[]:n;var h=this.mainGroup.selectAll(\".force-bar-blocks\").data(n);h.enter().append(\"path\").attr(\"class\",\"force-bar-blocks\").merge(h).attr(\"d\",(function(e,t){var n=o(e.x)+u,r=o(Math.abs(e.effect)),a=e.effect<0?-4:4,i=a;return t===c&&(a=0),t===c-1&&(i=0),p([[n,56],[n+r,56],[n+r+i,64.5],[n+r,73],[n,73],[n+a,64.5]])})).attr(\"fill\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})).on(\"mouseover\",(function(t){if(o(Math.abs(t.effect))<o(r)/50||o(Math.abs(t.effect))<10){var n=o(t.x)+u,a=o(Math.abs(t.effect));e.hoverLabel.attr(\"opacity\",1).attr(\"x\",n+a/2).attr(\"y\",50.5).attr(\"fill\",t.effect>0?e.colors[0]:e.colors[1]).text(d(t)),e.hoverLabelBacking.attr(\"opacity\",1).attr(\"x\",n+a/2).attr(\"y\",50.5).text(d(t))}})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)})),h.exit().remove();var v=(0,Re.filter)(n,(function(e){return o(Math.abs(e.effect))>o(r)/50&&o(Math.abs(e.effect))>10})),g=this.onTopGroup.selectAll(\".force-bar-labels\").data(v);if(g.exit().remove(),g=g.enter().append(\"text\").attr(\"class\",\"force-bar-labels\").attr(\"font-size\",\"12px\").attr(\"y\",98).merge(g).text((function(t){return void 0!==t.value&&null!==t.value&&\"\"!==t.value?t.name+\" = \"+(isNaN(t.value)?t.value:e.tickFormat(t.value)):t.name})).attr(\"fill\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})).attr(\"stroke\",(function(e){return e.textWidth=Math.max(this.getComputedTextLength(),o(Math.abs(e.effect))-10),e.innerTextWidth=this.getComputedTextLength(),\"none\"})),this.filteredData=v,n.length>0){f=s+o.invert(5);for(var y=c;y<n.length;++y)n[y].textx=f,f+=o.invert(n[y].textWidth+10);f=s-o.invert(5);for(var m=c-1;m>=0;--m)n[m].textx=f,f-=o.invert(n[m].textWidth+10)}g.attr(\"x\",(function(e){return o(e.textx)+u+(e.effect>0?-e.textWidth/2:e.textWidth/2)})).attr(\"text-anchor\",\"middle\"),v=(0,Re.filter)(v,(function(n){return o(n.textx)+u>e.props.labelMargin&&o(n.textx)+u<t-e.props.labelMargin})),this.filteredData2=v;var b=v.slice(),_=(0,Re.findIndex)(n,v[0])-1;_>=0&&b.unshift(n[_]);var w=this.mainGroup.selectAll(\".force-bar-labelBacking\").data(v);w.enter().append(\"path\").attr(\"class\",\"force-bar-labelBacking\").attr(\"stroke\",\"none\").attr(\"opacity\",.2).merge(w).attr(\"d\",(function(e){return p([[o(e.x)+o(Math.abs(e.effect))+u,73],[(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5,83],[(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5,104],[(e.effect>0?o(e.textx)-e.textWidth:o(e.textx))+u-5,104],[(e.effect>0?o(e.textx)-e.textWidth:o(e.textx))+u-5,83],[o(e.x)+u,73]])})).attr(\"fill\",(function(e){return\"url(#linear-backgrad-\".concat(e.effect>0?0:1,\")\")})),w.exit().remove();var x=this.mainGroup.selectAll(\".force-bar-labelDividers\").data(v.slice(0,-1));x.enter().append(\"rect\").attr(\"class\",\"force-bar-labelDividers\").attr(\"height\",\"21px\").attr(\"width\",\"1px\").attr(\"y\",83).merge(x).attr(\"x\",(function(e){return(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+4.5})).attr(\"fill\",(function(e){return\"url(#linear-grad-\".concat(e.effect>0?0:1,\")\")})),x.exit().remove();var k=this.mainGroup.selectAll(\".force-bar-labelLinks\").data(v.slice(0,-1));k.enter().append(\"line\").attr(\"class\",\"force-bar-labelLinks\").attr(\"y1\",73).attr(\"y2\",83).attr(\"stroke-opacity\",.5).attr(\"stroke-width\",1).merge(k).attr(\"x1\",(function(e){return o(e.x)+o(Math.abs(e.effect))+u})).attr(\"x2\",(function(e){return(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5})).attr(\"stroke\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})),k.exit().remove();var S=this.mainGroup.selectAll(\".force-bar-blockDividers\").data(n.slice(0,-1));S.enter().append(\"path\").attr(\"class\",\"force-bar-blockDividers\").attr(\"stroke-width\",2).attr(\"fill\",\"none\").merge(S).attr(\"d\",(function(e){var t=o(e.x)+o(Math.abs(e.effect))+u;return p([[t,56],[t+(e.effect<0?-4:4),64.5],[t,73]])})).attr(\"stroke\",(function(t,n){return c===n+1||Math.abs(t.effect)<1e-8?\"#rgba(0,0,0,0)\":t.effect>0?e.brighterColors[0]:e.brighterColors[1]})),S.exit().remove(),this.joinPointLine.attr(\"x1\",o(s)+u).attr(\"x2\",o(s)+u).attr(\"y1\",50).attr(\"y2\",56).attr(\"stroke\",\"#F2F2F2\").attr(\"stroke-width\",1).attr(\"opacity\",1),this.joinPointLabelOutline.attr(\"x\",o(s)+u).attr(\"y\",45).attr(\"color\",\"#fff\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",6).text(ze(\",.2f\")(this.invLinkFunction(s-i))).attr(\"opacity\",1),console.log(\"joinPoint\",s,u,50,i),this.joinPointLabel.attr(\"x\",o(s)+u).attr(\"y\",45).attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").text(ze(\",.2f\")(this.invLinkFunction(s-i))).attr(\"opacity\",1),this.joinPointTitle.attr(\"x\",o(s)+u).attr(\"y\",28).attr(\"text-anchor\",\"middle\").attr(\"font-size\",\"12\").attr(\"fill\",\"#000\").text(this.props.outNames[0]).attr(\"opacity\",.5),this.props.hideBars||(this.joinPointTitleLeft.attr(\"x\",o(s)+u-16).attr(\"y\",12).attr(\"text-anchor\",\"end\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[0]).text(\"higher\").attr(\"opacity\",1),this.joinPointTitleRight.attr(\"x\",o(s)+u+16).attr(\"y\",12).attr(\"text-anchor\",\"start\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[1]).text(\"lower\").attr(\"opacity\",1),this.joinPointTitleLeftArrow.attr(\"x\",o(s)+u+7).attr(\"y\",8).attr(\"text-anchor\",\"end\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[0]).text(\"→\").attr(\"opacity\",1),this.joinPointTitleRightArrow.attr(\"x\",o(s)+u-7).attr(\"y\",14).attr(\"text-anchor\",\"start\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[1]).text(\"←\").attr(\"opacity\",1)),this.props.hideBaseValueLabel||this.baseValueTitle.attr(\"x\",this.scaleCentered(0)).attr(\"y\",28).attr(\"text-anchor\",\"middle\").attr(\"font-size\",\"12\").attr(\"fill\",\"#000\").text(\"base value\").attr(\"opacity\",.5)}},{key:\"componentWillUnmount\",value:function(){window.removeEventListener(\"resize\",this.redraw)}},{key:\"render\",value:function(){var t=this;return e.createElement(\"svg\",{ref:function(e){return t.svg=Jt(e)},style:{userSelect:\"none\",display:\"block\",fontFamily:\"arial\",sansSerif:!0}},e.createElement(\"style\",{dangerouslySetInnerHTML:{__html:\"\\n          .force-bar-axis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-axis paths {\\n            display: none;\\n          }\\n          .tick line {\\n            stroke: #000;\\n            stroke-width: 1px;\\n            opacity: 0.4;\\n          }\\n          .tick text {\\n            fill: #000;\\n            opacity: 0.5;\\n            font-size: 12px;\\n            padding: 0px;\\n          }\"}}))}}])&&Tn(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);zn.defaultProps={plot_cmap:\"RdBu\"};const Ln=zn,On=1e3,An=6e4,Fn=36e5,Dn=864e5,Rn=6048e5,jn=31536e6,Un=new Date,In=new Date;function $n(e,t,n,r){function a(t){return e(t=0===arguments.length?new Date:new Date(+t)),t}return a.floor=t=>(e(t=new Date(+t)),t),a.ceil=n=>(e(n=new Date(n-1)),t(n,1),e(n),n),a.round=e=>{const t=a(e),n=a.ceil(e);return e-t<n-e?t:n},a.offset=(e,n)=>(t(e=new Date(+e),null==n?1:Math.floor(n)),e),a.range=(n,r,i)=>{const o=[];if(n=a.ceil(n),i=null==i?1:Math.floor(i),!(n<r&&i>0))return o;let u;do{o.push(u=new Date(+n)),t(n,i),e(n)}while(u<n&&n<r);return o},a.filter=n=>$n((t=>{if(t>=t)for(;e(t),!n(t);)t.setTime(t-1)}),((e,r)=>{if(e>=e)if(r<0)for(;++r<=0;)for(;t(e,-1),!n(e););else for(;--r>=0;)for(;t(e,1),!n(e););})),n&&(a.count=(t,r)=>(Un.setTime(+t),In.setTime(+r),e(Un),e(In),Math.floor(n(Un,In))),a.every=e=>(e=Math.floor(e),isFinite(e)&&e>0?e>1?a.filter(r?t=>r(t)%e==0:t=>a.count(0,t)%e==0):a:null)),a}const Bn=$n((()=>{}),((e,t)=>{e.setTime(+e+t)}),((e,t)=>t-e));Bn.every=e=>(e=Math.floor(e),isFinite(e)&&e>0?e>1?$n((t=>{t.setTime(Math.floor(t/e)*e)}),((t,n)=>{t.setTime(+t+n*e)}),((t,n)=>(n-t)/e)):Bn:null),Bn.range;const Wn=$n((e=>{e.setTime(e-e.getMilliseconds())}),((e,t)=>{e.setTime(+e+t*On)}),((e,t)=>(t-e)/On),(e=>e.getUTCSeconds())),Vn=(Wn.range,$n((e=>{e.setTime(e-e.getMilliseconds()-e.getSeconds()*On)}),((e,t)=>{e.setTime(+e+t*An)}),((e,t)=>(t-e)/An),(e=>e.getMinutes()))),Hn=(Vn.range,$n((e=>{e.setUTCSeconds(0,0)}),((e,t)=>{e.setTime(+e+t*An)}),((e,t)=>(t-e)/An),(e=>e.getUTCMinutes()))),qn=(Hn.range,$n((e=>{e.setTime(e-e.getMilliseconds()-e.getSeconds()*On-e.getMinutes()*An)}),((e,t)=>{e.setTime(+e+t*Fn)}),((e,t)=>(t-e)/Fn),(e=>e.getHours()))),Qn=(qn.range,$n((e=>{e.setUTCMinutes(0,0,0)}),((e,t)=>{e.setTime(+e+t*Fn)}),((e,t)=>(t-e)/Fn),(e=>e.getUTCHours()))),Yn=(Qn.range,$n((e=>e.setHours(0,0,0,0)),((e,t)=>e.setDate(e.getDate()+t)),((e,t)=>(t-e-(t.getTimezoneOffset()-e.getTimezoneOffset())*An)/Dn),(e=>e.getDate()-1))),Gn=(Yn.range,$n((e=>{e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+t)}),((e,t)=>(t-e)/Dn),(e=>e.getUTCDate()-1))),Kn=(Gn.range,$n((e=>{e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+t)}),((e,t)=>(t-e)/Dn),(e=>Math.floor(e/Dn))));function Zn(e){return $n((t=>{t.setDate(t.getDate()-(t.getDay()+7-e)%7),t.setHours(0,0,0,0)}),((e,t)=>{e.setDate(e.getDate()+7*t)}),((e,t)=>(t-e-(t.getTimezoneOffset()-e.getTimezoneOffset())*An)/Rn))}Kn.range;const Xn=Zn(0),Jn=Zn(1),er=Zn(2),tr=Zn(3),nr=Zn(4),rr=Zn(5),ar=Zn(6);function ir(e){return $n((t=>{t.setUTCDate(t.getUTCDate()-(t.getUTCDay()+7-e)%7),t.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+7*t)}),((e,t)=>(t-e)/Rn))}Xn.range,Jn.range,er.range,tr.range,nr.range,rr.range,ar.range;const or=ir(0),ur=ir(1),lr=ir(2),sr=ir(3),cr=ir(4),fr=ir(5),pr=ir(6),dr=(or.range,ur.range,lr.range,sr.range,cr.range,fr.range,pr.range,$n((e=>{e.setDate(1),e.setHours(0,0,0,0)}),((e,t)=>{e.setMonth(e.getMonth()+t)}),((e,t)=>t.getMonth()-e.getMonth()+12*(t.getFullYear()-e.getFullYear())),(e=>e.getMonth()))),hr=(dr.range,$n((e=>{e.setUTCDate(1),e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCMonth(e.getUTCMonth()+t)}),((e,t)=>t.getUTCMonth()-e.getUTCMonth()+12*(t.getUTCFullYear()-e.getUTCFullYear())),(e=>e.getUTCMonth()))),vr=(hr.range,$n((e=>{e.setMonth(0,1),e.setHours(0,0,0,0)}),((e,t)=>{e.setFullYear(e.getFullYear()+t)}),((e,t)=>t.getFullYear()-e.getFullYear()),(e=>e.getFullYear())));vr.every=e=>isFinite(e=Math.floor(e))&&e>0?$n((t=>{t.setFullYear(Math.floor(t.getFullYear()/e)*e),t.setMonth(0,1),t.setHours(0,0,0,0)}),((t,n)=>{t.setFullYear(t.getFullYear()+n*e)})):null,vr.range;const gr=$n((e=>{e.setUTCMonth(0,1),e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCFullYear(e.getUTCFullYear()+t)}),((e,t)=>t.getUTCFullYear()-e.getUTCFullYear()),(e=>e.getUTCFullYear()));function yr(e,t,n,r,a,i){const o=[[Wn,1,On],[Wn,5,5e3],[Wn,15,15e3],[Wn,30,3e4],[i,1,An],[i,5,3e5],[i,15,9e5],[i,30,18e5],[a,1,Fn],[a,3,108e5],[a,6,216e5],[a,12,432e5],[r,1,Dn],[r,2,1728e5],[n,1,Rn],[t,1,2592e6],[t,3,7776e6],[e,1,jn]];function u(t,n,r){const a=Math.abs(n-t)/r,i=f((([,,e])=>e)).right(o,a);if(i===o.length)return e.every(l(t/jn,n/jn,r));if(0===i)return Bn.every(Math.max(l(t,n,r),1));const[u,s]=o[a/o[i-1][2]<o[i][2]/a?i-1:i];return u.every(s)}return[function(e,t,n){const r=t<e;r&&([e,t]=[t,e]);const a=n&&\"function\"==typeof n.range?n:u(e,t,n),i=a?a.range(e,+t+1):[];return r?i.reverse():i},u]}gr.every=e=>isFinite(e=Math.floor(e))&&e>0?$n((t=>{t.setUTCFullYear(Math.floor(t.getUTCFullYear()/e)*e),t.setUTCMonth(0,1),t.setUTCHours(0,0,0,0)}),((t,n)=>{t.setUTCFullYear(t.getUTCFullYear()+n*e)})):null,gr.range;const[mr,br]=yr(gr,hr,or,Kn,Qn,Hn),[_r,wr]=yr(vr,dr,Xn,Yn,qn,Vn);function xr(e){if(0<=e.y&&e.y<100){var t=new Date(-1,e.m,e.d,e.H,e.M,e.S,e.L);return t.setFullYear(e.y),t}return new Date(e.y,e.m,e.d,e.H,e.M,e.S,e.L)}function kr(e){if(0<=e.y&&e.y<100){var t=new Date(Date.UTC(-1,e.m,e.d,e.H,e.M,e.S,e.L));return t.setUTCFullYear(e.y),t}return new Date(Date.UTC(e.y,e.m,e.d,e.H,e.M,e.S,e.L))}function Sr(e,t,n){return{y:e,m:t,d:n,H:0,M:0,S:0,L:0}}var Er,Cr,Tr,Mr={\"-\":\"\",_:\" \",0:\"0\"},Nr=/^\\s*\\d+/,Pr=/^%/,zr=/[\\\\^$*+?|[\\]().{}]/g;function Lr(e,t,n){var r=e<0?\"-\":\"\",a=(r?-e:e)+\"\",i=a.length;return r+(i<n?new Array(n-i+1).join(t)+a:a)}function Or(e){return e.replace(zr,\"\\\\$&\")}function Ar(e){return new RegExp(\"^(?:\"+e.map(Or).join(\"|\")+\")\",\"i\")}function Fr(e){return new Map(e.map(((e,t)=>[e.toLowerCase(),t])))}function Dr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.w=+r[0],n+r[0].length):-1}function Rr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.u=+r[0],n+r[0].length):-1}function jr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.U=+r[0],n+r[0].length):-1}function Ur(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.V=+r[0],n+r[0].length):-1}function Ir(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.W=+r[0],n+r[0].length):-1}function $r(e,t,n){var r=Nr.exec(t.slice(n,n+4));return r?(e.y=+r[0],n+r[0].length):-1}function Br(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.y=+r[0]+(+r[0]>68?1900:2e3),n+r[0].length):-1}function Wr(e,t,n){var r=/^(Z)|([+-]\\d\\d)(?::?(\\d\\d))?/.exec(t.slice(n,n+6));return r?(e.Z=r[1]?0:-(r[2]+(r[3]||\"00\")),n+r[0].length):-1}function Vr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.q=3*r[0]-3,n+r[0].length):-1}function Hr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.m=r[0]-1,n+r[0].length):-1}function qr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.d=+r[0],n+r[0].length):-1}function Qr(e,t,n){var r=Nr.exec(t.slice(n,n+3));return r?(e.m=0,e.d=+r[0],n+r[0].length):-1}function Yr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.H=+r[0],n+r[0].length):-1}function Gr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.M=+r[0],n+r[0].length):-1}function Kr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.S=+r[0],n+r[0].length):-1}function Zr(e,t,n){var r=Nr.exec(t.slice(n,n+3));return r?(e.L=+r[0],n+r[0].length):-1}function Xr(e,t,n){var r=Nr.exec(t.slice(n,n+6));return r?(e.L=Math.floor(r[0]/1e3),n+r[0].length):-1}function Jr(e,t,n){var r=Pr.exec(t.slice(n,n+1));return r?n+r[0].length:-1}function ea(e,t,n){var r=Nr.exec(t.slice(n));return r?(e.Q=+r[0],n+r[0].length):-1}function ta(e,t,n){var r=Nr.exec(t.slice(n));return r?(e.s=+r[0],n+r[0].length):-1}function na(e,t){return Lr(e.getDate(),t,2)}function ra(e,t){return Lr(e.getHours(),t,2)}function aa(e,t){return Lr(e.getHours()%12||12,t,2)}function ia(e,t){return Lr(1+Yn.count(vr(e),e),t,3)}function oa(e,t){return Lr(e.getMilliseconds(),t,3)}function ua(e,t){return oa(e,t)+\"000\"}function la(e,t){return Lr(e.getMonth()+1,t,2)}function sa(e,t){return Lr(e.getMinutes(),t,2)}function ca(e,t){return Lr(e.getSeconds(),t,2)}function fa(e){var t=e.getDay();return 0===t?7:t}function pa(e,t){return Lr(Xn.count(vr(e)-1,e),t,2)}function da(e){var t=e.getDay();return t>=4||0===t?nr(e):nr.ceil(e)}function ha(e,t){return e=da(e),Lr(nr.count(vr(e),e)+(4===vr(e).getDay()),t,2)}function va(e){return e.getDay()}function ga(e,t){return Lr(Jn.count(vr(e)-1,e),t,2)}function ya(e,t){return Lr(e.getFullYear()%100,t,2)}function ma(e,t){return Lr((e=da(e)).getFullYear()%100,t,2)}function ba(e,t){return Lr(e.getFullYear()%1e4,t,4)}function _a(e,t){var n=e.getDay();return Lr((e=n>=4||0===n?nr(e):nr.ceil(e)).getFullYear()%1e4,t,4)}function wa(e){var t=e.getTimezoneOffset();return(t>0?\"-\":(t*=-1,\"+\"))+Lr(t/60|0,\"0\",2)+Lr(t%60,\"0\",2)}function xa(e,t){return Lr(e.getUTCDate(),t,2)}function ka(e,t){return Lr(e.getUTCHours(),t,2)}function Sa(e,t){return Lr(e.getUTCHours()%12||12,t,2)}function Ea(e,t){return Lr(1+Gn.count(gr(e),e),t,3)}function Ca(e,t){return Lr(e.getUTCMilliseconds(),t,3)}function Ta(e,t){return Ca(e,t)+\"000\"}function Ma(e,t){return Lr(e.getUTCMonth()+1,t,2)}function Na(e,t){return Lr(e.getUTCMinutes(),t,2)}function Pa(e,t){return Lr(e.getUTCSeconds(),t,2)}function za(e){var t=e.getUTCDay();return 0===t?7:t}function La(e,t){return Lr(or.count(gr(e)-1,e),t,2)}function Oa(e){var t=e.getUTCDay();return t>=4||0===t?cr(e):cr.ceil(e)}function Aa(e,t){return e=Oa(e),Lr(cr.count(gr(e),e)+(4===gr(e).getUTCDay()),t,2)}function Fa(e){return e.getUTCDay()}function Da(e,t){return Lr(ur.count(gr(e)-1,e),t,2)}function Ra(e,t){return Lr(e.getUTCFullYear()%100,t,2)}function ja(e,t){return Lr((e=Oa(e)).getUTCFullYear()%100,t,2)}function Ua(e,t){return Lr(e.getUTCFullYear()%1e4,t,4)}function Ia(e,t){var n=e.getUTCDay();return Lr((e=n>=4||0===n?cr(e):cr.ceil(e)).getUTCFullYear()%1e4,t,4)}function $a(){return\"+0000\"}function Ba(){return\"%\"}function Wa(e){return+e}function Va(e){return Math.floor(+e/1e3)}function Ha(e){return new Date(e)}function qa(e){return e instanceof Date?+e:+new Date(+e)}function Qa(e,t,n,r,a,i,o,u,l,s){var c=be(),f=c.invert,p=c.domain,d=s(\".%L\"),h=s(\":%S\"),v=s(\"%I:%M\"),g=s(\"%I %p\"),y=s(\"%a %d\"),m=s(\"%b %d\"),b=s(\"%B\"),_=s(\"%Y\");function w(e){return(l(e)<e?d:u(e)<e?h:o(e)<e?v:i(e)<e?g:r(e)<e?a(e)<e?y:m:n(e)<e?b:_)(e)}return c.invert=function(e){return new Date(f(e))},c.domain=function(e){return arguments.length?p(Array.from(e,qa)):p().map(Ha)},c.ticks=function(t){var n=p();return e(n[0],n[n.length-1],null==t?10:t)},c.tickFormat=function(e,t){return null==t?w:s(t)},c.nice=function(e){var n=p();return e&&\"function\"==typeof e.range||(e=t(n[0],n[n.length-1],null==e?10:e)),e?p(function(e,t){var n,r=0,a=(e=e.slice()).length-1,i=e[r],o=e[a];return o<i&&(n=r,r=a,a=n,n=i,i=o,o=n),e[r]=t.floor(i),e[a]=t.ceil(o),e}(n,e)):c},c.copy=function(){return me(c,Qa(e,t,n,r,a,i,o,u,l,s))},c}function Ya(){return _e.apply(Qa(_r,wr,vr,dr,Xn,Yn,qn,Vn,Wn,Cr).domain([new Date(2e3,0,1),new Date(2e3,0,2)]),arguments)}function Ga(e,t){var n=\"undefined\"!=typeof Symbol&&e[Symbol.iterator]||e[\"@@iterator\"];if(!n){if(Array.isArray(e)||(n=function(e,t){if(e){if(\"string\"==typeof e)return Ka(e,t);var n=Object.prototype.toString.call(e).slice(8,-1);return\"Object\"===n&&e.constructor&&(n=e.constructor.name),\"Map\"===n||\"Set\"===n?Array.from(e):\"Arguments\"===n||/^(?:Ui|I)nt(?:8|16|32)(?:Clamped)?Array$/.test(n)?Ka(e,t):void 0}}(e))||t&&e&&\"number\"==typeof e.length){n&&(e=n);var r=0,a=function(){};return{s:a,n:function(){return r>=e.length?{done:!0}:{done:!1,value:e[r++]}},e:function(e){throw e},f:a}}throw new TypeError(\"Invalid attempt to iterate non-iterable instance.\\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.\")}var i,o=!0,u=!1;return{s:function(){n=n.call(e)},n:function(){var e=n.next();return o=e.done,e},e:function(e){u=!0,i=e},f:function(){try{o||null==n.return||n.return()}finally{if(u)throw i}}}}function Ka(e,t){(null==t||t>e.length)&&(t=e.length);for(var n=0,r=new Array(t);n<t;n++)r[n]=e[n];return r}function Za(e){return Za=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Za(e)}function Xa(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Za(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Za(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Za(a)?a:String(a)),r)}var a}function Ja(e,t){return Ja=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},Ja(e,t)}function ei(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function ti(e){return ti=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},ti(e)}Er=function(e){var t=e.dateTime,n=e.date,r=e.time,a=e.periods,i=e.days,o=e.shortDays,u=e.months,l=e.shortMonths,s=Ar(a),c=Fr(a),f=Ar(i),p=Fr(i),d=Ar(o),h=Fr(o),v=Ar(u),g=Fr(u),y=Ar(l),m=Fr(l),b={a:function(e){return o[e.getDay()]},A:function(e){return i[e.getDay()]},b:function(e){return l[e.getMonth()]},B:function(e){return u[e.getMonth()]},c:null,d:na,e:na,f:ua,g:ma,G:_a,H:ra,I:aa,j:ia,L:oa,m:la,M:sa,p:function(e){return a[+(e.getHours()>=12)]},q:function(e){return 1+~~(e.getMonth()/3)},Q:Wa,s:Va,S:ca,u:fa,U:pa,V:ha,w:va,W:ga,x:null,X:null,y:ya,Y:ba,Z:wa,\"%\":Ba},_={a:function(e){return o[e.getUTCDay()]},A:function(e){return i[e.getUTCDay()]},b:function(e){return l[e.getUTCMonth()]},B:function(e){return u[e.getUTCMonth()]},c:null,d:xa,e:xa,f:Ta,g:ja,G:Ia,H:ka,I:Sa,j:Ea,L:Ca,m:Ma,M:Na,p:function(e){return a[+(e.getUTCHours()>=12)]},q:function(e){return 1+~~(e.getUTCMonth()/3)},Q:Wa,s:Va,S:Pa,u:za,U:La,V:Aa,w:Fa,W:Da,x:null,X:null,y:Ra,Y:Ua,Z:$a,\"%\":Ba},w={a:function(e,t,n){var r=d.exec(t.slice(n));return r?(e.w=h.get(r[0].toLowerCase()),n+r[0].length):-1},A:function(e,t,n){var r=f.exec(t.slice(n));return r?(e.w=p.get(r[0].toLowerCase()),n+r[0].length):-1},b:function(e,t,n){var r=y.exec(t.slice(n));return r?(e.m=m.get(r[0].toLowerCase()),n+r[0].length):-1},B:function(e,t,n){var r=v.exec(t.slice(n));return r?(e.m=g.get(r[0].toLowerCase()),n+r[0].length):-1},c:function(e,n,r){return S(e,t,n,r)},d:qr,e:qr,f:Xr,g:Br,G:$r,H:Yr,I:Yr,j:Qr,L:Zr,m:Hr,M:Gr,p:function(e,t,n){var r=s.exec(t.slice(n));return r?(e.p=c.get(r[0].toLowerCase()),n+r[0].length):-1},q:Vr,Q:ea,s:ta,S:Kr,u:Rr,U:jr,V:Ur,w:Dr,W:Ir,x:function(e,t,r){return S(e,n,t,r)},X:function(e,t,n){return S(e,r,t,n)},y:Br,Y:$r,Z:Wr,\"%\":Jr};function x(e,t){return function(n){var r,a,i,o=[],u=-1,l=0,s=e.length;for(n instanceof Date||(n=new Date(+n));++u<s;)37===e.charCodeAt(u)&&(o.push(e.slice(l,u)),null!=(a=Mr[r=e.charAt(++u)])?r=e.charAt(++u):a=\"e\"===r?\" \":\"0\",(i=t[r])&&(r=i(n,a)),o.push(r),l=u+1);return o.push(e.slice(l,u)),o.join(\"\")}}function k(e,t){return function(n){var r,a,i=Sr(1900,void 0,1);if(S(i,e,n+=\"\",0)!=n.length)return null;if(\"Q\"in i)return new Date(i.Q);if(\"s\"in i)return new Date(1e3*i.s+(\"L\"in i?i.L:0));if(t&&!(\"Z\"in i)&&(i.Z=0),\"p\"in i&&(i.H=i.H%12+12*i.p),void 0===i.m&&(i.m=\"q\"in i?i.q:0),\"V\"in i){if(i.V<1||i.V>53)return null;\"w\"in i||(i.w=1),\"Z\"in i?(a=(r=kr(Sr(i.y,0,1))).getUTCDay(),r=a>4||0===a?ur.ceil(r):ur(r),r=Gn.offset(r,7*(i.V-1)),i.y=r.getUTCFullYear(),i.m=r.getUTCMonth(),i.d=r.getUTCDate()+(i.w+6)%7):(a=(r=xr(Sr(i.y,0,1))).getDay(),r=a>4||0===a?Jn.ceil(r):Jn(r),r=Yn.offset(r,7*(i.V-1)),i.y=r.getFullYear(),i.m=r.getMonth(),i.d=r.getDate()+(i.w+6)%7)}else(\"W\"in i||\"U\"in i)&&(\"w\"in i||(i.w=\"u\"in i?i.u%7:\"W\"in i?1:0),a=\"Z\"in i?kr(Sr(i.y,0,1)).getUTCDay():xr(Sr(i.y,0,1)).getDay(),i.m=0,i.d=\"W\"in i?(i.w+6)%7+7*i.W-(a+5)%7:i.w+7*i.U-(a+6)%7);return\"Z\"in i?(i.H+=i.Z/100|0,i.M+=i.Z%100,kr(i)):xr(i)}}function S(e,t,n,r){for(var a,i,o=0,u=t.length,l=n.length;o<u;){if(r>=l)return-1;if(37===(a=t.charCodeAt(o++))){if(a=t.charAt(o++),!(i=w[a in Mr?t.charAt(o++):a])||(r=i(e,n,r))<0)return-1}else if(a!=n.charCodeAt(r++))return-1}return r}return b.x=x(n,b),b.X=x(r,b),b.c=x(t,b),_.x=x(n,_),_.X=x(r,_),_.c=x(t,_),{format:function(e){var t=x(e+=\"\",b);return t.toString=function(){return e},t},parse:function(e){var t=k(e+=\"\",!1);return t.toString=function(){return e},t},utcFormat:function(e){var t=x(e+=\"\",_);return t.toString=function(){return e},t},utcParse:function(e){var t=k(e+=\"\",!0);return t.toString=function(){return e},t}}}({dateTime:\"%x, %X\",date:\"%-m/%-d/%Y\",time:\"%-I:%M:%S %p\",periods:[\"AM\",\"PM\"],days:[\"Sunday\",\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\",\"Saturday\"],shortDays:[\"Sun\",\"Mon\",\"Tue\",\"Wed\",\"Thu\",\"Fri\",\"Sat\"],months:[\"January\",\"February\",\"March\",\"April\",\"May\",\"June\",\"July\",\"August\",\"September\",\"October\",\"November\",\"December\"],shortMonths:[\"Jan\",\"Feb\",\"Mar\",\"Apr\",\"May\",\"Jun\",\"Jul\",\"Aug\",\"Sep\",\"Oct\",\"Nov\",\"Dec\"]}),Cr=Er.format,Tr=Er.parse,Er.utcFormat,Er.utcParse;var ni=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&Ja(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=ti(a);if(i){var n=ti(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Za(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return ei(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),e=o.call(this),window.lastAdditiveForceArrayVisualizer=ei(e),e.topOffset=28,e.leftOffset=80,e.height=350,e.effectFormat=ze(\".2\"),e.redraw=(0,Re.debounce)((function(){return e.draw()}),200),e}return n=u,(r=[{key:\"componentDidMount\",value:function(){var e=this;this.mainGroup=this.svg.append(\"g\"),this.onTopGroup=this.svg.append(\"g\"),this.xaxisElement=this.onTopGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-array-xaxis\"),this.yaxisElement=this.onTopGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-array-yaxis\"),this.hoverGroup1=this.svg.append(\"g\"),this.hoverGroup2=this.svg.append(\"g\"),this.baseValueTitle=this.svg.append(\"text\"),this.hoverLine=this.svg.append(\"line\"),this.hoverxOutline=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#fff\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",\"6\").attr(\"font-size\",\"12px\"),this.hoverx=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").attr(\"font-size\",\"12px\"),this.hoverxTitle=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"opacity\",.6).attr(\"font-size\",\"12px\"),this.hoveryOutline=this.svg.append(\"text\").attr(\"text-anchor\",\"end\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#fff\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",\"6\").attr(\"font-size\",\"12px\"),this.hovery=this.svg.append(\"text\").attr(\"text-anchor\",\"end\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").attr(\"font-size\",\"12px\"),this.xlabel=this.wrapper.select(\".additive-force-array-xlabel\"),this.ylabel=this.wrapper.select(\".additive-force-array-ylabel\");var t=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?t=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),t=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(t=this.props.plot_cmap),this.colors=t.map((function(e){return q(e)})),this.brighterColors=[1.45,1.6].map((function(t,n){return e.colors[n].brighter(t)}));var n=ze(\",.4\");null!=this.props.ordering_keys&&null!=this.props.ordering_keys_time_format?(this.parseTime=Tr(this.props.ordering_keys_time_format),this.formatTime=Cr(this.props.ordering_keys_time_format),this.xtickFormat=function(e){return\"object\"==Za(e)?this.formatTime(e):n(e)}):(this.parseTime=null,this.formatTime=null,this.xtickFormat=n),this.xscale=De(),this.xaxis=dn().scale(this.xscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.xtickFormat(t)})).tickPadding(-18),this.ytickFormat=n,this.yscale=De(),this.yaxis=pn(an,undefined).scale(this.yscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.ytickFormat(e.invLinkFunction(t))})).tickPadding(2),this.xlabel.node().onchange=function(){return e.internalDraw()},this.ylabel.node().onchange=function(){return e.internalDraw()},this.svg.on(\"mousemove\",(function(t){return e.mouseMoved(t)})),this.svg.on(\"click\",(function(){return alert(\"This original index of the sample you clicked is \"+e.nearestExpIndex)})),this.svg.on(\"mouseout\",(function(t){return e.mouseOut(t)})),window.addEventListener(\"resize\",this.redraw),window.setTimeout(this.redraw,50)}},{key:\"componentDidUpdate\",value:function(){this.draw()}},{key:\"mouseOut\",value:function(){this.hoverLine.attr(\"display\",\"none\"),this.hoverx.attr(\"display\",\"none\"),this.hoverxOutline.attr(\"display\",\"none\"),this.hoverxTitle.attr(\"display\",\"none\"),this.hovery.attr(\"display\",\"none\"),this.hoveryOutline.attr(\"display\",\"none\"),this.hoverGroup1.attr(\"display\",\"none\"),this.hoverGroup2.attr(\"display\",\"none\")}},{key:\"mouseMoved\",value:function(e){var t,n,r=this;this.hoverLine.attr(\"display\",\"\"),this.hoverx.attr(\"display\",\"\"),this.hoverxOutline.attr(\"display\",\"\"),this.hoverxTitle.attr(\"display\",\"\"),this.hovery.attr(\"display\",\"\"),this.hoveryOutline.attr(\"display\",\"\"),this.hoverGroup1.attr(\"display\",\"\"),this.hoverGroup2.attr(\"display\",\"\");var a=function(e,t){if(e=function(e){let t;for(;t=e.sourceEvent;)e=t;return e}(e),void 0===t&&(t=e.currentTarget),t){var n=t.ownerSVGElement||t;if(n.createSVGPoint){var r=n.createSVGPoint();return r.x=e.clientX,r.y=e.clientY,[(r=r.matrixTransform(t.getScreenCTM().inverse())).x,r.y]}if(t.getBoundingClientRect){var a=t.getBoundingClientRect();return[e.clientX-a.left-t.clientLeft,e.clientY-a.top-t.clientTop]}}return[e.pageX,e.pageY]}(e,this.svg.node())[0];if(this.props.explanations){for(t=0;t<this.currExplanations.length;++t)(!n||Math.abs(n.xmapScaled-a)>Math.abs(this.currExplanations[t].xmapScaled-a))&&(n=this.currExplanations[t]);this.nearestExpIndex=n.origInd,this.hoverLine.attr(\"x1\",n.xmapScaled).attr(\"x2\",n.xmapScaled).attr(\"y1\",0+this.topOffset).attr(\"y2\",this.height),this.hoverx.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-5).text(this.xtickFormat(n.xmap)),this.hoverxOutline.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-5).text(this.xtickFormat(n.xmap)),this.hoverxTitle.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-18).text(n.count>1?n.count+\" averaged samples\":\"\"),this.hovery.attr(\"x\",this.leftOffset-6).attr(\"y\",n.joinPointy).text(this.ytickFormat(this.invLinkFunction(n.joinPoint))),this.hoveryOutline.attr(\"x\",this.leftOffset-6).attr(\"y\",n.joinPointy).text(this.ytickFormat(this.invLinkFunction(n.joinPoint)));for(var i,o,u=[],l=this.currPosOrderedFeatures.length-1;l>=0;--l){var s=this.currPosOrderedFeatures[l],c=n.features[s];o=5+(c.posyTop+c.posyBottom)/2,(!i||o-i>=15)&&c.posyTop-c.posyBottom>=6&&(u.push(c),i=o)}var f=[];i=void 0;var p,d=Ga(this.currNegOrderedFeatures);try{for(d.s();!(p=d.n()).done;){var h=p.value,v=n.features[h];o=5+(v.negyTop+v.negyBottom)/2,(!i||i-o>=15)&&v.negyTop-v.negyBottom>=6&&(f.push(v),i=o)}}catch(e){d.e(e)}finally{d.f()}var g=function(e){var t=\"\";return null!==e.value&&void 0!==e.value&&(t=\" = \"+(isNaN(e.value)?e.value:r.ytickFormat(e.value))),n.count>1?\"mean(\"+r.props.featureNames[e.ind]+\")\"+t:r.props.featureNames[e.ind]+t},y=this.hoverGroup1.selectAll(\".pos-values\").data(u);y.enter().append(\"text\").attr(\"class\",\"pos-values\").merge(y).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.posyTop+e.posyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").attr(\"opacity\",1).text(g),y.exit().remove();var m=this.hoverGroup2.selectAll(\".pos-values\").data(u);m.enter().append(\"text\").attr(\"class\",\"pos-values\").merge(m).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.posyTop+e.posyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"fill\",this.colors[0]).text(g),m.exit().remove();var b=this.hoverGroup1.selectAll(\".neg-values\").data(f);b.enter().append(\"text\").attr(\"class\",\"neg-values\").merge(b).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.negyTop+e.negyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").attr(\"opacity\",1).text(g),b.exit().remove();var _=this.hoverGroup2.selectAll(\".neg-values\").data(f);_.enter().append(\"text\").attr(\"class\",\"neg-values\").merge(_).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.negyTop+e.negyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"fill\",this.colors[1]).text(g),_.exit().remove()}}},{key:\"draw\",value:function(){var e=this;if(this.props.explanations&&0!==this.props.explanations.length){(0,Re.each)(this.props.explanations,(function(e,t){return e.origInd=t}));var t,n={},r={},a={},i=Ga(this.props.explanations);try{for(i.s();!(t=i.n()).done;){var o=t.value;for(var u in o.features)void 0===n[u]&&(n[u]=0,r[u]=0,a[u]=0),o.features[u].effect>0?n[u]+=o.features[u].effect:r[u]-=o.features[u].effect,null!==o.features[u].value&&void 0!==o.features[u].value&&(a[u]+=1)}}catch(e){i.e(e)}finally{i.f()}this.usedFeatures=(0,Re.sortBy)((0,Re.keys)(n),(function(e){return-(n[e]+r[e])})),console.log(\"found \",this.usedFeatures.length,\" used features\"),this.posOrderedFeatures=(0,Re.sortBy)(this.usedFeatures,(function(e){return n[e]})),this.negOrderedFeatures=(0,Re.sortBy)(this.usedFeatures,(function(e){return-r[e]})),this.singleValueFeatures=(0,Re.filter)(this.usedFeatures,(function(e){return a[e]>0}));var l=[\"sample order by similarity\",\"sample order by output value\",\"original sample ordering\"].concat(this.singleValueFeatures.map((function(t){return e.props.featureNames[t]})));null!=this.props.ordering_keys&&l.unshift(\"sample order by key\");var s=this.xlabel.selectAll(\"option\").data(l);s.enter().append(\"option\").merge(s).attr(\"value\",(function(e){return e})).text((function(e){return e})),s.exit().remove();var c=this.props.outNames[0]?this.props.outNames[0]:\"model output value\";(l=(0,Re.map)(this.usedFeatures,(function(t){return[e.props.featureNames[t],e.props.featureNames[t]+\" effects\"]}))).unshift([\"model output value\",c]);var f=this.ylabel.selectAll(\"option\").data(l);f.enter().append(\"option\").merge(f).attr(\"value\",(function(e){return e[0]})).text((function(e){return e[1]})),f.exit().remove(),this.ylabel.style(\"top\",(this.height-10-this.topOffset)/2+this.topOffset+\"px\").style(\"left\",10-this.ylabel.node().offsetWidth/2+\"px\"),this.internalDraw()}}},{key:\"internalDraw\",value:function(){var e,t,n=this,r=Ga(this.props.explanations);try{for(r.s();!(e=r.n()).done;){var a,i=e.value,o=Ga(this.usedFeatures);try{for(o.s();!(a=o.n()).done;){var u=a.value;i.features.hasOwnProperty(u)||(i.features[u]={effect:0,value:0}),i.features[u].ind=u}}catch(e){o.e(e)}finally{o.f()}}}catch(e){r.e(e)}finally{r.f()}var l=this.xlabel.node().value,s=\"sample order by key\"===l&&null!=this.props.ordering_keys_time_format;if(this.xscale=s?Ya():De(),this.xaxis.scale(this.xscale),\"sample order by similarity\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return e.simIndex})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"sample order by output value\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return-e.outValue})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"original sample ordering\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return e.origInd})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"sample order by key\"===l)t=this.props.explanations,s?(0,Re.each)(t,(function(e,t){return e.xmap=n.parseTime(n.props.ordering_keys[t])})):(0,Re.each)(t,(function(e,t){return e.xmap=n.props.ordering_keys[t]})),t=(0,Re.sortBy)(t,(function(e){return e.xmap}));else{var c=(0,Re.findKey)(this.props.featureNames,(function(e){return e===l}));(0,Re.each)(this.props.explanations,(function(e,t){return e.xmap=e.features[c].value}));var f=(0,Re.sortBy)(this.props.explanations,(function(e){return e.xmap})),p=(0,Re.map)(f,(function(e){return e.xmap}));if(\"string\"==typeof p[0])return void alert(\"Ordering by category names is not yet supported.\");var d,h,v=(0,Re.min)(p),g=((0,Re.max)(p)-v)/100;t=[];for(var y=0;y<f.length;++y){var m=f[y];if(d&&!h&&m.xmap-d.xmap<=g||h&&m.xmap-h.xmap<=g){h||((h=(0,Re.cloneDeep)(d)).count=1);var b,_=Ga(this.usedFeatures);try{for(_.s();!(b=_.n()).done;){var w=b.value;h.features[w].effect+=m.features[w].effect,h.features[w].value+=m.features[w].value}}catch(e){_.e(e)}finally{_.f()}h.count+=1}else if(d)if(h){var x,k=Ga(this.usedFeatures);try{for(k.s();!(x=k.n()).done;){var S=x.value;h.features[S].effect/=h.count,h.features[S].value/=h.count}}catch(e){k.e(e)}finally{k.f()}t.push(h),h=void 0}else t.push(d);d=m}d.xmap-t[t.length-1].xmap>g&&t.push(d)}this.currUsedFeatures=this.usedFeatures,this.currPosOrderedFeatures=this.posOrderedFeatures,this.currNegOrderedFeatures=this.negOrderedFeatures;var E=this.ylabel.node().value;if(\"model output value\"!==E){var C=t;t=(0,Re.cloneDeep)(t);for(var T=(0,Re.findKey)(this.props.featureNames,(function(e){return e===E})),M=0;M<t.length;++M){var N=t[M].features[T];t[M].features={},t[M].features[T]=N,C[M].remapped_version=t[M]}this.currUsedFeatures=[T],this.currPosOrderedFeatures=[T],this.currNegOrderedFeatures=[T]}this.currExplanations=t,\"identity\"===this.props.link?this.invLinkFunction=function(e){return n.props.baseValue+e}:\"logit\"===this.props.link?this.invLinkFunction=function(e){return 1/(1+Math.exp(-(n.props.baseValue+e)))}:console.log(\"ERROR: Unrecognized link function: \",this.props.link),this.predValues=(0,Re.map)(t,(function(e){return(0,Re.sum)((0,Re.map)(e.features,(function(e){return e.effect})))}));var P=this.wrapper.node().offsetWidth;if(0==P)return setTimeout((function(){return n.draw(t)}),500);this.svg.style(\"height\",this.height+\"px\"),this.svg.style(\"width\",P+\"px\");var z=(0,Re.map)(t,(function(e){return e.xmap}));this.xscale.domain([(0,Re.min)(z),(0,Re.max)(z)]).range([this.leftOffset,P]).clamp(!0),this.xaxisElement.attr(\"transform\",\"translate(0,\"+this.topOffset+\")\").call(this.xaxis);for(var L=0;L<this.currExplanations.length;++L)this.currExplanations[L].xmapScaled=this.xscale(this.currExplanations[L].xmap);for(var O=t.length,A=0,F=0;F<O;++F){var D=t[F].features,R=(0,Re.sum)((0,Re.map)((0,Re.filter)(D,(function(e){return e.effect>0})),(function(e){return e.effect})))||0,j=(0,Re.sum)((0,Re.map)((0,Re.filter)(D,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0;A=Math.max(A,2.2*Math.max(R,j))}this.yscale.domain([-A/2,A/2]).range([this.height-10,this.topOffset]),this.yaxisElement.attr(\"transform\",\"translate(\"+this.leftOffset+\",0)\").call(this.yaxis);for(var U=0;U<O;++U){var I,$=t[U].features,B=-((0,Re.sum)((0,Re.map)((0,Re.filter)($,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0),W=void 0,V=Ga(this.currPosOrderedFeatures);try{for(V.s();!(I=V.n()).done;)$[W=I.value].posyTop=this.yscale(B),$[W].effect>0&&(B+=$[W].effect),$[W].posyBottom=this.yscale(B),$[W].ind=W}catch(e){V.e(e)}finally{V.f()}var H,q=B,Q=Ga(this.currNegOrderedFeatures);try{for(Q.s();!(H=Q.n()).done;)$[W=H.value].negyTop=this.yscale(B),$[W].effect<0&&(B-=$[W].effect),$[W].negyBottom=this.yscale(B)}catch(e){Q.e(e)}finally{Q.f()}t[U].joinPoint=q,t[U].joinPointy=this.yscale(q)}var Y=En().x((function(e){return e[0]})).y((function(e){return e[1]})),G=this.mainGroup.selectAll(\".force-bar-array-area-pos\").data(this.currUsedFeatures);G.enter().append(\"path\").attr(\"class\",\"force-bar-array-area-pos\").merge(G).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyTop]})),r=(0,Re.map)((0,Re.rangeRight)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyBottom]}));return Y(n.concat(r))})).attr(\"fill\",this.colors[0]),G.exit().remove();var K=this.mainGroup.selectAll(\".force-bar-array-area-neg\").data(this.currUsedFeatures);K.enter().append(\"path\").attr(\"class\",\"force-bar-array-area-neg\").merge(K).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyTop]})),r=(0,Re.map)((0,Re.rangeRight)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyBottom]}));return Y(n.concat(r))})).attr(\"fill\",this.colors[1]),K.exit().remove();var Z=this.mainGroup.selectAll(\".force-bar-array-divider-pos\").data(this.currUsedFeatures);Z.enter().append(\"path\").attr(\"class\",\"force-bar-array-divider-pos\").merge(Z).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyBottom]}));return Y(n)})).attr(\"fill\",\"none\").attr(\"stroke-width\",1).attr(\"stroke\",(function(){return n.colors[0].brighter(1.2)})),Z.exit().remove();var X=this.mainGroup.selectAll(\".force-bar-array-divider-neg\").data(this.currUsedFeatures);X.enter().append(\"path\").attr(\"class\",\"force-bar-array-divider-neg\").merge(X).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyTop]}));return Y(n)})).attr(\"fill\",\"none\").attr(\"stroke-width\",1).attr(\"stroke\",(function(){return n.colors[1].brighter(1.5)})),X.exit().remove();for(var J=function(e,t,n,r,a){var i,o,u,l;\"pos\"===a?(i=e[n].features[t].posyBottom,o=e[n].features[t].posyTop):(i=e[n].features[t].negyBottom,o=e[n].features[t].negyTop);for(var s=n+1;s<=r;++s)\"pos\"===a?(u=e[s].features[t].posyBottom,l=e[s].features[t].posyTop):(u=e[s].features[t].negyBottom,l=e[s].features[t].negyTop),u>i&&(i=u),l<o&&(o=l);return{top:i,bottom:o}},ee=[],te=0,ne=[\"pos\",\"neg\"];te<ne.length;te++){var re,ae=ne[te],ie=Ga(this.currUsedFeatures);try{for(ie.s();!(re=ie.n()).done;)for(var oe=re.value,ue=0,le=0,se=0,ce={top:0,bottom:0},fe=void 0;le<O-1;){for(;se<100&&le<O-1;)++le,se=t[le].xmapScaled-t[ue].xmapScaled;for(ce=J(t,oe,ue,le,ae);ce.bottom-ce.top<20&&ue<le;)++ue,ce=J(t,oe,ue,le,ae);if(se=t[le].xmapScaled-t[ue].xmapScaled,ce.bottom-ce.top>=20&&se>=100){for(;le<O-1;){if(++le,!((fe=J(t,oe,ue,le,ae)).bottom-fe.top>20)){--le;break}ce=fe}se=t[le].xmapScaled-t[ue].xmapScaled,ee.push([(t[le].xmapScaled+t[ue].xmapScaled)/2,(ce.top+ce.bottom)/2,this.props.featureNames[oe]]);var pe=t[le].xmapScaled;for(ue=le;pe+100>t[ue].xmapScaled&&ue<O-1;)++ue;le=ue}}}catch(e){ie.e(e)}finally{ie.f()}}var de=this.onTopGroup.selectAll(\".force-bar-array-flabels\").data(ee);de.enter().append(\"text\").attr(\"class\",\"force-bar-array-flabels\").merge(de).attr(\"x\",(function(e){return e[0]})).attr(\"y\",(function(e){return e[1]+4})).text((function(e){return e[2]})),de.exit().remove()}},{key:\"componentWillUnmount\",value:function(){window.removeEventListener(\"resize\",this.redraw)}},{key:\"render\",value:function(){var t=this;return e.createElement(\"div\",{ref:function(e){return t.wrapper=Jt(e)},style:{textAlign:\"center\"}},e.createElement(\"style\",{dangerouslySetInnerHTML:{__html:\"\\n          .force-bar-array-wrapper {\\n            text-align: center;\\n          }\\n          .force-bar-array-xaxis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-array-xaxis .domain {\\n            opacity: 0;\\n          }\\n          .force-bar-array-xaxis paths {\\n            display: none;\\n          }\\n          .force-bar-array-yaxis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-array-yaxis paths {\\n            display: none;\\n          }\\n          .tick line {\\n            stroke: #000;\\n            stroke-width: 1px;\\n            opacity: 0.4;\\n          }\\n          .tick text {\\n            fill: #000;\\n            opacity: 0.5;\\n            font-size: 12px;\\n            padding: 0px;\\n          }\\n          .force-bar-array-flabels {\\n            font-size: 12px;\\n            fill: #fff;\\n            text-anchor: middle;\\n          }\\n          .additive-force-array-xlabel {\\n            background: none;\\n            border: 1px solid #ccc;\\n            opacity: 0.5;\\n            margin-bottom: 0px;\\n            font-size: 12px;\\n            font-family: arial;\\n            margin-left: 80px;\\n            max-width: 300px;\\n          }\\n          .additive-force-array-xlabel:focus {\\n            outline: none;\\n          }\\n          .additive-force-array-ylabel {\\n            position: relative;\\n            top: 0px;\\n            left: 0px;\\n            transform: rotate(-90deg);\\n            background: none;\\n            border: 1px solid #ccc;\\n            opacity: 0.5;\\n            margin-bottom: 0px;\\n            font-size: 12px;\\n            font-family: arial;\\n            max-width: 150px;\\n          }\\n          .additive-force-array-ylabel:focus {\\n            outline: none;\\n          }\\n          .additive-force-array-hoverLine {\\n            stroke-width: 1px;\\n            stroke: #fff;\\n            opacity: 1;\\n          }\"}}),e.createElement(\"select\",{className:\"additive-force-array-xlabel\"}),e.createElement(\"div\",{style:{height:\"0px\",textAlign:\"left\"}},e.createElement(\"select\",{className:\"additive-force-array-ylabel\"})),e.createElement(\"svg\",{ref:function(e){return t.svg=Jt(e)},style:{userSelect:\"none\",display:\"block\",fontFamily:\"arial\",sansSerif:!0}}))}}])&&Xa(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);ni.defaultProps={plot_cmap:\"RdBu\",ordering_keys:null,ordering_keys_time_format:null};const ri=ni;window.SHAP={SimpleListVisualizer:He,AdditiveForceVisualizer:Ln,AdditiveForceArrayVisualizer:ri,React:e,ReactDom:t}})()})();\n",
              "</script>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "<div id='i0CQ65ZT4WN6ISIGQ8JTG'>\n",
              "<div style='color: #900; text-align: center;'>\n",
              "  <b>Visualization omitted, Javascript library not loaded!</b><br>\n",
              "  Have you run `initjs()` in this notebook? If this notebook was from another\n",
              "  user you must also trust this notebook (File -> Trust notebook). If you are viewing\n",
              "  this notebook on github the Javascript has been stripped for security. If you are using\n",
              "  JupyterLab this error is because a JupyterLab extension has not yet been written.\n",
              "</div></div>\n",
              " <script>\n",
              "   if (window.SHAP) SHAP.ReactDom.render(\n",
              "    SHAP.React.createElement(SHAP.AdditiveForceVisualizer, {\"outNames\": [\"f(x)\"], \"baseValue\": 0.5164396574068814, \"outValue\": 0.7735953011154308, \"link\": \"identity\", \"featureNames\": [\"PC1\", \"PC2\", \"PC3\", \"PC4\", \"PC5\", \"PC6\", \"PC7\", \"exclamation_flag\", \"Sentiment_VADER\", \"Drama\", \"Comedy\", \"ActionAdventure\", \"Crime\", \"ThrillerHorror\", \"Fantasy\", \"Sci-Fi\", \"Animation\", \"Romance\"], \"features\": {\"0\": {\"effect\": 0.10451705557201788, \"value\": -1.4448147107444298}, \"1\": {\"effect\": 0.035532639729951285, \"value\": -1.0659451103229876}, \"2\": {\"effect\": 0.04868773557967348, \"value\": -0.9320341147483494}, \"3\": {\"effect\": 0.01655256157927139, \"value\": 0.11097095807949081}, \"4\": {\"effect\": 0.0047522360628679435, \"value\": -0.19496367635845652}, \"5\": {\"effect\": -0.0907949117373856, \"value\": 1.9139831268837881}, \"6\": {\"effect\": 0.04542816458871328, \"value\": -0.9717059171821956}, \"7\": {\"effect\": 0.023981519835223893, \"value\": -0.8877700114445963}, \"8\": {\"effect\": -0.0022976775599372187, \"value\": -0.662296280856509}, \"9\": {\"effect\": -0.06842759715819258, \"value\": 1.1085455513584543}, \"10\": {\"effect\": -0.006757805886867922, \"value\": -0.43314808182421005}, \"11\": {\"effect\": -0.021219437386609915, \"value\": -0.4484846930766964}, \"12\": {\"effect\": -0.06587261751238155, \"value\": 2.3086792761230397}, \"13\": {\"effect\": 0.008655018608998455, \"value\": -0.40711897076111686}, \"14\": {\"effect\": 0.023563186067516506, \"value\": -0.3235924008452868}, \"15\": {\"effect\": 0.01104677449546975, \"value\": -0.33537744652025486}, \"16\": {\"effect\": 0.21993271807880133, \"value\": 1.9481926356124402}, \"17\": {\"effect\": -0.030123919248580933, \"value\": -0.10323610881072226}}, \"plot_cmap\": \"RdBu\", \"labelMargin\": 20}),\n",
              "    document.getElementById('i0CQ65ZT4WN6ISIGQ8JTG')\n",
              "  );\n",
              "</script>"
            ],
            "text/plain": [
              "<shap.plots._force.AdditiveForceVisualizer at 0x7bf48293b340>"
            ]
          },
          "execution_count": 637,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Look at the decision plot of the TP quote with the highest predicted probability in the test data\n",
        "shap.initjs()\n",
        "\n",
        "shap_value = explainer.shap_values(test_SHAP[TP_indices[1]])[ :, 1]\n",
        "\n",
        "shap.force_plot(explainer.expected_value[1], features = test_SHAP[TP_indices[1], ], shap_values = shap_value, feature_names = feature_names)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J9GTi_nH2VQd",
      "metadata": {
        "id": "J9GTi_nH2VQd"
      },
      "source": [
        "**False Negative Quotes**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 638,
      "id": "Hnx1Zso0qjMX",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "Hnx1Zso0qjMX",
        "outputId": "e234a135-df87-4997-dca9-2265f42f243c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"FN_quotes\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"Movie Quote\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"Oh, behave!\",\n          \"Better dead and cool then alive and uncool.\",\n          \"I'm not a big fat panda. I'm THE big fat panda.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label_ChatGPT_reliable\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"4. rude\",\n          \"3. politics\",\n          \"5. insensitive\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "FN_quotes"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-46dfb253-0aaa-4df4-b311-e4193f8a387c\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Movie Quote</th>\n",
              "      <th>label_ChatGPT_reliable</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>622</th>\n",
              "      <td>I'm not a big fat panda. I'm THE big fat panda.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>147</th>\n",
              "      <td>Oh, behave!</td>\n",
              "      <td>3. politics</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>743</th>\n",
              "      <td>So many times in my life I screwed up: I've talked when I should've listened, I've been harsh when I should've been tender.</td>\n",
              "      <td>4. rude</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>322</th>\n",
              "      <td>Better dead and cool then alive and uncool.</td>\n",
              "      <td>5. insensitive</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-46dfb253-0aaa-4df4-b311-e4193f8a387c')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-46dfb253-0aaa-4df4-b311-e4193f8a387c button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-46dfb253-0aaa-4df4-b311-e4193f8a387c');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9ae55883-8bde-416c-8807-c0339161b0f1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9ae55883-8bde-416c-8807-c0339161b0f1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9ae55883-8bde-416c-8807-c0339161b0f1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_ec789133-bcad-4be6-9541-34805cea103c\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('FN_quotes')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_ec789133-bcad-4be6-9541-34805cea103c button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('FN_quotes');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                                                                                                     Movie Quote  \\\n",
              "622                                                                              I'm not a big fat panda. I'm THE big fat panda.   \n",
              "147                                                                                                                  Oh, behave!   \n",
              "743  So many times in my life I screwed up: I've talked when I should've listened, I've been harsh when I should've been tender.   \n",
              "322                                                                                  Better dead and cool then alive and uncool.   \n",
              "\n",
              "    label_ChatGPT_reliable  \n",
              "622                4. rude  \n",
              "147            3. politics  \n",
              "743                4. rude  \n",
              "322         5. insensitive  "
            ]
          },
          "execution_count": 638,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Get problematic quotes that were incorrectly predicted:\n",
        "FN_indices = np.where((y_test_pred_class_20 == 0) & (y_test_actual_class == 1))[0]\n",
        "\n",
        "FN_quotes = test.iloc[FN_indices, ][['Movie Quote', 'label_ChatGPT_reliable']]\n",
        "\n",
        "FN_quotes"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "60U09nYc2iBK",
      "metadata": {
        "id": "60U09nYc2iBK"
      },
      "source": [
        "The test set quotes above are problematic quotes that were incorrectly labelled as 'non-problematic' by our model. We see that they come from a range of 'problematic' categories. This corroborates the point made earlier that our model may struggle with the more nuanced categories such as 'politics'.\n",
        "\n",
        "Furthermore our test set was relatively small and so maybe does not cover the breadth of possible problematic quotes necessary to fairly judge the model. Our initial data was limited and the data imbalanced so this issue is partly unavoidable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 639,
      "id": "uweX5A4Wz6Op",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "id": "uweX5A4Wz6Op",
        "outputId": "d814b70c-681d-4518-bd2c-d54ae332b58d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div align='center'><img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABkAAAAWCAYAAAA1vze2AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAdxJREFUeNq0Vt1Rg0AQJjcpgBJiBWIFkgoMFYhPPAIVECogPuYpdJBYgXQQrMCUkA50V7+d2ZwXuXPGm9khHLu3f9+3l1nkWNvtNqfHLgpfQ1EUS3tz5nAQ0+NIsiAZSc6eDlI8M3J00B/mDuUKDk6kfOebAgW3pkdD0pFcODGW4gKKvOrAUm04MA4QDt1OEIXU9hDigfS5rC1eS5T90gltck1Xrizo257kgySZcNRzgCSxCvgiE9nckPJo2b/B2AcEkk2OwL8bD8gmOKR1GPbaCUqxEgTq0tLvgb6zfo7+DgYGkkWL2tqLDV4RSITfbHPPfJKIrWz4nJQTMPAWA7IbD6imcNaDeDfgk+4No+wZr40BL3g9eQJJCFqRQ54KiSt72lsLpE3o3MCBSxDuq4yOckU2hKXRuwBH3OyMR4g1UpyTYw6mlmBqNdUXRM1NfyF5EPI6JkcpIDBIX8jX6DR/6ckAZJ0wEAdLR8DEk6OfC1Pp8BKo6TQIwPJbvJ6toK5lmuvJoRtfK6Ym1iRYIarRo2UyYHvRN5qpakR3yoizWrouoyuXXQqI185LCw07op5ZyCRGL99h24InP0e9xdQukEKVmhzrqZuRIfwISB//cP3Wk3f8f/yR+BRgAHu00HjLcEQBAAAAAElFTkSuQmCC' /></div><script charset='utf-8'>/*! For license information please see bundle.js.LICENSE.txt */\n",
              "(()=>{var e={486:function(e,t,n){var r;e=n.nmd(e),function(){var a,i=\"Expected a function\",o=\"__lodash_hash_undefined__\",u=\"__lodash_placeholder__\",l=32,s=128,c=1/0,f=9007199254740991,p=NaN,d=4294967295,h=[[\"ary\",s],[\"bind\",1],[\"bindKey\",2],[\"curry\",8],[\"curryRight\",16],[\"flip\",512],[\"partial\",l],[\"partialRight\",64],[\"rearg\",256]],v=\"[object Arguments]\",g=\"[object Array]\",y=\"[object Boolean]\",m=\"[object Date]\",b=\"[object Error]\",_=\"[object Function]\",w=\"[object GeneratorFunction]\",x=\"[object Map]\",k=\"[object Number]\",S=\"[object Object]\",E=\"[object Promise]\",C=\"[object RegExp]\",T=\"[object Set]\",M=\"[object String]\",N=\"[object Symbol]\",P=\"[object WeakMap]\",z=\"[object ArrayBuffer]\",L=\"[object DataView]\",O=\"[object Float32Array]\",A=\"[object Float64Array]\",F=\"[object Int8Array]\",D=\"[object Int16Array]\",R=\"[object Int32Array]\",j=\"[object Uint8Array]\",U=\"[object Uint8ClampedArray]\",I=\"[object Uint16Array]\",$=\"[object Uint32Array]\",B=/\\b__p \\+= '';/g,W=/\\b(__p \\+=) '' \\+/g,V=/(__e\\(.*?\\)|\\b__t\\)) \\+\\n'';/g,H=/&(?:amp|lt|gt|quot|#39);/g,q=/[&<>\"']/g,Q=RegExp(H.source),Y=RegExp(q.source),G=/<%-([\\s\\S]+?)%>/g,K=/<%([\\s\\S]+?)%>/g,Z=/<%=([\\s\\S]+?)%>/g,X=/\\.|\\[(?:[^[\\]]*|([\"'])(?:(?!\\1)[^\\\\]|\\\\.)*?\\1)\\]/,J=/^\\w*$/,ee=/[^.[\\]]+|\\[(?:(-?\\d+(?:\\.\\d+)?)|([\"'])((?:(?!\\2)[^\\\\]|\\\\.)*?)\\2)\\]|(?=(?:\\.|\\[\\])(?:\\.|\\[\\]|$))/g,te=/[\\\\^$.*+?()[\\]{}|]/g,ne=RegExp(te.source),re=/^\\s+/,ae=/\\s/,ie=/\\{(?:\\n\\/\\* \\[wrapped with .+\\] \\*\\/)?\\n?/,oe=/\\{\\n\\/\\* \\[wrapped with (.+)\\] \\*/,ue=/,? & /,le=/[^\\x00-\\x2f\\x3a-\\x40\\x5b-\\x60\\x7b-\\x7f]+/g,se=/[()=,{}\\[\\]\\/\\s]/,ce=/\\\\(\\\\)?/g,fe=/\\$\\{([^\\\\}]*(?:\\\\.[^\\\\}]*)*)\\}/g,pe=/\\w*$/,de=/^[-+]0x[0-9a-f]+$/i,he=/^0b[01]+$/i,ve=/^\\[object .+?Constructor\\]$/,ge=/^0o[0-7]+$/i,ye=/^(?:0|[1-9]\\d*)$/,me=/[\\xc0-\\xd6\\xd8-\\xf6\\xf8-\\xff\\u0100-\\u017f]/g,be=/($^)/,_e=/['\\n\\r\\u2028\\u2029\\\\]/g,we=\"\\\\ud800-\\\\udfff\",xe=\"\\\\u0300-\\\\u036f\\\\ufe20-\\\\ufe2f\\\\u20d0-\\\\u20ff\",ke=\"\\\\u2700-\\\\u27bf\",Se=\"a-z\\\\xdf-\\\\xf6\\\\xf8-\\\\xff\",Ee=\"A-Z\\\\xc0-\\\\xd6\\\\xd8-\\\\xde\",Ce=\"\\\\ufe0e\\\\ufe0f\",Te=\"\\\\xac\\\\xb1\\\\xd7\\\\xf7\\\\x00-\\\\x2f\\\\x3a-\\\\x40\\\\x5b-\\\\x60\\\\x7b-\\\\xbf\\\\u2000-\\\\u206f \\\\t\\\\x0b\\\\f\\\\xa0\\\\ufeff\\\\n\\\\r\\\\u2028\\\\u2029\\\\u1680\\\\u180e\\\\u2000\\\\u2001\\\\u2002\\\\u2003\\\\u2004\\\\u2005\\\\u2006\\\\u2007\\\\u2008\\\\u2009\\\\u200a\\\\u202f\\\\u205f\\\\u3000\",Me=\"[\"+we+\"]\",Ne=\"[\"+Te+\"]\",Pe=\"[\"+xe+\"]\",ze=\"\\\\d+\",Le=\"[\"+ke+\"]\",Oe=\"[\"+Se+\"]\",Ae=\"[^\"+we+Te+ze+ke+Se+Ee+\"]\",Fe=\"\\\\ud83c[\\\\udffb-\\\\udfff]\",De=\"[^\"+we+\"]\",Re=\"(?:\\\\ud83c[\\\\udde6-\\\\uddff]){2}\",je=\"[\\\\ud800-\\\\udbff][\\\\udc00-\\\\udfff]\",Ue=\"[\"+Ee+\"]\",Ie=\"\\\\u200d\",$e=\"(?:\"+Oe+\"|\"+Ae+\")\",Be=\"(?:\"+Ue+\"|\"+Ae+\")\",We=\"(?:['’](?:d|ll|m|re|s|t|ve))?\",Ve=\"(?:['’](?:D|LL|M|RE|S|T|VE))?\",He=\"(?:\"+Pe+\"|\"+Fe+\")?\",qe=\"[\"+Ce+\"]?\",Qe=qe+He+\"(?:\"+Ie+\"(?:\"+[De,Re,je].join(\"|\")+\")\"+qe+He+\")*\",Ye=\"(?:\"+[Le,Re,je].join(\"|\")+\")\"+Qe,Ge=\"(?:\"+[De+Pe+\"?\",Pe,Re,je,Me].join(\"|\")+\")\",Ke=RegExp(\"['’]\",\"g\"),Ze=RegExp(Pe,\"g\"),Xe=RegExp(Fe+\"(?=\"+Fe+\")|\"+Ge+Qe,\"g\"),Je=RegExp([Ue+\"?\"+Oe+\"+\"+We+\"(?=\"+[Ne,Ue,\"$\"].join(\"|\")+\")\",Be+\"+\"+Ve+\"(?=\"+[Ne,Ue+$e,\"$\"].join(\"|\")+\")\",Ue+\"?\"+$e+\"+\"+We,Ue+\"+\"+Ve,\"\\\\d*(?:1ST|2ND|3RD|(?![123])\\\\dTH)(?=\\\\b|[a-z_])\",\"\\\\d*(?:1st|2nd|3rd|(?![123])\\\\dth)(?=\\\\b|[A-Z_])\",ze,Ye].join(\"|\"),\"g\"),et=RegExp(\"[\"+Ie+we+xe+Ce+\"]\"),tt=/[a-z][A-Z]|[A-Z]{2}[a-z]|[0-9][a-zA-Z]|[a-zA-Z][0-9]|[^a-zA-Z0-9 ]/,nt=[\"Array\",\"Buffer\",\"DataView\",\"Date\",\"Error\",\"Float32Array\",\"Float64Array\",\"Function\",\"Int8Array\",\"Int16Array\",\"Int32Array\",\"Map\",\"Math\",\"Object\",\"Promise\",\"RegExp\",\"Set\",\"String\",\"Symbol\",\"TypeError\",\"Uint8Array\",\"Uint8ClampedArray\",\"Uint16Array\",\"Uint32Array\",\"WeakMap\",\"_\",\"clearTimeout\",\"isFinite\",\"parseInt\",\"setTimeout\"],rt=-1,at={};at[O]=at[A]=at[F]=at[D]=at[R]=at[j]=at[U]=at[I]=at[$]=!0,at[v]=at[g]=at[z]=at[y]=at[L]=at[m]=at[b]=at[_]=at[x]=at[k]=at[S]=at[C]=at[T]=at[M]=at[P]=!1;var it={};it[v]=it[g]=it[z]=it[L]=it[y]=it[m]=it[O]=it[A]=it[F]=it[D]=it[R]=it[x]=it[k]=it[S]=it[C]=it[T]=it[M]=it[N]=it[j]=it[U]=it[I]=it[$]=!0,it[b]=it[_]=it[P]=!1;var ot={\"\\\\\":\"\\\\\",\"'\":\"'\",\"\\n\":\"n\",\"\\r\":\"r\",\"\\u2028\":\"u2028\",\"\\u2029\":\"u2029\"},ut=parseFloat,lt=parseInt,st=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,ct=\"object\"==typeof self&&self&&self.Object===Object&&self,ft=st||ct||Function(\"return this\")(),pt=t&&!t.nodeType&&t,dt=pt&&e&&!e.nodeType&&e,ht=dt&&dt.exports===pt,vt=ht&&st.process,gt=function(){try{return dt&&dt.require&&dt.require(\"util\").types||vt&&vt.binding&&vt.binding(\"util\")}catch(e){}}(),yt=gt&&gt.isArrayBuffer,mt=gt&&gt.isDate,bt=gt&&gt.isMap,_t=gt&&gt.isRegExp,wt=gt&&gt.isSet,xt=gt&&gt.isTypedArray;function kt(e,t,n){switch(n.length){case 0:return e.call(t);case 1:return e.call(t,n[0]);case 2:return e.call(t,n[0],n[1]);case 3:return e.call(t,n[0],n[1],n[2])}return e.apply(t,n)}function St(e,t,n,r){for(var a=-1,i=null==e?0:e.length;++a<i;){var o=e[a];t(r,o,n(o),e)}return r}function Et(e,t){for(var n=-1,r=null==e?0:e.length;++n<r&&!1!==t(e[n],n,e););return e}function Ct(e,t){for(var n=null==e?0:e.length;n--&&!1!==t(e[n],n,e););return e}function Tt(e,t){for(var n=-1,r=null==e?0:e.length;++n<r;)if(!t(e[n],n,e))return!1;return!0}function Mt(e,t){for(var n=-1,r=null==e?0:e.length,a=0,i=[];++n<r;){var o=e[n];t(o,n,e)&&(i[a++]=o)}return i}function Nt(e,t){return!(null==e||!e.length)&&Ut(e,t,0)>-1}function Pt(e,t,n){for(var r=-1,a=null==e?0:e.length;++r<a;)if(n(t,e[r]))return!0;return!1}function zt(e,t){for(var n=-1,r=null==e?0:e.length,a=Array(r);++n<r;)a[n]=t(e[n],n,e);return a}function Lt(e,t){for(var n=-1,r=t.length,a=e.length;++n<r;)e[a+n]=t[n];return e}function Ot(e,t,n,r){var a=-1,i=null==e?0:e.length;for(r&&i&&(n=e[++a]);++a<i;)n=t(n,e[a],a,e);return n}function At(e,t,n,r){var a=null==e?0:e.length;for(r&&a&&(n=e[--a]);a--;)n=t(n,e[a],a,e);return n}function Ft(e,t){for(var n=-1,r=null==e?0:e.length;++n<r;)if(t(e[n],n,e))return!0;return!1}var Dt=Wt(\"length\");function Rt(e,t,n){var r;return n(e,(function(e,n,a){if(t(e,n,a))return r=n,!1})),r}function jt(e,t,n,r){for(var a=e.length,i=n+(r?1:-1);r?i--:++i<a;)if(t(e[i],i,e))return i;return-1}function Ut(e,t,n){return t==t?function(e,t,n){for(var r=n-1,a=e.length;++r<a;)if(e[r]===t)return r;return-1}(e,t,n):jt(e,$t,n)}function It(e,t,n,r){for(var a=n-1,i=e.length;++a<i;)if(r(e[a],t))return a;return-1}function $t(e){return e!=e}function Bt(e,t){var n=null==e?0:e.length;return n?qt(e,t)/n:p}function Wt(e){return function(t){return null==t?a:t[e]}}function Vt(e){return function(t){return null==e?a:e[t]}}function Ht(e,t,n,r,a){return a(e,(function(e,a,i){n=r?(r=!1,e):t(n,e,a,i)})),n}function qt(e,t){for(var n,r=-1,i=e.length;++r<i;){var o=t(e[r]);o!==a&&(n=n===a?o:n+o)}return n}function Qt(e,t){for(var n=-1,r=Array(e);++n<e;)r[n]=t(n);return r}function Yt(e){return e?e.slice(0,pn(e)+1).replace(re,\"\"):e}function Gt(e){return function(t){return e(t)}}function Kt(e,t){return zt(t,(function(t){return e[t]}))}function Zt(e,t){return e.has(t)}function Xt(e,t){for(var n=-1,r=e.length;++n<r&&Ut(t,e[n],0)>-1;);return n}function Jt(e,t){for(var n=e.length;n--&&Ut(t,e[n],0)>-1;);return n}var en=Vt({À:\"A\",Á:\"A\",Â:\"A\",Ã:\"A\",Ä:\"A\",Å:\"A\",à:\"a\",á:\"a\",â:\"a\",ã:\"a\",ä:\"a\",å:\"a\",Ç:\"C\",ç:\"c\",Ð:\"D\",ð:\"d\",È:\"E\",É:\"E\",Ê:\"E\",Ë:\"E\",è:\"e\",é:\"e\",ê:\"e\",ë:\"e\",Ì:\"I\",Í:\"I\",Î:\"I\",Ï:\"I\",ì:\"i\",í:\"i\",î:\"i\",ï:\"i\",Ñ:\"N\",ñ:\"n\",Ò:\"O\",Ó:\"O\",Ô:\"O\",Õ:\"O\",Ö:\"O\",Ø:\"O\",ò:\"o\",ó:\"o\",ô:\"o\",õ:\"o\",ö:\"o\",ø:\"o\",Ù:\"U\",Ú:\"U\",Û:\"U\",Ü:\"U\",ù:\"u\",ú:\"u\",û:\"u\",ü:\"u\",Ý:\"Y\",ý:\"y\",ÿ:\"y\",Æ:\"Ae\",æ:\"ae\",Þ:\"Th\",þ:\"th\",ß:\"ss\",Ā:\"A\",Ă:\"A\",Ą:\"A\",ā:\"a\",ă:\"a\",ą:\"a\",Ć:\"C\",Ĉ:\"C\",Ċ:\"C\",Č:\"C\",ć:\"c\",ĉ:\"c\",ċ:\"c\",č:\"c\",Ď:\"D\",Đ:\"D\",ď:\"d\",đ:\"d\",Ē:\"E\",Ĕ:\"E\",Ė:\"E\",Ę:\"E\",Ě:\"E\",ē:\"e\",ĕ:\"e\",ė:\"e\",ę:\"e\",ě:\"e\",Ĝ:\"G\",Ğ:\"G\",Ġ:\"G\",Ģ:\"G\",ĝ:\"g\",ğ:\"g\",ġ:\"g\",ģ:\"g\",Ĥ:\"H\",Ħ:\"H\",ĥ:\"h\",ħ:\"h\",Ĩ:\"I\",Ī:\"I\",Ĭ:\"I\",Į:\"I\",İ:\"I\",ĩ:\"i\",ī:\"i\",ĭ:\"i\",į:\"i\",ı:\"i\",Ĵ:\"J\",ĵ:\"j\",Ķ:\"K\",ķ:\"k\",ĸ:\"k\",Ĺ:\"L\",Ļ:\"L\",Ľ:\"L\",Ŀ:\"L\",Ł:\"L\",ĺ:\"l\",ļ:\"l\",ľ:\"l\",ŀ:\"l\",ł:\"l\",Ń:\"N\",Ņ:\"N\",Ň:\"N\",Ŋ:\"N\",ń:\"n\",ņ:\"n\",ň:\"n\",ŋ:\"n\",Ō:\"O\",Ŏ:\"O\",Ő:\"O\",ō:\"o\",ŏ:\"o\",ő:\"o\",Ŕ:\"R\",Ŗ:\"R\",Ř:\"R\",ŕ:\"r\",ŗ:\"r\",ř:\"r\",Ś:\"S\",Ŝ:\"S\",Ş:\"S\",Š:\"S\",ś:\"s\",ŝ:\"s\",ş:\"s\",š:\"s\",Ţ:\"T\",Ť:\"T\",Ŧ:\"T\",ţ:\"t\",ť:\"t\",ŧ:\"t\",Ũ:\"U\",Ū:\"U\",Ŭ:\"U\",Ů:\"U\",Ű:\"U\",Ų:\"U\",ũ:\"u\",ū:\"u\",ŭ:\"u\",ů:\"u\",ű:\"u\",ų:\"u\",Ŵ:\"W\",ŵ:\"w\",Ŷ:\"Y\",ŷ:\"y\",Ÿ:\"Y\",Ź:\"Z\",Ż:\"Z\",Ž:\"Z\",ź:\"z\",ż:\"z\",ž:\"z\",Ĳ:\"IJ\",ĳ:\"ij\",Œ:\"Oe\",œ:\"oe\",ŉ:\"'n\",ſ:\"s\"}),tn=Vt({\"&\":\"&amp;\",\"<\":\"&lt;\",\">\":\"&gt;\",'\"':\"&quot;\",\"'\":\"&#39;\"});function nn(e){return\"\\\\\"+ot[e]}function rn(e){return et.test(e)}function an(e){var t=-1,n=Array(e.size);return e.forEach((function(e,r){n[++t]=[r,e]})),n}function on(e,t){return function(n){return e(t(n))}}function un(e,t){for(var n=-1,r=e.length,a=0,i=[];++n<r;){var o=e[n];o!==t&&o!==u||(e[n]=u,i[a++]=n)}return i}function ln(e){var t=-1,n=Array(e.size);return e.forEach((function(e){n[++t]=e})),n}function sn(e){var t=-1,n=Array(e.size);return e.forEach((function(e){n[++t]=[e,e]})),n}function cn(e){return rn(e)?function(e){for(var t=Xe.lastIndex=0;Xe.test(e);)++t;return t}(e):Dt(e)}function fn(e){return rn(e)?function(e){return e.match(Xe)||[]}(e):function(e){return e.split(\"\")}(e)}function pn(e){for(var t=e.length;t--&&ae.test(e.charAt(t)););return t}var dn=Vt({\"&amp;\":\"&\",\"&lt;\":\"<\",\"&gt;\":\">\",\"&quot;\":'\"',\"&#39;\":\"'\"}),hn=function e(t){var n,r=(t=null==t?ft:hn.defaults(ft.Object(),t,hn.pick(ft,nt))).Array,ae=t.Date,we=t.Error,xe=t.Function,ke=t.Math,Se=t.Object,Ee=t.RegExp,Ce=t.String,Te=t.TypeError,Me=r.prototype,Ne=xe.prototype,Pe=Se.prototype,ze=t[\"__core-js_shared__\"],Le=Ne.toString,Oe=Pe.hasOwnProperty,Ae=0,Fe=(n=/[^.]+$/.exec(ze&&ze.keys&&ze.keys.IE_PROTO||\"\"))?\"Symbol(src)_1.\"+n:\"\",De=Pe.toString,Re=Le.call(Se),je=ft._,Ue=Ee(\"^\"+Le.call(Oe).replace(te,\"\\\\$&\").replace(/hasOwnProperty|(function).*?(?=\\\\\\()| for .+?(?=\\\\\\])/g,\"$1.*?\")+\"$\"),Ie=ht?t.Buffer:a,$e=t.Symbol,Be=t.Uint8Array,We=Ie?Ie.allocUnsafe:a,Ve=on(Se.getPrototypeOf,Se),He=Se.create,qe=Pe.propertyIsEnumerable,Qe=Me.splice,Ye=$e?$e.isConcatSpreadable:a,Ge=$e?$e.iterator:a,Xe=$e?$e.toStringTag:a,et=function(){try{var e=li(Se,\"defineProperty\");return e({},\"\",{}),e}catch(e){}}(),ot=t.clearTimeout!==ft.clearTimeout&&t.clearTimeout,st=ae&&ae.now!==ft.Date.now&&ae.now,ct=t.setTimeout!==ft.setTimeout&&t.setTimeout,pt=ke.ceil,dt=ke.floor,vt=Se.getOwnPropertySymbols,gt=Ie?Ie.isBuffer:a,Dt=t.isFinite,Vt=Me.join,vn=on(Se.keys,Se),gn=ke.max,yn=ke.min,mn=ae.now,bn=t.parseInt,_n=ke.random,wn=Me.reverse,xn=li(t,\"DataView\"),kn=li(t,\"Map\"),Sn=li(t,\"Promise\"),En=li(t,\"Set\"),Cn=li(t,\"WeakMap\"),Tn=li(Se,\"create\"),Mn=Cn&&new Cn,Nn={},Pn=Di(xn),zn=Di(kn),Ln=Di(Sn),On=Di(En),An=Di(Cn),Fn=$e?$e.prototype:a,Dn=Fn?Fn.valueOf:a,Rn=Fn?Fn.toString:a;function jn(e){if(eu(e)&&!Wo(e)&&!(e instanceof Bn)){if(e instanceof $n)return e;if(Oe.call(e,\"__wrapped__\"))return Ri(e)}return new $n(e)}var Un=function(){function e(){}return function(t){if(!Jo(t))return{};if(He)return He(t);e.prototype=t;var n=new e;return e.prototype=a,n}}();function In(){}function $n(e,t){this.__wrapped__=e,this.__actions__=[],this.__chain__=!!t,this.__index__=0,this.__values__=a}function Bn(e){this.__wrapped__=e,this.__actions__=[],this.__dir__=1,this.__filtered__=!1,this.__iteratees__=[],this.__takeCount__=d,this.__views__=[]}function Wn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function Vn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function Hn(e){var t=-1,n=null==e?0:e.length;for(this.clear();++t<n;){var r=e[t];this.set(r[0],r[1])}}function qn(e){var t=-1,n=null==e?0:e.length;for(this.__data__=new Hn;++t<n;)this.add(e[t])}function Qn(e){var t=this.__data__=new Vn(e);this.size=t.size}function Yn(e,t){var n=Wo(e),r=!n&&Bo(e),a=!n&&!r&&Qo(e),i=!n&&!r&&!a&&lu(e),o=n||r||a||i,u=o?Qt(e.length,Ce):[],l=u.length;for(var s in e)!t&&!Oe.call(e,s)||o&&(\"length\"==s||a&&(\"offset\"==s||\"parent\"==s)||i&&(\"buffer\"==s||\"byteLength\"==s||\"byteOffset\"==s)||vi(s,l))||u.push(s);return u}function Gn(e){var t=e.length;return t?e[Hr(0,t-1)]:a}function Kn(e,t){return zi(Ca(e),ir(t,0,e.length))}function Zn(e){return zi(Ca(e))}function Xn(e,t,n){(n!==a&&!Uo(e[t],n)||n===a&&!(t in e))&&rr(e,t,n)}function Jn(e,t,n){var r=e[t];Oe.call(e,t)&&Uo(r,n)&&(n!==a||t in e)||rr(e,t,n)}function er(e,t){for(var n=e.length;n--;)if(Uo(e[n][0],t))return n;return-1}function tr(e,t,n,r){return cr(e,(function(e,a,i){t(r,e,n(e),i)})),r}function nr(e,t){return e&&Ta(t,Pu(t),e)}function rr(e,t,n){\"__proto__\"==t&&et?et(e,t,{configurable:!0,enumerable:!0,value:n,writable:!0}):e[t]=n}function ar(e,t){for(var n=-1,i=t.length,o=r(i),u=null==e;++n<i;)o[n]=u?a:Eu(e,t[n]);return o}function ir(e,t,n){return e==e&&(n!==a&&(e=e<=n?e:n),t!==a&&(e=e>=t?e:t)),e}function or(e,t,n,r,i,o){var u,l=1&t,s=2&t,c=4&t;if(n&&(u=i?n(e,r,i,o):n(e)),u!==a)return u;if(!Jo(e))return e;var f=Wo(e);if(f){if(u=function(e){var t=e.length,n=new e.constructor(t);return t&&\"string\"==typeof e[0]&&Oe.call(e,\"index\")&&(n.index=e.index,n.input=e.input),n}(e),!l)return Ca(e,u)}else{var p=fi(e),d=p==_||p==w;if(Qo(e))return _a(e,l);if(p==S||p==v||d&&!i){if(u=s||d?{}:di(e),!l)return s?function(e,t){return Ta(e,ci(e),t)}(e,function(e,t){return e&&Ta(t,zu(t),e)}(u,e)):function(e,t){return Ta(e,si(e),t)}(e,nr(u,e))}else{if(!it[p])return i?e:{};u=function(e,t,n){var r,a=e.constructor;switch(t){case z:return wa(e);case y:case m:return new a(+e);case L:return function(e,t){var n=t?wa(e.buffer):e.buffer;return new e.constructor(n,e.byteOffset,e.byteLength)}(e,n);case O:case A:case F:case D:case R:case j:case U:case I:case $:return xa(e,n);case x:return new a;case k:case M:return new a(e);case C:return function(e){var t=new e.constructor(e.source,pe.exec(e));return t.lastIndex=e.lastIndex,t}(e);case T:return new a;case N:return r=e,Dn?Se(Dn.call(r)):{}}}(e,p,l)}}o||(o=new Qn);var h=o.get(e);if(h)return h;o.set(e,u),iu(e)?e.forEach((function(r){u.add(or(r,t,n,r,e,o))})):tu(e)&&e.forEach((function(r,a){u.set(a,or(r,t,n,a,e,o))}));var g=f?a:(c?s?ti:ei:s?zu:Pu)(e);return Et(g||e,(function(r,a){g&&(r=e[a=r]),Jn(u,a,or(r,t,n,a,e,o))})),u}function ur(e,t,n){var r=n.length;if(null==e)return!r;for(e=Se(e);r--;){var i=n[r],o=t[i],u=e[i];if(u===a&&!(i in e)||!o(u))return!1}return!0}function lr(e,t,n){if(\"function\"!=typeof e)throw new Te(i);return Ti((function(){e.apply(a,n)}),t)}function sr(e,t,n,r){var a=-1,i=Nt,o=!0,u=e.length,l=[],s=t.length;if(!u)return l;n&&(t=zt(t,Gt(n))),r?(i=Pt,o=!1):t.length>=200&&(i=Zt,o=!1,t=new qn(t));e:for(;++a<u;){var c=e[a],f=null==n?c:n(c);if(c=r||0!==c?c:0,o&&f==f){for(var p=s;p--;)if(t[p]===f)continue e;l.push(c)}else i(t,f,r)||l.push(c)}return l}jn.templateSettings={escape:G,evaluate:K,interpolate:Z,variable:\"\",imports:{_:jn}},jn.prototype=In.prototype,jn.prototype.constructor=jn,$n.prototype=Un(In.prototype),$n.prototype.constructor=$n,Bn.prototype=Un(In.prototype),Bn.prototype.constructor=Bn,Wn.prototype.clear=function(){this.__data__=Tn?Tn(null):{},this.size=0},Wn.prototype.delete=function(e){var t=this.has(e)&&delete this.__data__[e];return this.size-=t?1:0,t},Wn.prototype.get=function(e){var t=this.__data__;if(Tn){var n=t[e];return n===o?a:n}return Oe.call(t,e)?t[e]:a},Wn.prototype.has=function(e){var t=this.__data__;return Tn?t[e]!==a:Oe.call(t,e)},Wn.prototype.set=function(e,t){var n=this.__data__;return this.size+=this.has(e)?0:1,n[e]=Tn&&t===a?o:t,this},Vn.prototype.clear=function(){this.__data__=[],this.size=0},Vn.prototype.delete=function(e){var t=this.__data__,n=er(t,e);return!(n<0||(n==t.length-1?t.pop():Qe.call(t,n,1),--this.size,0))},Vn.prototype.get=function(e){var t=this.__data__,n=er(t,e);return n<0?a:t[n][1]},Vn.prototype.has=function(e){return er(this.__data__,e)>-1},Vn.prototype.set=function(e,t){var n=this.__data__,r=er(n,e);return r<0?(++this.size,n.push([e,t])):n[r][1]=t,this},Hn.prototype.clear=function(){this.size=0,this.__data__={hash:new Wn,map:new(kn||Vn),string:new Wn}},Hn.prototype.delete=function(e){var t=oi(this,e).delete(e);return this.size-=t?1:0,t},Hn.prototype.get=function(e){return oi(this,e).get(e)},Hn.prototype.has=function(e){return oi(this,e).has(e)},Hn.prototype.set=function(e,t){var n=oi(this,e),r=n.size;return n.set(e,t),this.size+=n.size==r?0:1,this},qn.prototype.add=qn.prototype.push=function(e){return this.__data__.set(e,o),this},qn.prototype.has=function(e){return this.__data__.has(e)},Qn.prototype.clear=function(){this.__data__=new Vn,this.size=0},Qn.prototype.delete=function(e){var t=this.__data__,n=t.delete(e);return this.size=t.size,n},Qn.prototype.get=function(e){return this.__data__.get(e)},Qn.prototype.has=function(e){return this.__data__.has(e)},Qn.prototype.set=function(e,t){var n=this.__data__;if(n instanceof Vn){var r=n.__data__;if(!kn||r.length<199)return r.push([e,t]),this.size=++n.size,this;n=this.__data__=new Hn(r)}return n.set(e,t),this.size=n.size,this};var cr=Pa(mr),fr=Pa(br,!0);function pr(e,t){var n=!0;return cr(e,(function(e,r,a){return n=!!t(e,r,a)})),n}function dr(e,t,n){for(var r=-1,i=e.length;++r<i;){var o=e[r],u=t(o);if(null!=u&&(l===a?u==u&&!uu(u):n(u,l)))var l=u,s=o}return s}function hr(e,t){var n=[];return cr(e,(function(e,r,a){t(e,r,a)&&n.push(e)})),n}function vr(e,t,n,r,a){var i=-1,o=e.length;for(n||(n=hi),a||(a=[]);++i<o;){var u=e[i];t>0&&n(u)?t>1?vr(u,t-1,n,r,a):Lt(a,u):r||(a[a.length]=u)}return a}var gr=za(),yr=za(!0);function mr(e,t){return e&&gr(e,t,Pu)}function br(e,t){return e&&yr(e,t,Pu)}function _r(e,t){return Mt(t,(function(t){return Ko(e[t])}))}function wr(e,t){for(var n=0,r=(t=ga(t,e)).length;null!=e&&n<r;)e=e[Fi(t[n++])];return n&&n==r?e:a}function xr(e,t,n){var r=t(e);return Wo(e)?r:Lt(r,n(e))}function kr(e){return null==e?e===a?\"[object Undefined]\":\"[object Null]\":Xe&&Xe in Se(e)?function(e){var t=Oe.call(e,Xe),n=e[Xe];try{e[Xe]=a;var r=!0}catch(e){}var i=De.call(e);return r&&(t?e[Xe]=n:delete e[Xe]),i}(e):function(e){return De.call(e)}(e)}function Sr(e,t){return e>t}function Er(e,t){return null!=e&&Oe.call(e,t)}function Cr(e,t){return null!=e&&t in Se(e)}function Tr(e,t,n){for(var i=n?Pt:Nt,o=e[0].length,u=e.length,l=u,s=r(u),c=1/0,f=[];l--;){var p=e[l];l&&t&&(p=zt(p,Gt(t))),c=yn(p.length,c),s[l]=!n&&(t||o>=120&&p.length>=120)?new qn(l&&p):a}p=e[0];var d=-1,h=s[0];e:for(;++d<o&&f.length<c;){var v=p[d],g=t?t(v):v;if(v=n||0!==v?v:0,!(h?Zt(h,g):i(f,g,n))){for(l=u;--l;){var y=s[l];if(!(y?Zt(y,g):i(e[l],g,n)))continue e}h&&h.push(g),f.push(v)}}return f}function Mr(e,t,n){var r=null==(e=Si(e,t=ga(t,e)))?e:e[Fi(Yi(t))];return null==r?a:kt(r,e,n)}function Nr(e){return eu(e)&&kr(e)==v}function Pr(e,t,n,r,i){return e===t||(null==e||null==t||!eu(e)&&!eu(t)?e!=e&&t!=t:function(e,t,n,r,i,o){var u=Wo(e),l=Wo(t),s=u?g:fi(e),c=l?g:fi(t),f=(s=s==v?S:s)==S,p=(c=c==v?S:c)==S,d=s==c;if(d&&Qo(e)){if(!Qo(t))return!1;u=!0,f=!1}if(d&&!f)return o||(o=new Qn),u||lu(e)?Xa(e,t,n,r,i,o):function(e,t,n,r,a,i,o){switch(n){case L:if(e.byteLength!=t.byteLength||e.byteOffset!=t.byteOffset)return!1;e=e.buffer,t=t.buffer;case z:return!(e.byteLength!=t.byteLength||!i(new Be(e),new Be(t)));case y:case m:case k:return Uo(+e,+t);case b:return e.name==t.name&&e.message==t.message;case C:case M:return e==t+\"\";case x:var u=an;case T:var l=1&r;if(u||(u=ln),e.size!=t.size&&!l)return!1;var s=o.get(e);if(s)return s==t;r|=2,o.set(e,t);var c=Xa(u(e),u(t),r,a,i,o);return o.delete(e),c;case N:if(Dn)return Dn.call(e)==Dn.call(t)}return!1}(e,t,s,n,r,i,o);if(!(1&n)){var h=f&&Oe.call(e,\"__wrapped__\"),_=p&&Oe.call(t,\"__wrapped__\");if(h||_){var w=h?e.value():e,E=_?t.value():t;return o||(o=new Qn),i(w,E,n,r,o)}}return!!d&&(o||(o=new Qn),function(e,t,n,r,i,o){var u=1&n,l=ei(e),s=l.length;if(s!=ei(t).length&&!u)return!1;for(var c=s;c--;){var f=l[c];if(!(u?f in t:Oe.call(t,f)))return!1}var p=o.get(e),d=o.get(t);if(p&&d)return p==t&&d==e;var h=!0;o.set(e,t),o.set(t,e);for(var v=u;++c<s;){var g=e[f=l[c]],y=t[f];if(r)var m=u?r(y,g,f,t,e,o):r(g,y,f,e,t,o);if(!(m===a?g===y||i(g,y,n,r,o):m)){h=!1;break}v||(v=\"constructor\"==f)}if(h&&!v){var b=e.constructor,_=t.constructor;b==_||!(\"constructor\"in e)||!(\"constructor\"in t)||\"function\"==typeof b&&b instanceof b&&\"function\"==typeof _&&_ instanceof _||(h=!1)}return o.delete(e),o.delete(t),h}(e,t,n,r,i,o))}(e,t,n,r,Pr,i))}function zr(e,t,n,r){var i=n.length,o=i,u=!r;if(null==e)return!o;for(e=Se(e);i--;){var l=n[i];if(u&&l[2]?l[1]!==e[l[0]]:!(l[0]in e))return!1}for(;++i<o;){var s=(l=n[i])[0],c=e[s],f=l[1];if(u&&l[2]){if(c===a&&!(s in e))return!1}else{var p=new Qn;if(r)var d=r(c,f,s,e,t,p);if(!(d===a?Pr(f,c,3,r,p):d))return!1}}return!0}function Lr(e){return!(!Jo(e)||(t=e,Fe&&Fe in t))&&(Ko(e)?Ue:ve).test(Di(e));var t}function Or(e){return\"function\"==typeof e?e:null==e?nl:\"object\"==typeof e?Wo(e)?jr(e[0],e[1]):Rr(e):fl(e)}function Ar(e){if(!_i(e))return vn(e);var t=[];for(var n in Se(e))Oe.call(e,n)&&\"constructor\"!=n&&t.push(n);return t}function Fr(e,t){return e<t}function Dr(e,t){var n=-1,a=Ho(e)?r(e.length):[];return cr(e,(function(e,r,i){a[++n]=t(e,r,i)})),a}function Rr(e){var t=ui(e);return 1==t.length&&t[0][2]?xi(t[0][0],t[0][1]):function(n){return n===e||zr(n,e,t)}}function jr(e,t){return yi(e)&&wi(t)?xi(Fi(e),t):function(n){var r=Eu(n,e);return r===a&&r===t?Cu(n,e):Pr(t,r,3)}}function Ur(e,t,n,r,i){e!==t&&gr(t,(function(o,u){if(i||(i=new Qn),Jo(o))!function(e,t,n,r,i,o,u){var l=Ei(e,n),s=Ei(t,n),c=u.get(s);if(c)Xn(e,n,c);else{var f=o?o(l,s,n+\"\",e,t,u):a,p=f===a;if(p){var d=Wo(s),h=!d&&Qo(s),v=!d&&!h&&lu(s);f=s,d||h||v?Wo(l)?f=l:qo(l)?f=Ca(l):h?(p=!1,f=_a(s,!0)):v?(p=!1,f=xa(s,!0)):f=[]:ru(s)||Bo(s)?(f=l,Bo(l)?f=gu(l):Jo(l)&&!Ko(l)||(f=di(s))):p=!1}p&&(u.set(s,f),i(f,s,r,o,u),u.delete(s)),Xn(e,n,f)}}(e,t,u,n,Ur,r,i);else{var l=r?r(Ei(e,u),o,u+\"\",e,t,i):a;l===a&&(l=o),Xn(e,u,l)}}),zu)}function Ir(e,t){var n=e.length;if(n)return vi(t+=t<0?n:0,n)?e[t]:a}function $r(e,t,n){t=t.length?zt(t,(function(e){return Wo(e)?function(t){return wr(t,1===e.length?e[0]:e)}:e})):[nl];var r=-1;t=zt(t,Gt(ii()));var a=Dr(e,(function(e,n,a){var i=zt(t,(function(t){return t(e)}));return{criteria:i,index:++r,value:e}}));return function(e,t){var r=e.length;for(e.sort((function(e,t){return function(e,t,n){for(var r=-1,a=e.criteria,i=t.criteria,o=a.length,u=n.length;++r<o;){var l=ka(a[r],i[r]);if(l)return r>=u?l:l*(\"desc\"==n[r]?-1:1)}return e.index-t.index}(e,t,n)}));r--;)e[r]=e[r].value;return e}(a)}function Br(e,t,n){for(var r=-1,a=t.length,i={};++r<a;){var o=t[r],u=wr(e,o);n(u,o)&&Kr(i,ga(o,e),u)}return i}function Wr(e,t,n,r){var a=r?It:Ut,i=-1,o=t.length,u=e;for(e===t&&(t=Ca(t)),n&&(u=zt(e,Gt(n)));++i<o;)for(var l=0,s=t[i],c=n?n(s):s;(l=a(u,c,l,r))>-1;)u!==e&&Qe.call(u,l,1),Qe.call(e,l,1);return e}function Vr(e,t){for(var n=e?t.length:0,r=n-1;n--;){var a=t[n];if(n==r||a!==i){var i=a;vi(a)?Qe.call(e,a,1):la(e,a)}}return e}function Hr(e,t){return e+dt(_n()*(t-e+1))}function qr(e,t){var n=\"\";if(!e||t<1||t>f)return n;do{t%2&&(n+=e),(t=dt(t/2))&&(e+=e)}while(t);return n}function Qr(e,t){return Mi(ki(e,t,nl),e+\"\")}function Yr(e){return Gn(Uu(e))}function Gr(e,t){var n=Uu(e);return zi(n,ir(t,0,n.length))}function Kr(e,t,n,r){if(!Jo(e))return e;for(var i=-1,o=(t=ga(t,e)).length,u=o-1,l=e;null!=l&&++i<o;){var s=Fi(t[i]),c=n;if(\"__proto__\"===s||\"constructor\"===s||\"prototype\"===s)return e;if(i!=u){var f=l[s];(c=r?r(f,s,l):a)===a&&(c=Jo(f)?f:vi(t[i+1])?[]:{})}Jn(l,s,c),l=l[s]}return e}var Zr=Mn?function(e,t){return Mn.set(e,t),e}:nl,Xr=et?function(e,t){return et(e,\"toString\",{configurable:!0,enumerable:!1,value:Ju(t),writable:!0})}:nl;function Jr(e){return zi(Uu(e))}function ea(e,t,n){var a=-1,i=e.length;t<0&&(t=-t>i?0:i+t),(n=n>i?i:n)<0&&(n+=i),i=t>n?0:n-t>>>0,t>>>=0;for(var o=r(i);++a<i;)o[a]=e[a+t];return o}function ta(e,t){var n;return cr(e,(function(e,r,a){return!(n=t(e,r,a))})),!!n}function na(e,t,n){var r=0,a=null==e?r:e.length;if(\"number\"==typeof t&&t==t&&a<=2147483647){for(;r<a;){var i=r+a>>>1,o=e[i];null!==o&&!uu(o)&&(n?o<=t:o<t)?r=i+1:a=i}return a}return ra(e,t,nl,n)}function ra(e,t,n,r){var i=0,o=null==e?0:e.length;if(0===o)return 0;for(var u=(t=n(t))!=t,l=null===t,s=uu(t),c=t===a;i<o;){var f=dt((i+o)/2),p=n(e[f]),d=p!==a,h=null===p,v=p==p,g=uu(p);if(u)var y=r||v;else y=c?v&&(r||d):l?v&&d&&(r||!h):s?v&&d&&!h&&(r||!g):!h&&!g&&(r?p<=t:p<t);y?i=f+1:o=f}return yn(o,4294967294)}function aa(e,t){for(var n=-1,r=e.length,a=0,i=[];++n<r;){var o=e[n],u=t?t(o):o;if(!n||!Uo(u,l)){var l=u;i[a++]=0===o?0:o}}return i}function ia(e){return\"number\"==typeof e?e:uu(e)?p:+e}function oa(e){if(\"string\"==typeof e)return e;if(Wo(e))return zt(e,oa)+\"\";if(uu(e))return Rn?Rn.call(e):\"\";var t=e+\"\";return\"0\"==t&&1/e==-1/0?\"-0\":t}function ua(e,t,n){var r=-1,a=Nt,i=e.length,o=!0,u=[],l=u;if(n)o=!1,a=Pt;else if(i>=200){var s=t?null:qa(e);if(s)return ln(s);o=!1,a=Zt,l=new qn}else l=t?[]:u;e:for(;++r<i;){var c=e[r],f=t?t(c):c;if(c=n||0!==c?c:0,o&&f==f){for(var p=l.length;p--;)if(l[p]===f)continue e;t&&l.push(f),u.push(c)}else a(l,f,n)||(l!==u&&l.push(f),u.push(c))}return u}function la(e,t){return null==(e=Si(e,t=ga(t,e)))||delete e[Fi(Yi(t))]}function sa(e,t,n,r){return Kr(e,t,n(wr(e,t)),r)}function ca(e,t,n,r){for(var a=e.length,i=r?a:-1;(r?i--:++i<a)&&t(e[i],i,e););return n?ea(e,r?0:i,r?i+1:a):ea(e,r?i+1:0,r?a:i)}function fa(e,t){var n=e;return n instanceof Bn&&(n=n.value()),Ot(t,(function(e,t){return t.func.apply(t.thisArg,Lt([e],t.args))}),n)}function pa(e,t,n){var a=e.length;if(a<2)return a?ua(e[0]):[];for(var i=-1,o=r(a);++i<a;)for(var u=e[i],l=-1;++l<a;)l!=i&&(o[i]=sr(o[i]||u,e[l],t,n));return ua(vr(o,1),t,n)}function da(e,t,n){for(var r=-1,i=e.length,o=t.length,u={};++r<i;){var l=r<o?t[r]:a;n(u,e[r],l)}return u}function ha(e){return qo(e)?e:[]}function va(e){return\"function\"==typeof e?e:nl}function ga(e,t){return Wo(e)?e:yi(e,t)?[e]:Ai(yu(e))}var ya=Qr;function ma(e,t,n){var r=e.length;return n=n===a?r:n,!t&&n>=r?e:ea(e,t,n)}var ba=ot||function(e){return ft.clearTimeout(e)};function _a(e,t){if(t)return e.slice();var n=e.length,r=We?We(n):new e.constructor(n);return e.copy(r),r}function wa(e){var t=new e.constructor(e.byteLength);return new Be(t).set(new Be(e)),t}function xa(e,t){var n=t?wa(e.buffer):e.buffer;return new e.constructor(n,e.byteOffset,e.length)}function ka(e,t){if(e!==t){var n=e!==a,r=null===e,i=e==e,o=uu(e),u=t!==a,l=null===t,s=t==t,c=uu(t);if(!l&&!c&&!o&&e>t||o&&u&&s&&!l&&!c||r&&u&&s||!n&&s||!i)return 1;if(!r&&!o&&!c&&e<t||c&&n&&i&&!r&&!o||l&&n&&i||!u&&i||!s)return-1}return 0}function Sa(e,t,n,a){for(var i=-1,o=e.length,u=n.length,l=-1,s=t.length,c=gn(o-u,0),f=r(s+c),p=!a;++l<s;)f[l]=t[l];for(;++i<u;)(p||i<o)&&(f[n[i]]=e[i]);for(;c--;)f[l++]=e[i++];return f}function Ea(e,t,n,a){for(var i=-1,o=e.length,u=-1,l=n.length,s=-1,c=t.length,f=gn(o-l,0),p=r(f+c),d=!a;++i<f;)p[i]=e[i];for(var h=i;++s<c;)p[h+s]=t[s];for(;++u<l;)(d||i<o)&&(p[h+n[u]]=e[i++]);return p}function Ca(e,t){var n=-1,a=e.length;for(t||(t=r(a));++n<a;)t[n]=e[n];return t}function Ta(e,t,n,r){var i=!n;n||(n={});for(var o=-1,u=t.length;++o<u;){var l=t[o],s=r?r(n[l],e[l],l,n,e):a;s===a&&(s=e[l]),i?rr(n,l,s):Jn(n,l,s)}return n}function Ma(e,t){return function(n,r){var a=Wo(n)?St:tr,i=t?t():{};return a(n,e,ii(r,2),i)}}function Na(e){return Qr((function(t,n){var r=-1,i=n.length,o=i>1?n[i-1]:a,u=i>2?n[2]:a;for(o=e.length>3&&\"function\"==typeof o?(i--,o):a,u&&gi(n[0],n[1],u)&&(o=i<3?a:o,i=1),t=Se(t);++r<i;){var l=n[r];l&&e(t,l,r,o)}return t}))}function Pa(e,t){return function(n,r){if(null==n)return n;if(!Ho(n))return e(n,r);for(var a=n.length,i=t?a:-1,o=Se(n);(t?i--:++i<a)&&!1!==r(o[i],i,o););return n}}function za(e){return function(t,n,r){for(var a=-1,i=Se(t),o=r(t),u=o.length;u--;){var l=o[e?u:++a];if(!1===n(i[l],l,i))break}return t}}function La(e){return function(t){var n=rn(t=yu(t))?fn(t):a,r=n?n[0]:t.charAt(0),i=n?ma(n,1).join(\"\"):t.slice(1);return r[e]()+i}}function Oa(e){return function(t){return Ot(Ku(Bu(t).replace(Ke,\"\")),e,\"\")}}function Aa(e){return function(){var t=arguments;switch(t.length){case 0:return new e;case 1:return new e(t[0]);case 2:return new e(t[0],t[1]);case 3:return new e(t[0],t[1],t[2]);case 4:return new e(t[0],t[1],t[2],t[3]);case 5:return new e(t[0],t[1],t[2],t[3],t[4]);case 6:return new e(t[0],t[1],t[2],t[3],t[4],t[5]);case 7:return new e(t[0],t[1],t[2],t[3],t[4],t[5],t[6])}var n=Un(e.prototype),r=e.apply(n,t);return Jo(r)?r:n}}function Fa(e){return function(t,n,r){var i=Se(t);if(!Ho(t)){var o=ii(n,3);t=Pu(t),n=function(e){return o(i[e],e,i)}}var u=e(t,n,r);return u>-1?i[o?t[u]:u]:a}}function Da(e){return Ja((function(t){var n=t.length,r=n,o=$n.prototype.thru;for(e&&t.reverse();r--;){var u=t[r];if(\"function\"!=typeof u)throw new Te(i);if(o&&!l&&\"wrapper\"==ri(u))var l=new $n([],!0)}for(r=l?r:n;++r<n;){var s=ri(u=t[r]),c=\"wrapper\"==s?ni(u):a;l=c&&mi(c[0])&&424==c[1]&&!c[4].length&&1==c[9]?l[ri(c[0])].apply(l,c[3]):1==u.length&&mi(u)?l[s]():l.thru(u)}return function(){var e=arguments,r=e[0];if(l&&1==e.length&&Wo(r))return l.plant(r).value();for(var a=0,i=n?t[a].apply(this,e):r;++a<n;)i=t[a].call(this,i);return i}}))}function Ra(e,t,n,i,o,u,l,c,f,p){var d=t&s,h=1&t,v=2&t,g=24&t,y=512&t,m=v?a:Aa(e);return function s(){for(var b=arguments.length,_=r(b),w=b;w--;)_[w]=arguments[w];if(g)var x=ai(s),k=function(e,t){for(var n=e.length,r=0;n--;)e[n]===t&&++r;return r}(_,x);if(i&&(_=Sa(_,i,o,g)),u&&(_=Ea(_,u,l,g)),b-=k,g&&b<p){var S=un(_,x);return Va(e,t,Ra,s.placeholder,n,_,S,c,f,p-b)}var E=h?n:this,C=v?E[e]:e;return b=_.length,c?_=function(e,t){for(var n=e.length,r=yn(t.length,n),i=Ca(e);r--;){var o=t[r];e[r]=vi(o,n)?i[o]:a}return e}(_,c):y&&b>1&&_.reverse(),d&&f<b&&(_.length=f),this&&this!==ft&&this instanceof s&&(C=m||Aa(C)),C.apply(E,_)}}function ja(e,t){return function(n,r){return function(e,t,n,r){return mr(e,(function(e,a,i){t(r,n(e),a,i)})),r}(n,e,t(r),{})}}function Ua(e,t){return function(n,r){var i;if(n===a&&r===a)return t;if(n!==a&&(i=n),r!==a){if(i===a)return r;\"string\"==typeof n||\"string\"==typeof r?(n=oa(n),r=oa(r)):(n=ia(n),r=ia(r)),i=e(n,r)}return i}}function Ia(e){return Ja((function(t){return t=zt(t,Gt(ii())),Qr((function(n){var r=this;return e(t,(function(e){return kt(e,r,n)}))}))}))}function $a(e,t){var n=(t=t===a?\" \":oa(t)).length;if(n<2)return n?qr(t,e):t;var r=qr(t,pt(e/cn(t)));return rn(t)?ma(fn(r),0,e).join(\"\"):r.slice(0,e)}function Ba(e){return function(t,n,i){return i&&\"number\"!=typeof i&&gi(t,n,i)&&(n=i=a),t=pu(t),n===a?(n=t,t=0):n=pu(n),function(e,t,n,a){for(var i=-1,o=gn(pt((t-e)/(n||1)),0),u=r(o);o--;)u[a?o:++i]=e,e+=n;return u}(t,n,i=i===a?t<n?1:-1:pu(i),e)}}function Wa(e){return function(t,n){return\"string\"==typeof t&&\"string\"==typeof n||(t=vu(t),n=vu(n)),e(t,n)}}function Va(e,t,n,r,i,o,u,s,c,f){var p=8&t;t|=p?l:64,4&(t&=~(p?64:l))||(t&=-4);var d=[e,t,i,p?o:a,p?u:a,p?a:o,p?a:u,s,c,f],h=n.apply(a,d);return mi(e)&&Ci(h,d),h.placeholder=r,Ni(h,e,t)}function Ha(e){var t=ke[e];return function(e,n){if(e=vu(e),(n=null==n?0:yn(du(n),292))&&Dt(e)){var r=(yu(e)+\"e\").split(\"e\");return+((r=(yu(t(r[0]+\"e\"+(+r[1]+n)))+\"e\").split(\"e\"))[0]+\"e\"+(+r[1]-n))}return t(e)}}var qa=En&&1/ln(new En([,-0]))[1]==c?function(e){return new En(e)}:ul;function Qa(e){return function(t){var n=fi(t);return n==x?an(t):n==T?sn(t):function(e,t){return zt(t,(function(t){return[t,e[t]]}))}(t,e(t))}}function Ya(e,t,n,o,c,f,p,d){var h=2&t;if(!h&&\"function\"!=typeof e)throw new Te(i);var v=o?o.length:0;if(v||(t&=-97,o=c=a),p=p===a?p:gn(du(p),0),d=d===a?d:du(d),v-=c?c.length:0,64&t){var g=o,y=c;o=c=a}var m=h?a:ni(e),b=[e,t,n,o,c,g,y,f,p,d];if(m&&function(e,t){var n=e[1],r=t[1],a=n|r,i=a<131,o=r==s&&8==n||r==s&&256==n&&e[7].length<=t[8]||384==r&&t[7].length<=t[8]&&8==n;if(!i&&!o)return e;1&r&&(e[2]=t[2],a|=1&n?0:4);var l=t[3];if(l){var c=e[3];e[3]=c?Sa(c,l,t[4]):l,e[4]=c?un(e[3],u):t[4]}(l=t[5])&&(c=e[5],e[5]=c?Ea(c,l,t[6]):l,e[6]=c?un(e[5],u):t[6]),(l=t[7])&&(e[7]=l),r&s&&(e[8]=null==e[8]?t[8]:yn(e[8],t[8])),null==e[9]&&(e[9]=t[9]),e[0]=t[0],e[1]=a}(b,m),e=b[0],t=b[1],n=b[2],o=b[3],c=b[4],!(d=b[9]=b[9]===a?h?0:e.length:gn(b[9]-v,0))&&24&t&&(t&=-25),t&&1!=t)_=8==t||16==t?function(e,t,n){var i=Aa(e);return function o(){for(var u=arguments.length,l=r(u),s=u,c=ai(o);s--;)l[s]=arguments[s];var f=u<3&&l[0]!==c&&l[u-1]!==c?[]:un(l,c);return(u-=f.length)<n?Va(e,t,Ra,o.placeholder,a,l,f,a,a,n-u):kt(this&&this!==ft&&this instanceof o?i:e,this,l)}}(e,t,d):t!=l&&33!=t||c.length?Ra.apply(a,b):function(e,t,n,a){var i=1&t,o=Aa(e);return function t(){for(var u=-1,l=arguments.length,s=-1,c=a.length,f=r(c+l),p=this&&this!==ft&&this instanceof t?o:e;++s<c;)f[s]=a[s];for(;l--;)f[s++]=arguments[++u];return kt(p,i?n:this,f)}}(e,t,n,o);else var _=function(e,t,n){var r=1&t,a=Aa(e);return function t(){return(this&&this!==ft&&this instanceof t?a:e).apply(r?n:this,arguments)}}(e,t,n);return Ni((m?Zr:Ci)(_,b),e,t)}function Ga(e,t,n,r){return e===a||Uo(e,Pe[n])&&!Oe.call(r,n)?t:e}function Ka(e,t,n,r,i,o){return Jo(e)&&Jo(t)&&(o.set(t,e),Ur(e,t,a,Ka,o),o.delete(t)),e}function Za(e){return ru(e)?a:e}function Xa(e,t,n,r,i,o){var u=1&n,l=e.length,s=t.length;if(l!=s&&!(u&&s>l))return!1;var c=o.get(e),f=o.get(t);if(c&&f)return c==t&&f==e;var p=-1,d=!0,h=2&n?new qn:a;for(o.set(e,t),o.set(t,e);++p<l;){var v=e[p],g=t[p];if(r)var y=u?r(g,v,p,t,e,o):r(v,g,p,e,t,o);if(y!==a){if(y)continue;d=!1;break}if(h){if(!Ft(t,(function(e,t){if(!Zt(h,t)&&(v===e||i(v,e,n,r,o)))return h.push(t)}))){d=!1;break}}else if(v!==g&&!i(v,g,n,r,o)){d=!1;break}}return o.delete(e),o.delete(t),d}function Ja(e){return Mi(ki(e,a,Wi),e+\"\")}function ei(e){return xr(e,Pu,si)}function ti(e){return xr(e,zu,ci)}var ni=Mn?function(e){return Mn.get(e)}:ul;function ri(e){for(var t=e.name+\"\",n=Nn[t],r=Oe.call(Nn,t)?n.length:0;r--;){var a=n[r],i=a.func;if(null==i||i==e)return a.name}return t}function ai(e){return(Oe.call(jn,\"placeholder\")?jn:e).placeholder}function ii(){var e=jn.iteratee||rl;return e=e===rl?Or:e,arguments.length?e(arguments[0],arguments[1]):e}function oi(e,t){var n,r,a=e.__data__;return(\"string\"==(r=typeof(n=t))||\"number\"==r||\"symbol\"==r||\"boolean\"==r?\"__proto__\"!==n:null===n)?a[\"string\"==typeof t?\"string\":\"hash\"]:a.map}function ui(e){for(var t=Pu(e),n=t.length;n--;){var r=t[n],a=e[r];t[n]=[r,a,wi(a)]}return t}function li(e,t){var n=function(e,t){return null==e?a:e[t]}(e,t);return Lr(n)?n:a}var si=vt?function(e){return null==e?[]:(e=Se(e),Mt(vt(e),(function(t){return qe.call(e,t)})))}:hl,ci=vt?function(e){for(var t=[];e;)Lt(t,si(e)),e=Ve(e);return t}:hl,fi=kr;function pi(e,t,n){for(var r=-1,a=(t=ga(t,e)).length,i=!1;++r<a;){var o=Fi(t[r]);if(!(i=null!=e&&n(e,o)))break;e=e[o]}return i||++r!=a?i:!!(a=null==e?0:e.length)&&Xo(a)&&vi(o,a)&&(Wo(e)||Bo(e))}function di(e){return\"function\"!=typeof e.constructor||_i(e)?{}:Un(Ve(e))}function hi(e){return Wo(e)||Bo(e)||!!(Ye&&e&&e[Ye])}function vi(e,t){var n=typeof e;return!!(t=null==t?f:t)&&(\"number\"==n||\"symbol\"!=n&&ye.test(e))&&e>-1&&e%1==0&&e<t}function gi(e,t,n){if(!Jo(n))return!1;var r=typeof t;return!!(\"number\"==r?Ho(n)&&vi(t,n.length):\"string\"==r&&t in n)&&Uo(n[t],e)}function yi(e,t){if(Wo(e))return!1;var n=typeof e;return!(\"number\"!=n&&\"symbol\"!=n&&\"boolean\"!=n&&null!=e&&!uu(e))||J.test(e)||!X.test(e)||null!=t&&e in Se(t)}function mi(e){var t=ri(e),n=jn[t];if(\"function\"!=typeof n||!(t in Bn.prototype))return!1;if(e===n)return!0;var r=ni(n);return!!r&&e===r[0]}(xn&&fi(new xn(new ArrayBuffer(1)))!=L||kn&&fi(new kn)!=x||Sn&&fi(Sn.resolve())!=E||En&&fi(new En)!=T||Cn&&fi(new Cn)!=P)&&(fi=function(e){var t=kr(e),n=t==S?e.constructor:a,r=n?Di(n):\"\";if(r)switch(r){case Pn:return L;case zn:return x;case Ln:return E;case On:return T;case An:return P}return t});var bi=ze?Ko:vl;function _i(e){var t=e&&e.constructor;return e===(\"function\"==typeof t&&t.prototype||Pe)}function wi(e){return e==e&&!Jo(e)}function xi(e,t){return function(n){return null!=n&&n[e]===t&&(t!==a||e in Se(n))}}function ki(e,t,n){return t=gn(t===a?e.length-1:t,0),function(){for(var a=arguments,i=-1,o=gn(a.length-t,0),u=r(o);++i<o;)u[i]=a[t+i];i=-1;for(var l=r(t+1);++i<t;)l[i]=a[i];return l[t]=n(u),kt(e,this,l)}}function Si(e,t){return t.length<2?e:wr(e,ea(t,0,-1))}function Ei(e,t){if((\"constructor\"!==t||\"function\"!=typeof e[t])&&\"__proto__\"!=t)return e[t]}var Ci=Pi(Zr),Ti=ct||function(e,t){return ft.setTimeout(e,t)},Mi=Pi(Xr);function Ni(e,t,n){var r=t+\"\";return Mi(e,function(e,t){var n=t.length;if(!n)return e;var r=n-1;return t[r]=(n>1?\"& \":\"\")+t[r],t=t.join(n>2?\", \":\" \"),e.replace(ie,\"{\\n/* [wrapped with \"+t+\"] */\\n\")}(r,function(e,t){return Et(h,(function(n){var r=\"_.\"+n[0];t&n[1]&&!Nt(e,r)&&e.push(r)})),e.sort()}(function(e){var t=e.match(oe);return t?t[1].split(ue):[]}(r),n)))}function Pi(e){var t=0,n=0;return function(){var r=mn(),i=16-(r-n);if(n=r,i>0){if(++t>=800)return arguments[0]}else t=0;return e.apply(a,arguments)}}function zi(e,t){var n=-1,r=e.length,i=r-1;for(t=t===a?r:t;++n<t;){var o=Hr(n,i),u=e[o];e[o]=e[n],e[n]=u}return e.length=t,e}var Li,Oi,Ai=(Li=Oo((function(e){var t=[];return 46===e.charCodeAt(0)&&t.push(\"\"),e.replace(ee,(function(e,n,r,a){t.push(r?a.replace(ce,\"$1\"):n||e)})),t}),(function(e){return 500===Oi.size&&Oi.clear(),e})),Oi=Li.cache,Li);function Fi(e){if(\"string\"==typeof e||uu(e))return e;var t=e+\"\";return\"0\"==t&&1/e==-1/0?\"-0\":t}function Di(e){if(null!=e){try{return Le.call(e)}catch(e){}try{return e+\"\"}catch(e){}}return\"\"}function Ri(e){if(e instanceof Bn)return e.clone();var t=new $n(e.__wrapped__,e.__chain__);return t.__actions__=Ca(e.__actions__),t.__index__=e.__index__,t.__values__=e.__values__,t}var ji=Qr((function(e,t){return qo(e)?sr(e,vr(t,1,qo,!0)):[]})),Ui=Qr((function(e,t){var n=Yi(t);return qo(n)&&(n=a),qo(e)?sr(e,vr(t,1,qo,!0),ii(n,2)):[]})),Ii=Qr((function(e,t){var n=Yi(t);return qo(n)&&(n=a),qo(e)?sr(e,vr(t,1,qo,!0),a,n):[]}));function $i(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var a=null==n?0:du(n);return a<0&&(a=gn(r+a,0)),jt(e,ii(t,3),a)}function Bi(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var i=r-1;return n!==a&&(i=du(n),i=n<0?gn(r+i,0):yn(i,r-1)),jt(e,ii(t,3),i,!0)}function Wi(e){return null!=e&&e.length?vr(e,1):[]}function Vi(e){return e&&e.length?e[0]:a}var Hi=Qr((function(e){var t=zt(e,ha);return t.length&&t[0]===e[0]?Tr(t):[]})),qi=Qr((function(e){var t=Yi(e),n=zt(e,ha);return t===Yi(n)?t=a:n.pop(),n.length&&n[0]===e[0]?Tr(n,ii(t,2)):[]})),Qi=Qr((function(e){var t=Yi(e),n=zt(e,ha);return(t=\"function\"==typeof t?t:a)&&n.pop(),n.length&&n[0]===e[0]?Tr(n,a,t):[]}));function Yi(e){var t=null==e?0:e.length;return t?e[t-1]:a}var Gi=Qr(Ki);function Ki(e,t){return e&&e.length&&t&&t.length?Wr(e,t):e}var Zi=Ja((function(e,t){var n=null==e?0:e.length,r=ar(e,t);return Vr(e,zt(t,(function(e){return vi(e,n)?+e:e})).sort(ka)),r}));function Xi(e){return null==e?e:wn.call(e)}var Ji=Qr((function(e){return ua(vr(e,1,qo,!0))})),eo=Qr((function(e){var t=Yi(e);return qo(t)&&(t=a),ua(vr(e,1,qo,!0),ii(t,2))})),to=Qr((function(e){var t=Yi(e);return t=\"function\"==typeof t?t:a,ua(vr(e,1,qo,!0),a,t)}));function no(e){if(!e||!e.length)return[];var t=0;return e=Mt(e,(function(e){if(qo(e))return t=gn(e.length,t),!0})),Qt(t,(function(t){return zt(e,Wt(t))}))}function ro(e,t){if(!e||!e.length)return[];var n=no(e);return null==t?n:zt(n,(function(e){return kt(t,a,e)}))}var ao=Qr((function(e,t){return qo(e)?sr(e,t):[]})),io=Qr((function(e){return pa(Mt(e,qo))})),oo=Qr((function(e){var t=Yi(e);return qo(t)&&(t=a),pa(Mt(e,qo),ii(t,2))})),uo=Qr((function(e){var t=Yi(e);return t=\"function\"==typeof t?t:a,pa(Mt(e,qo),a,t)})),lo=Qr(no),so=Qr((function(e){var t=e.length,n=t>1?e[t-1]:a;return n=\"function\"==typeof n?(e.pop(),n):a,ro(e,n)}));function co(e){var t=jn(e);return t.__chain__=!0,t}function fo(e,t){return t(e)}var po=Ja((function(e){var t=e.length,n=t?e[0]:0,r=this.__wrapped__,i=function(t){return ar(t,e)};return!(t>1||this.__actions__.length)&&r instanceof Bn&&vi(n)?((r=r.slice(n,+n+(t?1:0))).__actions__.push({func:fo,args:[i],thisArg:a}),new $n(r,this.__chain__).thru((function(e){return t&&!e.length&&e.push(a),e}))):this.thru(i)})),ho=Ma((function(e,t,n){Oe.call(e,n)?++e[n]:rr(e,n,1)})),vo=Fa($i),go=Fa(Bi);function yo(e,t){return(Wo(e)?Et:cr)(e,ii(t,3))}function mo(e,t){return(Wo(e)?Ct:fr)(e,ii(t,3))}var bo=Ma((function(e,t,n){Oe.call(e,n)?e[n].push(t):rr(e,n,[t])})),_o=Qr((function(e,t,n){var a=-1,i=\"function\"==typeof t,o=Ho(e)?r(e.length):[];return cr(e,(function(e){o[++a]=i?kt(t,e,n):Mr(e,t,n)})),o})),wo=Ma((function(e,t,n){rr(e,n,t)}));function xo(e,t){return(Wo(e)?zt:Dr)(e,ii(t,3))}var ko=Ma((function(e,t,n){e[n?0:1].push(t)}),(function(){return[[],[]]})),So=Qr((function(e,t){if(null==e)return[];var n=t.length;return n>1&&gi(e,t[0],t[1])?t=[]:n>2&&gi(t[0],t[1],t[2])&&(t=[t[0]]),$r(e,vr(t,1),[])})),Eo=st||function(){return ft.Date.now()};function Co(e,t,n){return t=n?a:t,t=e&&null==t?e.length:t,Ya(e,s,a,a,a,a,t)}function To(e,t){var n;if(\"function\"!=typeof t)throw new Te(i);return e=du(e),function(){return--e>0&&(n=t.apply(this,arguments)),e<=1&&(t=a),n}}var Mo=Qr((function(e,t,n){var r=1;if(n.length){var a=un(n,ai(Mo));r|=l}return Ya(e,r,t,n,a)})),No=Qr((function(e,t,n){var r=3;if(n.length){var a=un(n,ai(No));r|=l}return Ya(t,r,e,n,a)}));function Po(e,t,n){var r,o,u,l,s,c,f=0,p=!1,d=!1,h=!0;if(\"function\"!=typeof e)throw new Te(i);function v(t){var n=r,i=o;return r=o=a,f=t,l=e.apply(i,n)}function g(e){var n=e-c;return c===a||n>=t||n<0||d&&e-f>=u}function y(){var e=Eo();if(g(e))return m(e);s=Ti(y,function(e){var n=t-(e-c);return d?yn(n,u-(e-f)):n}(e))}function m(e){return s=a,h&&r?v(e):(r=o=a,l)}function b(){var e=Eo(),n=g(e);if(r=arguments,o=this,c=e,n){if(s===a)return function(e){return f=e,s=Ti(y,t),p?v(e):l}(c);if(d)return ba(s),s=Ti(y,t),v(c)}return s===a&&(s=Ti(y,t)),l}return t=vu(t)||0,Jo(n)&&(p=!!n.leading,u=(d=\"maxWait\"in n)?gn(vu(n.maxWait)||0,t):u,h=\"trailing\"in n?!!n.trailing:h),b.cancel=function(){s!==a&&ba(s),f=0,r=c=o=s=a},b.flush=function(){return s===a?l:m(Eo())},b}var zo=Qr((function(e,t){return lr(e,1,t)})),Lo=Qr((function(e,t,n){return lr(e,vu(t)||0,n)}));function Oo(e,t){if(\"function\"!=typeof e||null!=t&&\"function\"!=typeof t)throw new Te(i);var n=function(){var r=arguments,a=t?t.apply(this,r):r[0],i=n.cache;if(i.has(a))return i.get(a);var o=e.apply(this,r);return n.cache=i.set(a,o)||i,o};return n.cache=new(Oo.Cache||Hn),n}function Ao(e){if(\"function\"!=typeof e)throw new Te(i);return function(){var t=arguments;switch(t.length){case 0:return!e.call(this);case 1:return!e.call(this,t[0]);case 2:return!e.call(this,t[0],t[1]);case 3:return!e.call(this,t[0],t[1],t[2])}return!e.apply(this,t)}}Oo.Cache=Hn;var Fo=ya((function(e,t){var n=(t=1==t.length&&Wo(t[0])?zt(t[0],Gt(ii())):zt(vr(t,1),Gt(ii()))).length;return Qr((function(r){for(var a=-1,i=yn(r.length,n);++a<i;)r[a]=t[a].call(this,r[a]);return kt(e,this,r)}))})),Do=Qr((function(e,t){var n=un(t,ai(Do));return Ya(e,l,a,t,n)})),Ro=Qr((function(e,t){var n=un(t,ai(Ro));return Ya(e,64,a,t,n)})),jo=Ja((function(e,t){return Ya(e,256,a,a,a,t)}));function Uo(e,t){return e===t||e!=e&&t!=t}var Io=Wa(Sr),$o=Wa((function(e,t){return e>=t})),Bo=Nr(function(){return arguments}())?Nr:function(e){return eu(e)&&Oe.call(e,\"callee\")&&!qe.call(e,\"callee\")},Wo=r.isArray,Vo=yt?Gt(yt):function(e){return eu(e)&&kr(e)==z};function Ho(e){return null!=e&&Xo(e.length)&&!Ko(e)}function qo(e){return eu(e)&&Ho(e)}var Qo=gt||vl,Yo=mt?Gt(mt):function(e){return eu(e)&&kr(e)==m};function Go(e){if(!eu(e))return!1;var t=kr(e);return t==b||\"[object DOMException]\"==t||\"string\"==typeof e.message&&\"string\"==typeof e.name&&!ru(e)}function Ko(e){if(!Jo(e))return!1;var t=kr(e);return t==_||t==w||\"[object AsyncFunction]\"==t||\"[object Proxy]\"==t}function Zo(e){return\"number\"==typeof e&&e==du(e)}function Xo(e){return\"number\"==typeof e&&e>-1&&e%1==0&&e<=f}function Jo(e){var t=typeof e;return null!=e&&(\"object\"==t||\"function\"==t)}function eu(e){return null!=e&&\"object\"==typeof e}var tu=bt?Gt(bt):function(e){return eu(e)&&fi(e)==x};function nu(e){return\"number\"==typeof e||eu(e)&&kr(e)==k}function ru(e){if(!eu(e)||kr(e)!=S)return!1;var t=Ve(e);if(null===t)return!0;var n=Oe.call(t,\"constructor\")&&t.constructor;return\"function\"==typeof n&&n instanceof n&&Le.call(n)==Re}var au=_t?Gt(_t):function(e){return eu(e)&&kr(e)==C},iu=wt?Gt(wt):function(e){return eu(e)&&fi(e)==T};function ou(e){return\"string\"==typeof e||!Wo(e)&&eu(e)&&kr(e)==M}function uu(e){return\"symbol\"==typeof e||eu(e)&&kr(e)==N}var lu=xt?Gt(xt):function(e){return eu(e)&&Xo(e.length)&&!!at[kr(e)]},su=Wa(Fr),cu=Wa((function(e,t){return e<=t}));function fu(e){if(!e)return[];if(Ho(e))return ou(e)?fn(e):Ca(e);if(Ge&&e[Ge])return function(e){for(var t,n=[];!(t=e.next()).done;)n.push(t.value);return n}(e[Ge]());var t=fi(e);return(t==x?an:t==T?ln:Uu)(e)}function pu(e){return e?(e=vu(e))===c||e===-1/0?17976931348623157e292*(e<0?-1:1):e==e?e:0:0===e?e:0}function du(e){var t=pu(e),n=t%1;return t==t?n?t-n:t:0}function hu(e){return e?ir(du(e),0,d):0}function vu(e){if(\"number\"==typeof e)return e;if(uu(e))return p;if(Jo(e)){var t=\"function\"==typeof e.valueOf?e.valueOf():e;e=Jo(t)?t+\"\":t}if(\"string\"!=typeof e)return 0===e?e:+e;e=Yt(e);var n=he.test(e);return n||ge.test(e)?lt(e.slice(2),n?2:8):de.test(e)?p:+e}function gu(e){return Ta(e,zu(e))}function yu(e){return null==e?\"\":oa(e)}var mu=Na((function(e,t){if(_i(t)||Ho(t))Ta(t,Pu(t),e);else for(var n in t)Oe.call(t,n)&&Jn(e,n,t[n])})),bu=Na((function(e,t){Ta(t,zu(t),e)})),_u=Na((function(e,t,n,r){Ta(t,zu(t),e,r)})),wu=Na((function(e,t,n,r){Ta(t,Pu(t),e,r)})),xu=Ja(ar),ku=Qr((function(e,t){e=Se(e);var n=-1,r=t.length,i=r>2?t[2]:a;for(i&&gi(t[0],t[1],i)&&(r=1);++n<r;)for(var o=t[n],u=zu(o),l=-1,s=u.length;++l<s;){var c=u[l],f=e[c];(f===a||Uo(f,Pe[c])&&!Oe.call(e,c))&&(e[c]=o[c])}return e})),Su=Qr((function(e){return e.push(a,Ka),kt(Ou,a,e)}));function Eu(e,t,n){var r=null==e?a:wr(e,t);return r===a?n:r}function Cu(e,t){return null!=e&&pi(e,t,Cr)}var Tu=ja((function(e,t,n){null!=t&&\"function\"!=typeof t.toString&&(t=De.call(t)),e[t]=n}),Ju(nl)),Mu=ja((function(e,t,n){null!=t&&\"function\"!=typeof t.toString&&(t=De.call(t)),Oe.call(e,t)?e[t].push(n):e[t]=[n]}),ii),Nu=Qr(Mr);function Pu(e){return Ho(e)?Yn(e):Ar(e)}function zu(e){return Ho(e)?Yn(e,!0):function(e){if(!Jo(e))return function(e){var t=[];if(null!=e)for(var n in Se(e))t.push(n);return t}(e);var t=_i(e),n=[];for(var r in e)(\"constructor\"!=r||!t&&Oe.call(e,r))&&n.push(r);return n}(e)}var Lu=Na((function(e,t,n){Ur(e,t,n)})),Ou=Na((function(e,t,n,r){Ur(e,t,n,r)})),Au=Ja((function(e,t){var n={};if(null==e)return n;var r=!1;t=zt(t,(function(t){return t=ga(t,e),r||(r=t.length>1),t})),Ta(e,ti(e),n),r&&(n=or(n,7,Za));for(var a=t.length;a--;)la(n,t[a]);return n})),Fu=Ja((function(e,t){return null==e?{}:function(e,t){return Br(e,t,(function(t,n){return Cu(e,n)}))}(e,t)}));function Du(e,t){if(null==e)return{};var n=zt(ti(e),(function(e){return[e]}));return t=ii(t),Br(e,n,(function(e,n){return t(e,n[0])}))}var Ru=Qa(Pu),ju=Qa(zu);function Uu(e){return null==e?[]:Kt(e,Pu(e))}var Iu=Oa((function(e,t,n){return t=t.toLowerCase(),e+(n?$u(t):t)}));function $u(e){return Gu(yu(e).toLowerCase())}function Bu(e){return(e=yu(e))&&e.replace(me,en).replace(Ze,\"\")}var Wu=Oa((function(e,t,n){return e+(n?\"-\":\"\")+t.toLowerCase()})),Vu=Oa((function(e,t,n){return e+(n?\" \":\"\")+t.toLowerCase()})),Hu=La(\"toLowerCase\"),qu=Oa((function(e,t,n){return e+(n?\"_\":\"\")+t.toLowerCase()})),Qu=Oa((function(e,t,n){return e+(n?\" \":\"\")+Gu(t)})),Yu=Oa((function(e,t,n){return e+(n?\" \":\"\")+t.toUpperCase()})),Gu=La(\"toUpperCase\");function Ku(e,t,n){return e=yu(e),(t=n?a:t)===a?function(e){return tt.test(e)}(e)?function(e){return e.match(Je)||[]}(e):function(e){return e.match(le)||[]}(e):e.match(t)||[]}var Zu=Qr((function(e,t){try{return kt(e,a,t)}catch(e){return Go(e)?e:new we(e)}})),Xu=Ja((function(e,t){return Et(t,(function(t){t=Fi(t),rr(e,t,Mo(e[t],e))})),e}));function Ju(e){return function(){return e}}var el=Da(),tl=Da(!0);function nl(e){return e}function rl(e){return Or(\"function\"==typeof e?e:or(e,1))}var al=Qr((function(e,t){return function(n){return Mr(n,e,t)}})),il=Qr((function(e,t){return function(n){return Mr(e,n,t)}}));function ol(e,t,n){var r=Pu(t),a=_r(t,r);null!=n||Jo(t)&&(a.length||!r.length)||(n=t,t=e,e=this,a=_r(t,Pu(t)));var i=!(Jo(n)&&\"chain\"in n&&!n.chain),o=Ko(e);return Et(a,(function(n){var r=t[n];e[n]=r,o&&(e.prototype[n]=function(){var t=this.__chain__;if(i||t){var n=e(this.__wrapped__);return(n.__actions__=Ca(this.__actions__)).push({func:r,args:arguments,thisArg:e}),n.__chain__=t,n}return r.apply(e,Lt([this.value()],arguments))})})),e}function ul(){}var ll=Ia(zt),sl=Ia(Tt),cl=Ia(Ft);function fl(e){return yi(e)?Wt(Fi(e)):function(e){return function(t){return wr(t,e)}}(e)}var pl=Ba(),dl=Ba(!0);function hl(){return[]}function vl(){return!1}var gl,yl=Ua((function(e,t){return e+t}),0),ml=Ha(\"ceil\"),bl=Ua((function(e,t){return e/t}),1),_l=Ha(\"floor\"),wl=Ua((function(e,t){return e*t}),1),xl=Ha(\"round\"),kl=Ua((function(e,t){return e-t}),0);return jn.after=function(e,t){if(\"function\"!=typeof t)throw new Te(i);return e=du(e),function(){if(--e<1)return t.apply(this,arguments)}},jn.ary=Co,jn.assign=mu,jn.assignIn=bu,jn.assignInWith=_u,jn.assignWith=wu,jn.at=xu,jn.before=To,jn.bind=Mo,jn.bindAll=Xu,jn.bindKey=No,jn.castArray=function(){if(!arguments.length)return[];var e=arguments[0];return Wo(e)?e:[e]},jn.chain=co,jn.chunk=function(e,t,n){t=(n?gi(e,t,n):t===a)?1:gn(du(t),0);var i=null==e?0:e.length;if(!i||t<1)return[];for(var o=0,u=0,l=r(pt(i/t));o<i;)l[u++]=ea(e,o,o+=t);return l},jn.compact=function(e){for(var t=-1,n=null==e?0:e.length,r=0,a=[];++t<n;){var i=e[t];i&&(a[r++]=i)}return a},jn.concat=function(){var e=arguments.length;if(!e)return[];for(var t=r(e-1),n=arguments[0],a=e;a--;)t[a-1]=arguments[a];return Lt(Wo(n)?Ca(n):[n],vr(t,1))},jn.cond=function(e){var t=null==e?0:e.length,n=ii();return e=t?zt(e,(function(e){if(\"function\"!=typeof e[1])throw new Te(i);return[n(e[0]),e[1]]})):[],Qr((function(n){for(var r=-1;++r<t;){var a=e[r];if(kt(a[0],this,n))return kt(a[1],this,n)}}))},jn.conforms=function(e){return function(e){var t=Pu(e);return function(n){return ur(n,e,t)}}(or(e,1))},jn.constant=Ju,jn.countBy=ho,jn.create=function(e,t){var n=Un(e);return null==t?n:nr(n,t)},jn.curry=function e(t,n,r){var i=Ya(t,8,a,a,a,a,a,n=r?a:n);return i.placeholder=e.placeholder,i},jn.curryRight=function e(t,n,r){var i=Ya(t,16,a,a,a,a,a,n=r?a:n);return i.placeholder=e.placeholder,i},jn.debounce=Po,jn.defaults=ku,jn.defaultsDeep=Su,jn.defer=zo,jn.delay=Lo,jn.difference=ji,jn.differenceBy=Ui,jn.differenceWith=Ii,jn.drop=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,(t=n||t===a?1:du(t))<0?0:t,r):[]},jn.dropRight=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,0,(t=r-(t=n||t===a?1:du(t)))<0?0:t):[]},jn.dropRightWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!0,!0):[]},jn.dropWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!0):[]},jn.fill=function(e,t,n,r){var i=null==e?0:e.length;return i?(n&&\"number\"!=typeof n&&gi(e,t,n)&&(n=0,r=i),function(e,t,n,r){var i=e.length;for((n=du(n))<0&&(n=-n>i?0:i+n),(r=r===a||r>i?i:du(r))<0&&(r+=i),r=n>r?0:hu(r);n<r;)e[n++]=t;return e}(e,t,n,r)):[]},jn.filter=function(e,t){return(Wo(e)?Mt:hr)(e,ii(t,3))},jn.flatMap=function(e,t){return vr(xo(e,t),1)},jn.flatMapDeep=function(e,t){return vr(xo(e,t),c)},jn.flatMapDepth=function(e,t,n){return n=n===a?1:du(n),vr(xo(e,t),n)},jn.flatten=Wi,jn.flattenDeep=function(e){return null!=e&&e.length?vr(e,c):[]},jn.flattenDepth=function(e,t){return null!=e&&e.length?vr(e,t=t===a?1:du(t)):[]},jn.flip=function(e){return Ya(e,512)},jn.flow=el,jn.flowRight=tl,jn.fromPairs=function(e){for(var t=-1,n=null==e?0:e.length,r={};++t<n;){var a=e[t];r[a[0]]=a[1]}return r},jn.functions=function(e){return null==e?[]:_r(e,Pu(e))},jn.functionsIn=function(e){return null==e?[]:_r(e,zu(e))},jn.groupBy=bo,jn.initial=function(e){return null!=e&&e.length?ea(e,0,-1):[]},jn.intersection=Hi,jn.intersectionBy=qi,jn.intersectionWith=Qi,jn.invert=Tu,jn.invertBy=Mu,jn.invokeMap=_o,jn.iteratee=rl,jn.keyBy=wo,jn.keys=Pu,jn.keysIn=zu,jn.map=xo,jn.mapKeys=function(e,t){var n={};return t=ii(t,3),mr(e,(function(e,r,a){rr(n,t(e,r,a),e)})),n},jn.mapValues=function(e,t){var n={};return t=ii(t,3),mr(e,(function(e,r,a){rr(n,r,t(e,r,a))})),n},jn.matches=function(e){return Rr(or(e,1))},jn.matchesProperty=function(e,t){return jr(e,or(t,1))},jn.memoize=Oo,jn.merge=Lu,jn.mergeWith=Ou,jn.method=al,jn.methodOf=il,jn.mixin=ol,jn.negate=Ao,jn.nthArg=function(e){return e=du(e),Qr((function(t){return Ir(t,e)}))},jn.omit=Au,jn.omitBy=function(e,t){return Du(e,Ao(ii(t)))},jn.once=function(e){return To(2,e)},jn.orderBy=function(e,t,n,r){return null==e?[]:(Wo(t)||(t=null==t?[]:[t]),Wo(n=r?a:n)||(n=null==n?[]:[n]),$r(e,t,n))},jn.over=ll,jn.overArgs=Fo,jn.overEvery=sl,jn.overSome=cl,jn.partial=Do,jn.partialRight=Ro,jn.partition=ko,jn.pick=Fu,jn.pickBy=Du,jn.property=fl,jn.propertyOf=function(e){return function(t){return null==e?a:wr(e,t)}},jn.pull=Gi,jn.pullAll=Ki,jn.pullAllBy=function(e,t,n){return e&&e.length&&t&&t.length?Wr(e,t,ii(n,2)):e},jn.pullAllWith=function(e,t,n){return e&&e.length&&t&&t.length?Wr(e,t,a,n):e},jn.pullAt=Zi,jn.range=pl,jn.rangeRight=dl,jn.rearg=jo,jn.reject=function(e,t){return(Wo(e)?Mt:hr)(e,Ao(ii(t,3)))},jn.remove=function(e,t){var n=[];if(!e||!e.length)return n;var r=-1,a=[],i=e.length;for(t=ii(t,3);++r<i;){var o=e[r];t(o,r,e)&&(n.push(o),a.push(r))}return Vr(e,a),n},jn.rest=function(e,t){if(\"function\"!=typeof e)throw new Te(i);return Qr(e,t=t===a?t:du(t))},jn.reverse=Xi,jn.sampleSize=function(e,t,n){return t=(n?gi(e,t,n):t===a)?1:du(t),(Wo(e)?Kn:Gr)(e,t)},jn.set=function(e,t,n){return null==e?e:Kr(e,t,n)},jn.setWith=function(e,t,n,r){return r=\"function\"==typeof r?r:a,null==e?e:Kr(e,t,n,r)},jn.shuffle=function(e){return(Wo(e)?Zn:Jr)(e)},jn.slice=function(e,t,n){var r=null==e?0:e.length;return r?(n&&\"number\"!=typeof n&&gi(e,t,n)?(t=0,n=r):(t=null==t?0:du(t),n=n===a?r:du(n)),ea(e,t,n)):[]},jn.sortBy=So,jn.sortedUniq=function(e){return e&&e.length?aa(e):[]},jn.sortedUniqBy=function(e,t){return e&&e.length?aa(e,ii(t,2)):[]},jn.split=function(e,t,n){return n&&\"number\"!=typeof n&&gi(e,t,n)&&(t=n=a),(n=n===a?d:n>>>0)?(e=yu(e))&&(\"string\"==typeof t||null!=t&&!au(t))&&!(t=oa(t))&&rn(e)?ma(fn(e),0,n):e.split(t,n):[]},jn.spread=function(e,t){if(\"function\"!=typeof e)throw new Te(i);return t=null==t?0:gn(du(t),0),Qr((function(n){var r=n[t],a=ma(n,0,t);return r&&Lt(a,r),kt(e,this,a)}))},jn.tail=function(e){var t=null==e?0:e.length;return t?ea(e,1,t):[]},jn.take=function(e,t,n){return e&&e.length?ea(e,0,(t=n||t===a?1:du(t))<0?0:t):[]},jn.takeRight=function(e,t,n){var r=null==e?0:e.length;return r?ea(e,(t=r-(t=n||t===a?1:du(t)))<0?0:t,r):[]},jn.takeRightWhile=function(e,t){return e&&e.length?ca(e,ii(t,3),!1,!0):[]},jn.takeWhile=function(e,t){return e&&e.length?ca(e,ii(t,3)):[]},jn.tap=function(e,t){return t(e),e},jn.throttle=function(e,t,n){var r=!0,a=!0;if(\"function\"!=typeof e)throw new Te(i);return Jo(n)&&(r=\"leading\"in n?!!n.leading:r,a=\"trailing\"in n?!!n.trailing:a),Po(e,t,{leading:r,maxWait:t,trailing:a})},jn.thru=fo,jn.toArray=fu,jn.toPairs=Ru,jn.toPairsIn=ju,jn.toPath=function(e){return Wo(e)?zt(e,Fi):uu(e)?[e]:Ca(Ai(yu(e)))},jn.toPlainObject=gu,jn.transform=function(e,t,n){var r=Wo(e),a=r||Qo(e)||lu(e);if(t=ii(t,4),null==n){var i=e&&e.constructor;n=a?r?new i:[]:Jo(e)&&Ko(i)?Un(Ve(e)):{}}return(a?Et:mr)(e,(function(e,r,a){return t(n,e,r,a)})),n},jn.unary=function(e){return Co(e,1)},jn.union=Ji,jn.unionBy=eo,jn.unionWith=to,jn.uniq=function(e){return e&&e.length?ua(e):[]},jn.uniqBy=function(e,t){return e&&e.length?ua(e,ii(t,2)):[]},jn.uniqWith=function(e,t){return t=\"function\"==typeof t?t:a,e&&e.length?ua(e,a,t):[]},jn.unset=function(e,t){return null==e||la(e,t)},jn.unzip=no,jn.unzipWith=ro,jn.update=function(e,t,n){return null==e?e:sa(e,t,va(n))},jn.updateWith=function(e,t,n,r){return r=\"function\"==typeof r?r:a,null==e?e:sa(e,t,va(n),r)},jn.values=Uu,jn.valuesIn=function(e){return null==e?[]:Kt(e,zu(e))},jn.without=ao,jn.words=Ku,jn.wrap=function(e,t){return Do(va(t),e)},jn.xor=io,jn.xorBy=oo,jn.xorWith=uo,jn.zip=lo,jn.zipObject=function(e,t){return da(e||[],t||[],Jn)},jn.zipObjectDeep=function(e,t){return da(e||[],t||[],Kr)},jn.zipWith=so,jn.entries=Ru,jn.entriesIn=ju,jn.extend=bu,jn.extendWith=_u,ol(jn,jn),jn.add=yl,jn.attempt=Zu,jn.camelCase=Iu,jn.capitalize=$u,jn.ceil=ml,jn.clamp=function(e,t,n){return n===a&&(n=t,t=a),n!==a&&(n=(n=vu(n))==n?n:0),t!==a&&(t=(t=vu(t))==t?t:0),ir(vu(e),t,n)},jn.clone=function(e){return or(e,4)},jn.cloneDeep=function(e){return or(e,5)},jn.cloneDeepWith=function(e,t){return or(e,5,t=\"function\"==typeof t?t:a)},jn.cloneWith=function(e,t){return or(e,4,t=\"function\"==typeof t?t:a)},jn.conformsTo=function(e,t){return null==t||ur(e,t,Pu(t))},jn.deburr=Bu,jn.defaultTo=function(e,t){return null==e||e!=e?t:e},jn.divide=bl,jn.endsWith=function(e,t,n){e=yu(e),t=oa(t);var r=e.length,i=n=n===a?r:ir(du(n),0,r);return(n-=t.length)>=0&&e.slice(n,i)==t},jn.eq=Uo,jn.escape=function(e){return(e=yu(e))&&Y.test(e)?e.replace(q,tn):e},jn.escapeRegExp=function(e){return(e=yu(e))&&ne.test(e)?e.replace(te,\"\\\\$&\"):e},jn.every=function(e,t,n){var r=Wo(e)?Tt:pr;return n&&gi(e,t,n)&&(t=a),r(e,ii(t,3))},jn.find=vo,jn.findIndex=$i,jn.findKey=function(e,t){return Rt(e,ii(t,3),mr)},jn.findLast=go,jn.findLastIndex=Bi,jn.findLastKey=function(e,t){return Rt(e,ii(t,3),br)},jn.floor=_l,jn.forEach=yo,jn.forEachRight=mo,jn.forIn=function(e,t){return null==e?e:gr(e,ii(t,3),zu)},jn.forInRight=function(e,t){return null==e?e:yr(e,ii(t,3),zu)},jn.forOwn=function(e,t){return e&&mr(e,ii(t,3))},jn.forOwnRight=function(e,t){return e&&br(e,ii(t,3))},jn.get=Eu,jn.gt=Io,jn.gte=$o,jn.has=function(e,t){return null!=e&&pi(e,t,Er)},jn.hasIn=Cu,jn.head=Vi,jn.identity=nl,jn.includes=function(e,t,n,r){e=Ho(e)?e:Uu(e),n=n&&!r?du(n):0;var a=e.length;return n<0&&(n=gn(a+n,0)),ou(e)?n<=a&&e.indexOf(t,n)>-1:!!a&&Ut(e,t,n)>-1},jn.indexOf=function(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var a=null==n?0:du(n);return a<0&&(a=gn(r+a,0)),Ut(e,t,a)},jn.inRange=function(e,t,n){return t=pu(t),n===a?(n=t,t=0):n=pu(n),function(e,t,n){return e>=yn(t,n)&&e<gn(t,n)}(e=vu(e),t,n)},jn.invoke=Nu,jn.isArguments=Bo,jn.isArray=Wo,jn.isArrayBuffer=Vo,jn.isArrayLike=Ho,jn.isArrayLikeObject=qo,jn.isBoolean=function(e){return!0===e||!1===e||eu(e)&&kr(e)==y},jn.isBuffer=Qo,jn.isDate=Yo,jn.isElement=function(e){return eu(e)&&1===e.nodeType&&!ru(e)},jn.isEmpty=function(e){if(null==e)return!0;if(Ho(e)&&(Wo(e)||\"string\"==typeof e||\"function\"==typeof e.splice||Qo(e)||lu(e)||Bo(e)))return!e.length;var t=fi(e);if(t==x||t==T)return!e.size;if(_i(e))return!Ar(e).length;for(var n in e)if(Oe.call(e,n))return!1;return!0},jn.isEqual=function(e,t){return Pr(e,t)},jn.isEqualWith=function(e,t,n){var r=(n=\"function\"==typeof n?n:a)?n(e,t):a;return r===a?Pr(e,t,a,n):!!r},jn.isError=Go,jn.isFinite=function(e){return\"number\"==typeof e&&Dt(e)},jn.isFunction=Ko,jn.isInteger=Zo,jn.isLength=Xo,jn.isMap=tu,jn.isMatch=function(e,t){return e===t||zr(e,t,ui(t))},jn.isMatchWith=function(e,t,n){return n=\"function\"==typeof n?n:a,zr(e,t,ui(t),n)},jn.isNaN=function(e){return nu(e)&&e!=+e},jn.isNative=function(e){if(bi(e))throw new we(\"Unsupported core-js use. Try https://npms.io/search?q=ponyfill.\");return Lr(e)},jn.isNil=function(e){return null==e},jn.isNull=function(e){return null===e},jn.isNumber=nu,jn.isObject=Jo,jn.isObjectLike=eu,jn.isPlainObject=ru,jn.isRegExp=au,jn.isSafeInteger=function(e){return Zo(e)&&e>=-9007199254740991&&e<=f},jn.isSet=iu,jn.isString=ou,jn.isSymbol=uu,jn.isTypedArray=lu,jn.isUndefined=function(e){return e===a},jn.isWeakMap=function(e){return eu(e)&&fi(e)==P},jn.isWeakSet=function(e){return eu(e)&&\"[object WeakSet]\"==kr(e)},jn.join=function(e,t){return null==e?\"\":Vt.call(e,t)},jn.kebabCase=Wu,jn.last=Yi,jn.lastIndexOf=function(e,t,n){var r=null==e?0:e.length;if(!r)return-1;var i=r;return n!==a&&(i=(i=du(n))<0?gn(r+i,0):yn(i,r-1)),t==t?function(e,t,n){for(var r=n+1;r--;)if(e[r]===t)return r;return r}(e,t,i):jt(e,$t,i,!0)},jn.lowerCase=Vu,jn.lowerFirst=Hu,jn.lt=su,jn.lte=cu,jn.max=function(e){return e&&e.length?dr(e,nl,Sr):a},jn.maxBy=function(e,t){return e&&e.length?dr(e,ii(t,2),Sr):a},jn.mean=function(e){return Bt(e,nl)},jn.meanBy=function(e,t){return Bt(e,ii(t,2))},jn.min=function(e){return e&&e.length?dr(e,nl,Fr):a},jn.minBy=function(e,t){return e&&e.length?dr(e,ii(t,2),Fr):a},jn.stubArray=hl,jn.stubFalse=vl,jn.stubObject=function(){return{}},jn.stubString=function(){return\"\"},jn.stubTrue=function(){return!0},jn.multiply=wl,jn.nth=function(e,t){return e&&e.length?Ir(e,du(t)):a},jn.noConflict=function(){return ft._===this&&(ft._=je),this},jn.noop=ul,jn.now=Eo,jn.pad=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;if(!t||r>=t)return e;var a=(t-r)/2;return $a(dt(a),n)+e+$a(pt(a),n)},jn.padEnd=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;return t&&r<t?e+$a(t-r,n):e},jn.padStart=function(e,t,n){e=yu(e);var r=(t=du(t))?cn(e):0;return t&&r<t?$a(t-r,n)+e:e},jn.parseInt=function(e,t,n){return n||null==t?t=0:t&&(t=+t),bn(yu(e).replace(re,\"\"),t||0)},jn.random=function(e,t,n){if(n&&\"boolean\"!=typeof n&&gi(e,t,n)&&(t=n=a),n===a&&(\"boolean\"==typeof t?(n=t,t=a):\"boolean\"==typeof e&&(n=e,e=a)),e===a&&t===a?(e=0,t=1):(e=pu(e),t===a?(t=e,e=0):t=pu(t)),e>t){var r=e;e=t,t=r}if(n||e%1||t%1){var i=_n();return yn(e+i*(t-e+ut(\"1e-\"+((i+\"\").length-1))),t)}return Hr(e,t)},jn.reduce=function(e,t,n){var r=Wo(e)?Ot:Ht,a=arguments.length<3;return r(e,ii(t,4),n,a,cr)},jn.reduceRight=function(e,t,n){var r=Wo(e)?At:Ht,a=arguments.length<3;return r(e,ii(t,4),n,a,fr)},jn.repeat=function(e,t,n){return t=(n?gi(e,t,n):t===a)?1:du(t),qr(yu(e),t)},jn.replace=function(){var e=arguments,t=yu(e[0]);return e.length<3?t:t.replace(e[1],e[2])},jn.result=function(e,t,n){var r=-1,i=(t=ga(t,e)).length;for(i||(i=1,e=a);++r<i;){var o=null==e?a:e[Fi(t[r])];o===a&&(r=i,o=n),e=Ko(o)?o.call(e):o}return e},jn.round=xl,jn.runInContext=e,jn.sample=function(e){return(Wo(e)?Gn:Yr)(e)},jn.size=function(e){if(null==e)return 0;if(Ho(e))return ou(e)?cn(e):e.length;var t=fi(e);return t==x||t==T?e.size:Ar(e).length},jn.snakeCase=qu,jn.some=function(e,t,n){var r=Wo(e)?Ft:ta;return n&&gi(e,t,n)&&(t=a),r(e,ii(t,3))},jn.sortedIndex=function(e,t){return na(e,t)},jn.sortedIndexBy=function(e,t,n){return ra(e,t,ii(n,2))},jn.sortedIndexOf=function(e,t){var n=null==e?0:e.length;if(n){var r=na(e,t);if(r<n&&Uo(e[r],t))return r}return-1},jn.sortedLastIndex=function(e,t){return na(e,t,!0)},jn.sortedLastIndexBy=function(e,t,n){return ra(e,t,ii(n,2),!0)},jn.sortedLastIndexOf=function(e,t){if(null!=e&&e.length){var n=na(e,t,!0)-1;if(Uo(e[n],t))return n}return-1},jn.startCase=Qu,jn.startsWith=function(e,t,n){return e=yu(e),n=null==n?0:ir(du(n),0,e.length),t=oa(t),e.slice(n,n+t.length)==t},jn.subtract=kl,jn.sum=function(e){return e&&e.length?qt(e,nl):0},jn.sumBy=function(e,t){return e&&e.length?qt(e,ii(t,2)):0},jn.template=function(e,t,n){var r=jn.templateSettings;n&&gi(e,t,n)&&(t=a),e=yu(e),t=_u({},t,r,Ga);var i,o,u=_u({},t.imports,r.imports,Ga),l=Pu(u),s=Kt(u,l),c=0,f=t.interpolate||be,p=\"__p += '\",d=Ee((t.escape||be).source+\"|\"+f.source+\"|\"+(f===Z?fe:be).source+\"|\"+(t.evaluate||be).source+\"|$\",\"g\"),h=\"//# sourceURL=\"+(Oe.call(t,\"sourceURL\")?(t.sourceURL+\"\").replace(/\\s/g,\" \"):\"lodash.templateSources[\"+ ++rt+\"]\")+\"\\n\";e.replace(d,(function(t,n,r,a,u,l){return r||(r=a),p+=e.slice(c,l).replace(_e,nn),n&&(i=!0,p+=\"' +\\n__e(\"+n+\") +\\n'\"),u&&(o=!0,p+=\"';\\n\"+u+\";\\n__p += '\"),r&&(p+=\"' +\\n((__t = (\"+r+\")) == null ? '' : __t) +\\n'\"),c=l+t.length,t})),p+=\"';\\n\";var v=Oe.call(t,\"variable\")&&t.variable;if(v){if(se.test(v))throw new we(\"Invalid `variable` option passed into `_.template`\")}else p=\"with (obj) {\\n\"+p+\"\\n}\\n\";p=(o?p.replace(B,\"\"):p).replace(W,\"$1\").replace(V,\"$1;\"),p=\"function(\"+(v||\"obj\")+\") {\\n\"+(v?\"\":\"obj || (obj = {});\\n\")+\"var __t, __p = ''\"+(i?\", __e = _.escape\":\"\")+(o?\", __j = Array.prototype.join;\\nfunction print() { __p += __j.call(arguments, '') }\\n\":\";\\n\")+p+\"return __p\\n}\";var g=Zu((function(){return xe(l,h+\"return \"+p).apply(a,s)}));if(g.source=p,Go(g))throw g;return g},jn.times=function(e,t){if((e=du(e))<1||e>f)return[];var n=d,r=yn(e,d);t=ii(t),e-=d;for(var a=Qt(r,t);++n<e;)t(n);return a},jn.toFinite=pu,jn.toInteger=du,jn.toLength=hu,jn.toLower=function(e){return yu(e).toLowerCase()},jn.toNumber=vu,jn.toSafeInteger=function(e){return e?ir(du(e),-9007199254740991,f):0===e?e:0},jn.toString=yu,jn.toUpper=function(e){return yu(e).toUpperCase()},jn.trim=function(e,t,n){if((e=yu(e))&&(n||t===a))return Yt(e);if(!e||!(t=oa(t)))return e;var r=fn(e),i=fn(t);return ma(r,Xt(r,i),Jt(r,i)+1).join(\"\")},jn.trimEnd=function(e,t,n){if((e=yu(e))&&(n||t===a))return e.slice(0,pn(e)+1);if(!e||!(t=oa(t)))return e;var r=fn(e);return ma(r,0,Jt(r,fn(t))+1).join(\"\")},jn.trimStart=function(e,t,n){if((e=yu(e))&&(n||t===a))return e.replace(re,\"\");if(!e||!(t=oa(t)))return e;var r=fn(e);return ma(r,Xt(r,fn(t))).join(\"\")},jn.truncate=function(e,t){var n=30,r=\"...\";if(Jo(t)){var i=\"separator\"in t?t.separator:i;n=\"length\"in t?du(t.length):n,r=\"omission\"in t?oa(t.omission):r}var o=(e=yu(e)).length;if(rn(e)){var u=fn(e);o=u.length}if(n>=o)return e;var l=n-cn(r);if(l<1)return r;var s=u?ma(u,0,l).join(\"\"):e.slice(0,l);if(i===a)return s+r;if(u&&(l+=s.length-l),au(i)){if(e.slice(l).search(i)){var c,f=s;for(i.global||(i=Ee(i.source,yu(pe.exec(i))+\"g\")),i.lastIndex=0;c=i.exec(f);)var p=c.index;s=s.slice(0,p===a?l:p)}}else if(e.indexOf(oa(i),l)!=l){var d=s.lastIndexOf(i);d>-1&&(s=s.slice(0,d))}return s+r},jn.unescape=function(e){return(e=yu(e))&&Q.test(e)?e.replace(H,dn):e},jn.uniqueId=function(e){var t=++Ae;return yu(e)+t},jn.upperCase=Yu,jn.upperFirst=Gu,jn.each=yo,jn.eachRight=mo,jn.first=Vi,ol(jn,(gl={},mr(jn,(function(e,t){Oe.call(jn.prototype,t)||(gl[t]=e)})),gl),{chain:!1}),jn.VERSION=\"4.17.21\",Et([\"bind\",\"bindKey\",\"curry\",\"curryRight\",\"partial\",\"partialRight\"],(function(e){jn[e].placeholder=jn})),Et([\"drop\",\"take\"],(function(e,t){Bn.prototype[e]=function(n){n=n===a?1:gn(du(n),0);var r=this.__filtered__&&!t?new Bn(this):this.clone();return r.__filtered__?r.__takeCount__=yn(n,r.__takeCount__):r.__views__.push({size:yn(n,d),type:e+(r.__dir__<0?\"Right\":\"\")}),r},Bn.prototype[e+\"Right\"]=function(t){return this.reverse()[e](t).reverse()}})),Et([\"filter\",\"map\",\"takeWhile\"],(function(e,t){var n=t+1,r=1==n||3==n;Bn.prototype[e]=function(e){var t=this.clone();return t.__iteratees__.push({iteratee:ii(e,3),type:n}),t.__filtered__=t.__filtered__||r,t}})),Et([\"head\",\"last\"],(function(e,t){var n=\"take\"+(t?\"Right\":\"\");Bn.prototype[e]=function(){return this[n](1).value()[0]}})),Et([\"initial\",\"tail\"],(function(e,t){var n=\"drop\"+(t?\"\":\"Right\");Bn.prototype[e]=function(){return this.__filtered__?new Bn(this):this[n](1)}})),Bn.prototype.compact=function(){return this.filter(nl)},Bn.prototype.find=function(e){return this.filter(e).head()},Bn.prototype.findLast=function(e){return this.reverse().find(e)},Bn.prototype.invokeMap=Qr((function(e,t){return\"function\"==typeof e?new Bn(this):this.map((function(n){return Mr(n,e,t)}))})),Bn.prototype.reject=function(e){return this.filter(Ao(ii(e)))},Bn.prototype.slice=function(e,t){e=du(e);var n=this;return n.__filtered__&&(e>0||t<0)?new Bn(n):(e<0?n=n.takeRight(-e):e&&(n=n.drop(e)),t!==a&&(n=(t=du(t))<0?n.dropRight(-t):n.take(t-e)),n)},Bn.prototype.takeRightWhile=function(e){return this.reverse().takeWhile(e).reverse()},Bn.prototype.toArray=function(){return this.take(d)},mr(Bn.prototype,(function(e,t){var n=/^(?:filter|find|map|reject)|While$/.test(t),r=/^(?:head|last)$/.test(t),i=jn[r?\"take\"+(\"last\"==t?\"Right\":\"\"):t],o=r||/^find/.test(t);i&&(jn.prototype[t]=function(){var t=this.__wrapped__,u=r?[1]:arguments,l=t instanceof Bn,s=u[0],c=l||Wo(t),f=function(e){var t=i.apply(jn,Lt([e],u));return r&&p?t[0]:t};c&&n&&\"function\"==typeof s&&1!=s.length&&(l=c=!1);var p=this.__chain__,d=!!this.__actions__.length,h=o&&!p,v=l&&!d;if(!o&&c){t=v?t:new Bn(this);var g=e.apply(t,u);return g.__actions__.push({func:fo,args:[f],thisArg:a}),new $n(g,p)}return h&&v?e.apply(this,u):(g=this.thru(f),h?r?g.value()[0]:g.value():g)})})),Et([\"pop\",\"push\",\"shift\",\"sort\",\"splice\",\"unshift\"],(function(e){var t=Me[e],n=/^(?:push|sort|unshift)$/.test(e)?\"tap\":\"thru\",r=/^(?:pop|shift)$/.test(e);jn.prototype[e]=function(){var e=arguments;if(r&&!this.__chain__){var a=this.value();return t.apply(Wo(a)?a:[],e)}return this[n]((function(n){return t.apply(Wo(n)?n:[],e)}))}})),mr(Bn.prototype,(function(e,t){var n=jn[t];if(n){var r=n.name+\"\";Oe.call(Nn,r)||(Nn[r]=[]),Nn[r].push({name:t,func:n})}})),Nn[Ra(a,2).name]=[{name:\"wrapper\",func:a}],Bn.prototype.clone=function(){var e=new Bn(this.__wrapped__);return e.__actions__=Ca(this.__actions__),e.__dir__=this.__dir__,e.__filtered__=this.__filtered__,e.__iteratees__=Ca(this.__iteratees__),e.__takeCount__=this.__takeCount__,e.__views__=Ca(this.__views__),e},Bn.prototype.reverse=function(){if(this.__filtered__){var e=new Bn(this);e.__dir__=-1,e.__filtered__=!0}else(e=this.clone()).__dir__*=-1;return e},Bn.prototype.value=function(){var e=this.__wrapped__.value(),t=this.__dir__,n=Wo(e),r=t<0,a=n?e.length:0,i=function(e,t,n){for(var r=-1,a=n.length;++r<a;){var i=n[r],o=i.size;switch(i.type){case\"drop\":e+=o;break;case\"dropRight\":t-=o;break;case\"take\":t=yn(t,e+o);break;case\"takeRight\":e=gn(e,t-o)}}return{start:e,end:t}}(0,a,this.__views__),o=i.start,u=i.end,l=u-o,s=r?u:o-1,c=this.__iteratees__,f=c.length,p=0,d=yn(l,this.__takeCount__);if(!n||!r&&a==l&&d==l)return fa(e,this.__actions__);var h=[];e:for(;l--&&p<d;){for(var v=-1,g=e[s+=t];++v<f;){var y=c[v],m=y.iteratee,b=y.type,_=m(g);if(2==b)g=_;else if(!_){if(1==b)continue e;break e}}h[p++]=g}return h},jn.prototype.at=po,jn.prototype.chain=function(){return co(this)},jn.prototype.commit=function(){return new $n(this.value(),this.__chain__)},jn.prototype.next=function(){this.__values__===a&&(this.__values__=fu(this.value()));var e=this.__index__>=this.__values__.length;return{done:e,value:e?a:this.__values__[this.__index__++]}},jn.prototype.plant=function(e){for(var t,n=this;n instanceof In;){var r=Ri(n);r.__index__=0,r.__values__=a,t?i.__wrapped__=r:t=r;var i=r;n=n.__wrapped__}return i.__wrapped__=e,t},jn.prototype.reverse=function(){var e=this.__wrapped__;if(e instanceof Bn){var t=e;return this.__actions__.length&&(t=new Bn(this)),(t=t.reverse()).__actions__.push({func:fo,args:[Xi],thisArg:a}),new $n(t,this.__chain__)}return this.thru(Xi)},jn.prototype.toJSON=jn.prototype.valueOf=jn.prototype.value=function(){return fa(this.__wrapped__,this.__actions__)},jn.prototype.first=jn.prototype.head,Ge&&(jn.prototype[Ge]=function(){return this}),jn}();ft._=hn,(r=function(){return hn}.call(t,n,t,e))===a||(e.exports=r)}.call(this)},448:(e,t,n)=>{\"use strict\";var r=n(294),a=n(840);function i(e){for(var t=\"https://reactjs.org/docs/error-decoder.html?invariant=\"+e,n=1;n<arguments.length;n++)t+=\"&args[]=\"+encodeURIComponent(arguments[n]);return\"Minified React error #\"+e+\"; visit \"+t+\" for the full message or use the non-minified dev environment for full errors and additional helpful warnings.\"}var o=new Set,u={};function l(e,t){s(e,t),s(e+\"Capture\",t)}function s(e,t){for(u[e]=t,e=0;e<t.length;e++)o.add(t[e])}var c=!(\"undefined\"==typeof window||void 0===window.document||void 0===window.document.createElement),f=Object.prototype.hasOwnProperty,p=/^[:A-Z_a-z\\u00C0-\\u00D6\\u00D8-\\u00F6\\u00F8-\\u02FF\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF\\uF900-\\uFDCF\\uFDF0-\\uFFFD][:A-Z_a-z\\u00C0-\\u00D6\\u00D8-\\u00F6\\u00F8-\\u02FF\\u0370-\\u037D\\u037F-\\u1FFF\\u200C-\\u200D\\u2070-\\u218F\\u2C00-\\u2FEF\\u3001-\\uD7FF\\uF900-\\uFDCF\\uFDF0-\\uFFFD\\-.0-9\\u00B7\\u0300-\\u036F\\u203F-\\u2040]*$/,d={},h={};function v(e,t,n,r,a,i,o){this.acceptsBooleans=2===t||3===t||4===t,this.attributeName=r,this.attributeNamespace=a,this.mustUseProperty=n,this.propertyName=e,this.type=t,this.sanitizeURL=i,this.removeEmptyString=o}var g={};\"children dangerouslySetInnerHTML defaultValue defaultChecked innerHTML suppressContentEditableWarning suppressHydrationWarning style\".split(\" \").forEach((function(e){g[e]=new v(e,0,!1,e,null,!1,!1)})),[[\"acceptCharset\",\"accept-charset\"],[\"className\",\"class\"],[\"htmlFor\",\"for\"],[\"httpEquiv\",\"http-equiv\"]].forEach((function(e){var t=e[0];g[t]=new v(t,1,!1,e[1],null,!1,!1)})),[\"contentEditable\",\"draggable\",\"spellCheck\",\"value\"].forEach((function(e){g[e]=new v(e,2,!1,e.toLowerCase(),null,!1,!1)})),[\"autoReverse\",\"externalResourcesRequired\",\"focusable\",\"preserveAlpha\"].forEach((function(e){g[e]=new v(e,2,!1,e,null,!1,!1)})),\"allowFullScreen async autoFocus autoPlay controls default defer disabled disablePictureInPicture disableRemotePlayback formNoValidate hidden loop noModule noValidate open playsInline readOnly required reversed scoped seamless itemScope\".split(\" \").forEach((function(e){g[e]=new v(e,3,!1,e.toLowerCase(),null,!1,!1)})),[\"checked\",\"multiple\",\"muted\",\"selected\"].forEach((function(e){g[e]=new v(e,3,!0,e,null,!1,!1)})),[\"capture\",\"download\"].forEach((function(e){g[e]=new v(e,4,!1,e,null,!1,!1)})),[\"cols\",\"rows\",\"size\",\"span\"].forEach((function(e){g[e]=new v(e,6,!1,e,null,!1,!1)})),[\"rowSpan\",\"start\"].forEach((function(e){g[e]=new v(e,5,!1,e.toLowerCase(),null,!1,!1)}));var y=/[\\-:]([a-z])/g;function m(e){return e[1].toUpperCase()}function b(e,t,n,r){var a=g.hasOwnProperty(t)?g[t]:null;(null!==a?0!==a.type:r||!(2<t.length)||\"o\"!==t[0]&&\"O\"!==t[0]||\"n\"!==t[1]&&\"N\"!==t[1])&&(function(e,t,n,r){if(null==t||function(e,t,n,r){if(null!==n&&0===n.type)return!1;switch(typeof t){case\"function\":case\"symbol\":return!0;case\"boolean\":return!r&&(null!==n?!n.acceptsBooleans:\"data-\"!==(e=e.toLowerCase().slice(0,5))&&\"aria-\"!==e);default:return!1}}(e,t,n,r))return!0;if(r)return!1;if(null!==n)switch(n.type){case 3:return!t;case 4:return!1===t;case 5:return isNaN(t);case 6:return isNaN(t)||1>t}return!1}(t,n,a,r)&&(n=null),r||null===a?function(e){return!!f.call(h,e)||!f.call(d,e)&&(p.test(e)?h[e]=!0:(d[e]=!0,!1))}(t)&&(null===n?e.removeAttribute(t):e.setAttribute(t,\"\"+n)):a.mustUseProperty?e[a.propertyName]=null===n?3!==a.type&&\"\":n:(t=a.attributeName,r=a.attributeNamespace,null===n?e.removeAttribute(t):(n=3===(a=a.type)||4===a&&!0===n?\"\":\"\"+n,r?e.setAttributeNS(r,t,n):e.setAttribute(t,n))))}\"accent-height alignment-baseline arabic-form baseline-shift cap-height clip-path clip-rule color-interpolation color-interpolation-filters color-profile color-rendering dominant-baseline enable-background fill-opacity fill-rule flood-color flood-opacity font-family font-size font-size-adjust font-stretch font-style font-variant font-weight glyph-name glyph-orientation-horizontal glyph-orientation-vertical horiz-adv-x horiz-origin-x image-rendering letter-spacing lighting-color marker-end marker-mid marker-start overline-position overline-thickness paint-order panose-1 pointer-events rendering-intent shape-rendering stop-color stop-opacity strikethrough-position strikethrough-thickness stroke-dasharray stroke-dashoffset stroke-linecap stroke-linejoin stroke-miterlimit stroke-opacity stroke-width text-anchor text-decoration text-rendering underline-position underline-thickness unicode-bidi unicode-range units-per-em v-alphabetic v-hanging v-ideographic v-mathematical vector-effect vert-adv-y vert-origin-x vert-origin-y word-spacing writing-mode xmlns:xlink x-height\".split(\" \").forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,null,!1,!1)})),\"xlink:actuate xlink:arcrole xlink:role xlink:show xlink:title xlink:type\".split(\" \").forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,\"http://www.w3.org/1999/xlink\",!1,!1)})),[\"xml:base\",\"xml:lang\",\"xml:space\"].forEach((function(e){var t=e.replace(y,m);g[t]=new v(t,1,!1,e,\"http://www.w3.org/XML/1998/namespace\",!1,!1)})),[\"tabIndex\",\"crossOrigin\"].forEach((function(e){g[e]=new v(e,1,!1,e.toLowerCase(),null,!1,!1)})),g.xlinkHref=new v(\"xlinkHref\",1,!1,\"xlink:href\",\"http://www.w3.org/1999/xlink\",!0,!1),[\"src\",\"href\",\"action\",\"formAction\"].forEach((function(e){g[e]=new v(e,1,!1,e.toLowerCase(),null,!0,!0)}));var _=r.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED,w=Symbol.for(\"react.element\"),x=Symbol.for(\"react.portal\"),k=Symbol.for(\"react.fragment\"),S=Symbol.for(\"react.strict_mode\"),E=Symbol.for(\"react.profiler\"),C=Symbol.for(\"react.provider\"),T=Symbol.for(\"react.context\"),M=Symbol.for(\"react.forward_ref\"),N=Symbol.for(\"react.suspense\"),P=Symbol.for(\"react.suspense_list\"),z=Symbol.for(\"react.memo\"),L=Symbol.for(\"react.lazy\");Symbol.for(\"react.scope\"),Symbol.for(\"react.debug_trace_mode\");var O=Symbol.for(\"react.offscreen\");Symbol.for(\"react.legacy_hidden\"),Symbol.for(\"react.cache\"),Symbol.for(\"react.tracing_marker\");var A=Symbol.iterator;function F(e){return null===e||\"object\"!=typeof e?null:\"function\"==typeof(e=A&&e[A]||e[\"@@iterator\"])?e:null}var D,R=Object.assign;function j(e){if(void 0===D)try{throw Error()}catch(e){var t=e.stack.trim().match(/\\n( *(at )?)/);D=t&&t[1]||\"\"}return\"\\n\"+D+e}var U=!1;function I(e,t){if(!e||U)return\"\";U=!0;var n=Error.prepareStackTrace;Error.prepareStackTrace=void 0;try{if(t)if(t=function(){throw Error()},Object.defineProperty(t.prototype,\"props\",{set:function(){throw Error()}}),\"object\"==typeof Reflect&&Reflect.construct){try{Reflect.construct(t,[])}catch(e){var r=e}Reflect.construct(e,[],t)}else{try{t.call()}catch(e){r=e}e.call(t.prototype)}else{try{throw Error()}catch(e){r=e}e()}}catch(t){if(t&&r&&\"string\"==typeof t.stack){for(var a=t.stack.split(\"\\n\"),i=r.stack.split(\"\\n\"),o=a.length-1,u=i.length-1;1<=o&&0<=u&&a[o]!==i[u];)u--;for(;1<=o&&0<=u;o--,u--)if(a[o]!==i[u]){if(1!==o||1!==u)do{if(o--,0>--u||a[o]!==i[u]){var l=\"\\n\"+a[o].replace(\" at new \",\" at \");return e.displayName&&l.includes(\"<anonymous>\")&&(l=l.replace(\"<anonymous>\",e.displayName)),l}}while(1<=o&&0<=u);break}}}finally{U=!1,Error.prepareStackTrace=n}return(e=e?e.displayName||e.name:\"\")?j(e):\"\"}function $(e){switch(e.tag){case 5:return j(e.type);case 16:return j(\"Lazy\");case 13:return j(\"Suspense\");case 19:return j(\"SuspenseList\");case 0:case 2:case 15:return I(e.type,!1);case 11:return I(e.type.render,!1);case 1:return I(e.type,!0);default:return\"\"}}function B(e){if(null==e)return null;if(\"function\"==typeof e)return e.displayName||e.name||null;if(\"string\"==typeof e)return e;switch(e){case k:return\"Fragment\";case x:return\"Portal\";case E:return\"Profiler\";case S:return\"StrictMode\";case N:return\"Suspense\";case P:return\"SuspenseList\"}if(\"object\"==typeof e)switch(e.$$typeof){case T:return(e.displayName||\"Context\")+\".Consumer\";case C:return(e._context.displayName||\"Context\")+\".Provider\";case M:var t=e.render;return(e=e.displayName)||(e=\"\"!==(e=t.displayName||t.name||\"\")?\"ForwardRef(\"+e+\")\":\"ForwardRef\"),e;case z:return null!==(t=e.displayName||null)?t:B(e.type)||\"Memo\";case L:t=e._payload,e=e._init;try{return B(e(t))}catch(e){}}return null}function W(e){var t=e.type;switch(e.tag){case 24:return\"Cache\";case 9:return(t.displayName||\"Context\")+\".Consumer\";case 10:return(t._context.displayName||\"Context\")+\".Provider\";case 18:return\"DehydratedFragment\";case 11:return e=(e=t.render).displayName||e.name||\"\",t.displayName||(\"\"!==e?\"ForwardRef(\"+e+\")\":\"ForwardRef\");case 7:return\"Fragment\";case 5:return t;case 4:return\"Portal\";case 3:return\"Root\";case 6:return\"Text\";case 16:return B(t);case 8:return t===S?\"StrictMode\":\"Mode\";case 22:return\"Offscreen\";case 12:return\"Profiler\";case 21:return\"Scope\";case 13:return\"Suspense\";case 19:return\"SuspenseList\";case 25:return\"TracingMarker\";case 1:case 0:case 17:case 2:case 14:case 15:if(\"function\"==typeof t)return t.displayName||t.name||null;if(\"string\"==typeof t)return t}return null}function V(e){switch(typeof e){case\"boolean\":case\"number\":case\"string\":case\"undefined\":case\"object\":return e;default:return\"\"}}function H(e){var t=e.type;return(e=e.nodeName)&&\"input\"===e.toLowerCase()&&(\"checkbox\"===t||\"radio\"===t)}function q(e){e._valueTracker||(e._valueTracker=function(e){var t=H(e)?\"checked\":\"value\",n=Object.getOwnPropertyDescriptor(e.constructor.prototype,t),r=\"\"+e[t];if(!e.hasOwnProperty(t)&&void 0!==n&&\"function\"==typeof n.get&&\"function\"==typeof n.set){var a=n.get,i=n.set;return Object.defineProperty(e,t,{configurable:!0,get:function(){return a.call(this)},set:function(e){r=\"\"+e,i.call(this,e)}}),Object.defineProperty(e,t,{enumerable:n.enumerable}),{getValue:function(){return r},setValue:function(e){r=\"\"+e},stopTracking:function(){e._valueTracker=null,delete e[t]}}}}(e))}function Q(e){if(!e)return!1;var t=e._valueTracker;if(!t)return!0;var n=t.getValue(),r=\"\";return e&&(r=H(e)?e.checked?\"true\":\"false\":e.value),(e=r)!==n&&(t.setValue(e),!0)}function Y(e){if(void 0===(e=e||(\"undefined\"!=typeof document?document:void 0)))return null;try{return e.activeElement||e.body}catch(t){return e.body}}function G(e,t){var n=t.checked;return R({},t,{defaultChecked:void 0,defaultValue:void 0,value:void 0,checked:null!=n?n:e._wrapperState.initialChecked})}function K(e,t){var n=null==t.defaultValue?\"\":t.defaultValue,r=null!=t.checked?t.checked:t.defaultChecked;n=V(null!=t.value?t.value:n),e._wrapperState={initialChecked:r,initialValue:n,controlled:\"checkbox\"===t.type||\"radio\"===t.type?null!=t.checked:null!=t.value}}function Z(e,t){null!=(t=t.checked)&&b(e,\"checked\",t,!1)}function X(e,t){Z(e,t);var n=V(t.value),r=t.type;if(null!=n)\"number\"===r?(0===n&&\"\"===e.value||e.value!=n)&&(e.value=\"\"+n):e.value!==\"\"+n&&(e.value=\"\"+n);else if(\"submit\"===r||\"reset\"===r)return void e.removeAttribute(\"value\");t.hasOwnProperty(\"value\")?ee(e,t.type,n):t.hasOwnProperty(\"defaultValue\")&&ee(e,t.type,V(t.defaultValue)),null==t.checked&&null!=t.defaultChecked&&(e.defaultChecked=!!t.defaultChecked)}function J(e,t,n){if(t.hasOwnProperty(\"value\")||t.hasOwnProperty(\"defaultValue\")){var r=t.type;if(!(\"submit\"!==r&&\"reset\"!==r||void 0!==t.value&&null!==t.value))return;t=\"\"+e._wrapperState.initialValue,n||t===e.value||(e.value=t),e.defaultValue=t}\"\"!==(n=e.name)&&(e.name=\"\"),e.defaultChecked=!!e._wrapperState.initialChecked,\"\"!==n&&(e.name=n)}function ee(e,t,n){\"number\"===t&&Y(e.ownerDocument)===e||(null==n?e.defaultValue=\"\"+e._wrapperState.initialValue:e.defaultValue!==\"\"+n&&(e.defaultValue=\"\"+n))}var te=Array.isArray;function ne(e,t,n,r){if(e=e.options,t){t={};for(var a=0;a<n.length;a++)t[\"$\"+n[a]]=!0;for(n=0;n<e.length;n++)a=t.hasOwnProperty(\"$\"+e[n].value),e[n].selected!==a&&(e[n].selected=a),a&&r&&(e[n].defaultSelected=!0)}else{for(n=\"\"+V(n),t=null,a=0;a<e.length;a++){if(e[a].value===n)return e[a].selected=!0,void(r&&(e[a].defaultSelected=!0));null!==t||e[a].disabled||(t=e[a])}null!==t&&(t.selected=!0)}}function re(e,t){if(null!=t.dangerouslySetInnerHTML)throw Error(i(91));return R({},t,{value:void 0,defaultValue:void 0,children:\"\"+e._wrapperState.initialValue})}function ae(e,t){var n=t.value;if(null==n){if(n=t.children,t=t.defaultValue,null!=n){if(null!=t)throw Error(i(92));if(te(n)){if(1<n.length)throw Error(i(93));n=n[0]}t=n}null==t&&(t=\"\"),n=t}e._wrapperState={initialValue:V(n)}}function ie(e,t){var n=V(t.value),r=V(t.defaultValue);null!=n&&((n=\"\"+n)!==e.value&&(e.value=n),null==t.defaultValue&&e.defaultValue!==n&&(e.defaultValue=n)),null!=r&&(e.defaultValue=\"\"+r)}function oe(e){var t=e.textContent;t===e._wrapperState.initialValue&&\"\"!==t&&null!==t&&(e.value=t)}function ue(e){switch(e){case\"svg\":return\"http://www.w3.org/2000/svg\";case\"math\":return\"http://www.w3.org/1998/Math/MathML\";default:return\"http://www.w3.org/1999/xhtml\"}}function le(e,t){return null==e||\"http://www.w3.org/1999/xhtml\"===e?ue(t):\"http://www.w3.org/2000/svg\"===e&&\"foreignObject\"===t?\"http://www.w3.org/1999/xhtml\":e}var se,ce,fe=(ce=function(e,t){if(\"http://www.w3.org/2000/svg\"!==e.namespaceURI||\"innerHTML\"in e)e.innerHTML=t;else{for((se=se||document.createElement(\"div\")).innerHTML=\"<svg>\"+t.valueOf().toString()+\"</svg>\",t=se.firstChild;e.firstChild;)e.removeChild(e.firstChild);for(;t.firstChild;)e.appendChild(t.firstChild)}},\"undefined\"!=typeof MSApp&&MSApp.execUnsafeLocalFunction?function(e,t,n,r){MSApp.execUnsafeLocalFunction((function(){return ce(e,t)}))}:ce);function pe(e,t){if(t){var n=e.firstChild;if(n&&n===e.lastChild&&3===n.nodeType)return void(n.nodeValue=t)}e.textContent=t}var de={animationIterationCount:!0,aspectRatio:!0,borderImageOutset:!0,borderImageSlice:!0,borderImageWidth:!0,boxFlex:!0,boxFlexGroup:!0,boxOrdinalGroup:!0,columnCount:!0,columns:!0,flex:!0,flexGrow:!0,flexPositive:!0,flexShrink:!0,flexNegative:!0,flexOrder:!0,gridArea:!0,gridRow:!0,gridRowEnd:!0,gridRowSpan:!0,gridRowStart:!0,gridColumn:!0,gridColumnEnd:!0,gridColumnSpan:!0,gridColumnStart:!0,fontWeight:!0,lineClamp:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,tabSize:!0,widows:!0,zIndex:!0,zoom:!0,fillOpacity:!0,floodOpacity:!0,stopOpacity:!0,strokeDasharray:!0,strokeDashoffset:!0,strokeMiterlimit:!0,strokeOpacity:!0,strokeWidth:!0},he=[\"Webkit\",\"ms\",\"Moz\",\"O\"];function ve(e,t,n){return null==t||\"boolean\"==typeof t||\"\"===t?\"\":n||\"number\"!=typeof t||0===t||de.hasOwnProperty(e)&&de[e]?(\"\"+t).trim():t+\"px\"}function ge(e,t){for(var n in e=e.style,t)if(t.hasOwnProperty(n)){var r=0===n.indexOf(\"--\"),a=ve(n,t[n],r);\"float\"===n&&(n=\"cssFloat\"),r?e.setProperty(n,a):e[n]=a}}Object.keys(de).forEach((function(e){he.forEach((function(t){t=t+e.charAt(0).toUpperCase()+e.substring(1),de[t]=de[e]}))}));var ye=R({menuitem:!0},{area:!0,base:!0,br:!0,col:!0,embed:!0,hr:!0,img:!0,input:!0,keygen:!0,link:!0,meta:!0,param:!0,source:!0,track:!0,wbr:!0});function me(e,t){if(t){if(ye[e]&&(null!=t.children||null!=t.dangerouslySetInnerHTML))throw Error(i(137,e));if(null!=t.dangerouslySetInnerHTML){if(null!=t.children)throw Error(i(60));if(\"object\"!=typeof t.dangerouslySetInnerHTML||!(\"__html\"in t.dangerouslySetInnerHTML))throw Error(i(61))}if(null!=t.style&&\"object\"!=typeof t.style)throw Error(i(62))}}function be(e,t){if(-1===e.indexOf(\"-\"))return\"string\"==typeof t.is;switch(e){case\"annotation-xml\":case\"color-profile\":case\"font-face\":case\"font-face-src\":case\"font-face-uri\":case\"font-face-format\":case\"font-face-name\":case\"missing-glyph\":return!1;default:return!0}}var _e=null;function we(e){return(e=e.target||e.srcElement||window).correspondingUseElement&&(e=e.correspondingUseElement),3===e.nodeType?e.parentNode:e}var xe=null,ke=null,Se=null;function Ee(e){if(e=ba(e)){if(\"function\"!=typeof xe)throw Error(i(280));var t=e.stateNode;t&&(t=wa(t),xe(e.stateNode,e.type,t))}}function Ce(e){ke?Se?Se.push(e):Se=[e]:ke=e}function Te(){if(ke){var e=ke,t=Se;if(Se=ke=null,Ee(e),t)for(e=0;e<t.length;e++)Ee(t[e])}}function Me(e,t){return e(t)}function Ne(){}var Pe=!1;function ze(e,t,n){if(Pe)return e(t,n);Pe=!0;try{return Me(e,t,n)}finally{Pe=!1,(null!==ke||null!==Se)&&(Ne(),Te())}}function Le(e,t){var n=e.stateNode;if(null===n)return null;var r=wa(n);if(null===r)return null;n=r[t];e:switch(t){case\"onClick\":case\"onClickCapture\":case\"onDoubleClick\":case\"onDoubleClickCapture\":case\"onMouseDown\":case\"onMouseDownCapture\":case\"onMouseMove\":case\"onMouseMoveCapture\":case\"onMouseUp\":case\"onMouseUpCapture\":case\"onMouseEnter\":(r=!r.disabled)||(r=!(\"button\"===(e=e.type)||\"input\"===e||\"select\"===e||\"textarea\"===e)),e=!r;break e;default:e=!1}if(e)return null;if(n&&\"function\"!=typeof n)throw Error(i(231,t,typeof n));return n}var Oe=!1;if(c)try{var Ae={};Object.defineProperty(Ae,\"passive\",{get:function(){Oe=!0}}),window.addEventListener(\"test\",Ae,Ae),window.removeEventListener(\"test\",Ae,Ae)}catch(ce){Oe=!1}function Fe(e,t,n,r,a,i,o,u,l){var s=Array.prototype.slice.call(arguments,3);try{t.apply(n,s)}catch(e){this.onError(e)}}var De=!1,Re=null,je=!1,Ue=null,Ie={onError:function(e){De=!0,Re=e}};function $e(e,t,n,r,a,i,o,u,l){De=!1,Re=null,Fe.apply(Ie,arguments)}function Be(e){var t=e,n=e;if(e.alternate)for(;t.return;)t=t.return;else{e=t;do{0!=(4098&(t=e).flags)&&(n=t.return),e=t.return}while(e)}return 3===t.tag?n:null}function We(e){if(13===e.tag){var t=e.memoizedState;if(null===t&&null!==(e=e.alternate)&&(t=e.memoizedState),null!==t)return t.dehydrated}return null}function Ve(e){if(Be(e)!==e)throw Error(i(188))}function He(e){return null!==(e=function(e){var t=e.alternate;if(!t){if(null===(t=Be(e)))throw Error(i(188));return t!==e?null:e}for(var n=e,r=t;;){var a=n.return;if(null===a)break;var o=a.alternate;if(null===o){if(null!==(r=a.return)){n=r;continue}break}if(a.child===o.child){for(o=a.child;o;){if(o===n)return Ve(a),e;if(o===r)return Ve(a),t;o=o.sibling}throw Error(i(188))}if(n.return!==r.return)n=a,r=o;else{for(var u=!1,l=a.child;l;){if(l===n){u=!0,n=a,r=o;break}if(l===r){u=!0,r=a,n=o;break}l=l.sibling}if(!u){for(l=o.child;l;){if(l===n){u=!0,n=o,r=a;break}if(l===r){u=!0,r=o,n=a;break}l=l.sibling}if(!u)throw Error(i(189))}}if(n.alternate!==r)throw Error(i(190))}if(3!==n.tag)throw Error(i(188));return n.stateNode.current===n?e:t}(e))?qe(e):null}function qe(e){if(5===e.tag||6===e.tag)return e;for(e=e.child;null!==e;){var t=qe(e);if(null!==t)return t;e=e.sibling}return null}var Qe=a.unstable_scheduleCallback,Ye=a.unstable_cancelCallback,Ge=a.unstable_shouldYield,Ke=a.unstable_requestPaint,Ze=a.unstable_now,Xe=a.unstable_getCurrentPriorityLevel,Je=a.unstable_ImmediatePriority,et=a.unstable_UserBlockingPriority,tt=a.unstable_NormalPriority,nt=a.unstable_LowPriority,rt=a.unstable_IdlePriority,at=null,it=null,ot=Math.clz32?Math.clz32:function(e){return 0===(e>>>=0)?32:31-(ut(e)/lt|0)|0},ut=Math.log,lt=Math.LN2,st=64,ct=4194304;function ft(e){switch(e&-e){case 1:return 1;case 2:return 2;case 4:return 4;case 8:return 8;case 16:return 16;case 32:return 32;case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return 4194240&e;case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:return 130023424&e;case 134217728:return 134217728;case 268435456:return 268435456;case 536870912:return 536870912;case 1073741824:return 1073741824;default:return e}}function pt(e,t){var n=e.pendingLanes;if(0===n)return 0;var r=0,a=e.suspendedLanes,i=e.pingedLanes,o=268435455&n;if(0!==o){var u=o&~a;0!==u?r=ft(u):0!=(i&=o)&&(r=ft(i))}else 0!=(o=n&~a)?r=ft(o):0!==i&&(r=ft(i));if(0===r)return 0;if(0!==t&&t!==r&&0==(t&a)&&((a=r&-r)>=(i=t&-t)||16===a&&0!=(4194240&i)))return t;if(0!=(4&r)&&(r|=16&n),0!==(t=e.entangledLanes))for(e=e.entanglements,t&=r;0<t;)a=1<<(n=31-ot(t)),r|=e[n],t&=~a;return r}function dt(e,t){switch(e){case 1:case 2:case 4:return t+250;case 8:case 16:case 32:case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:return t+5e3;default:return-1}}function ht(e){return 0!=(e=-1073741825&e.pendingLanes)?e:1073741824&e?1073741824:0}function vt(){var e=st;return 0==(4194240&(st<<=1))&&(st=64),e}function gt(e){for(var t=[],n=0;31>n;n++)t.push(e);return t}function yt(e,t,n){e.pendingLanes|=t,536870912!==t&&(e.suspendedLanes=0,e.pingedLanes=0),(e=e.eventTimes)[t=31-ot(t)]=n}function mt(e,t){var n=e.entangledLanes|=t;for(e=e.entanglements;n;){var r=31-ot(n),a=1<<r;a&t|e[r]&t&&(e[r]|=t),n&=~a}}var bt=0;function _t(e){return 1<(e&=-e)?4<e?0!=(268435455&e)?16:536870912:4:1}var wt,xt,kt,St,Et,Ct=!1,Tt=[],Mt=null,Nt=null,Pt=null,zt=new Map,Lt=new Map,Ot=[],At=\"mousedown mouseup touchcancel touchend touchstart auxclick dblclick pointercancel pointerdown pointerup dragend dragstart drop compositionend compositionstart keydown keypress keyup input textInput copy cut paste click change contextmenu reset submit\".split(\" \");function Ft(e,t){switch(e){case\"focusin\":case\"focusout\":Mt=null;break;case\"dragenter\":case\"dragleave\":Nt=null;break;case\"mouseover\":case\"mouseout\":Pt=null;break;case\"pointerover\":case\"pointerout\":zt.delete(t.pointerId);break;case\"gotpointercapture\":case\"lostpointercapture\":Lt.delete(t.pointerId)}}function Dt(e,t,n,r,a,i){return null===e||e.nativeEvent!==i?(e={blockedOn:t,domEventName:n,eventSystemFlags:r,nativeEvent:i,targetContainers:[a]},null!==t&&null!==(t=ba(t))&&xt(t),e):(e.eventSystemFlags|=r,t=e.targetContainers,null!==a&&-1===t.indexOf(a)&&t.push(a),e)}function Rt(e){var t=ma(e.target);if(null!==t){var n=Be(t);if(null!==n)if(13===(t=n.tag)){if(null!==(t=We(n)))return e.blockedOn=t,void Et(e.priority,(function(){kt(n)}))}else if(3===t&&n.stateNode.current.memoizedState.isDehydrated)return void(e.blockedOn=3===n.tag?n.stateNode.containerInfo:null)}e.blockedOn=null}function jt(e){if(null!==e.blockedOn)return!1;for(var t=e.targetContainers;0<t.length;){var n=Gt(e.domEventName,e.eventSystemFlags,t[0],e.nativeEvent);if(null!==n)return null!==(t=ba(n))&&xt(t),e.blockedOn=n,!1;var r=new(n=e.nativeEvent).constructor(n.type,n);_e=r,n.target.dispatchEvent(r),_e=null,t.shift()}return!0}function Ut(e,t,n){jt(e)&&n.delete(t)}function It(){Ct=!1,null!==Mt&&jt(Mt)&&(Mt=null),null!==Nt&&jt(Nt)&&(Nt=null),null!==Pt&&jt(Pt)&&(Pt=null),zt.forEach(Ut),Lt.forEach(Ut)}function $t(e,t){e.blockedOn===t&&(e.blockedOn=null,Ct||(Ct=!0,a.unstable_scheduleCallback(a.unstable_NormalPriority,It)))}function Bt(e){function t(t){return $t(t,e)}if(0<Tt.length){$t(Tt[0],e);for(var n=1;n<Tt.length;n++){var r=Tt[n];r.blockedOn===e&&(r.blockedOn=null)}}for(null!==Mt&&$t(Mt,e),null!==Nt&&$t(Nt,e),null!==Pt&&$t(Pt,e),zt.forEach(t),Lt.forEach(t),n=0;n<Ot.length;n++)(r=Ot[n]).blockedOn===e&&(r.blockedOn=null);for(;0<Ot.length&&null===(n=Ot[0]).blockedOn;)Rt(n),null===n.blockedOn&&Ot.shift()}var Wt=_.ReactCurrentBatchConfig,Vt=!0;function Ht(e,t,n,r){var a=bt,i=Wt.transition;Wt.transition=null;try{bt=1,Qt(e,t,n,r)}finally{bt=a,Wt.transition=i}}function qt(e,t,n,r){var a=bt,i=Wt.transition;Wt.transition=null;try{bt=4,Qt(e,t,n,r)}finally{bt=a,Wt.transition=i}}function Qt(e,t,n,r){if(Vt){var a=Gt(e,t,n,r);if(null===a)Vr(e,t,r,Yt,n),Ft(e,r);else if(function(e,t,n,r,a){switch(t){case\"focusin\":return Mt=Dt(Mt,e,t,n,r,a),!0;case\"dragenter\":return Nt=Dt(Nt,e,t,n,r,a),!0;case\"mouseover\":return Pt=Dt(Pt,e,t,n,r,a),!0;case\"pointerover\":var i=a.pointerId;return zt.set(i,Dt(zt.get(i)||null,e,t,n,r,a)),!0;case\"gotpointercapture\":return i=a.pointerId,Lt.set(i,Dt(Lt.get(i)||null,e,t,n,r,a)),!0}return!1}(a,e,t,n,r))r.stopPropagation();else if(Ft(e,r),4&t&&-1<At.indexOf(e)){for(;null!==a;){var i=ba(a);if(null!==i&&wt(i),null===(i=Gt(e,t,n,r))&&Vr(e,t,r,Yt,n),i===a)break;a=i}null!==a&&r.stopPropagation()}else Vr(e,t,r,null,n)}}var Yt=null;function Gt(e,t,n,r){if(Yt=null,null!==(e=ma(e=we(r))))if(null===(t=Be(e)))e=null;else if(13===(n=t.tag)){if(null!==(e=We(t)))return e;e=null}else if(3===n){if(t.stateNode.current.memoizedState.isDehydrated)return 3===t.tag?t.stateNode.containerInfo:null;e=null}else t!==e&&(e=null);return Yt=e,null}function Kt(e){switch(e){case\"cancel\":case\"click\":case\"close\":case\"contextmenu\":case\"copy\":case\"cut\":case\"auxclick\":case\"dblclick\":case\"dragend\":case\"dragstart\":case\"drop\":case\"focusin\":case\"focusout\":case\"input\":case\"invalid\":case\"keydown\":case\"keypress\":case\"keyup\":case\"mousedown\":case\"mouseup\":case\"paste\":case\"pause\":case\"play\":case\"pointercancel\":case\"pointerdown\":case\"pointerup\":case\"ratechange\":case\"reset\":case\"resize\":case\"seeked\":case\"submit\":case\"touchcancel\":case\"touchend\":case\"touchstart\":case\"volumechange\":case\"change\":case\"selectionchange\":case\"textInput\":case\"compositionstart\":case\"compositionend\":case\"compositionupdate\":case\"beforeblur\":case\"afterblur\":case\"beforeinput\":case\"blur\":case\"fullscreenchange\":case\"focus\":case\"hashchange\":case\"popstate\":case\"select\":case\"selectstart\":return 1;case\"drag\":case\"dragenter\":case\"dragexit\":case\"dragleave\":case\"dragover\":case\"mousemove\":case\"mouseout\":case\"mouseover\":case\"pointermove\":case\"pointerout\":case\"pointerover\":case\"scroll\":case\"toggle\":case\"touchmove\":case\"wheel\":case\"mouseenter\":case\"mouseleave\":case\"pointerenter\":case\"pointerleave\":return 4;case\"message\":switch(Xe()){case Je:return 1;case et:return 4;case tt:case nt:return 16;case rt:return 536870912;default:return 16}default:return 16}}var Zt=null,Xt=null,Jt=null;function en(){if(Jt)return Jt;var e,t,n=Xt,r=n.length,a=\"value\"in Zt?Zt.value:Zt.textContent,i=a.length;for(e=0;e<r&&n[e]===a[e];e++);var o=r-e;for(t=1;t<=o&&n[r-t]===a[i-t];t++);return Jt=a.slice(e,1<t?1-t:void 0)}function tn(e){var t=e.keyCode;return\"charCode\"in e?0===(e=e.charCode)&&13===t&&(e=13):e=t,10===e&&(e=13),32<=e||13===e?e:0}function nn(){return!0}function rn(){return!1}function an(e){function t(t,n,r,a,i){for(var o in this._reactName=t,this._targetInst=r,this.type=n,this.nativeEvent=a,this.target=i,this.currentTarget=null,e)e.hasOwnProperty(o)&&(t=e[o],this[o]=t?t(a):a[o]);return this.isDefaultPrevented=(null!=a.defaultPrevented?a.defaultPrevented:!1===a.returnValue)?nn:rn,this.isPropagationStopped=rn,this}return R(t.prototype,{preventDefault:function(){this.defaultPrevented=!0;var e=this.nativeEvent;e&&(e.preventDefault?e.preventDefault():\"unknown\"!=typeof e.returnValue&&(e.returnValue=!1),this.isDefaultPrevented=nn)},stopPropagation:function(){var e=this.nativeEvent;e&&(e.stopPropagation?e.stopPropagation():\"unknown\"!=typeof e.cancelBubble&&(e.cancelBubble=!0),this.isPropagationStopped=nn)},persist:function(){},isPersistent:nn}),t}var on,un,ln,sn={eventPhase:0,bubbles:0,cancelable:0,timeStamp:function(e){return e.timeStamp||Date.now()},defaultPrevented:0,isTrusted:0},cn=an(sn),fn=R({},sn,{view:0,detail:0}),pn=an(fn),dn=R({},fn,{screenX:0,screenY:0,clientX:0,clientY:0,pageX:0,pageY:0,ctrlKey:0,shiftKey:0,altKey:0,metaKey:0,getModifierState:En,button:0,buttons:0,relatedTarget:function(e){return void 0===e.relatedTarget?e.fromElement===e.srcElement?e.toElement:e.fromElement:e.relatedTarget},movementX:function(e){return\"movementX\"in e?e.movementX:(e!==ln&&(ln&&\"mousemove\"===e.type?(on=e.screenX-ln.screenX,un=e.screenY-ln.screenY):un=on=0,ln=e),on)},movementY:function(e){return\"movementY\"in e?e.movementY:un}}),hn=an(dn),vn=an(R({},dn,{dataTransfer:0})),gn=an(R({},fn,{relatedTarget:0})),yn=an(R({},sn,{animationName:0,elapsedTime:0,pseudoElement:0})),mn=R({},sn,{clipboardData:function(e){return\"clipboardData\"in e?e.clipboardData:window.clipboardData}}),bn=an(mn),_n=an(R({},sn,{data:0})),wn={Esc:\"Escape\",Spacebar:\" \",Left:\"ArrowLeft\",Up:\"ArrowUp\",Right:\"ArrowRight\",Down:\"ArrowDown\",Del:\"Delete\",Win:\"OS\",Menu:\"ContextMenu\",Apps:\"ContextMenu\",Scroll:\"ScrollLock\",MozPrintableKey:\"Unidentified\"},xn={8:\"Backspace\",9:\"Tab\",12:\"Clear\",13:\"Enter\",16:\"Shift\",17:\"Control\",18:\"Alt\",19:\"Pause\",20:\"CapsLock\",27:\"Escape\",32:\" \",33:\"PageUp\",34:\"PageDown\",35:\"End\",36:\"Home\",37:\"ArrowLeft\",38:\"ArrowUp\",39:\"ArrowRight\",40:\"ArrowDown\",45:\"Insert\",46:\"Delete\",112:\"F1\",113:\"F2\",114:\"F3\",115:\"F4\",116:\"F5\",117:\"F6\",118:\"F7\",119:\"F8\",120:\"F9\",121:\"F10\",122:\"F11\",123:\"F12\",144:\"NumLock\",145:\"ScrollLock\",224:\"Meta\"},kn={Alt:\"altKey\",Control:\"ctrlKey\",Meta:\"metaKey\",Shift:\"shiftKey\"};function Sn(e){var t=this.nativeEvent;return t.getModifierState?t.getModifierState(e):!!(e=kn[e])&&!!t[e]}function En(){return Sn}var Cn=R({},fn,{key:function(e){if(e.key){var t=wn[e.key]||e.key;if(\"Unidentified\"!==t)return t}return\"keypress\"===e.type?13===(e=tn(e))?\"Enter\":String.fromCharCode(e):\"keydown\"===e.type||\"keyup\"===e.type?xn[e.keyCode]||\"Unidentified\":\"\"},code:0,location:0,ctrlKey:0,shiftKey:0,altKey:0,metaKey:0,repeat:0,locale:0,getModifierState:En,charCode:function(e){return\"keypress\"===e.type?tn(e):0},keyCode:function(e){return\"keydown\"===e.type||\"keyup\"===e.type?e.keyCode:0},which:function(e){return\"keypress\"===e.type?tn(e):\"keydown\"===e.type||\"keyup\"===e.type?e.keyCode:0}}),Tn=an(Cn),Mn=an(R({},dn,{pointerId:0,width:0,height:0,pressure:0,tangentialPressure:0,tiltX:0,tiltY:0,twist:0,pointerType:0,isPrimary:0})),Nn=an(R({},fn,{touches:0,targetTouches:0,changedTouches:0,altKey:0,metaKey:0,ctrlKey:0,shiftKey:0,getModifierState:En})),Pn=an(R({},sn,{propertyName:0,elapsedTime:0,pseudoElement:0})),zn=R({},dn,{deltaX:function(e){return\"deltaX\"in e?e.deltaX:\"wheelDeltaX\"in e?-e.wheelDeltaX:0},deltaY:function(e){return\"deltaY\"in e?e.deltaY:\"wheelDeltaY\"in e?-e.wheelDeltaY:\"wheelDelta\"in e?-e.wheelDelta:0},deltaZ:0,deltaMode:0}),Ln=an(zn),On=[9,13,27,32],An=c&&\"CompositionEvent\"in window,Fn=null;c&&\"documentMode\"in document&&(Fn=document.documentMode);var Dn=c&&\"TextEvent\"in window&&!Fn,Rn=c&&(!An||Fn&&8<Fn&&11>=Fn),jn=String.fromCharCode(32),Un=!1;function In(e,t){switch(e){case\"keyup\":return-1!==On.indexOf(t.keyCode);case\"keydown\":return 229!==t.keyCode;case\"keypress\":case\"mousedown\":case\"focusout\":return!0;default:return!1}}function $n(e){return\"object\"==typeof(e=e.detail)&&\"data\"in e?e.data:null}var Bn=!1,Wn={color:!0,date:!0,datetime:!0,\"datetime-local\":!0,email:!0,month:!0,number:!0,password:!0,range:!0,search:!0,tel:!0,text:!0,time:!0,url:!0,week:!0};function Vn(e){var t=e&&e.nodeName&&e.nodeName.toLowerCase();return\"input\"===t?!!Wn[e.type]:\"textarea\"===t}function Hn(e,t,n,r){Ce(r),0<(t=qr(t,\"onChange\")).length&&(n=new cn(\"onChange\",\"change\",null,n,r),e.push({event:n,listeners:t}))}var qn=null,Qn=null;function Yn(e){jr(e,0)}function Gn(e){if(Q(_a(e)))return e}function Kn(e,t){if(\"change\"===e)return t}var Zn=!1;if(c){var Xn;if(c){var Jn=\"oninput\"in document;if(!Jn){var er=document.createElement(\"div\");er.setAttribute(\"oninput\",\"return;\"),Jn=\"function\"==typeof er.oninput}Xn=Jn}else Xn=!1;Zn=Xn&&(!document.documentMode||9<document.documentMode)}function tr(){qn&&(qn.detachEvent(\"onpropertychange\",nr),Qn=qn=null)}function nr(e){if(\"value\"===e.propertyName&&Gn(Qn)){var t=[];Hn(t,Qn,e,we(e)),ze(Yn,t)}}function rr(e,t,n){\"focusin\"===e?(tr(),Qn=n,(qn=t).attachEvent(\"onpropertychange\",nr)):\"focusout\"===e&&tr()}function ar(e){if(\"selectionchange\"===e||\"keyup\"===e||\"keydown\"===e)return Gn(Qn)}function ir(e,t){if(\"click\"===e)return Gn(t)}function or(e,t){if(\"input\"===e||\"change\"===e)return Gn(t)}var ur=\"function\"==typeof Object.is?Object.is:function(e,t){return e===t&&(0!==e||1/e==1/t)||e!=e&&t!=t};function lr(e,t){if(ur(e,t))return!0;if(\"object\"!=typeof e||null===e||\"object\"!=typeof t||null===t)return!1;var n=Object.keys(e),r=Object.keys(t);if(n.length!==r.length)return!1;for(r=0;r<n.length;r++){var a=n[r];if(!f.call(t,a)||!ur(e[a],t[a]))return!1}return!0}function sr(e){for(;e&&e.firstChild;)e=e.firstChild;return e}function cr(e,t){var n,r=sr(e);for(e=0;r;){if(3===r.nodeType){if(n=e+r.textContent.length,e<=t&&n>=t)return{node:r,offset:t-e};e=n}e:{for(;r;){if(r.nextSibling){r=r.nextSibling;break e}r=r.parentNode}r=void 0}r=sr(r)}}function fr(e,t){return!(!e||!t)&&(e===t||(!e||3!==e.nodeType)&&(t&&3===t.nodeType?fr(e,t.parentNode):\"contains\"in e?e.contains(t):!!e.compareDocumentPosition&&!!(16&e.compareDocumentPosition(t))))}function pr(){for(var e=window,t=Y();t instanceof e.HTMLIFrameElement;){try{var n=\"string\"==typeof t.contentWindow.location.href}catch(e){n=!1}if(!n)break;t=Y((e=t.contentWindow).document)}return t}function dr(e){var t=e&&e.nodeName&&e.nodeName.toLowerCase();return t&&(\"input\"===t&&(\"text\"===e.type||\"search\"===e.type||\"tel\"===e.type||\"url\"===e.type||\"password\"===e.type)||\"textarea\"===t||\"true\"===e.contentEditable)}function hr(e){var t=pr(),n=e.focusedElem,r=e.selectionRange;if(t!==n&&n&&n.ownerDocument&&fr(n.ownerDocument.documentElement,n)){if(null!==r&&dr(n))if(t=r.start,void 0===(e=r.end)&&(e=t),\"selectionStart\"in n)n.selectionStart=t,n.selectionEnd=Math.min(e,n.value.length);else if((e=(t=n.ownerDocument||document)&&t.defaultView||window).getSelection){e=e.getSelection();var a=n.textContent.length,i=Math.min(r.start,a);r=void 0===r.end?i:Math.min(r.end,a),!e.extend&&i>r&&(a=r,r=i,i=a),a=cr(n,i);var o=cr(n,r);a&&o&&(1!==e.rangeCount||e.anchorNode!==a.node||e.anchorOffset!==a.offset||e.focusNode!==o.node||e.focusOffset!==o.offset)&&((t=t.createRange()).setStart(a.node,a.offset),e.removeAllRanges(),i>r?(e.addRange(t),e.extend(o.node,o.offset)):(t.setEnd(o.node,o.offset),e.addRange(t)))}for(t=[],e=n;e=e.parentNode;)1===e.nodeType&&t.push({element:e,left:e.scrollLeft,top:e.scrollTop});for(\"function\"==typeof n.focus&&n.focus(),n=0;n<t.length;n++)(e=t[n]).element.scrollLeft=e.left,e.element.scrollTop=e.top}}var vr=c&&\"documentMode\"in document&&11>=document.documentMode,gr=null,yr=null,mr=null,br=!1;function _r(e,t,n){var r=n.window===n?n.document:9===n.nodeType?n:n.ownerDocument;br||null==gr||gr!==Y(r)||(r=\"selectionStart\"in(r=gr)&&dr(r)?{start:r.selectionStart,end:r.selectionEnd}:{anchorNode:(r=(r.ownerDocument&&r.ownerDocument.defaultView||window).getSelection()).anchorNode,anchorOffset:r.anchorOffset,focusNode:r.focusNode,focusOffset:r.focusOffset},mr&&lr(mr,r)||(mr=r,0<(r=qr(yr,\"onSelect\")).length&&(t=new cn(\"onSelect\",\"select\",null,t,n),e.push({event:t,listeners:r}),t.target=gr)))}function wr(e,t){var n={};return n[e.toLowerCase()]=t.toLowerCase(),n[\"Webkit\"+e]=\"webkit\"+t,n[\"Moz\"+e]=\"moz\"+t,n}var xr={animationend:wr(\"Animation\",\"AnimationEnd\"),animationiteration:wr(\"Animation\",\"AnimationIteration\"),animationstart:wr(\"Animation\",\"AnimationStart\"),transitionend:wr(\"Transition\",\"TransitionEnd\")},kr={},Sr={};function Er(e){if(kr[e])return kr[e];if(!xr[e])return e;var t,n=xr[e];for(t in n)if(n.hasOwnProperty(t)&&t in Sr)return kr[e]=n[t];return e}c&&(Sr=document.createElement(\"div\").style,\"AnimationEvent\"in window||(delete xr.animationend.animation,delete xr.animationiteration.animation,delete xr.animationstart.animation),\"TransitionEvent\"in window||delete xr.transitionend.transition);var Cr=Er(\"animationend\"),Tr=Er(\"animationiteration\"),Mr=Er(\"animationstart\"),Nr=Er(\"transitionend\"),Pr=new Map,zr=\"abort auxClick cancel canPlay canPlayThrough click close contextMenu copy cut drag dragEnd dragEnter dragExit dragLeave dragOver dragStart drop durationChange emptied encrypted ended error gotPointerCapture input invalid keyDown keyPress keyUp load loadedData loadedMetadata loadStart lostPointerCapture mouseDown mouseMove mouseOut mouseOver mouseUp paste pause play playing pointerCancel pointerDown pointerMove pointerOut pointerOver pointerUp progress rateChange reset resize seeked seeking stalled submit suspend timeUpdate touchCancel touchEnd touchStart volumeChange scroll toggle touchMove waiting wheel\".split(\" \");function Lr(e,t){Pr.set(e,t),l(t,[e])}for(var Or=0;Or<zr.length;Or++){var Ar=zr[Or];Lr(Ar.toLowerCase(),\"on\"+(Ar[0].toUpperCase()+Ar.slice(1)))}Lr(Cr,\"onAnimationEnd\"),Lr(Tr,\"onAnimationIteration\"),Lr(Mr,\"onAnimationStart\"),Lr(\"dblclick\",\"onDoubleClick\"),Lr(\"focusin\",\"onFocus\"),Lr(\"focusout\",\"onBlur\"),Lr(Nr,\"onTransitionEnd\"),s(\"onMouseEnter\",[\"mouseout\",\"mouseover\"]),s(\"onMouseLeave\",[\"mouseout\",\"mouseover\"]),s(\"onPointerEnter\",[\"pointerout\",\"pointerover\"]),s(\"onPointerLeave\",[\"pointerout\",\"pointerover\"]),l(\"onChange\",\"change click focusin focusout input keydown keyup selectionchange\".split(\" \")),l(\"onSelect\",\"focusout contextmenu dragend focusin keydown keyup mousedown mouseup selectionchange\".split(\" \")),l(\"onBeforeInput\",[\"compositionend\",\"keypress\",\"textInput\",\"paste\"]),l(\"onCompositionEnd\",\"compositionend focusout keydown keypress keyup mousedown\".split(\" \")),l(\"onCompositionStart\",\"compositionstart focusout keydown keypress keyup mousedown\".split(\" \")),l(\"onCompositionUpdate\",\"compositionupdate focusout keydown keypress keyup mousedown\".split(\" \"));var Fr=\"abort canplay canplaythrough durationchange emptied encrypted ended error loadeddata loadedmetadata loadstart pause play playing progress ratechange resize seeked seeking stalled suspend timeupdate volumechange waiting\".split(\" \"),Dr=new Set(\"cancel close invalid load scroll toggle\".split(\" \").concat(Fr));function Rr(e,t,n){var r=e.type||\"unknown-event\";e.currentTarget=n,function(e,t,n,r,a,o,u,l,s){if($e.apply(this,arguments),De){if(!De)throw Error(i(198));var c=Re;De=!1,Re=null,je||(je=!0,Ue=c)}}(r,t,void 0,e),e.currentTarget=null}function jr(e,t){t=0!=(4&t);for(var n=0;n<e.length;n++){var r=e[n],a=r.event;r=r.listeners;e:{var i=void 0;if(t)for(var o=r.length-1;0<=o;o--){var u=r[o],l=u.instance,s=u.currentTarget;if(u=u.listener,l!==i&&a.isPropagationStopped())break e;Rr(a,u,s),i=l}else for(o=0;o<r.length;o++){if(l=(u=r[o]).instance,s=u.currentTarget,u=u.listener,l!==i&&a.isPropagationStopped())break e;Rr(a,u,s),i=l}}}if(je)throw e=Ue,je=!1,Ue=null,e}function Ur(e,t){var n=t[va];void 0===n&&(n=t[va]=new Set);var r=e+\"__bubble\";n.has(r)||(Wr(t,e,2,!1),n.add(r))}function Ir(e,t,n){var r=0;t&&(r|=4),Wr(n,e,r,t)}var $r=\"_reactListening\"+Math.random().toString(36).slice(2);function Br(e){if(!e[$r]){e[$r]=!0,o.forEach((function(t){\"selectionchange\"!==t&&(Dr.has(t)||Ir(t,!1,e),Ir(t,!0,e))}));var t=9===e.nodeType?e:e.ownerDocument;null===t||t[$r]||(t[$r]=!0,Ir(\"selectionchange\",!1,t))}}function Wr(e,t,n,r){switch(Kt(t)){case 1:var a=Ht;break;case 4:a=qt;break;default:a=Qt}n=a.bind(null,t,n,e),a=void 0,!Oe||\"touchstart\"!==t&&\"touchmove\"!==t&&\"wheel\"!==t||(a=!0),r?void 0!==a?e.addEventListener(t,n,{capture:!0,passive:a}):e.addEventListener(t,n,!0):void 0!==a?e.addEventListener(t,n,{passive:a}):e.addEventListener(t,n,!1)}function Vr(e,t,n,r,a){var i=r;if(0==(1&t)&&0==(2&t)&&null!==r)e:for(;;){if(null===r)return;var o=r.tag;if(3===o||4===o){var u=r.stateNode.containerInfo;if(u===a||8===u.nodeType&&u.parentNode===a)break;if(4===o)for(o=r.return;null!==o;){var l=o.tag;if((3===l||4===l)&&((l=o.stateNode.containerInfo)===a||8===l.nodeType&&l.parentNode===a))return;o=o.return}for(;null!==u;){if(null===(o=ma(u)))return;if(5===(l=o.tag)||6===l){r=i=o;continue e}u=u.parentNode}}r=r.return}ze((function(){var r=i,a=we(n),o=[];e:{var u=Pr.get(e);if(void 0!==u){var l=cn,s=e;switch(e){case\"keypress\":if(0===tn(n))break e;case\"keydown\":case\"keyup\":l=Tn;break;case\"focusin\":s=\"focus\",l=gn;break;case\"focusout\":s=\"blur\",l=gn;break;case\"beforeblur\":case\"afterblur\":l=gn;break;case\"click\":if(2===n.button)break e;case\"auxclick\":case\"dblclick\":case\"mousedown\":case\"mousemove\":case\"mouseup\":case\"mouseout\":case\"mouseover\":case\"contextmenu\":l=hn;break;case\"drag\":case\"dragend\":case\"dragenter\":case\"dragexit\":case\"dragleave\":case\"dragover\":case\"dragstart\":case\"drop\":l=vn;break;case\"touchcancel\":case\"touchend\":case\"touchmove\":case\"touchstart\":l=Nn;break;case Cr:case Tr:case Mr:l=yn;break;case Nr:l=Pn;break;case\"scroll\":l=pn;break;case\"wheel\":l=Ln;break;case\"copy\":case\"cut\":case\"paste\":l=bn;break;case\"gotpointercapture\":case\"lostpointercapture\":case\"pointercancel\":case\"pointerdown\":case\"pointermove\":case\"pointerout\":case\"pointerover\":case\"pointerup\":l=Mn}var c=0!=(4&t),f=!c&&\"scroll\"===e,p=c?null!==u?u+\"Capture\":null:u;c=[];for(var d,h=r;null!==h;){var v=(d=h).stateNode;if(5===d.tag&&null!==v&&(d=v,null!==p&&null!=(v=Le(h,p))&&c.push(Hr(h,v,d))),f)break;h=h.return}0<c.length&&(u=new l(u,s,null,n,a),o.push({event:u,listeners:c}))}}if(0==(7&t)){if(l=\"mouseout\"===e||\"pointerout\"===e,(!(u=\"mouseover\"===e||\"pointerover\"===e)||n===_e||!(s=n.relatedTarget||n.fromElement)||!ma(s)&&!s[ha])&&(l||u)&&(u=a.window===a?a:(u=a.ownerDocument)?u.defaultView||u.parentWindow:window,l?(l=r,null!==(s=(s=n.relatedTarget||n.toElement)?ma(s):null)&&(s!==(f=Be(s))||5!==s.tag&&6!==s.tag)&&(s=null)):(l=null,s=r),l!==s)){if(c=hn,v=\"onMouseLeave\",p=\"onMouseEnter\",h=\"mouse\",\"pointerout\"!==e&&\"pointerover\"!==e||(c=Mn,v=\"onPointerLeave\",p=\"onPointerEnter\",h=\"pointer\"),f=null==l?u:_a(l),d=null==s?u:_a(s),(u=new c(v,h+\"leave\",l,n,a)).target=f,u.relatedTarget=d,v=null,ma(a)===r&&((c=new c(p,h+\"enter\",s,n,a)).target=d,c.relatedTarget=f,v=c),f=v,l&&s)e:{for(p=s,h=0,d=c=l;d;d=Qr(d))h++;for(d=0,v=p;v;v=Qr(v))d++;for(;0<h-d;)c=Qr(c),h--;for(;0<d-h;)p=Qr(p),d--;for(;h--;){if(c===p||null!==p&&c===p.alternate)break e;c=Qr(c),p=Qr(p)}c=null}else c=null;null!==l&&Yr(o,u,l,c,!1),null!==s&&null!==f&&Yr(o,f,s,c,!0)}if(\"select\"===(l=(u=r?_a(r):window).nodeName&&u.nodeName.toLowerCase())||\"input\"===l&&\"file\"===u.type)var g=Kn;else if(Vn(u))if(Zn)g=or;else{g=ar;var y=rr}else(l=u.nodeName)&&\"input\"===l.toLowerCase()&&(\"checkbox\"===u.type||\"radio\"===u.type)&&(g=ir);switch(g&&(g=g(e,r))?Hn(o,g,n,a):(y&&y(e,u,r),\"focusout\"===e&&(y=u._wrapperState)&&y.controlled&&\"number\"===u.type&&ee(u,\"number\",u.value)),y=r?_a(r):window,e){case\"focusin\":(Vn(y)||\"true\"===y.contentEditable)&&(gr=y,yr=r,mr=null);break;case\"focusout\":mr=yr=gr=null;break;case\"mousedown\":br=!0;break;case\"contextmenu\":case\"mouseup\":case\"dragend\":br=!1,_r(o,n,a);break;case\"selectionchange\":if(vr)break;case\"keydown\":case\"keyup\":_r(o,n,a)}var m;if(An)e:{switch(e){case\"compositionstart\":var b=\"onCompositionStart\";break e;case\"compositionend\":b=\"onCompositionEnd\";break e;case\"compositionupdate\":b=\"onCompositionUpdate\";break e}b=void 0}else Bn?In(e,n)&&(b=\"onCompositionEnd\"):\"keydown\"===e&&229===n.keyCode&&(b=\"onCompositionStart\");b&&(Rn&&\"ko\"!==n.locale&&(Bn||\"onCompositionStart\"!==b?\"onCompositionEnd\"===b&&Bn&&(m=en()):(Xt=\"value\"in(Zt=a)?Zt.value:Zt.textContent,Bn=!0)),0<(y=qr(r,b)).length&&(b=new _n(b,e,null,n,a),o.push({event:b,listeners:y}),(m||null!==(m=$n(n)))&&(b.data=m))),(m=Dn?function(e,t){switch(e){case\"compositionend\":return $n(t);case\"keypress\":return 32!==t.which?null:(Un=!0,jn);case\"textInput\":return(e=t.data)===jn&&Un?null:e;default:return null}}(e,n):function(e,t){if(Bn)return\"compositionend\"===e||!An&&In(e,t)?(e=en(),Jt=Xt=Zt=null,Bn=!1,e):null;switch(e){case\"paste\":default:return null;case\"keypress\":if(!(t.ctrlKey||t.altKey||t.metaKey)||t.ctrlKey&&t.altKey){if(t.char&&1<t.char.length)return t.char;if(t.which)return String.fromCharCode(t.which)}return null;case\"compositionend\":return Rn&&\"ko\"!==t.locale?null:t.data}}(e,n))&&0<(r=qr(r,\"onBeforeInput\")).length&&(a=new _n(\"onBeforeInput\",\"beforeinput\",null,n,a),o.push({event:a,listeners:r}),a.data=m)}jr(o,t)}))}function Hr(e,t,n){return{instance:e,listener:t,currentTarget:n}}function qr(e,t){for(var n=t+\"Capture\",r=[];null!==e;){var a=e,i=a.stateNode;5===a.tag&&null!==i&&(a=i,null!=(i=Le(e,n))&&r.unshift(Hr(e,i,a)),null!=(i=Le(e,t))&&r.push(Hr(e,i,a))),e=e.return}return r}function Qr(e){if(null===e)return null;do{e=e.return}while(e&&5!==e.tag);return e||null}function Yr(e,t,n,r,a){for(var i=t._reactName,o=[];null!==n&&n!==r;){var u=n,l=u.alternate,s=u.stateNode;if(null!==l&&l===r)break;5===u.tag&&null!==s&&(u=s,a?null!=(l=Le(n,i))&&o.unshift(Hr(n,l,u)):a||null!=(l=Le(n,i))&&o.push(Hr(n,l,u))),n=n.return}0!==o.length&&e.push({event:t,listeners:o})}var Gr=/\\r\\n?/g,Kr=/\\u0000|\\uFFFD/g;function Zr(e){return(\"string\"==typeof e?e:\"\"+e).replace(Gr,\"\\n\").replace(Kr,\"\")}function Xr(e,t,n){if(t=Zr(t),Zr(e)!==t&&n)throw Error(i(425))}function Jr(){}var ea=null,ta=null;function na(e,t){return\"textarea\"===e||\"noscript\"===e||\"string\"==typeof t.children||\"number\"==typeof t.children||\"object\"==typeof t.dangerouslySetInnerHTML&&null!==t.dangerouslySetInnerHTML&&null!=t.dangerouslySetInnerHTML.__html}var ra=\"function\"==typeof setTimeout?setTimeout:void 0,aa=\"function\"==typeof clearTimeout?clearTimeout:void 0,ia=\"function\"==typeof Promise?Promise:void 0,oa=\"function\"==typeof queueMicrotask?queueMicrotask:void 0!==ia?function(e){return ia.resolve(null).then(e).catch(ua)}:ra;function ua(e){setTimeout((function(){throw e}))}function la(e,t){var n=t,r=0;do{var a=n.nextSibling;if(e.removeChild(n),a&&8===a.nodeType)if(\"/$\"===(n=a.data)){if(0===r)return e.removeChild(a),void Bt(t);r--}else\"$\"!==n&&\"$?\"!==n&&\"$!\"!==n||r++;n=a}while(n);Bt(t)}function sa(e){for(;null!=e;e=e.nextSibling){var t=e.nodeType;if(1===t||3===t)break;if(8===t){if(\"$\"===(t=e.data)||\"$!\"===t||\"$?\"===t)break;if(\"/$\"===t)return null}}return e}function ca(e){e=e.previousSibling;for(var t=0;e;){if(8===e.nodeType){var n=e.data;if(\"$\"===n||\"$!\"===n||\"$?\"===n){if(0===t)return e;t--}else\"/$\"===n&&t++}e=e.previousSibling}return null}var fa=Math.random().toString(36).slice(2),pa=\"__reactFiber$\"+fa,da=\"__reactProps$\"+fa,ha=\"__reactContainer$\"+fa,va=\"__reactEvents$\"+fa,ga=\"__reactListeners$\"+fa,ya=\"__reactHandles$\"+fa;function ma(e){var t=e[pa];if(t)return t;for(var n=e.parentNode;n;){if(t=n[ha]||n[pa]){if(n=t.alternate,null!==t.child||null!==n&&null!==n.child)for(e=ca(e);null!==e;){if(n=e[pa])return n;e=ca(e)}return t}n=(e=n).parentNode}return null}function ba(e){return!(e=e[pa]||e[ha])||5!==e.tag&&6!==e.tag&&13!==e.tag&&3!==e.tag?null:e}function _a(e){if(5===e.tag||6===e.tag)return e.stateNode;throw Error(i(33))}function wa(e){return e[da]||null}var xa=[],ka=-1;function Sa(e){return{current:e}}function Ea(e){0>ka||(e.current=xa[ka],xa[ka]=null,ka--)}function Ca(e,t){ka++,xa[ka]=e.current,e.current=t}var Ta={},Ma=Sa(Ta),Na=Sa(!1),Pa=Ta;function za(e,t){var n=e.type.contextTypes;if(!n)return Ta;var r=e.stateNode;if(r&&r.__reactInternalMemoizedUnmaskedChildContext===t)return r.__reactInternalMemoizedMaskedChildContext;var a,i={};for(a in n)i[a]=t[a];return r&&((e=e.stateNode).__reactInternalMemoizedUnmaskedChildContext=t,e.__reactInternalMemoizedMaskedChildContext=i),i}function La(e){return null!=e.childContextTypes}function Oa(){Ea(Na),Ea(Ma)}function Aa(e,t,n){if(Ma.current!==Ta)throw Error(i(168));Ca(Ma,t),Ca(Na,n)}function Fa(e,t,n){var r=e.stateNode;if(t=t.childContextTypes,\"function\"!=typeof r.getChildContext)return n;for(var a in r=r.getChildContext())if(!(a in t))throw Error(i(108,W(e)||\"Unknown\",a));return R({},n,r)}function Da(e){return e=(e=e.stateNode)&&e.__reactInternalMemoizedMergedChildContext||Ta,Pa=Ma.current,Ca(Ma,e),Ca(Na,Na.current),!0}function Ra(e,t,n){var r=e.stateNode;if(!r)throw Error(i(169));n?(e=Fa(e,t,Pa),r.__reactInternalMemoizedMergedChildContext=e,Ea(Na),Ea(Ma),Ca(Ma,e)):Ea(Na),Ca(Na,n)}var ja=null,Ua=!1,Ia=!1;function $a(e){null===ja?ja=[e]:ja.push(e)}function Ba(){if(!Ia&&null!==ja){Ia=!0;var e=0,t=bt;try{var n=ja;for(bt=1;e<n.length;e++){var r=n[e];do{r=r(!0)}while(null!==r)}ja=null,Ua=!1}catch(t){throw null!==ja&&(ja=ja.slice(e+1)),Qe(Je,Ba),t}finally{bt=t,Ia=!1}}return null}var Wa=[],Va=0,Ha=null,qa=0,Qa=[],Ya=0,Ga=null,Ka=1,Za=\"\";function Xa(e,t){Wa[Va++]=qa,Wa[Va++]=Ha,Ha=e,qa=t}function Ja(e,t,n){Qa[Ya++]=Ka,Qa[Ya++]=Za,Qa[Ya++]=Ga,Ga=e;var r=Ka;e=Za;var a=32-ot(r)-1;r&=~(1<<a),n+=1;var i=32-ot(t)+a;if(30<i){var o=a-a%5;i=(r&(1<<o)-1).toString(32),r>>=o,a-=o,Ka=1<<32-ot(t)+a|n<<a|r,Za=i+e}else Ka=1<<i|n<<a|r,Za=e}function ei(e){null!==e.return&&(Xa(e,1),Ja(e,1,0))}function ti(e){for(;e===Ha;)Ha=Wa[--Va],Wa[Va]=null,qa=Wa[--Va],Wa[Va]=null;for(;e===Ga;)Ga=Qa[--Ya],Qa[Ya]=null,Za=Qa[--Ya],Qa[Ya]=null,Ka=Qa[--Ya],Qa[Ya]=null}var ni=null,ri=null,ai=!1,ii=null;function oi(e,t){var n=Ls(5,null,null,0);n.elementType=\"DELETED\",n.stateNode=t,n.return=e,null===(t=e.deletions)?(e.deletions=[n],e.flags|=16):t.push(n)}function ui(e,t){switch(e.tag){case 5:var n=e.type;return null!==(t=1!==t.nodeType||n.toLowerCase()!==t.nodeName.toLowerCase()?null:t)&&(e.stateNode=t,ni=e,ri=sa(t.firstChild),!0);case 6:return null!==(t=\"\"===e.pendingProps||3!==t.nodeType?null:t)&&(e.stateNode=t,ni=e,ri=null,!0);case 13:return null!==(t=8!==t.nodeType?null:t)&&(n=null!==Ga?{id:Ka,overflow:Za}:null,e.memoizedState={dehydrated:t,treeContext:n,retryLane:1073741824},(n=Ls(18,null,null,0)).stateNode=t,n.return=e,e.child=n,ni=e,ri=null,!0);default:return!1}}function li(e){return 0!=(1&e.mode)&&0==(128&e.flags)}function si(e){if(ai){var t=ri;if(t){var n=t;if(!ui(e,t)){if(li(e))throw Error(i(418));t=sa(n.nextSibling);var r=ni;t&&ui(e,t)?oi(r,n):(e.flags=-4097&e.flags|2,ai=!1,ni=e)}}else{if(li(e))throw Error(i(418));e.flags=-4097&e.flags|2,ai=!1,ni=e}}}function ci(e){for(e=e.return;null!==e&&5!==e.tag&&3!==e.tag&&13!==e.tag;)e=e.return;ni=e}function fi(e){if(e!==ni)return!1;if(!ai)return ci(e),ai=!0,!1;var t;if((t=3!==e.tag)&&!(t=5!==e.tag)&&(t=\"head\"!==(t=e.type)&&\"body\"!==t&&!na(e.type,e.memoizedProps)),t&&(t=ri)){if(li(e))throw pi(),Error(i(418));for(;t;)oi(e,t),t=sa(t.nextSibling)}if(ci(e),13===e.tag){if(!(e=null!==(e=e.memoizedState)?e.dehydrated:null))throw Error(i(317));e:{for(e=e.nextSibling,t=0;e;){if(8===e.nodeType){var n=e.data;if(\"/$\"===n){if(0===t){ri=sa(e.nextSibling);break e}t--}else\"$\"!==n&&\"$!\"!==n&&\"$?\"!==n||t++}e=e.nextSibling}ri=null}}else ri=ni?sa(e.stateNode.nextSibling):null;return!0}function pi(){for(var e=ri;e;)e=sa(e.nextSibling)}function di(){ri=ni=null,ai=!1}function hi(e){null===ii?ii=[e]:ii.push(e)}var vi=_.ReactCurrentBatchConfig;function gi(e,t){if(e&&e.defaultProps){for(var n in t=R({},t),e=e.defaultProps)void 0===t[n]&&(t[n]=e[n]);return t}return t}var yi=Sa(null),mi=null,bi=null,_i=null;function wi(){_i=bi=mi=null}function xi(e){var t=yi.current;Ea(yi),e._currentValue=t}function ki(e,t,n){for(;null!==e;){var r=e.alternate;if((e.childLanes&t)!==t?(e.childLanes|=t,null!==r&&(r.childLanes|=t)):null!==r&&(r.childLanes&t)!==t&&(r.childLanes|=t),e===n)break;e=e.return}}function Si(e,t){mi=e,_i=bi=null,null!==(e=e.dependencies)&&null!==e.firstContext&&(0!=(e.lanes&t)&&(_u=!0),e.firstContext=null)}function Ei(e){var t=e._currentValue;if(_i!==e)if(e={context:e,memoizedValue:t,next:null},null===bi){if(null===mi)throw Error(i(308));bi=e,mi.dependencies={lanes:0,firstContext:e}}else bi=bi.next=e;return t}var Ci=null;function Ti(e){null===Ci?Ci=[e]:Ci.push(e)}function Mi(e,t,n,r){var a=t.interleaved;return null===a?(n.next=n,Ti(t)):(n.next=a.next,a.next=n),t.interleaved=n,Ni(e,r)}function Ni(e,t){e.lanes|=t;var n=e.alternate;for(null!==n&&(n.lanes|=t),n=e,e=e.return;null!==e;)e.childLanes|=t,null!==(n=e.alternate)&&(n.childLanes|=t),n=e,e=e.return;return 3===n.tag?n.stateNode:null}var Pi=!1;function zi(e){e.updateQueue={baseState:e.memoizedState,firstBaseUpdate:null,lastBaseUpdate:null,shared:{pending:null,interleaved:null,lanes:0},effects:null}}function Li(e,t){e=e.updateQueue,t.updateQueue===e&&(t.updateQueue={baseState:e.baseState,firstBaseUpdate:e.firstBaseUpdate,lastBaseUpdate:e.lastBaseUpdate,shared:e.shared,effects:e.effects})}function Oi(e,t){return{eventTime:e,lane:t,tag:0,payload:null,callback:null,next:null}}function Ai(e,t,n){var r=e.updateQueue;if(null===r)return null;if(r=r.shared,0!=(2&Nl)){var a=r.pending;return null===a?t.next=t:(t.next=a.next,a.next=t),r.pending=t,Ni(e,n)}return null===(a=r.interleaved)?(t.next=t,Ti(r)):(t.next=a.next,a.next=t),r.interleaved=t,Ni(e,n)}function Fi(e,t,n){if(null!==(t=t.updateQueue)&&(t=t.shared,0!=(4194240&n))){var r=t.lanes;n|=r&=e.pendingLanes,t.lanes=n,mt(e,n)}}function Di(e,t){var n=e.updateQueue,r=e.alternate;if(null!==r&&n===(r=r.updateQueue)){var a=null,i=null;if(null!==(n=n.firstBaseUpdate)){do{var o={eventTime:n.eventTime,lane:n.lane,tag:n.tag,payload:n.payload,callback:n.callback,next:null};null===i?a=i=o:i=i.next=o,n=n.next}while(null!==n);null===i?a=i=t:i=i.next=t}else a=i=t;return n={baseState:r.baseState,firstBaseUpdate:a,lastBaseUpdate:i,shared:r.shared,effects:r.effects},void(e.updateQueue=n)}null===(e=n.lastBaseUpdate)?n.firstBaseUpdate=t:e.next=t,n.lastBaseUpdate=t}function Ri(e,t,n,r){var a=e.updateQueue;Pi=!1;var i=a.firstBaseUpdate,o=a.lastBaseUpdate,u=a.shared.pending;if(null!==u){a.shared.pending=null;var l=u,s=l.next;l.next=null,null===o?i=s:o.next=s,o=l;var c=e.alternate;null!==c&&(u=(c=c.updateQueue).lastBaseUpdate)!==o&&(null===u?c.firstBaseUpdate=s:u.next=s,c.lastBaseUpdate=l)}if(null!==i){var f=a.baseState;for(o=0,c=s=l=null,u=i;;){var p=u.lane,d=u.eventTime;if((r&p)===p){null!==c&&(c=c.next={eventTime:d,lane:0,tag:u.tag,payload:u.payload,callback:u.callback,next:null});e:{var h=e,v=u;switch(p=t,d=n,v.tag){case 1:if(\"function\"==typeof(h=v.payload)){f=h.call(d,f,p);break e}f=h;break e;case 3:h.flags=-65537&h.flags|128;case 0:if(null==(p=\"function\"==typeof(h=v.payload)?h.call(d,f,p):h))break e;f=R({},f,p);break e;case 2:Pi=!0}}null!==u.callback&&0!==u.lane&&(e.flags|=64,null===(p=a.effects)?a.effects=[u]:p.push(u))}else d={eventTime:d,lane:p,tag:u.tag,payload:u.payload,callback:u.callback,next:null},null===c?(s=c=d,l=f):c=c.next=d,o|=p;if(null===(u=u.next)){if(null===(u=a.shared.pending))break;u=(p=u).next,p.next=null,a.lastBaseUpdate=p,a.shared.pending=null}}if(null===c&&(l=f),a.baseState=l,a.firstBaseUpdate=s,a.lastBaseUpdate=c,null!==(t=a.shared.interleaved)){a=t;do{o|=a.lane,a=a.next}while(a!==t)}else null===i&&(a.shared.lanes=0);Rl|=o,e.lanes=o,e.memoizedState=f}}function ji(e,t,n){if(e=t.effects,t.effects=null,null!==e)for(t=0;t<e.length;t++){var r=e[t],a=r.callback;if(null!==a){if(r.callback=null,r=n,\"function\"!=typeof a)throw Error(i(191,a));a.call(r)}}}var Ui=(new r.Component).refs;function Ii(e,t,n,r){n=null==(n=n(r,t=e.memoizedState))?t:R({},t,n),e.memoizedState=n,0===e.lanes&&(e.updateQueue.baseState=n)}var $i={isMounted:function(e){return!!(e=e._reactInternals)&&Be(e)===e},enqueueSetState:function(e,t,n){e=e._reactInternals;var r=ts(),a=ns(e),i=Oi(r,a);i.payload=t,null!=n&&(i.callback=n),null!==(t=Ai(e,i,a))&&(rs(t,e,a,r),Fi(t,e,a))},enqueueReplaceState:function(e,t,n){e=e._reactInternals;var r=ts(),a=ns(e),i=Oi(r,a);i.tag=1,i.payload=t,null!=n&&(i.callback=n),null!==(t=Ai(e,i,a))&&(rs(t,e,a,r),Fi(t,e,a))},enqueueForceUpdate:function(e,t){e=e._reactInternals;var n=ts(),r=ns(e),a=Oi(n,r);a.tag=2,null!=t&&(a.callback=t),null!==(t=Ai(e,a,r))&&(rs(t,e,r,n),Fi(t,e,r))}};function Bi(e,t,n,r,a,i,o){return\"function\"==typeof(e=e.stateNode).shouldComponentUpdate?e.shouldComponentUpdate(r,i,o):!(t.prototype&&t.prototype.isPureReactComponent&&lr(n,r)&&lr(a,i))}function Wi(e,t,n){var r=!1,a=Ta,i=t.contextType;return\"object\"==typeof i&&null!==i?i=Ei(i):(a=La(t)?Pa:Ma.current,i=(r=null!=(r=t.contextTypes))?za(e,a):Ta),t=new t(n,i),e.memoizedState=null!==t.state&&void 0!==t.state?t.state:null,t.updater=$i,e.stateNode=t,t._reactInternals=e,r&&((e=e.stateNode).__reactInternalMemoizedUnmaskedChildContext=a,e.__reactInternalMemoizedMaskedChildContext=i),t}function Vi(e,t,n,r){e=t.state,\"function\"==typeof t.componentWillReceiveProps&&t.componentWillReceiveProps(n,r),\"function\"==typeof t.UNSAFE_componentWillReceiveProps&&t.UNSAFE_componentWillReceiveProps(n,r),t.state!==e&&$i.enqueueReplaceState(t,t.state,null)}function Hi(e,t,n,r){var a=e.stateNode;a.props=n,a.state=e.memoizedState,a.refs=Ui,zi(e);var i=t.contextType;\"object\"==typeof i&&null!==i?a.context=Ei(i):(i=La(t)?Pa:Ma.current,a.context=za(e,i)),a.state=e.memoizedState,\"function\"==typeof(i=t.getDerivedStateFromProps)&&(Ii(e,t,i,n),a.state=e.memoizedState),\"function\"==typeof t.getDerivedStateFromProps||\"function\"==typeof a.getSnapshotBeforeUpdate||\"function\"!=typeof a.UNSAFE_componentWillMount&&\"function\"!=typeof a.componentWillMount||(t=a.state,\"function\"==typeof a.componentWillMount&&a.componentWillMount(),\"function\"==typeof a.UNSAFE_componentWillMount&&a.UNSAFE_componentWillMount(),t!==a.state&&$i.enqueueReplaceState(a,a.state,null),Ri(e,n,a,r),a.state=e.memoizedState),\"function\"==typeof a.componentDidMount&&(e.flags|=4194308)}function qi(e,t,n){if(null!==(e=n.ref)&&\"function\"!=typeof e&&\"object\"!=typeof e){if(n._owner){if(n=n._owner){if(1!==n.tag)throw Error(i(309));var r=n.stateNode}if(!r)throw Error(i(147,e));var a=r,o=\"\"+e;return null!==t&&null!==t.ref&&\"function\"==typeof t.ref&&t.ref._stringRef===o?t.ref:(t=function(e){var t=a.refs;t===Ui&&(t=a.refs={}),null===e?delete t[o]:t[o]=e},t._stringRef=o,t)}if(\"string\"!=typeof e)throw Error(i(284));if(!n._owner)throw Error(i(290,e))}return e}function Qi(e,t){throw e=Object.prototype.toString.call(t),Error(i(31,\"[object Object]\"===e?\"object with keys {\"+Object.keys(t).join(\", \")+\"}\":e))}function Yi(e){return(0,e._init)(e._payload)}function Gi(e){function t(t,n){if(e){var r=t.deletions;null===r?(t.deletions=[n],t.flags|=16):r.push(n)}}function n(n,r){if(!e)return null;for(;null!==r;)t(n,r),r=r.sibling;return null}function r(e,t){for(e=new Map;null!==t;)null!==t.key?e.set(t.key,t):e.set(t.index,t),t=t.sibling;return e}function a(e,t){return(e=As(e,t)).index=0,e.sibling=null,e}function o(t,n,r){return t.index=r,e?null!==(r=t.alternate)?(r=r.index)<n?(t.flags|=2,n):r:(t.flags|=2,n):(t.flags|=1048576,n)}function u(t){return e&&null===t.alternate&&(t.flags|=2),t}function l(e,t,n,r){return null===t||6!==t.tag?((t=js(n,e.mode,r)).return=e,t):((t=a(t,n)).return=e,t)}function s(e,t,n,r){var i=n.type;return i===k?f(e,t,n.props.children,r,n.key):null!==t&&(t.elementType===i||\"object\"==typeof i&&null!==i&&i.$$typeof===L&&Yi(i)===t.type)?((r=a(t,n.props)).ref=qi(e,t,n),r.return=e,r):((r=Fs(n.type,n.key,n.props,null,e.mode,r)).ref=qi(e,t,n),r.return=e,r)}function c(e,t,n,r){return null===t||4!==t.tag||t.stateNode.containerInfo!==n.containerInfo||t.stateNode.implementation!==n.implementation?((t=Us(n,e.mode,r)).return=e,t):((t=a(t,n.children||[])).return=e,t)}function f(e,t,n,r,i){return null===t||7!==t.tag?((t=Ds(n,e.mode,r,i)).return=e,t):((t=a(t,n)).return=e,t)}function p(e,t,n){if(\"string\"==typeof t&&\"\"!==t||\"number\"==typeof t)return(t=js(\"\"+t,e.mode,n)).return=e,t;if(\"object\"==typeof t&&null!==t){switch(t.$$typeof){case w:return(n=Fs(t.type,t.key,t.props,null,e.mode,n)).ref=qi(e,null,t),n.return=e,n;case x:return(t=Us(t,e.mode,n)).return=e,t;case L:return p(e,(0,t._init)(t._payload),n)}if(te(t)||F(t))return(t=Ds(t,e.mode,n,null)).return=e,t;Qi(e,t)}return null}function d(e,t,n,r){var a=null!==t?t.key:null;if(\"string\"==typeof n&&\"\"!==n||\"number\"==typeof n)return null!==a?null:l(e,t,\"\"+n,r);if(\"object\"==typeof n&&null!==n){switch(n.$$typeof){case w:return n.key===a?s(e,t,n,r):null;case x:return n.key===a?c(e,t,n,r):null;case L:return d(e,t,(a=n._init)(n._payload),r)}if(te(n)||F(n))return null!==a?null:f(e,t,n,r,null);Qi(e,n)}return null}function h(e,t,n,r,a){if(\"string\"==typeof r&&\"\"!==r||\"number\"==typeof r)return l(t,e=e.get(n)||null,\"\"+r,a);if(\"object\"==typeof r&&null!==r){switch(r.$$typeof){case w:return s(t,e=e.get(null===r.key?n:r.key)||null,r,a);case x:return c(t,e=e.get(null===r.key?n:r.key)||null,r,a);case L:return h(e,t,n,(0,r._init)(r._payload),a)}if(te(r)||F(r))return f(t,e=e.get(n)||null,r,a,null);Qi(t,r)}return null}function v(a,i,u,l){for(var s=null,c=null,f=i,v=i=0,g=null;null!==f&&v<u.length;v++){f.index>v?(g=f,f=null):g=f.sibling;var y=d(a,f,u[v],l);if(null===y){null===f&&(f=g);break}e&&f&&null===y.alternate&&t(a,f),i=o(y,i,v),null===c?s=y:c.sibling=y,c=y,f=g}if(v===u.length)return n(a,f),ai&&Xa(a,v),s;if(null===f){for(;v<u.length;v++)null!==(f=p(a,u[v],l))&&(i=o(f,i,v),null===c?s=f:c.sibling=f,c=f);return ai&&Xa(a,v),s}for(f=r(a,f);v<u.length;v++)null!==(g=h(f,a,v,u[v],l))&&(e&&null!==g.alternate&&f.delete(null===g.key?v:g.key),i=o(g,i,v),null===c?s=g:c.sibling=g,c=g);return e&&f.forEach((function(e){return t(a,e)})),ai&&Xa(a,v),s}function g(a,u,l,s){var c=F(l);if(\"function\"!=typeof c)throw Error(i(150));if(null==(l=c.call(l)))throw Error(i(151));for(var f=c=null,v=u,g=u=0,y=null,m=l.next();null!==v&&!m.done;g++,m=l.next()){v.index>g?(y=v,v=null):y=v.sibling;var b=d(a,v,m.value,s);if(null===b){null===v&&(v=y);break}e&&v&&null===b.alternate&&t(a,v),u=o(b,u,g),null===f?c=b:f.sibling=b,f=b,v=y}if(m.done)return n(a,v),ai&&Xa(a,g),c;if(null===v){for(;!m.done;g++,m=l.next())null!==(m=p(a,m.value,s))&&(u=o(m,u,g),null===f?c=m:f.sibling=m,f=m);return ai&&Xa(a,g),c}for(v=r(a,v);!m.done;g++,m=l.next())null!==(m=h(v,a,g,m.value,s))&&(e&&null!==m.alternate&&v.delete(null===m.key?g:m.key),u=o(m,u,g),null===f?c=m:f.sibling=m,f=m);return e&&v.forEach((function(e){return t(a,e)})),ai&&Xa(a,g),c}return function e(r,i,o,l){if(\"object\"==typeof o&&null!==o&&o.type===k&&null===o.key&&(o=o.props.children),\"object\"==typeof o&&null!==o){switch(o.$$typeof){case w:e:{for(var s=o.key,c=i;null!==c;){if(c.key===s){if((s=o.type)===k){if(7===c.tag){n(r,c.sibling),(i=a(c,o.props.children)).return=r,r=i;break e}}else if(c.elementType===s||\"object\"==typeof s&&null!==s&&s.$$typeof===L&&Yi(s)===c.type){n(r,c.sibling),(i=a(c,o.props)).ref=qi(r,c,o),i.return=r,r=i;break e}n(r,c);break}t(r,c),c=c.sibling}o.type===k?((i=Ds(o.props.children,r.mode,l,o.key)).return=r,r=i):((l=Fs(o.type,o.key,o.props,null,r.mode,l)).ref=qi(r,i,o),l.return=r,r=l)}return u(r);case x:e:{for(c=o.key;null!==i;){if(i.key===c){if(4===i.tag&&i.stateNode.containerInfo===o.containerInfo&&i.stateNode.implementation===o.implementation){n(r,i.sibling),(i=a(i,o.children||[])).return=r,r=i;break e}n(r,i);break}t(r,i),i=i.sibling}(i=Us(o,r.mode,l)).return=r,r=i}return u(r);case L:return e(r,i,(c=o._init)(o._payload),l)}if(te(o))return v(r,i,o,l);if(F(o))return g(r,i,o,l);Qi(r,o)}return\"string\"==typeof o&&\"\"!==o||\"number\"==typeof o?(o=\"\"+o,null!==i&&6===i.tag?(n(r,i.sibling),(i=a(i,o)).return=r,r=i):(n(r,i),(i=js(o,r.mode,l)).return=r,r=i),u(r)):n(r,i)}}var Ki=Gi(!0),Zi=Gi(!1),Xi={},Ji=Sa(Xi),eo=Sa(Xi),to=Sa(Xi);function no(e){if(e===Xi)throw Error(i(174));return e}function ro(e,t){switch(Ca(to,t),Ca(eo,e),Ca(Ji,Xi),e=t.nodeType){case 9:case 11:t=(t=t.documentElement)?t.namespaceURI:le(null,\"\");break;default:t=le(t=(e=8===e?t.parentNode:t).namespaceURI||null,e=e.tagName)}Ea(Ji),Ca(Ji,t)}function ao(){Ea(Ji),Ea(eo),Ea(to)}function io(e){no(to.current);var t=no(Ji.current),n=le(t,e.type);t!==n&&(Ca(eo,e),Ca(Ji,n))}function oo(e){eo.current===e&&(Ea(Ji),Ea(eo))}var uo=Sa(0);function lo(e){for(var t=e;null!==t;){if(13===t.tag){var n=t.memoizedState;if(null!==n&&(null===(n=n.dehydrated)||\"$?\"===n.data||\"$!\"===n.data))return t}else if(19===t.tag&&void 0!==t.memoizedProps.revealOrder){if(0!=(128&t.flags))return t}else if(null!==t.child){t.child.return=t,t=t.child;continue}if(t===e)break;for(;null===t.sibling;){if(null===t.return||t.return===e)return null;t=t.return}t.sibling.return=t.return,t=t.sibling}return null}var so=[];function co(){for(var e=0;e<so.length;e++)so[e]._workInProgressVersionPrimary=null;so.length=0}var fo=_.ReactCurrentDispatcher,po=_.ReactCurrentBatchConfig,ho=0,vo=null,go=null,yo=null,mo=!1,bo=!1,_o=0,wo=0;function xo(){throw Error(i(321))}function ko(e,t){if(null===t)return!1;for(var n=0;n<t.length&&n<e.length;n++)if(!ur(e[n],t[n]))return!1;return!0}function So(e,t,n,r,a,o){if(ho=o,vo=t,t.memoizedState=null,t.updateQueue=null,t.lanes=0,fo.current=null===e||null===e.memoizedState?uu:lu,e=n(r,a),bo){o=0;do{if(bo=!1,_o=0,25<=o)throw Error(i(301));o+=1,yo=go=null,t.updateQueue=null,fo.current=su,e=n(r,a)}while(bo)}if(fo.current=ou,t=null!==go&&null!==go.next,ho=0,yo=go=vo=null,mo=!1,t)throw Error(i(300));return e}function Eo(){var e=0!==_o;return _o=0,e}function Co(){var e={memoizedState:null,baseState:null,baseQueue:null,queue:null,next:null};return null===yo?vo.memoizedState=yo=e:yo=yo.next=e,yo}function To(){if(null===go){var e=vo.alternate;e=null!==e?e.memoizedState:null}else e=go.next;var t=null===yo?vo.memoizedState:yo.next;if(null!==t)yo=t,go=e;else{if(null===e)throw Error(i(310));e={memoizedState:(go=e).memoizedState,baseState:go.baseState,baseQueue:go.baseQueue,queue:go.queue,next:null},null===yo?vo.memoizedState=yo=e:yo=yo.next=e}return yo}function Mo(e,t){return\"function\"==typeof t?t(e):t}function No(e){var t=To(),n=t.queue;if(null===n)throw Error(i(311));n.lastRenderedReducer=e;var r=go,a=r.baseQueue,o=n.pending;if(null!==o){if(null!==a){var u=a.next;a.next=o.next,o.next=u}r.baseQueue=a=o,n.pending=null}if(null!==a){o=a.next,r=r.baseState;var l=u=null,s=null,c=o;do{var f=c.lane;if((ho&f)===f)null!==s&&(s=s.next={lane:0,action:c.action,hasEagerState:c.hasEagerState,eagerState:c.eagerState,next:null}),r=c.hasEagerState?c.eagerState:e(r,c.action);else{var p={lane:f,action:c.action,hasEagerState:c.hasEagerState,eagerState:c.eagerState,next:null};null===s?(l=s=p,u=r):s=s.next=p,vo.lanes|=f,Rl|=f}c=c.next}while(null!==c&&c!==o);null===s?u=r:s.next=l,ur(r,t.memoizedState)||(_u=!0),t.memoizedState=r,t.baseState=u,t.baseQueue=s,n.lastRenderedState=r}if(null!==(e=n.interleaved)){a=e;do{o=a.lane,vo.lanes|=o,Rl|=o,a=a.next}while(a!==e)}else null===a&&(n.lanes=0);return[t.memoizedState,n.dispatch]}function Po(e){var t=To(),n=t.queue;if(null===n)throw Error(i(311));n.lastRenderedReducer=e;var r=n.dispatch,a=n.pending,o=t.memoizedState;if(null!==a){n.pending=null;var u=a=a.next;do{o=e(o,u.action),u=u.next}while(u!==a);ur(o,t.memoizedState)||(_u=!0),t.memoizedState=o,null===t.baseQueue&&(t.baseState=o),n.lastRenderedState=o}return[o,r]}function zo(){}function Lo(e,t){var n=vo,r=To(),a=t(),o=!ur(r.memoizedState,a);if(o&&(r.memoizedState=a,_u=!0),r=r.queue,Vo(Fo.bind(null,n,r,e),[e]),r.getSnapshot!==t||o||null!==yo&&1&yo.memoizedState.tag){if(n.flags|=2048,Uo(9,Ao.bind(null,n,r,a,t),void 0,null),null===Pl)throw Error(i(349));0!=(30&ho)||Oo(n,t,a)}return a}function Oo(e,t,n){e.flags|=16384,e={getSnapshot:t,value:n},null===(t=vo.updateQueue)?(t={lastEffect:null,stores:null},vo.updateQueue=t,t.stores=[e]):null===(n=t.stores)?t.stores=[e]:n.push(e)}function Ao(e,t,n,r){t.value=n,t.getSnapshot=r,Do(t)&&Ro(e)}function Fo(e,t,n){return n((function(){Do(t)&&Ro(e)}))}function Do(e){var t=e.getSnapshot;e=e.value;try{var n=t();return!ur(e,n)}catch(e){return!0}}function Ro(e){var t=Ni(e,1);null!==t&&rs(t,e,1,-1)}function jo(e){var t=Co();return\"function\"==typeof e&&(e=e()),t.memoizedState=t.baseState=e,e={pending:null,interleaved:null,lanes:0,dispatch:null,lastRenderedReducer:Mo,lastRenderedState:e},t.queue=e,e=e.dispatch=nu.bind(null,vo,e),[t.memoizedState,e]}function Uo(e,t,n,r){return e={tag:e,create:t,destroy:n,deps:r,next:null},null===(t=vo.updateQueue)?(t={lastEffect:null,stores:null},vo.updateQueue=t,t.lastEffect=e.next=e):null===(n=t.lastEffect)?t.lastEffect=e.next=e:(r=n.next,n.next=e,e.next=r,t.lastEffect=e),e}function Io(){return To().memoizedState}function $o(e,t,n,r){var a=Co();vo.flags|=e,a.memoizedState=Uo(1|t,n,void 0,void 0===r?null:r)}function Bo(e,t,n,r){var a=To();r=void 0===r?null:r;var i=void 0;if(null!==go){var o=go.memoizedState;if(i=o.destroy,null!==r&&ko(r,o.deps))return void(a.memoizedState=Uo(t,n,i,r))}vo.flags|=e,a.memoizedState=Uo(1|t,n,i,r)}function Wo(e,t){return $o(8390656,8,e,t)}function Vo(e,t){return Bo(2048,8,e,t)}function Ho(e,t){return Bo(4,2,e,t)}function qo(e,t){return Bo(4,4,e,t)}function Qo(e,t){return\"function\"==typeof t?(e=e(),t(e),function(){t(null)}):null!=t?(e=e(),t.current=e,function(){t.current=null}):void 0}function Yo(e,t,n){return n=null!=n?n.concat([e]):null,Bo(4,4,Qo.bind(null,t,e),n)}function Go(){}function Ko(e,t){var n=To();t=void 0===t?null:t;var r=n.memoizedState;return null!==r&&null!==t&&ko(t,r[1])?r[0]:(n.memoizedState=[e,t],e)}function Zo(e,t){var n=To();t=void 0===t?null:t;var r=n.memoizedState;return null!==r&&null!==t&&ko(t,r[1])?r[0]:(e=e(),n.memoizedState=[e,t],e)}function Xo(e,t,n){return 0==(21&ho)?(e.baseState&&(e.baseState=!1,_u=!0),e.memoizedState=n):(ur(n,t)||(n=vt(),vo.lanes|=n,Rl|=n,e.baseState=!0),t)}function Jo(e,t){var n=bt;bt=0!==n&&4>n?n:4,e(!0);var r=po.transition;po.transition={};try{e(!1),t()}finally{bt=n,po.transition=r}}function eu(){return To().memoizedState}function tu(e,t,n){var r=ns(e);n={lane:r,action:n,hasEagerState:!1,eagerState:null,next:null},ru(e)?au(t,n):null!==(n=Mi(e,t,n,r))&&(rs(n,e,r,ts()),iu(n,t,r))}function nu(e,t,n){var r=ns(e),a={lane:r,action:n,hasEagerState:!1,eagerState:null,next:null};if(ru(e))au(t,a);else{var i=e.alternate;if(0===e.lanes&&(null===i||0===i.lanes)&&null!==(i=t.lastRenderedReducer))try{var o=t.lastRenderedState,u=i(o,n);if(a.hasEagerState=!0,a.eagerState=u,ur(u,o)){var l=t.interleaved;return null===l?(a.next=a,Ti(t)):(a.next=l.next,l.next=a),void(t.interleaved=a)}}catch(e){}null!==(n=Mi(e,t,a,r))&&(rs(n,e,r,a=ts()),iu(n,t,r))}}function ru(e){var t=e.alternate;return e===vo||null!==t&&t===vo}function au(e,t){bo=mo=!0;var n=e.pending;null===n?t.next=t:(t.next=n.next,n.next=t),e.pending=t}function iu(e,t,n){if(0!=(4194240&n)){var r=t.lanes;n|=r&=e.pendingLanes,t.lanes=n,mt(e,n)}}var ou={readContext:Ei,useCallback:xo,useContext:xo,useEffect:xo,useImperativeHandle:xo,useInsertionEffect:xo,useLayoutEffect:xo,useMemo:xo,useReducer:xo,useRef:xo,useState:xo,useDebugValue:xo,useDeferredValue:xo,useTransition:xo,useMutableSource:xo,useSyncExternalStore:xo,useId:xo,unstable_isNewReconciler:!1},uu={readContext:Ei,useCallback:function(e,t){return Co().memoizedState=[e,void 0===t?null:t],e},useContext:Ei,useEffect:Wo,useImperativeHandle:function(e,t,n){return n=null!=n?n.concat([e]):null,$o(4194308,4,Qo.bind(null,t,e),n)},useLayoutEffect:function(e,t){return $o(4194308,4,e,t)},useInsertionEffect:function(e,t){return $o(4,2,e,t)},useMemo:function(e,t){var n=Co();return t=void 0===t?null:t,e=e(),n.memoizedState=[e,t],e},useReducer:function(e,t,n){var r=Co();return t=void 0!==n?n(t):t,r.memoizedState=r.baseState=t,e={pending:null,interleaved:null,lanes:0,dispatch:null,lastRenderedReducer:e,lastRenderedState:t},r.queue=e,e=e.dispatch=tu.bind(null,vo,e),[r.memoizedState,e]},useRef:function(e){return e={current:e},Co().memoizedState=e},useState:jo,useDebugValue:Go,useDeferredValue:function(e){return Co().memoizedState=e},useTransition:function(){var e=jo(!1),t=e[0];return e=Jo.bind(null,e[1]),Co().memoizedState=e,[t,e]},useMutableSource:function(){},useSyncExternalStore:function(e,t,n){var r=vo,a=Co();if(ai){if(void 0===n)throw Error(i(407));n=n()}else{if(n=t(),null===Pl)throw Error(i(349));0!=(30&ho)||Oo(r,t,n)}a.memoizedState=n;var o={value:n,getSnapshot:t};return a.queue=o,Wo(Fo.bind(null,r,o,e),[e]),r.flags|=2048,Uo(9,Ao.bind(null,r,o,n,t),void 0,null),n},useId:function(){var e=Co(),t=Pl.identifierPrefix;if(ai){var n=Za;t=\":\"+t+\"R\"+(n=(Ka&~(1<<32-ot(Ka)-1)).toString(32)+n),0<(n=_o++)&&(t+=\"H\"+n.toString(32)),t+=\":\"}else t=\":\"+t+\"r\"+(n=wo++).toString(32)+\":\";return e.memoizedState=t},unstable_isNewReconciler:!1},lu={readContext:Ei,useCallback:Ko,useContext:Ei,useEffect:Vo,useImperativeHandle:Yo,useInsertionEffect:Ho,useLayoutEffect:qo,useMemo:Zo,useReducer:No,useRef:Io,useState:function(){return No(Mo)},useDebugValue:Go,useDeferredValue:function(e){return Xo(To(),go.memoizedState,e)},useTransition:function(){return[No(Mo)[0],To().memoizedState]},useMutableSource:zo,useSyncExternalStore:Lo,useId:eu,unstable_isNewReconciler:!1},su={readContext:Ei,useCallback:Ko,useContext:Ei,useEffect:Vo,useImperativeHandle:Yo,useInsertionEffect:Ho,useLayoutEffect:qo,useMemo:Zo,useReducer:Po,useRef:Io,useState:function(){return Po(Mo)},useDebugValue:Go,useDeferredValue:function(e){var t=To();return null===go?t.memoizedState=e:Xo(t,go.memoizedState,e)},useTransition:function(){return[Po(Mo)[0],To().memoizedState]},useMutableSource:zo,useSyncExternalStore:Lo,useId:eu,unstable_isNewReconciler:!1};function cu(e,t){try{var n=\"\",r=t;do{n+=$(r),r=r.return}while(r);var a=n}catch(e){a=\"\\nError generating stack: \"+e.message+\"\\n\"+e.stack}return{value:e,source:t,stack:a,digest:null}}function fu(e,t,n){return{value:e,source:null,stack:null!=n?n:null,digest:null!=t?t:null}}function pu(e,t){try{console.error(t.value)}catch(e){setTimeout((function(){throw e}))}}var du=\"function\"==typeof WeakMap?WeakMap:Map;function hu(e,t,n){(n=Oi(-1,n)).tag=3,n.payload={element:null};var r=t.value;return n.callback=function(){Hl||(Hl=!0,ql=r),pu(0,t)},n}function vu(e,t,n){(n=Oi(-1,n)).tag=3;var r=e.type.getDerivedStateFromError;if(\"function\"==typeof r){var a=t.value;n.payload=function(){return r(a)},n.callback=function(){pu(0,t)}}var i=e.stateNode;return null!==i&&\"function\"==typeof i.componentDidCatch&&(n.callback=function(){pu(0,t),\"function\"!=typeof r&&(null===Ql?Ql=new Set([this]):Ql.add(this));var e=t.stack;this.componentDidCatch(t.value,{componentStack:null!==e?e:\"\"})}),n}function gu(e,t,n){var r=e.pingCache;if(null===r){r=e.pingCache=new du;var a=new Set;r.set(t,a)}else void 0===(a=r.get(t))&&(a=new Set,r.set(t,a));a.has(n)||(a.add(n),e=Cs.bind(null,e,t,n),t.then(e,e))}function yu(e){do{var t;if((t=13===e.tag)&&(t=null===(t=e.memoizedState)||null!==t.dehydrated),t)return e;e=e.return}while(null!==e);return null}function mu(e,t,n,r,a){return 0==(1&e.mode)?(e===t?e.flags|=65536:(e.flags|=128,n.flags|=131072,n.flags&=-52805,1===n.tag&&(null===n.alternate?n.tag=17:((t=Oi(-1,1)).tag=2,Ai(n,t,1))),n.lanes|=1),e):(e.flags|=65536,e.lanes=a,e)}var bu=_.ReactCurrentOwner,_u=!1;function wu(e,t,n,r){t.child=null===e?Zi(t,null,n,r):Ki(t,e.child,n,r)}function xu(e,t,n,r,a){n=n.render;var i=t.ref;return Si(t,a),r=So(e,t,n,r,i,a),n=Eo(),null===e||_u?(ai&&n&&ei(t),t.flags|=1,wu(e,t,r,a),t.child):(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~a,Hu(e,t,a))}function ku(e,t,n,r,a){if(null===e){var i=n.type;return\"function\"!=typeof i||Os(i)||void 0!==i.defaultProps||null!==n.compare||void 0!==n.defaultProps?((e=Fs(n.type,null,r,t,t.mode,a)).ref=t.ref,e.return=t,t.child=e):(t.tag=15,t.type=i,Su(e,t,i,r,a))}if(i=e.child,0==(e.lanes&a)){var o=i.memoizedProps;if((n=null!==(n=n.compare)?n:lr)(o,r)&&e.ref===t.ref)return Hu(e,t,a)}return t.flags|=1,(e=As(i,r)).ref=t.ref,e.return=t,t.child=e}function Su(e,t,n,r,a){if(null!==e){var i=e.memoizedProps;if(lr(i,r)&&e.ref===t.ref){if(_u=!1,t.pendingProps=r=i,0==(e.lanes&a))return t.lanes=e.lanes,Hu(e,t,a);0!=(131072&e.flags)&&(_u=!0)}}return Tu(e,t,n,r,a)}function Eu(e,t,n){var r=t.pendingProps,a=r.children,i=null!==e?e.memoizedState:null;if(\"hidden\"===r.mode)if(0==(1&t.mode))t.memoizedState={baseLanes:0,cachePool:null,transitions:null},Ca(Al,Ol),Ol|=n;else{if(0==(1073741824&n))return e=null!==i?i.baseLanes|n:n,t.lanes=t.childLanes=1073741824,t.memoizedState={baseLanes:e,cachePool:null,transitions:null},t.updateQueue=null,Ca(Al,Ol),Ol|=e,null;t.memoizedState={baseLanes:0,cachePool:null,transitions:null},r=null!==i?i.baseLanes:n,Ca(Al,Ol),Ol|=r}else null!==i?(r=i.baseLanes|n,t.memoizedState=null):r=n,Ca(Al,Ol),Ol|=r;return wu(e,t,a,n),t.child}function Cu(e,t){var n=t.ref;(null===e&&null!==n||null!==e&&e.ref!==n)&&(t.flags|=512,t.flags|=2097152)}function Tu(e,t,n,r,a){var i=La(n)?Pa:Ma.current;return i=za(t,i),Si(t,a),n=So(e,t,n,r,i,a),r=Eo(),null===e||_u?(ai&&r&&ei(t),t.flags|=1,wu(e,t,n,a),t.child):(t.updateQueue=e.updateQueue,t.flags&=-2053,e.lanes&=~a,Hu(e,t,a))}function Mu(e,t,n,r,a){if(La(n)){var i=!0;Da(t)}else i=!1;if(Si(t,a),null===t.stateNode)Vu(e,t),Wi(t,n,r),Hi(t,n,r,a),r=!0;else if(null===e){var o=t.stateNode,u=t.memoizedProps;o.props=u;var l=o.context,s=n.contextType;s=\"object\"==typeof s&&null!==s?Ei(s):za(t,s=La(n)?Pa:Ma.current);var c=n.getDerivedStateFromProps,f=\"function\"==typeof c||\"function\"==typeof o.getSnapshotBeforeUpdate;f||\"function\"!=typeof o.UNSAFE_componentWillReceiveProps&&\"function\"!=typeof o.componentWillReceiveProps||(u!==r||l!==s)&&Vi(t,o,r,s),Pi=!1;var p=t.memoizedState;o.state=p,Ri(t,r,o,a),l=t.memoizedState,u!==r||p!==l||Na.current||Pi?(\"function\"==typeof c&&(Ii(t,n,c,r),l=t.memoizedState),(u=Pi||Bi(t,n,u,r,p,l,s))?(f||\"function\"!=typeof o.UNSAFE_componentWillMount&&\"function\"!=typeof o.componentWillMount||(\"function\"==typeof o.componentWillMount&&o.componentWillMount(),\"function\"==typeof o.UNSAFE_componentWillMount&&o.UNSAFE_componentWillMount()),\"function\"==typeof o.componentDidMount&&(t.flags|=4194308)):(\"function\"==typeof o.componentDidMount&&(t.flags|=4194308),t.memoizedProps=r,t.memoizedState=l),o.props=r,o.state=l,o.context=s,r=u):(\"function\"==typeof o.componentDidMount&&(t.flags|=4194308),r=!1)}else{o=t.stateNode,Li(e,t),u=t.memoizedProps,s=t.type===t.elementType?u:gi(t.type,u),o.props=s,f=t.pendingProps,p=o.context,l=\"object\"==typeof(l=n.contextType)&&null!==l?Ei(l):za(t,l=La(n)?Pa:Ma.current);var d=n.getDerivedStateFromProps;(c=\"function\"==typeof d||\"function\"==typeof o.getSnapshotBeforeUpdate)||\"function\"!=typeof o.UNSAFE_componentWillReceiveProps&&\"function\"!=typeof o.componentWillReceiveProps||(u!==f||p!==l)&&Vi(t,o,r,l),Pi=!1,p=t.memoizedState,o.state=p,Ri(t,r,o,a);var h=t.memoizedState;u!==f||p!==h||Na.current||Pi?(\"function\"==typeof d&&(Ii(t,n,d,r),h=t.memoizedState),(s=Pi||Bi(t,n,s,r,p,h,l)||!1)?(c||\"function\"!=typeof o.UNSAFE_componentWillUpdate&&\"function\"!=typeof o.componentWillUpdate||(\"function\"==typeof o.componentWillUpdate&&o.componentWillUpdate(r,h,l),\"function\"==typeof o.UNSAFE_componentWillUpdate&&o.UNSAFE_componentWillUpdate(r,h,l)),\"function\"==typeof o.componentDidUpdate&&(t.flags|=4),\"function\"==typeof o.getSnapshotBeforeUpdate&&(t.flags|=1024)):(\"function\"!=typeof o.componentDidUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=4),\"function\"!=typeof o.getSnapshotBeforeUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=1024),t.memoizedProps=r,t.memoizedState=h),o.props=r,o.state=h,o.context=l,r=s):(\"function\"!=typeof o.componentDidUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=4),\"function\"!=typeof o.getSnapshotBeforeUpdate||u===e.memoizedProps&&p===e.memoizedState||(t.flags|=1024),r=!1)}return Nu(e,t,n,r,i,a)}function Nu(e,t,n,r,a,i){Cu(e,t);var o=0!=(128&t.flags);if(!r&&!o)return a&&Ra(t,n,!1),Hu(e,t,i);r=t.stateNode,bu.current=t;var u=o&&\"function\"!=typeof n.getDerivedStateFromError?null:r.render();return t.flags|=1,null!==e&&o?(t.child=Ki(t,e.child,null,i),t.child=Ki(t,null,u,i)):wu(e,t,u,i),t.memoizedState=r.state,a&&Ra(t,n,!0),t.child}function Pu(e){var t=e.stateNode;t.pendingContext?Aa(0,t.pendingContext,t.pendingContext!==t.context):t.context&&Aa(0,t.context,!1),ro(e,t.containerInfo)}function zu(e,t,n,r,a){return di(),hi(a),t.flags|=256,wu(e,t,n,r),t.child}var Lu,Ou,Au,Fu,Du={dehydrated:null,treeContext:null,retryLane:0};function Ru(e){return{baseLanes:e,cachePool:null,transitions:null}}function ju(e,t,n){var r,a=t.pendingProps,o=uo.current,u=!1,l=0!=(128&t.flags);if((r=l)||(r=(null===e||null!==e.memoizedState)&&0!=(2&o)),r?(u=!0,t.flags&=-129):null!==e&&null===e.memoizedState||(o|=1),Ca(uo,1&o),null===e)return si(t),null!==(e=t.memoizedState)&&null!==(e=e.dehydrated)?(0==(1&t.mode)?t.lanes=1:\"$!\"===e.data?t.lanes=8:t.lanes=1073741824,null):(l=a.children,e=a.fallback,u?(a=t.mode,u=t.child,l={mode:\"hidden\",children:l},0==(1&a)&&null!==u?(u.childLanes=0,u.pendingProps=l):u=Rs(l,a,0,null),e=Ds(e,a,n,null),u.return=t,e.return=t,u.sibling=e,t.child=u,t.child.memoizedState=Ru(n),t.memoizedState=Du,e):Uu(t,l));if(null!==(o=e.memoizedState)&&null!==(r=o.dehydrated))return function(e,t,n,r,a,o,u){if(n)return 256&t.flags?(t.flags&=-257,Iu(e,t,u,r=fu(Error(i(422))))):null!==t.memoizedState?(t.child=e.child,t.flags|=128,null):(o=r.fallback,a=t.mode,r=Rs({mode:\"visible\",children:r.children},a,0,null),(o=Ds(o,a,u,null)).flags|=2,r.return=t,o.return=t,r.sibling=o,t.child=r,0!=(1&t.mode)&&Ki(t,e.child,null,u),t.child.memoizedState=Ru(u),t.memoizedState=Du,o);if(0==(1&t.mode))return Iu(e,t,u,null);if(\"$!\"===a.data){if(r=a.nextSibling&&a.nextSibling.dataset)var l=r.dgst;return r=l,Iu(e,t,u,r=fu(o=Error(i(419)),r,void 0))}if(l=0!=(u&e.childLanes),_u||l){if(null!==(r=Pl)){switch(u&-u){case 4:a=2;break;case 16:a=8;break;case 64:case 128:case 256:case 512:case 1024:case 2048:case 4096:case 8192:case 16384:case 32768:case 65536:case 131072:case 262144:case 524288:case 1048576:case 2097152:case 4194304:case 8388608:case 16777216:case 33554432:case 67108864:a=32;break;case 536870912:a=268435456;break;default:a=0}0!==(a=0!=(a&(r.suspendedLanes|u))?0:a)&&a!==o.retryLane&&(o.retryLane=a,Ni(e,a),rs(r,e,a,-1))}return gs(),Iu(e,t,u,r=fu(Error(i(421))))}return\"$?\"===a.data?(t.flags|=128,t.child=e.child,t=Ms.bind(null,e),a._reactRetry=t,null):(e=o.treeContext,ri=sa(a.nextSibling),ni=t,ai=!0,ii=null,null!==e&&(Qa[Ya++]=Ka,Qa[Ya++]=Za,Qa[Ya++]=Ga,Ka=e.id,Za=e.overflow,Ga=t),(t=Uu(t,r.children)).flags|=4096,t)}(e,t,l,a,r,o,n);if(u){u=a.fallback,l=t.mode,r=(o=e.child).sibling;var s={mode:\"hidden\",children:a.children};return 0==(1&l)&&t.child!==o?((a=t.child).childLanes=0,a.pendingProps=s,t.deletions=null):(a=As(o,s)).subtreeFlags=14680064&o.subtreeFlags,null!==r?u=As(r,u):(u=Ds(u,l,n,null)).flags|=2,u.return=t,a.return=t,a.sibling=u,t.child=a,a=u,u=t.child,l=null===(l=e.child.memoizedState)?Ru(n):{baseLanes:l.baseLanes|n,cachePool:null,transitions:l.transitions},u.memoizedState=l,u.childLanes=e.childLanes&~n,t.memoizedState=Du,a}return e=(u=e.child).sibling,a=As(u,{mode:\"visible\",children:a.children}),0==(1&t.mode)&&(a.lanes=n),a.return=t,a.sibling=null,null!==e&&(null===(n=t.deletions)?(t.deletions=[e],t.flags|=16):n.push(e)),t.child=a,t.memoizedState=null,a}function Uu(e,t){return(t=Rs({mode:\"visible\",children:t},e.mode,0,null)).return=e,e.child=t}function Iu(e,t,n,r){return null!==r&&hi(r),Ki(t,e.child,null,n),(e=Uu(t,t.pendingProps.children)).flags|=2,t.memoizedState=null,e}function $u(e,t,n){e.lanes|=t;var r=e.alternate;null!==r&&(r.lanes|=t),ki(e.return,t,n)}function Bu(e,t,n,r,a){var i=e.memoizedState;null===i?e.memoizedState={isBackwards:t,rendering:null,renderingStartTime:0,last:r,tail:n,tailMode:a}:(i.isBackwards=t,i.rendering=null,i.renderingStartTime=0,i.last=r,i.tail=n,i.tailMode=a)}function Wu(e,t,n){var r=t.pendingProps,a=r.revealOrder,i=r.tail;if(wu(e,t,r.children,n),0!=(2&(r=uo.current)))r=1&r|2,t.flags|=128;else{if(null!==e&&0!=(128&e.flags))e:for(e=t.child;null!==e;){if(13===e.tag)null!==e.memoizedState&&$u(e,n,t);else if(19===e.tag)$u(e,n,t);else if(null!==e.child){e.child.return=e,e=e.child;continue}if(e===t)break e;for(;null===e.sibling;){if(null===e.return||e.return===t)break e;e=e.return}e.sibling.return=e.return,e=e.sibling}r&=1}if(Ca(uo,r),0==(1&t.mode))t.memoizedState=null;else switch(a){case\"forwards\":for(n=t.child,a=null;null!==n;)null!==(e=n.alternate)&&null===lo(e)&&(a=n),n=n.sibling;null===(n=a)?(a=t.child,t.child=null):(a=n.sibling,n.sibling=null),Bu(t,!1,a,n,i);break;case\"backwards\":for(n=null,a=t.child,t.child=null;null!==a;){if(null!==(e=a.alternate)&&null===lo(e)){t.child=a;break}e=a.sibling,a.sibling=n,n=a,a=e}Bu(t,!0,n,null,i);break;case\"together\":Bu(t,!1,null,null,void 0);break;default:t.memoizedState=null}return t.child}function Vu(e,t){0==(1&t.mode)&&null!==e&&(e.alternate=null,t.alternate=null,t.flags|=2)}function Hu(e,t,n){if(null!==e&&(t.dependencies=e.dependencies),Rl|=t.lanes,0==(n&t.childLanes))return null;if(null!==e&&t.child!==e.child)throw Error(i(153));if(null!==t.child){for(n=As(e=t.child,e.pendingProps),t.child=n,n.return=t;null!==e.sibling;)e=e.sibling,(n=n.sibling=As(e,e.pendingProps)).return=t;n.sibling=null}return t.child}function qu(e,t){if(!ai)switch(e.tailMode){case\"hidden\":t=e.tail;for(var n=null;null!==t;)null!==t.alternate&&(n=t),t=t.sibling;null===n?e.tail=null:n.sibling=null;break;case\"collapsed\":n=e.tail;for(var r=null;null!==n;)null!==n.alternate&&(r=n),n=n.sibling;null===r?t||null===e.tail?e.tail=null:e.tail.sibling=null:r.sibling=null}}function Qu(e){var t=null!==e.alternate&&e.alternate.child===e.child,n=0,r=0;if(t)for(var a=e.child;null!==a;)n|=a.lanes|a.childLanes,r|=14680064&a.subtreeFlags,r|=14680064&a.flags,a.return=e,a=a.sibling;else for(a=e.child;null!==a;)n|=a.lanes|a.childLanes,r|=a.subtreeFlags,r|=a.flags,a.return=e,a=a.sibling;return e.subtreeFlags|=r,e.childLanes=n,t}function Yu(e,t,n){var r=t.pendingProps;switch(ti(t),t.tag){case 2:case 16:case 15:case 0:case 11:case 7:case 8:case 12:case 9:case 14:return Qu(t),null;case 1:case 17:return La(t.type)&&Oa(),Qu(t),null;case 3:return r=t.stateNode,ao(),Ea(Na),Ea(Ma),co(),r.pendingContext&&(r.context=r.pendingContext,r.pendingContext=null),null!==e&&null!==e.child||(fi(t)?t.flags|=4:null===e||e.memoizedState.isDehydrated&&0==(256&t.flags)||(t.flags|=1024,null!==ii&&(us(ii),ii=null))),Ou(e,t),Qu(t),null;case 5:oo(t);var a=no(to.current);if(n=t.type,null!==e&&null!=t.stateNode)Au(e,t,n,r,a),e.ref!==t.ref&&(t.flags|=512,t.flags|=2097152);else{if(!r){if(null===t.stateNode)throw Error(i(166));return Qu(t),null}if(e=no(Ji.current),fi(t)){r=t.stateNode,n=t.type;var o=t.memoizedProps;switch(r[pa]=t,r[da]=o,e=0!=(1&t.mode),n){case\"dialog\":Ur(\"cancel\",r),Ur(\"close\",r);break;case\"iframe\":case\"object\":case\"embed\":Ur(\"load\",r);break;case\"video\":case\"audio\":for(a=0;a<Fr.length;a++)Ur(Fr[a],r);break;case\"source\":Ur(\"error\",r);break;case\"img\":case\"image\":case\"link\":Ur(\"error\",r),Ur(\"load\",r);break;case\"details\":Ur(\"toggle\",r);break;case\"input\":K(r,o),Ur(\"invalid\",r);break;case\"select\":r._wrapperState={wasMultiple:!!o.multiple},Ur(\"invalid\",r);break;case\"textarea\":ae(r,o),Ur(\"invalid\",r)}for(var l in me(n,o),a=null,o)if(o.hasOwnProperty(l)){var s=o[l];\"children\"===l?\"string\"==typeof s?r.textContent!==s&&(!0!==o.suppressHydrationWarning&&Xr(r.textContent,s,e),a=[\"children\",s]):\"number\"==typeof s&&r.textContent!==\"\"+s&&(!0!==o.suppressHydrationWarning&&Xr(r.textContent,s,e),a=[\"children\",\"\"+s]):u.hasOwnProperty(l)&&null!=s&&\"onScroll\"===l&&Ur(\"scroll\",r)}switch(n){case\"input\":q(r),J(r,o,!0);break;case\"textarea\":q(r),oe(r);break;case\"select\":case\"option\":break;default:\"function\"==typeof o.onClick&&(r.onclick=Jr)}r=a,t.updateQueue=r,null!==r&&(t.flags|=4)}else{l=9===a.nodeType?a:a.ownerDocument,\"http://www.w3.org/1999/xhtml\"===e&&(e=ue(n)),\"http://www.w3.org/1999/xhtml\"===e?\"script\"===n?((e=l.createElement(\"div\")).innerHTML=\"<script><\\/script>\",e=e.removeChild(e.firstChild)):\"string\"==typeof r.is?e=l.createElement(n,{is:r.is}):(e=l.createElement(n),\"select\"===n&&(l=e,r.multiple?l.multiple=!0:r.size&&(l.size=r.size))):e=l.createElementNS(e,n),e[pa]=t,e[da]=r,Lu(e,t,!1,!1),t.stateNode=e;e:{switch(l=be(n,r),n){case\"dialog\":Ur(\"cancel\",e),Ur(\"close\",e),a=r;break;case\"iframe\":case\"object\":case\"embed\":Ur(\"load\",e),a=r;break;case\"video\":case\"audio\":for(a=0;a<Fr.length;a++)Ur(Fr[a],e);a=r;break;case\"source\":Ur(\"error\",e),a=r;break;case\"img\":case\"image\":case\"link\":Ur(\"error\",e),Ur(\"load\",e),a=r;break;case\"details\":Ur(\"toggle\",e),a=r;break;case\"input\":K(e,r),a=G(e,r),Ur(\"invalid\",e);break;case\"option\":default:a=r;break;case\"select\":e._wrapperState={wasMultiple:!!r.multiple},a=R({},r,{value:void 0}),Ur(\"invalid\",e);break;case\"textarea\":ae(e,r),a=re(e,r),Ur(\"invalid\",e)}for(o in me(n,a),s=a)if(s.hasOwnProperty(o)){var c=s[o];\"style\"===o?ge(e,c):\"dangerouslySetInnerHTML\"===o?null!=(c=c?c.__html:void 0)&&fe(e,c):\"children\"===o?\"string\"==typeof c?(\"textarea\"!==n||\"\"!==c)&&pe(e,c):\"number\"==typeof c&&pe(e,\"\"+c):\"suppressContentEditableWarning\"!==o&&\"suppressHydrationWarning\"!==o&&\"autoFocus\"!==o&&(u.hasOwnProperty(o)?null!=c&&\"onScroll\"===o&&Ur(\"scroll\",e):null!=c&&b(e,o,c,l))}switch(n){case\"input\":q(e),J(e,r,!1);break;case\"textarea\":q(e),oe(e);break;case\"option\":null!=r.value&&e.setAttribute(\"value\",\"\"+V(r.value));break;case\"select\":e.multiple=!!r.multiple,null!=(o=r.value)?ne(e,!!r.multiple,o,!1):null!=r.defaultValue&&ne(e,!!r.multiple,r.defaultValue,!0);break;default:\"function\"==typeof a.onClick&&(e.onclick=Jr)}switch(n){case\"button\":case\"input\":case\"select\":case\"textarea\":r=!!r.autoFocus;break e;case\"img\":r=!0;break e;default:r=!1}}r&&(t.flags|=4)}null!==t.ref&&(t.flags|=512,t.flags|=2097152)}return Qu(t),null;case 6:if(e&&null!=t.stateNode)Fu(e,t,e.memoizedProps,r);else{if(\"string\"!=typeof r&&null===t.stateNode)throw Error(i(166));if(n=no(to.current),no(Ji.current),fi(t)){if(r=t.stateNode,n=t.memoizedProps,r[pa]=t,(o=r.nodeValue!==n)&&null!==(e=ni))switch(e.tag){case 3:Xr(r.nodeValue,n,0!=(1&e.mode));break;case 5:!0!==e.memoizedProps.suppressHydrationWarning&&Xr(r.nodeValue,n,0!=(1&e.mode))}o&&(t.flags|=4)}else(r=(9===n.nodeType?n:n.ownerDocument).createTextNode(r))[pa]=t,t.stateNode=r}return Qu(t),null;case 13:if(Ea(uo),r=t.memoizedState,null===e||null!==e.memoizedState&&null!==e.memoizedState.dehydrated){if(ai&&null!==ri&&0!=(1&t.mode)&&0==(128&t.flags))pi(),di(),t.flags|=98560,o=!1;else if(o=fi(t),null!==r&&null!==r.dehydrated){if(null===e){if(!o)throw Error(i(318));if(!(o=null!==(o=t.memoizedState)?o.dehydrated:null))throw Error(i(317));o[pa]=t}else di(),0==(128&t.flags)&&(t.memoizedState=null),t.flags|=4;Qu(t),o=!1}else null!==ii&&(us(ii),ii=null),o=!0;if(!o)return 65536&t.flags?t:null}return 0!=(128&t.flags)?(t.lanes=n,t):((r=null!==r)!=(null!==e&&null!==e.memoizedState)&&r&&(t.child.flags|=8192,0!=(1&t.mode)&&(null===e||0!=(1&uo.current)?0===Fl&&(Fl=3):gs())),null!==t.updateQueue&&(t.flags|=4),Qu(t),null);case 4:return ao(),Ou(e,t),null===e&&Br(t.stateNode.containerInfo),Qu(t),null;case 10:return xi(t.type._context),Qu(t),null;case 19:if(Ea(uo),null===(o=t.memoizedState))return Qu(t),null;if(r=0!=(128&t.flags),null===(l=o.rendering))if(r)qu(o,!1);else{if(0!==Fl||null!==e&&0!=(128&e.flags))for(e=t.child;null!==e;){if(null!==(l=lo(e))){for(t.flags|=128,qu(o,!1),null!==(r=l.updateQueue)&&(t.updateQueue=r,t.flags|=4),t.subtreeFlags=0,r=n,n=t.child;null!==n;)e=r,(o=n).flags&=14680066,null===(l=o.alternate)?(o.childLanes=0,o.lanes=e,o.child=null,o.subtreeFlags=0,o.memoizedProps=null,o.memoizedState=null,o.updateQueue=null,o.dependencies=null,o.stateNode=null):(o.childLanes=l.childLanes,o.lanes=l.lanes,o.child=l.child,o.subtreeFlags=0,o.deletions=null,o.memoizedProps=l.memoizedProps,o.memoizedState=l.memoizedState,o.updateQueue=l.updateQueue,o.type=l.type,e=l.dependencies,o.dependencies=null===e?null:{lanes:e.lanes,firstContext:e.firstContext}),n=n.sibling;return Ca(uo,1&uo.current|2),t.child}e=e.sibling}null!==o.tail&&Ze()>Wl&&(t.flags|=128,r=!0,qu(o,!1),t.lanes=4194304)}else{if(!r)if(null!==(e=lo(l))){if(t.flags|=128,r=!0,null!==(n=e.updateQueue)&&(t.updateQueue=n,t.flags|=4),qu(o,!0),null===o.tail&&\"hidden\"===o.tailMode&&!l.alternate&&!ai)return Qu(t),null}else 2*Ze()-o.renderingStartTime>Wl&&1073741824!==n&&(t.flags|=128,r=!0,qu(o,!1),t.lanes=4194304);o.isBackwards?(l.sibling=t.child,t.child=l):(null!==(n=o.last)?n.sibling=l:t.child=l,o.last=l)}return null!==o.tail?(t=o.tail,o.rendering=t,o.tail=t.sibling,o.renderingStartTime=Ze(),t.sibling=null,n=uo.current,Ca(uo,r?1&n|2:1&n),t):(Qu(t),null);case 22:case 23:return ps(),r=null!==t.memoizedState,null!==e&&null!==e.memoizedState!==r&&(t.flags|=8192),r&&0!=(1&t.mode)?0!=(1073741824&Ol)&&(Qu(t),6&t.subtreeFlags&&(t.flags|=8192)):Qu(t),null;case 24:case 25:return null}throw Error(i(156,t.tag))}function Gu(e,t){switch(ti(t),t.tag){case 1:return La(t.type)&&Oa(),65536&(e=t.flags)?(t.flags=-65537&e|128,t):null;case 3:return ao(),Ea(Na),Ea(Ma),co(),0!=(65536&(e=t.flags))&&0==(128&e)?(t.flags=-65537&e|128,t):null;case 5:return oo(t),null;case 13:if(Ea(uo),null!==(e=t.memoizedState)&&null!==e.dehydrated){if(null===t.alternate)throw Error(i(340));di()}return 65536&(e=t.flags)?(t.flags=-65537&e|128,t):null;case 19:return Ea(uo),null;case 4:return ao(),null;case 10:return xi(t.type._context),null;case 22:case 23:return ps(),null;default:return null}}Lu=function(e,t){for(var n=t.child;null!==n;){if(5===n.tag||6===n.tag)e.appendChild(n.stateNode);else if(4!==n.tag&&null!==n.child){n.child.return=n,n=n.child;continue}if(n===t)break;for(;null===n.sibling;){if(null===n.return||n.return===t)return;n=n.return}n.sibling.return=n.return,n=n.sibling}},Ou=function(){},Au=function(e,t,n,r){var a=e.memoizedProps;if(a!==r){e=t.stateNode,no(Ji.current);var i,o=null;switch(n){case\"input\":a=G(e,a),r=G(e,r),o=[];break;case\"select\":a=R({},a,{value:void 0}),r=R({},r,{value:void 0}),o=[];break;case\"textarea\":a=re(e,a),r=re(e,r),o=[];break;default:\"function\"!=typeof a.onClick&&\"function\"==typeof r.onClick&&(e.onclick=Jr)}for(c in me(n,r),n=null,a)if(!r.hasOwnProperty(c)&&a.hasOwnProperty(c)&&null!=a[c])if(\"style\"===c){var l=a[c];for(i in l)l.hasOwnProperty(i)&&(n||(n={}),n[i]=\"\")}else\"dangerouslySetInnerHTML\"!==c&&\"children\"!==c&&\"suppressContentEditableWarning\"!==c&&\"suppressHydrationWarning\"!==c&&\"autoFocus\"!==c&&(u.hasOwnProperty(c)?o||(o=[]):(o=o||[]).push(c,null));for(c in r){var s=r[c];if(l=null!=a?a[c]:void 0,r.hasOwnProperty(c)&&s!==l&&(null!=s||null!=l))if(\"style\"===c)if(l){for(i in l)!l.hasOwnProperty(i)||s&&s.hasOwnProperty(i)||(n||(n={}),n[i]=\"\");for(i in s)s.hasOwnProperty(i)&&l[i]!==s[i]&&(n||(n={}),n[i]=s[i])}else n||(o||(o=[]),o.push(c,n)),n=s;else\"dangerouslySetInnerHTML\"===c?(s=s?s.__html:void 0,l=l?l.__html:void 0,null!=s&&l!==s&&(o=o||[]).push(c,s)):\"children\"===c?\"string\"!=typeof s&&\"number\"!=typeof s||(o=o||[]).push(c,\"\"+s):\"suppressContentEditableWarning\"!==c&&\"suppressHydrationWarning\"!==c&&(u.hasOwnProperty(c)?(null!=s&&\"onScroll\"===c&&Ur(\"scroll\",e),o||l===s||(o=[])):(o=o||[]).push(c,s))}n&&(o=o||[]).push(\"style\",n);var c=o;(t.updateQueue=c)&&(t.flags|=4)}},Fu=function(e,t,n,r){n!==r&&(t.flags|=4)};var Ku=!1,Zu=!1,Xu=\"function\"==typeof WeakSet?WeakSet:Set,Ju=null;function el(e,t){var n=e.ref;if(null!==n)if(\"function\"==typeof n)try{n(null)}catch(n){Es(e,t,n)}else n.current=null}function tl(e,t,n){try{n()}catch(n){Es(e,t,n)}}var nl=!1;function rl(e,t,n){var r=t.updateQueue;if(null!==(r=null!==r?r.lastEffect:null)){var a=r=r.next;do{if((a.tag&e)===e){var i=a.destroy;a.destroy=void 0,void 0!==i&&tl(t,n,i)}a=a.next}while(a!==r)}}function al(e,t){if(null!==(t=null!==(t=t.updateQueue)?t.lastEffect:null)){var n=t=t.next;do{if((n.tag&e)===e){var r=n.create;n.destroy=r()}n=n.next}while(n!==t)}}function il(e){var t=e.ref;if(null!==t){var n=e.stateNode;e.tag,e=n,\"function\"==typeof t?t(e):t.current=e}}function ol(e){var t=e.alternate;null!==t&&(e.alternate=null,ol(t)),e.child=null,e.deletions=null,e.sibling=null,5===e.tag&&null!==(t=e.stateNode)&&(delete t[pa],delete t[da],delete t[va],delete t[ga],delete t[ya]),e.stateNode=null,e.return=null,e.dependencies=null,e.memoizedProps=null,e.memoizedState=null,e.pendingProps=null,e.stateNode=null,e.updateQueue=null}function ul(e){return 5===e.tag||3===e.tag||4===e.tag}function ll(e){e:for(;;){for(;null===e.sibling;){if(null===e.return||ul(e.return))return null;e=e.return}for(e.sibling.return=e.return,e=e.sibling;5!==e.tag&&6!==e.tag&&18!==e.tag;){if(2&e.flags)continue e;if(null===e.child||4===e.tag)continue e;e.child.return=e,e=e.child}if(!(2&e.flags))return e.stateNode}}function sl(e,t,n){var r=e.tag;if(5===r||6===r)e=e.stateNode,t?8===n.nodeType?n.parentNode.insertBefore(e,t):n.insertBefore(e,t):(8===n.nodeType?(t=n.parentNode).insertBefore(e,n):(t=n).appendChild(e),null!=(n=n._reactRootContainer)||null!==t.onclick||(t.onclick=Jr));else if(4!==r&&null!==(e=e.child))for(sl(e,t,n),e=e.sibling;null!==e;)sl(e,t,n),e=e.sibling}function cl(e,t,n){var r=e.tag;if(5===r||6===r)e=e.stateNode,t?n.insertBefore(e,t):n.appendChild(e);else if(4!==r&&null!==(e=e.child))for(cl(e,t,n),e=e.sibling;null!==e;)cl(e,t,n),e=e.sibling}var fl=null,pl=!1;function dl(e,t,n){for(n=n.child;null!==n;)hl(e,t,n),n=n.sibling}function hl(e,t,n){if(it&&\"function\"==typeof it.onCommitFiberUnmount)try{it.onCommitFiberUnmount(at,n)}catch(e){}switch(n.tag){case 5:Zu||el(n,t);case 6:var r=fl,a=pl;fl=null,dl(e,t,n),pl=a,null!==(fl=r)&&(pl?(e=fl,n=n.stateNode,8===e.nodeType?e.parentNode.removeChild(n):e.removeChild(n)):fl.removeChild(n.stateNode));break;case 18:null!==fl&&(pl?(e=fl,n=n.stateNode,8===e.nodeType?la(e.parentNode,n):1===e.nodeType&&la(e,n),Bt(e)):la(fl,n.stateNode));break;case 4:r=fl,a=pl,fl=n.stateNode.containerInfo,pl=!0,dl(e,t,n),fl=r,pl=a;break;case 0:case 11:case 14:case 15:if(!Zu&&null!==(r=n.updateQueue)&&null!==(r=r.lastEffect)){a=r=r.next;do{var i=a,o=i.destroy;i=i.tag,void 0!==o&&(0!=(2&i)||0!=(4&i))&&tl(n,t,o),a=a.next}while(a!==r)}dl(e,t,n);break;case 1:if(!Zu&&(el(n,t),\"function\"==typeof(r=n.stateNode).componentWillUnmount))try{r.props=n.memoizedProps,r.state=n.memoizedState,r.componentWillUnmount()}catch(e){Es(n,t,e)}dl(e,t,n);break;case 21:dl(e,t,n);break;case 22:1&n.mode?(Zu=(r=Zu)||null!==n.memoizedState,dl(e,t,n),Zu=r):dl(e,t,n);break;default:dl(e,t,n)}}function vl(e){var t=e.updateQueue;if(null!==t){e.updateQueue=null;var n=e.stateNode;null===n&&(n=e.stateNode=new Xu),t.forEach((function(t){var r=Ns.bind(null,e,t);n.has(t)||(n.add(t),t.then(r,r))}))}}function gl(e,t){var n=t.deletions;if(null!==n)for(var r=0;r<n.length;r++){var a=n[r];try{var o=e,u=t,l=u;e:for(;null!==l;){switch(l.tag){case 5:fl=l.stateNode,pl=!1;break e;case 3:case 4:fl=l.stateNode.containerInfo,pl=!0;break e}l=l.return}if(null===fl)throw Error(i(160));hl(o,u,a),fl=null,pl=!1;var s=a.alternate;null!==s&&(s.return=null),a.return=null}catch(e){Es(a,t,e)}}if(12854&t.subtreeFlags)for(t=t.child;null!==t;)yl(t,e),t=t.sibling}function yl(e,t){var n=e.alternate,r=e.flags;switch(e.tag){case 0:case 11:case 14:case 15:if(gl(t,e),ml(e),4&r){try{rl(3,e,e.return),al(3,e)}catch(t){Es(e,e.return,t)}try{rl(5,e,e.return)}catch(t){Es(e,e.return,t)}}break;case 1:gl(t,e),ml(e),512&r&&null!==n&&el(n,n.return);break;case 5:if(gl(t,e),ml(e),512&r&&null!==n&&el(n,n.return),32&e.flags){var a=e.stateNode;try{pe(a,\"\")}catch(t){Es(e,e.return,t)}}if(4&r&&null!=(a=e.stateNode)){var o=e.memoizedProps,u=null!==n?n.memoizedProps:o,l=e.type,s=e.updateQueue;if(e.updateQueue=null,null!==s)try{\"input\"===l&&\"radio\"===o.type&&null!=o.name&&Z(a,o),be(l,u);var c=be(l,o);for(u=0;u<s.length;u+=2){var f=s[u],p=s[u+1];\"style\"===f?ge(a,p):\"dangerouslySetInnerHTML\"===f?fe(a,p):\"children\"===f?pe(a,p):b(a,f,p,c)}switch(l){case\"input\":X(a,o);break;case\"textarea\":ie(a,o);break;case\"select\":var d=a._wrapperState.wasMultiple;a._wrapperState.wasMultiple=!!o.multiple;var h=o.value;null!=h?ne(a,!!o.multiple,h,!1):d!==!!o.multiple&&(null!=o.defaultValue?ne(a,!!o.multiple,o.defaultValue,!0):ne(a,!!o.multiple,o.multiple?[]:\"\",!1))}a[da]=o}catch(t){Es(e,e.return,t)}}break;case 6:if(gl(t,e),ml(e),4&r){if(null===e.stateNode)throw Error(i(162));a=e.stateNode,o=e.memoizedProps;try{a.nodeValue=o}catch(t){Es(e,e.return,t)}}break;case 3:if(gl(t,e),ml(e),4&r&&null!==n&&n.memoizedState.isDehydrated)try{Bt(t.containerInfo)}catch(t){Es(e,e.return,t)}break;case 4:default:gl(t,e),ml(e);break;case 13:gl(t,e),ml(e),8192&(a=e.child).flags&&(o=null!==a.memoizedState,a.stateNode.isHidden=o,!o||null!==a.alternate&&null!==a.alternate.memoizedState||(Bl=Ze())),4&r&&vl(e);break;case 22:if(f=null!==n&&null!==n.memoizedState,1&e.mode?(Zu=(c=Zu)||f,gl(t,e),Zu=c):gl(t,e),ml(e),8192&r){if(c=null!==e.memoizedState,(e.stateNode.isHidden=c)&&!f&&0!=(1&e.mode))for(Ju=e,f=e.child;null!==f;){for(p=Ju=f;null!==Ju;){switch(h=(d=Ju).child,d.tag){case 0:case 11:case 14:case 15:rl(4,d,d.return);break;case 1:el(d,d.return);var v=d.stateNode;if(\"function\"==typeof v.componentWillUnmount){r=d,n=d.return;try{t=r,v.props=t.memoizedProps,v.state=t.memoizedState,v.componentWillUnmount()}catch(e){Es(r,n,e)}}break;case 5:el(d,d.return);break;case 22:if(null!==d.memoizedState){xl(p);continue}}null!==h?(h.return=d,Ju=h):xl(p)}f=f.sibling}e:for(f=null,p=e;;){if(5===p.tag){if(null===f){f=p;try{a=p.stateNode,c?\"function\"==typeof(o=a.style).setProperty?o.setProperty(\"display\",\"none\",\"important\"):o.display=\"none\":(l=p.stateNode,u=null!=(s=p.memoizedProps.style)&&s.hasOwnProperty(\"display\")?s.display:null,l.style.display=ve(\"display\",u))}catch(t){Es(e,e.return,t)}}}else if(6===p.tag){if(null===f)try{p.stateNode.nodeValue=c?\"\":p.memoizedProps}catch(t){Es(e,e.return,t)}}else if((22!==p.tag&&23!==p.tag||null===p.memoizedState||p===e)&&null!==p.child){p.child.return=p,p=p.child;continue}if(p===e)break e;for(;null===p.sibling;){if(null===p.return||p.return===e)break e;f===p&&(f=null),p=p.return}f===p&&(f=null),p.sibling.return=p.return,p=p.sibling}}break;case 19:gl(t,e),ml(e),4&r&&vl(e);case 21:}}function ml(e){var t=e.flags;if(2&t){try{e:{for(var n=e.return;null!==n;){if(ul(n)){var r=n;break e}n=n.return}throw Error(i(160))}switch(r.tag){case 5:var a=r.stateNode;32&r.flags&&(pe(a,\"\"),r.flags&=-33),cl(e,ll(e),a);break;case 3:case 4:var o=r.stateNode.containerInfo;sl(e,ll(e),o);break;default:throw Error(i(161))}}catch(t){Es(e,e.return,t)}e.flags&=-3}4096&t&&(e.flags&=-4097)}function bl(e,t,n){Ju=e,_l(e,t,n)}function _l(e,t,n){for(var r=0!=(1&e.mode);null!==Ju;){var a=Ju,i=a.child;if(22===a.tag&&r){var o=null!==a.memoizedState||Ku;if(!o){var u=a.alternate,l=null!==u&&null!==u.memoizedState||Zu;u=Ku;var s=Zu;if(Ku=o,(Zu=l)&&!s)for(Ju=a;null!==Ju;)l=(o=Ju).child,22===o.tag&&null!==o.memoizedState?kl(a):null!==l?(l.return=o,Ju=l):kl(a);for(;null!==i;)Ju=i,_l(i,t,n),i=i.sibling;Ju=a,Ku=u,Zu=s}wl(e)}else 0!=(8772&a.subtreeFlags)&&null!==i?(i.return=a,Ju=i):wl(e)}}function wl(e){for(;null!==Ju;){var t=Ju;if(0!=(8772&t.flags)){var n=t.alternate;try{if(0!=(8772&t.flags))switch(t.tag){case 0:case 11:case 15:Zu||al(5,t);break;case 1:var r=t.stateNode;if(4&t.flags&&!Zu)if(null===n)r.componentDidMount();else{var a=t.elementType===t.type?n.memoizedProps:gi(t.type,n.memoizedProps);r.componentDidUpdate(a,n.memoizedState,r.__reactInternalSnapshotBeforeUpdate)}var o=t.updateQueue;null!==o&&ji(t,o,r);break;case 3:var u=t.updateQueue;if(null!==u){if(n=null,null!==t.child)switch(t.child.tag){case 5:case 1:n=t.child.stateNode}ji(t,u,n)}break;case 5:var l=t.stateNode;if(null===n&&4&t.flags){n=l;var s=t.memoizedProps;switch(t.type){case\"button\":case\"input\":case\"select\":case\"textarea\":s.autoFocus&&n.focus();break;case\"img\":s.src&&(n.src=s.src)}}break;case 6:case 4:case 12:case 19:case 17:case 21:case 22:case 23:case 25:break;case 13:if(null===t.memoizedState){var c=t.alternate;if(null!==c){var f=c.memoizedState;if(null!==f){var p=f.dehydrated;null!==p&&Bt(p)}}}break;default:throw Error(i(163))}Zu||512&t.flags&&il(t)}catch(e){Es(t,t.return,e)}}if(t===e){Ju=null;break}if(null!==(n=t.sibling)){n.return=t.return,Ju=n;break}Ju=t.return}}function xl(e){for(;null!==Ju;){var t=Ju;if(t===e){Ju=null;break}var n=t.sibling;if(null!==n){n.return=t.return,Ju=n;break}Ju=t.return}}function kl(e){for(;null!==Ju;){var t=Ju;try{switch(t.tag){case 0:case 11:case 15:var n=t.return;try{al(4,t)}catch(e){Es(t,n,e)}break;case 1:var r=t.stateNode;if(\"function\"==typeof r.componentDidMount){var a=t.return;try{r.componentDidMount()}catch(e){Es(t,a,e)}}var i=t.return;try{il(t)}catch(e){Es(t,i,e)}break;case 5:var o=t.return;try{il(t)}catch(e){Es(t,o,e)}}}catch(e){Es(t,t.return,e)}if(t===e){Ju=null;break}var u=t.sibling;if(null!==u){u.return=t.return,Ju=u;break}Ju=t.return}}var Sl,El=Math.ceil,Cl=_.ReactCurrentDispatcher,Tl=_.ReactCurrentOwner,Ml=_.ReactCurrentBatchConfig,Nl=0,Pl=null,zl=null,Ll=0,Ol=0,Al=Sa(0),Fl=0,Dl=null,Rl=0,jl=0,Ul=0,Il=null,$l=null,Bl=0,Wl=1/0,Vl=null,Hl=!1,ql=null,Ql=null,Yl=!1,Gl=null,Kl=0,Zl=0,Xl=null,Jl=-1,es=0;function ts(){return 0!=(6&Nl)?Ze():-1!==Jl?Jl:Jl=Ze()}function ns(e){return 0==(1&e.mode)?1:0!=(2&Nl)&&0!==Ll?Ll&-Ll:null!==vi.transition?(0===es&&(es=vt()),es):0!==(e=bt)?e:e=void 0===(e=window.event)?16:Kt(e.type)}function rs(e,t,n,r){if(50<Zl)throw Zl=0,Xl=null,Error(i(185));yt(e,n,r),0!=(2&Nl)&&e===Pl||(e===Pl&&(0==(2&Nl)&&(jl|=n),4===Fl&&ls(e,Ll)),as(e,r),1===n&&0===Nl&&0==(1&t.mode)&&(Wl=Ze()+500,Ua&&Ba()))}function as(e,t){var n=e.callbackNode;!function(e,t){for(var n=e.suspendedLanes,r=e.pingedLanes,a=e.expirationTimes,i=e.pendingLanes;0<i;){var o=31-ot(i),u=1<<o,l=a[o];-1===l?0!=(u&n)&&0==(u&r)||(a[o]=dt(u,t)):l<=t&&(e.expiredLanes|=u),i&=~u}}(e,t);var r=pt(e,e===Pl?Ll:0);if(0===r)null!==n&&Ye(n),e.callbackNode=null,e.callbackPriority=0;else if(t=r&-r,e.callbackPriority!==t){if(null!=n&&Ye(n),1===t)0===e.tag?function(e){Ua=!0,$a(e)}(ss.bind(null,e)):$a(ss.bind(null,e)),oa((function(){0==(6&Nl)&&Ba()})),n=null;else{switch(_t(r)){case 1:n=Je;break;case 4:n=et;break;case 16:default:n=tt;break;case 536870912:n=rt}n=Ps(n,is.bind(null,e))}e.callbackPriority=t,e.callbackNode=n}}function is(e,t){if(Jl=-1,es=0,0!=(6&Nl))throw Error(i(327));var n=e.callbackNode;if(ks()&&e.callbackNode!==n)return null;var r=pt(e,e===Pl?Ll:0);if(0===r)return null;if(0!=(30&r)||0!=(r&e.expiredLanes)||t)t=ys(e,r);else{t=r;var a=Nl;Nl|=2;var o=vs();for(Pl===e&&Ll===t||(Vl=null,Wl=Ze()+500,ds(e,t));;)try{bs();break}catch(t){hs(e,t)}wi(),Cl.current=o,Nl=a,null!==zl?t=0:(Pl=null,Ll=0,t=Fl)}if(0!==t){if(2===t&&0!==(a=ht(e))&&(r=a,t=os(e,a)),1===t)throw n=Dl,ds(e,0),ls(e,r),as(e,Ze()),n;if(6===t)ls(e,r);else{if(a=e.current.alternate,0==(30&r)&&!function(e){for(var t=e;;){if(16384&t.flags){var n=t.updateQueue;if(null!==n&&null!==(n=n.stores))for(var r=0;r<n.length;r++){var a=n[r],i=a.getSnapshot;a=a.value;try{if(!ur(i(),a))return!1}catch(e){return!1}}}if(n=t.child,16384&t.subtreeFlags&&null!==n)n.return=t,t=n;else{if(t===e)break;for(;null===t.sibling;){if(null===t.return||t.return===e)return!0;t=t.return}t.sibling.return=t.return,t=t.sibling}}return!0}(a)&&(2===(t=ys(e,r))&&0!==(o=ht(e))&&(r=o,t=os(e,o)),1===t))throw n=Dl,ds(e,0),ls(e,r),as(e,Ze()),n;switch(e.finishedWork=a,e.finishedLanes=r,t){case 0:case 1:throw Error(i(345));case 2:case 5:xs(e,$l,Vl);break;case 3:if(ls(e,r),(130023424&r)===r&&10<(t=Bl+500-Ze())){if(0!==pt(e,0))break;if(((a=e.suspendedLanes)&r)!==r){ts(),e.pingedLanes|=e.suspendedLanes&a;break}e.timeoutHandle=ra(xs.bind(null,e,$l,Vl),t);break}xs(e,$l,Vl);break;case 4:if(ls(e,r),(4194240&r)===r)break;for(t=e.eventTimes,a=-1;0<r;){var u=31-ot(r);o=1<<u,(u=t[u])>a&&(a=u),r&=~o}if(r=a,10<(r=(120>(r=Ze()-r)?120:480>r?480:1080>r?1080:1920>r?1920:3e3>r?3e3:4320>r?4320:1960*El(r/1960))-r)){e.timeoutHandle=ra(xs.bind(null,e,$l,Vl),r);break}xs(e,$l,Vl);break;default:throw Error(i(329))}}}return as(e,Ze()),e.callbackNode===n?is.bind(null,e):null}function os(e,t){var n=Il;return e.current.memoizedState.isDehydrated&&(ds(e,t).flags|=256),2!==(e=ys(e,t))&&(t=$l,$l=n,null!==t&&us(t)),e}function us(e){null===$l?$l=e:$l.push.apply($l,e)}function ls(e,t){for(t&=~Ul,t&=~jl,e.suspendedLanes|=t,e.pingedLanes&=~t,e=e.expirationTimes;0<t;){var n=31-ot(t),r=1<<n;e[n]=-1,t&=~r}}function ss(e){if(0!=(6&Nl))throw Error(i(327));ks();var t=pt(e,0);if(0==(1&t))return as(e,Ze()),null;var n=ys(e,t);if(0!==e.tag&&2===n){var r=ht(e);0!==r&&(t=r,n=os(e,r))}if(1===n)throw n=Dl,ds(e,0),ls(e,t),as(e,Ze()),n;if(6===n)throw Error(i(345));return e.finishedWork=e.current.alternate,e.finishedLanes=t,xs(e,$l,Vl),as(e,Ze()),null}function cs(e,t){var n=Nl;Nl|=1;try{return e(t)}finally{0===(Nl=n)&&(Wl=Ze()+500,Ua&&Ba())}}function fs(e){null!==Gl&&0===Gl.tag&&0==(6&Nl)&&ks();var t=Nl;Nl|=1;var n=Ml.transition,r=bt;try{if(Ml.transition=null,bt=1,e)return e()}finally{bt=r,Ml.transition=n,0==(6&(Nl=t))&&Ba()}}function ps(){Ol=Al.current,Ea(Al)}function ds(e,t){e.finishedWork=null,e.finishedLanes=0;var n=e.timeoutHandle;if(-1!==n&&(e.timeoutHandle=-1,aa(n)),null!==zl)for(n=zl.return;null!==n;){var r=n;switch(ti(r),r.tag){case 1:null!=(r=r.type.childContextTypes)&&Oa();break;case 3:ao(),Ea(Na),Ea(Ma),co();break;case 5:oo(r);break;case 4:ao();break;case 13:case 19:Ea(uo);break;case 10:xi(r.type._context);break;case 22:case 23:ps()}n=n.return}if(Pl=e,zl=e=As(e.current,null),Ll=Ol=t,Fl=0,Dl=null,Ul=jl=Rl=0,$l=Il=null,null!==Ci){for(t=0;t<Ci.length;t++)if(null!==(r=(n=Ci[t]).interleaved)){n.interleaved=null;var a=r.next,i=n.pending;if(null!==i){var o=i.next;i.next=a,r.next=o}n.pending=r}Ci=null}return e}function hs(e,t){for(;;){var n=zl;try{if(wi(),fo.current=ou,mo){for(var r=vo.memoizedState;null!==r;){var a=r.queue;null!==a&&(a.pending=null),r=r.next}mo=!1}if(ho=0,yo=go=vo=null,bo=!1,_o=0,Tl.current=null,null===n||null===n.return){Fl=1,Dl=t,zl=null;break}e:{var o=e,u=n.return,l=n,s=t;if(t=Ll,l.flags|=32768,null!==s&&\"object\"==typeof s&&\"function\"==typeof s.then){var c=s,f=l,p=f.tag;if(0==(1&f.mode)&&(0===p||11===p||15===p)){var d=f.alternate;d?(f.updateQueue=d.updateQueue,f.memoizedState=d.memoizedState,f.lanes=d.lanes):(f.updateQueue=null,f.memoizedState=null)}var h=yu(u);if(null!==h){h.flags&=-257,mu(h,u,l,0,t),1&h.mode&&gu(o,c,t),s=c;var v=(t=h).updateQueue;if(null===v){var g=new Set;g.add(s),t.updateQueue=g}else v.add(s);break e}if(0==(1&t)){gu(o,c,t),gs();break e}s=Error(i(426))}else if(ai&&1&l.mode){var y=yu(u);if(null!==y){0==(65536&y.flags)&&(y.flags|=256),mu(y,u,l,0,t),hi(cu(s,l));break e}}o=s=cu(s,l),4!==Fl&&(Fl=2),null===Il?Il=[o]:Il.push(o),o=u;do{switch(o.tag){case 3:o.flags|=65536,t&=-t,o.lanes|=t,Di(o,hu(0,s,t));break e;case 1:l=s;var m=o.type,b=o.stateNode;if(0==(128&o.flags)&&(\"function\"==typeof m.getDerivedStateFromError||null!==b&&\"function\"==typeof b.componentDidCatch&&(null===Ql||!Ql.has(b)))){o.flags|=65536,t&=-t,o.lanes|=t,Di(o,vu(o,l,t));break e}}o=o.return}while(null!==o)}ws(n)}catch(e){t=e,zl===n&&null!==n&&(zl=n=n.return);continue}break}}function vs(){var e=Cl.current;return Cl.current=ou,null===e?ou:e}function gs(){0!==Fl&&3!==Fl&&2!==Fl||(Fl=4),null===Pl||0==(268435455&Rl)&&0==(268435455&jl)||ls(Pl,Ll)}function ys(e,t){var n=Nl;Nl|=2;var r=vs();for(Pl===e&&Ll===t||(Vl=null,ds(e,t));;)try{ms();break}catch(t){hs(e,t)}if(wi(),Nl=n,Cl.current=r,null!==zl)throw Error(i(261));return Pl=null,Ll=0,Fl}function ms(){for(;null!==zl;)_s(zl)}function bs(){for(;null!==zl&&!Ge();)_s(zl)}function _s(e){var t=Sl(e.alternate,e,Ol);e.memoizedProps=e.pendingProps,null===t?ws(e):zl=t,Tl.current=null}function ws(e){var t=e;do{var n=t.alternate;if(e=t.return,0==(32768&t.flags)){if(null!==(n=Yu(n,t,Ol)))return void(zl=n)}else{if(null!==(n=Gu(n,t)))return n.flags&=32767,void(zl=n);if(null===e)return Fl=6,void(zl=null);e.flags|=32768,e.subtreeFlags=0,e.deletions=null}if(null!==(t=t.sibling))return void(zl=t);zl=t=e}while(null!==t);0===Fl&&(Fl=5)}function xs(e,t,n){var r=bt,a=Ml.transition;try{Ml.transition=null,bt=1,function(e,t,n,r){do{ks()}while(null!==Gl);if(0!=(6&Nl))throw Error(i(327));n=e.finishedWork;var a=e.finishedLanes;if(null===n)return null;if(e.finishedWork=null,e.finishedLanes=0,n===e.current)throw Error(i(177));e.callbackNode=null,e.callbackPriority=0;var o=n.lanes|n.childLanes;if(function(e,t){var n=e.pendingLanes&~t;e.pendingLanes=t,e.suspendedLanes=0,e.pingedLanes=0,e.expiredLanes&=t,e.mutableReadLanes&=t,e.entangledLanes&=t,t=e.entanglements;var r=e.eventTimes;for(e=e.expirationTimes;0<n;){var a=31-ot(n),i=1<<a;t[a]=0,r[a]=-1,e[a]=-1,n&=~i}}(e,o),e===Pl&&(zl=Pl=null,Ll=0),0==(2064&n.subtreeFlags)&&0==(2064&n.flags)||Yl||(Yl=!0,Ps(tt,(function(){return ks(),null}))),o=0!=(15990&n.flags),0!=(15990&n.subtreeFlags)||o){o=Ml.transition,Ml.transition=null;var u=bt;bt=1;var l=Nl;Nl|=4,Tl.current=null,function(e,t){if(ea=Vt,dr(e=pr())){if(\"selectionStart\"in e)var n={start:e.selectionStart,end:e.selectionEnd};else e:{var r=(n=(n=e.ownerDocument)&&n.defaultView||window).getSelection&&n.getSelection();if(r&&0!==r.rangeCount){n=r.anchorNode;var a=r.anchorOffset,o=r.focusNode;r=r.focusOffset;try{n.nodeType,o.nodeType}catch(e){n=null;break e}var u=0,l=-1,s=-1,c=0,f=0,p=e,d=null;t:for(;;){for(var h;p!==n||0!==a&&3!==p.nodeType||(l=u+a),p!==o||0!==r&&3!==p.nodeType||(s=u+r),3===p.nodeType&&(u+=p.nodeValue.length),null!==(h=p.firstChild);)d=p,p=h;for(;;){if(p===e)break t;if(d===n&&++c===a&&(l=u),d===o&&++f===r&&(s=u),null!==(h=p.nextSibling))break;d=(p=d).parentNode}p=h}n=-1===l||-1===s?null:{start:l,end:s}}else n=null}n=n||{start:0,end:0}}else n=null;for(ta={focusedElem:e,selectionRange:n},Vt=!1,Ju=t;null!==Ju;)if(e=(t=Ju).child,0!=(1028&t.subtreeFlags)&&null!==e)e.return=t,Ju=e;else for(;null!==Ju;){t=Ju;try{var v=t.alternate;if(0!=(1024&t.flags))switch(t.tag){case 0:case 11:case 15:case 5:case 6:case 4:case 17:break;case 1:if(null!==v){var g=v.memoizedProps,y=v.memoizedState,m=t.stateNode,b=m.getSnapshotBeforeUpdate(t.elementType===t.type?g:gi(t.type,g),y);m.__reactInternalSnapshotBeforeUpdate=b}break;case 3:var _=t.stateNode.containerInfo;1===_.nodeType?_.textContent=\"\":9===_.nodeType&&_.documentElement&&_.removeChild(_.documentElement);break;default:throw Error(i(163))}}catch(e){Es(t,t.return,e)}if(null!==(e=t.sibling)){e.return=t.return,Ju=e;break}Ju=t.return}v=nl,nl=!1}(e,n),yl(n,e),hr(ta),Vt=!!ea,ta=ea=null,e.current=n,bl(n,e,a),Ke(),Nl=l,bt=u,Ml.transition=o}else e.current=n;if(Yl&&(Yl=!1,Gl=e,Kl=a),0===(o=e.pendingLanes)&&(Ql=null),function(e){if(it&&\"function\"==typeof it.onCommitFiberRoot)try{it.onCommitFiberRoot(at,e,void 0,128==(128&e.current.flags))}catch(e){}}(n.stateNode),as(e,Ze()),null!==t)for(r=e.onRecoverableError,n=0;n<t.length;n++)r((a=t[n]).value,{componentStack:a.stack,digest:a.digest});if(Hl)throw Hl=!1,e=ql,ql=null,e;0!=(1&Kl)&&0!==e.tag&&ks(),0!=(1&(o=e.pendingLanes))?e===Xl?Zl++:(Zl=0,Xl=e):Zl=0,Ba()}(e,t,n,r)}finally{Ml.transition=a,bt=r}return null}function ks(){if(null!==Gl){var e=_t(Kl),t=Ml.transition,n=bt;try{if(Ml.transition=null,bt=16>e?16:e,null===Gl)var r=!1;else{if(e=Gl,Gl=null,Kl=0,0!=(6&Nl))throw Error(i(331));var a=Nl;for(Nl|=4,Ju=e.current;null!==Ju;){var o=Ju,u=o.child;if(0!=(16&Ju.flags)){var l=o.deletions;if(null!==l){for(var s=0;s<l.length;s++){var c=l[s];for(Ju=c;null!==Ju;){var f=Ju;switch(f.tag){case 0:case 11:case 15:rl(8,f,o)}var p=f.child;if(null!==p)p.return=f,Ju=p;else for(;null!==Ju;){var d=(f=Ju).sibling,h=f.return;if(ol(f),f===c){Ju=null;break}if(null!==d){d.return=h,Ju=d;break}Ju=h}}}var v=o.alternate;if(null!==v){var g=v.child;if(null!==g){v.child=null;do{var y=g.sibling;g.sibling=null,g=y}while(null!==g)}}Ju=o}}if(0!=(2064&o.subtreeFlags)&&null!==u)u.return=o,Ju=u;else e:for(;null!==Ju;){if(0!=(2048&(o=Ju).flags))switch(o.tag){case 0:case 11:case 15:rl(9,o,o.return)}var m=o.sibling;if(null!==m){m.return=o.return,Ju=m;break e}Ju=o.return}}var b=e.current;for(Ju=b;null!==Ju;){var _=(u=Ju).child;if(0!=(2064&u.subtreeFlags)&&null!==_)_.return=u,Ju=_;else e:for(u=b;null!==Ju;){if(0!=(2048&(l=Ju).flags))try{switch(l.tag){case 0:case 11:case 15:al(9,l)}}catch(e){Es(l,l.return,e)}if(l===u){Ju=null;break e}var w=l.sibling;if(null!==w){w.return=l.return,Ju=w;break e}Ju=l.return}}if(Nl=a,Ba(),it&&\"function\"==typeof it.onPostCommitFiberRoot)try{it.onPostCommitFiberRoot(at,e)}catch(e){}r=!0}return r}finally{bt=n,Ml.transition=t}}return!1}function Ss(e,t,n){e=Ai(e,t=hu(0,t=cu(n,t),1),1),t=ts(),null!==e&&(yt(e,1,t),as(e,t))}function Es(e,t,n){if(3===e.tag)Ss(e,e,n);else for(;null!==t;){if(3===t.tag){Ss(t,e,n);break}if(1===t.tag){var r=t.stateNode;if(\"function\"==typeof t.type.getDerivedStateFromError||\"function\"==typeof r.componentDidCatch&&(null===Ql||!Ql.has(r))){t=Ai(t,e=vu(t,e=cu(n,e),1),1),e=ts(),null!==t&&(yt(t,1,e),as(t,e));break}}t=t.return}}function Cs(e,t,n){var r=e.pingCache;null!==r&&r.delete(t),t=ts(),e.pingedLanes|=e.suspendedLanes&n,Pl===e&&(Ll&n)===n&&(4===Fl||3===Fl&&(130023424&Ll)===Ll&&500>Ze()-Bl?ds(e,0):Ul|=n),as(e,t)}function Ts(e,t){0===t&&(0==(1&e.mode)?t=1:(t=ct,0==(130023424&(ct<<=1))&&(ct=4194304)));var n=ts();null!==(e=Ni(e,t))&&(yt(e,t,n),as(e,n))}function Ms(e){var t=e.memoizedState,n=0;null!==t&&(n=t.retryLane),Ts(e,n)}function Ns(e,t){var n=0;switch(e.tag){case 13:var r=e.stateNode,a=e.memoizedState;null!==a&&(n=a.retryLane);break;case 19:r=e.stateNode;break;default:throw Error(i(314))}null!==r&&r.delete(t),Ts(e,n)}function Ps(e,t){return Qe(e,t)}function zs(e,t,n,r){this.tag=e,this.key=n,this.sibling=this.child=this.return=this.stateNode=this.type=this.elementType=null,this.index=0,this.ref=null,this.pendingProps=t,this.dependencies=this.memoizedState=this.updateQueue=this.memoizedProps=null,this.mode=r,this.subtreeFlags=this.flags=0,this.deletions=null,this.childLanes=this.lanes=0,this.alternate=null}function Ls(e,t,n,r){return new zs(e,t,n,r)}function Os(e){return!(!(e=e.prototype)||!e.isReactComponent)}function As(e,t){var n=e.alternate;return null===n?((n=Ls(e.tag,t,e.key,e.mode)).elementType=e.elementType,n.type=e.type,n.stateNode=e.stateNode,n.alternate=e,e.alternate=n):(n.pendingProps=t,n.type=e.type,n.flags=0,n.subtreeFlags=0,n.deletions=null),n.flags=14680064&e.flags,n.childLanes=e.childLanes,n.lanes=e.lanes,n.child=e.child,n.memoizedProps=e.memoizedProps,n.memoizedState=e.memoizedState,n.updateQueue=e.updateQueue,t=e.dependencies,n.dependencies=null===t?null:{lanes:t.lanes,firstContext:t.firstContext},n.sibling=e.sibling,n.index=e.index,n.ref=e.ref,n}function Fs(e,t,n,r,a,o){var u=2;if(r=e,\"function\"==typeof e)Os(e)&&(u=1);else if(\"string\"==typeof e)u=5;else e:switch(e){case k:return Ds(n.children,a,o,t);case S:u=8,a|=8;break;case E:return(e=Ls(12,n,t,2|a)).elementType=E,e.lanes=o,e;case N:return(e=Ls(13,n,t,a)).elementType=N,e.lanes=o,e;case P:return(e=Ls(19,n,t,a)).elementType=P,e.lanes=o,e;case O:return Rs(n,a,o,t);default:if(\"object\"==typeof e&&null!==e)switch(e.$$typeof){case C:u=10;break e;case T:u=9;break e;case M:u=11;break e;case z:u=14;break e;case L:u=16,r=null;break e}throw Error(i(130,null==e?e:typeof e,\"\"))}return(t=Ls(u,n,t,a)).elementType=e,t.type=r,t.lanes=o,t}function Ds(e,t,n,r){return(e=Ls(7,e,r,t)).lanes=n,e}function Rs(e,t,n,r){return(e=Ls(22,e,r,t)).elementType=O,e.lanes=n,e.stateNode={isHidden:!1},e}function js(e,t,n){return(e=Ls(6,e,null,t)).lanes=n,e}function Us(e,t,n){return(t=Ls(4,null!==e.children?e.children:[],e.key,t)).lanes=n,t.stateNode={containerInfo:e.containerInfo,pendingChildren:null,implementation:e.implementation},t}function Is(e,t,n,r,a){this.tag=t,this.containerInfo=e,this.finishedWork=this.pingCache=this.current=this.pendingChildren=null,this.timeoutHandle=-1,this.callbackNode=this.pendingContext=this.context=null,this.callbackPriority=0,this.eventTimes=gt(0),this.expirationTimes=gt(-1),this.entangledLanes=this.finishedLanes=this.mutableReadLanes=this.expiredLanes=this.pingedLanes=this.suspendedLanes=this.pendingLanes=0,this.entanglements=gt(0),this.identifierPrefix=r,this.onRecoverableError=a,this.mutableSourceEagerHydrationData=null}function $s(e,t,n,r,a,i,o,u,l){return e=new Is(e,t,n,u,l),1===t?(t=1,!0===i&&(t|=8)):t=0,i=Ls(3,null,null,t),e.current=i,i.stateNode=e,i.memoizedState={element:r,isDehydrated:n,cache:null,transitions:null,pendingSuspenseBoundaries:null},zi(i),e}function Bs(e){if(!e)return Ta;e:{if(Be(e=e._reactInternals)!==e||1!==e.tag)throw Error(i(170));var t=e;do{switch(t.tag){case 3:t=t.stateNode.context;break e;case 1:if(La(t.type)){t=t.stateNode.__reactInternalMemoizedMergedChildContext;break e}}t=t.return}while(null!==t);throw Error(i(171))}if(1===e.tag){var n=e.type;if(La(n))return Fa(e,n,t)}return t}function Ws(e,t,n,r,a,i,o,u,l){return(e=$s(n,r,!0,e,0,i,0,u,l)).context=Bs(null),n=e.current,(i=Oi(r=ts(),a=ns(n))).callback=null!=t?t:null,Ai(n,i,a),e.current.lanes=a,yt(e,a,r),as(e,r),e}function Vs(e,t,n,r){var a=t.current,i=ts(),o=ns(a);return n=Bs(n),null===t.context?t.context=n:t.pendingContext=n,(t=Oi(i,o)).payload={element:e},null!==(r=void 0===r?null:r)&&(t.callback=r),null!==(e=Ai(a,t,o))&&(rs(e,a,o,i),Fi(e,a,o)),o}function Hs(e){return(e=e.current).child?(e.child.tag,e.child.stateNode):null}function qs(e,t){if(null!==(e=e.memoizedState)&&null!==e.dehydrated){var n=e.retryLane;e.retryLane=0!==n&&n<t?n:t}}function Qs(e,t){qs(e,t),(e=e.alternate)&&qs(e,t)}Sl=function(e,t,n){if(null!==e)if(e.memoizedProps!==t.pendingProps||Na.current)_u=!0;else{if(0==(e.lanes&n)&&0==(128&t.flags))return _u=!1,function(e,t,n){switch(t.tag){case 3:Pu(t),di();break;case 5:io(t);break;case 1:La(t.type)&&Da(t);break;case 4:ro(t,t.stateNode.containerInfo);break;case 10:var r=t.type._context,a=t.memoizedProps.value;Ca(yi,r._currentValue),r._currentValue=a;break;case 13:if(null!==(r=t.memoizedState))return null!==r.dehydrated?(Ca(uo,1&uo.current),t.flags|=128,null):0!=(n&t.child.childLanes)?ju(e,t,n):(Ca(uo,1&uo.current),null!==(e=Hu(e,t,n))?e.sibling:null);Ca(uo,1&uo.current);break;case 19:if(r=0!=(n&t.childLanes),0!=(128&e.flags)){if(r)return Wu(e,t,n);t.flags|=128}if(null!==(a=t.memoizedState)&&(a.rendering=null,a.tail=null,a.lastEffect=null),Ca(uo,uo.current),r)break;return null;case 22:case 23:return t.lanes=0,Eu(e,t,n)}return Hu(e,t,n)}(e,t,n);_u=0!=(131072&e.flags)}else _u=!1,ai&&0!=(1048576&t.flags)&&Ja(t,qa,t.index);switch(t.lanes=0,t.tag){case 2:var r=t.type;Vu(e,t),e=t.pendingProps;var a=za(t,Ma.current);Si(t,n),a=So(null,t,r,e,a,n);var o=Eo();return t.flags|=1,\"object\"==typeof a&&null!==a&&\"function\"==typeof a.render&&void 0===a.$$typeof?(t.tag=1,t.memoizedState=null,t.updateQueue=null,La(r)?(o=!0,Da(t)):o=!1,t.memoizedState=null!==a.state&&void 0!==a.state?a.state:null,zi(t),a.updater=$i,t.stateNode=a,a._reactInternals=t,Hi(t,r,e,n),t=Nu(null,t,r,!0,o,n)):(t.tag=0,ai&&o&&ei(t),wu(null,t,a,n),t=t.child),t;case 16:r=t.elementType;e:{switch(Vu(e,t),e=t.pendingProps,r=(a=r._init)(r._payload),t.type=r,a=t.tag=function(e){if(\"function\"==typeof e)return Os(e)?1:0;if(null!=e){if((e=e.$$typeof)===M)return 11;if(e===z)return 14}return 2}(r),e=gi(r,e),a){case 0:t=Tu(null,t,r,e,n);break e;case 1:t=Mu(null,t,r,e,n);break e;case 11:t=xu(null,t,r,e,n);break e;case 14:t=ku(null,t,r,gi(r.type,e),n);break e}throw Error(i(306,r,\"\"))}return t;case 0:return r=t.type,a=t.pendingProps,Tu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 1:return r=t.type,a=t.pendingProps,Mu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 3:e:{if(Pu(t),null===e)throw Error(i(387));r=t.pendingProps,a=(o=t.memoizedState).element,Li(e,t),Ri(t,r,null,n);var u=t.memoizedState;if(r=u.element,o.isDehydrated){if(o={element:r,isDehydrated:!1,cache:u.cache,pendingSuspenseBoundaries:u.pendingSuspenseBoundaries,transitions:u.transitions},t.updateQueue.baseState=o,t.memoizedState=o,256&t.flags){t=zu(e,t,r,n,a=cu(Error(i(423)),t));break e}if(r!==a){t=zu(e,t,r,n,a=cu(Error(i(424)),t));break e}for(ri=sa(t.stateNode.containerInfo.firstChild),ni=t,ai=!0,ii=null,n=Zi(t,null,r,n),t.child=n;n;)n.flags=-3&n.flags|4096,n=n.sibling}else{if(di(),r===a){t=Hu(e,t,n);break e}wu(e,t,r,n)}t=t.child}return t;case 5:return io(t),null===e&&si(t),r=t.type,a=t.pendingProps,o=null!==e?e.memoizedProps:null,u=a.children,na(r,a)?u=null:null!==o&&na(r,o)&&(t.flags|=32),Cu(e,t),wu(e,t,u,n),t.child;case 6:return null===e&&si(t),null;case 13:return ju(e,t,n);case 4:return ro(t,t.stateNode.containerInfo),r=t.pendingProps,null===e?t.child=Ki(t,null,r,n):wu(e,t,r,n),t.child;case 11:return r=t.type,a=t.pendingProps,xu(e,t,r,a=t.elementType===r?a:gi(r,a),n);case 7:return wu(e,t,t.pendingProps,n),t.child;case 8:case 12:return wu(e,t,t.pendingProps.children,n),t.child;case 10:e:{if(r=t.type._context,a=t.pendingProps,o=t.memoizedProps,u=a.value,Ca(yi,r._currentValue),r._currentValue=u,null!==o)if(ur(o.value,u)){if(o.children===a.children&&!Na.current){t=Hu(e,t,n);break e}}else for(null!==(o=t.child)&&(o.return=t);null!==o;){var l=o.dependencies;if(null!==l){u=o.child;for(var s=l.firstContext;null!==s;){if(s.context===r){if(1===o.tag){(s=Oi(-1,n&-n)).tag=2;var c=o.updateQueue;if(null!==c){var f=(c=c.shared).pending;null===f?s.next=s:(s.next=f.next,f.next=s),c.pending=s}}o.lanes|=n,null!==(s=o.alternate)&&(s.lanes|=n),ki(o.return,n,t),l.lanes|=n;break}s=s.next}}else if(10===o.tag)u=o.type===t.type?null:o.child;else if(18===o.tag){if(null===(u=o.return))throw Error(i(341));u.lanes|=n,null!==(l=u.alternate)&&(l.lanes|=n),ki(u,n,t),u=o.sibling}else u=o.child;if(null!==u)u.return=o;else for(u=o;null!==u;){if(u===t){u=null;break}if(null!==(o=u.sibling)){o.return=u.return,u=o;break}u=u.return}o=u}wu(e,t,a.children,n),t=t.child}return t;case 9:return a=t.type,r=t.pendingProps.children,Si(t,n),r=r(a=Ei(a)),t.flags|=1,wu(e,t,r,n),t.child;case 14:return a=gi(r=t.type,t.pendingProps),ku(e,t,r,a=gi(r.type,a),n);case 15:return Su(e,t,t.type,t.pendingProps,n);case 17:return r=t.type,a=t.pendingProps,a=t.elementType===r?a:gi(r,a),Vu(e,t),t.tag=1,La(r)?(e=!0,Da(t)):e=!1,Si(t,n),Wi(t,r,a),Hi(t,r,a,n),Nu(null,t,r,!0,e,n);case 19:return Wu(e,t,n);case 22:return Eu(e,t,n)}throw Error(i(156,t.tag))};var Ys=\"function\"==typeof reportError?reportError:function(e){console.error(e)};function Gs(e){this._internalRoot=e}function Ks(e){this._internalRoot=e}function Zs(e){return!(!e||1!==e.nodeType&&9!==e.nodeType&&11!==e.nodeType)}function Xs(e){return!(!e||1!==e.nodeType&&9!==e.nodeType&&11!==e.nodeType&&(8!==e.nodeType||\" react-mount-point-unstable \"!==e.nodeValue))}function Js(){}function ec(e,t,n,r,a){var i=n._reactRootContainer;if(i){var o=i;if(\"function\"==typeof a){var u=a;a=function(){var e=Hs(o);u.call(e)}}Vs(t,o,e,a)}else o=function(e,t,n,r,a){if(a){if(\"function\"==typeof r){var i=r;r=function(){var e=Hs(o);i.call(e)}}var o=Ws(t,r,e,0,null,!1,0,\"\",Js);return e._reactRootContainer=o,e[ha]=o.current,Br(8===e.nodeType?e.parentNode:e),fs(),o}for(;a=e.lastChild;)e.removeChild(a);if(\"function\"==typeof r){var u=r;r=function(){var e=Hs(l);u.call(e)}}var l=$s(e,0,!1,null,0,!1,0,\"\",Js);return e._reactRootContainer=l,e[ha]=l.current,Br(8===e.nodeType?e.parentNode:e),fs((function(){Vs(t,l,n,r)})),l}(n,t,e,a,r);return Hs(o)}Ks.prototype.render=Gs.prototype.render=function(e){var t=this._internalRoot;if(null===t)throw Error(i(409));Vs(e,t,null,null)},Ks.prototype.unmount=Gs.prototype.unmount=function(){var e=this._internalRoot;if(null!==e){this._internalRoot=null;var t=e.containerInfo;fs((function(){Vs(null,e,null,null)})),t[ha]=null}},Ks.prototype.unstable_scheduleHydration=function(e){if(e){var t=St();e={blockedOn:null,target:e,priority:t};for(var n=0;n<Ot.length&&0!==t&&t<Ot[n].priority;n++);Ot.splice(n,0,e),0===n&&Rt(e)}},wt=function(e){switch(e.tag){case 3:var t=e.stateNode;if(t.current.memoizedState.isDehydrated){var n=ft(t.pendingLanes);0!==n&&(mt(t,1|n),as(t,Ze()),0==(6&Nl)&&(Wl=Ze()+500,Ba()))}break;case 13:fs((function(){var t=Ni(e,1);if(null!==t){var n=ts();rs(t,e,1,n)}})),Qs(e,1)}},xt=function(e){if(13===e.tag){var t=Ni(e,134217728);null!==t&&rs(t,e,134217728,ts()),Qs(e,134217728)}},kt=function(e){if(13===e.tag){var t=ns(e),n=Ni(e,t);null!==n&&rs(n,e,t,ts()),Qs(e,t)}},St=function(){return bt},Et=function(e,t){var n=bt;try{return bt=e,t()}finally{bt=n}},xe=function(e,t,n){switch(t){case\"input\":if(X(e,n),t=n.name,\"radio\"===n.type&&null!=t){for(n=e;n.parentNode;)n=n.parentNode;for(n=n.querySelectorAll(\"input[name=\"+JSON.stringify(\"\"+t)+'][type=\"radio\"]'),t=0;t<n.length;t++){var r=n[t];if(r!==e&&r.form===e.form){var a=wa(r);if(!a)throw Error(i(90));Q(r),X(r,a)}}}break;case\"textarea\":ie(e,n);break;case\"select\":null!=(t=n.value)&&ne(e,!!n.multiple,t,!1)}},Me=cs,Ne=fs;var tc={usingClientEntryPoint:!1,Events:[ba,_a,wa,Ce,Te,cs]},nc={findFiberByHostInstance:ma,bundleType:0,version:\"18.2.0\",rendererPackageName:\"react-dom\"},rc={bundleType:nc.bundleType,version:nc.version,rendererPackageName:nc.rendererPackageName,rendererConfig:nc.rendererConfig,overrideHookState:null,overrideHookStateDeletePath:null,overrideHookStateRenamePath:null,overrideProps:null,overridePropsDeletePath:null,overridePropsRenamePath:null,setErrorHandler:null,setSuspenseHandler:null,scheduleUpdate:null,currentDispatcherRef:_.ReactCurrentDispatcher,findHostInstanceByFiber:function(e){return null===(e=He(e))?null:e.stateNode},findFiberByHostInstance:nc.findFiberByHostInstance||function(){return null},findHostInstancesForRefresh:null,scheduleRefresh:null,scheduleRoot:null,setRefreshHandler:null,getCurrentFiber:null,reconcilerVersion:\"18.2.0-next-9e3b772b8-20220608\"};if(\"undefined\"!=typeof __REACT_DEVTOOLS_GLOBAL_HOOK__){var ac=__REACT_DEVTOOLS_GLOBAL_HOOK__;if(!ac.isDisabled&&ac.supportsFiber)try{at=ac.inject(rc),it=ac}catch(ce){}}t.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED=tc,t.createPortal=function(e,t){var n=2<arguments.length&&void 0!==arguments[2]?arguments[2]:null;if(!Zs(t))throw Error(i(200));return function(e,t,n){var r=3<arguments.length&&void 0!==arguments[3]?arguments[3]:null;return{$$typeof:x,key:null==r?null:\"\"+r,children:e,containerInfo:t,implementation:n}}(e,t,null,n)},t.createRoot=function(e,t){if(!Zs(e))throw Error(i(299));var n=!1,r=\"\",a=Ys;return null!=t&&(!0===t.unstable_strictMode&&(n=!0),void 0!==t.identifierPrefix&&(r=t.identifierPrefix),void 0!==t.onRecoverableError&&(a=t.onRecoverableError)),t=$s(e,1,!1,null,0,n,0,r,a),e[ha]=t.current,Br(8===e.nodeType?e.parentNode:e),new Gs(t)},t.findDOMNode=function(e){if(null==e)return null;if(1===e.nodeType)return e;var t=e._reactInternals;if(void 0===t){if(\"function\"==typeof e.render)throw Error(i(188));throw e=Object.keys(e).join(\",\"),Error(i(268,e))}return null===(e=He(t))?null:e.stateNode},t.flushSync=function(e){return fs(e)},t.hydrate=function(e,t,n){if(!Xs(t))throw Error(i(200));return ec(null,e,t,!0,n)},t.hydrateRoot=function(e,t,n){if(!Zs(e))throw Error(i(405));var r=null!=n&&n.hydratedSources||null,a=!1,o=\"\",u=Ys;if(null!=n&&(!0===n.unstable_strictMode&&(a=!0),void 0!==n.identifierPrefix&&(o=n.identifierPrefix),void 0!==n.onRecoverableError&&(u=n.onRecoverableError)),t=Ws(t,null,e,1,null!=n?n:null,a,0,o,u),e[ha]=t.current,Br(e),r)for(e=0;e<r.length;e++)a=(a=(n=r[e])._getVersion)(n._source),null==t.mutableSourceEagerHydrationData?t.mutableSourceEagerHydrationData=[n,a]:t.mutableSourceEagerHydrationData.push(n,a);return new Ks(t)},t.render=function(e,t,n){if(!Xs(t))throw Error(i(200));return ec(null,e,t,!1,n)},t.unmountComponentAtNode=function(e){if(!Xs(e))throw Error(i(40));return!!e._reactRootContainer&&(fs((function(){ec(null,null,e,!1,(function(){e._reactRootContainer=null,e[ha]=null}))})),!0)},t.unstable_batchedUpdates=cs,t.unstable_renderSubtreeIntoContainer=function(e,t,n,r){if(!Xs(n))throw Error(i(200));if(null==e||void 0===e._reactInternals)throw Error(i(38));return ec(e,t,n,!1,r)},t.version=\"18.2.0-next-9e3b772b8-20220608\"},935:(e,t,n)=>{\"use strict\";!function e(){if(\"undefined\"!=typeof __REACT_DEVTOOLS_GLOBAL_HOOK__&&\"function\"==typeof __REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE)try{__REACT_DEVTOOLS_GLOBAL_HOOK__.checkDCE(e)}catch(e){console.error(e)}}(),e.exports=n(448)},408:(e,t)=>{\"use strict\";var n=Symbol.for(\"react.element\"),r=Symbol.for(\"react.portal\"),a=Symbol.for(\"react.fragment\"),i=Symbol.for(\"react.strict_mode\"),o=Symbol.for(\"react.profiler\"),u=Symbol.for(\"react.provider\"),l=Symbol.for(\"react.context\"),s=Symbol.for(\"react.forward_ref\"),c=Symbol.for(\"react.suspense\"),f=Symbol.for(\"react.memo\"),p=Symbol.for(\"react.lazy\"),d=Symbol.iterator,h={isMounted:function(){return!1},enqueueForceUpdate:function(){},enqueueReplaceState:function(){},enqueueSetState:function(){}},v=Object.assign,g={};function y(e,t,n){this.props=e,this.context=t,this.refs=g,this.updater=n||h}function m(){}function b(e,t,n){this.props=e,this.context=t,this.refs=g,this.updater=n||h}y.prototype.isReactComponent={},y.prototype.setState=function(e,t){if(\"object\"!=typeof e&&\"function\"!=typeof e&&null!=e)throw Error(\"setState(...): takes an object of state variables to update or a function which returns an object of state variables.\");this.updater.enqueueSetState(this,e,t,\"setState\")},y.prototype.forceUpdate=function(e){this.updater.enqueueForceUpdate(this,e,\"forceUpdate\")},m.prototype=y.prototype;var _=b.prototype=new m;_.constructor=b,v(_,y.prototype),_.isPureReactComponent=!0;var w=Array.isArray,x=Object.prototype.hasOwnProperty,k={current:null},S={key:!0,ref:!0,__self:!0,__source:!0};function E(e,t,r){var a,i={},o=null,u=null;if(null!=t)for(a in void 0!==t.ref&&(u=t.ref),void 0!==t.key&&(o=\"\"+t.key),t)x.call(t,a)&&!S.hasOwnProperty(a)&&(i[a]=t[a]);var l=arguments.length-2;if(1===l)i.children=r;else if(1<l){for(var s=Array(l),c=0;c<l;c++)s[c]=arguments[c+2];i.children=s}if(e&&e.defaultProps)for(a in l=e.defaultProps)void 0===i[a]&&(i[a]=l[a]);return{$$typeof:n,type:e,key:o,ref:u,props:i,_owner:k.current}}function C(e){return\"object\"==typeof e&&null!==e&&e.$$typeof===n}var T=/\\/+/g;function M(e,t){return\"object\"==typeof e&&null!==e&&null!=e.key?function(e){var t={\"=\":\"=0\",\":\":\"=2\"};return\"$\"+e.replace(/[=:]/g,(function(e){return t[e]}))}(\"\"+e.key):t.toString(36)}function N(e,t,a,i,o){var u=typeof e;\"undefined\"!==u&&\"boolean\"!==u||(e=null);var l=!1;if(null===e)l=!0;else switch(u){case\"string\":case\"number\":l=!0;break;case\"object\":switch(e.$$typeof){case n:case r:l=!0}}if(l)return o=o(l=e),e=\"\"===i?\".\"+M(l,0):i,w(o)?(a=\"\",null!=e&&(a=e.replace(T,\"$&/\")+\"/\"),N(o,t,a,\"\",(function(e){return e}))):null!=o&&(C(o)&&(o=function(e,t){return{$$typeof:n,type:e.type,key:t,ref:e.ref,props:e.props,_owner:e._owner}}(o,a+(!o.key||l&&l.key===o.key?\"\":(\"\"+o.key).replace(T,\"$&/\")+\"/\")+e)),t.push(o)),1;if(l=0,i=\"\"===i?\".\":i+\":\",w(e))for(var s=0;s<e.length;s++){var c=i+M(u=e[s],s);l+=N(u,t,a,c,o)}else if(c=function(e){return null===e||\"object\"!=typeof e?null:\"function\"==typeof(e=d&&e[d]||e[\"@@iterator\"])?e:null}(e),\"function\"==typeof c)for(e=c.call(e),s=0;!(u=e.next()).done;)l+=N(u=u.value,t,a,c=i+M(u,s++),o);else if(\"object\"===u)throw t=String(e),Error(\"Objects are not valid as a React child (found: \"+(\"[object Object]\"===t?\"object with keys {\"+Object.keys(e).join(\", \")+\"}\":t)+\"). If you meant to render a collection of children, use an array instead.\");return l}function P(e,t,n){if(null==e)return e;var r=[],a=0;return N(e,r,\"\",\"\",(function(e){return t.call(n,e,a++)})),r}function z(e){if(-1===e._status){var t=e._result;(t=t()).then((function(t){0!==e._status&&-1!==e._status||(e._status=1,e._result=t)}),(function(t){0!==e._status&&-1!==e._status||(e._status=2,e._result=t)})),-1===e._status&&(e._status=0,e._result=t)}if(1===e._status)return e._result.default;throw e._result}var L={current:null},O={transition:null},A={ReactCurrentDispatcher:L,ReactCurrentBatchConfig:O,ReactCurrentOwner:k};t.Children={map:P,forEach:function(e,t,n){P(e,(function(){t.apply(this,arguments)}),n)},count:function(e){var t=0;return P(e,(function(){t++})),t},toArray:function(e){return P(e,(function(e){return e}))||[]},only:function(e){if(!C(e))throw Error(\"React.Children.only expected to receive a single React element child.\");return e}},t.Component=y,t.Fragment=a,t.Profiler=o,t.PureComponent=b,t.StrictMode=i,t.Suspense=c,t.__SECRET_INTERNALS_DO_NOT_USE_OR_YOU_WILL_BE_FIRED=A,t.cloneElement=function(e,t,r){if(null==e)throw Error(\"React.cloneElement(...): The argument must be a React element, but you passed \"+e+\".\");var a=v({},e.props),i=e.key,o=e.ref,u=e._owner;if(null!=t){if(void 0!==t.ref&&(o=t.ref,u=k.current),void 0!==t.key&&(i=\"\"+t.key),e.type&&e.type.defaultProps)var l=e.type.defaultProps;for(s in t)x.call(t,s)&&!S.hasOwnProperty(s)&&(a[s]=void 0===t[s]&&void 0!==l?l[s]:t[s])}var s=arguments.length-2;if(1===s)a.children=r;else if(1<s){l=Array(s);for(var c=0;c<s;c++)l[c]=arguments[c+2];a.children=l}return{$$typeof:n,type:e.type,key:i,ref:o,props:a,_owner:u}},t.createContext=function(e){return(e={$$typeof:l,_currentValue:e,_currentValue2:e,_threadCount:0,Provider:null,Consumer:null,_defaultValue:null,_globalName:null}).Provider={$$typeof:u,_context:e},e.Consumer=e},t.createElement=E,t.createFactory=function(e){var t=E.bind(null,e);return t.type=e,t},t.createRef=function(){return{current:null}},t.forwardRef=function(e){return{$$typeof:s,render:e}},t.isValidElement=C,t.lazy=function(e){return{$$typeof:p,_payload:{_status:-1,_result:e},_init:z}},t.memo=function(e,t){return{$$typeof:f,type:e,compare:void 0===t?null:t}},t.startTransition=function(e){var t=O.transition;O.transition={};try{e()}finally{O.transition=t}},t.unstable_act=function(){throw Error(\"act(...) is not supported in production builds of React.\")},t.useCallback=function(e,t){return L.current.useCallback(e,t)},t.useContext=function(e){return L.current.useContext(e)},t.useDebugValue=function(){},t.useDeferredValue=function(e){return L.current.useDeferredValue(e)},t.useEffect=function(e,t){return L.current.useEffect(e,t)},t.useId=function(){return L.current.useId()},t.useImperativeHandle=function(e,t,n){return L.current.useImperativeHandle(e,t,n)},t.useInsertionEffect=function(e,t){return L.current.useInsertionEffect(e,t)},t.useLayoutEffect=function(e,t){return L.current.useLayoutEffect(e,t)},t.useMemo=function(e,t){return L.current.useMemo(e,t)},t.useReducer=function(e,t,n){return L.current.useReducer(e,t,n)},t.useRef=function(e){return L.current.useRef(e)},t.useState=function(e){return L.current.useState(e)},t.useSyncExternalStore=function(e,t,n){return L.current.useSyncExternalStore(e,t,n)},t.useTransition=function(){return L.current.useTransition()},t.version=\"18.2.0\"},294:(e,t,n)=>{\"use strict\";e.exports=n(408)},53:(e,t)=>{\"use strict\";function n(e,t){var n=e.length;e.push(t);e:for(;0<n;){var r=n-1>>>1,a=e[r];if(!(0<i(a,t)))break e;e[r]=t,e[n]=a,n=r}}function r(e){return 0===e.length?null:e[0]}function a(e){if(0===e.length)return null;var t=e[0],n=e.pop();if(n!==t){e[0]=n;e:for(var r=0,a=e.length,o=a>>>1;r<o;){var u=2*(r+1)-1,l=e[u],s=u+1,c=e[s];if(0>i(l,n))s<a&&0>i(c,l)?(e[r]=c,e[s]=n,r=s):(e[r]=l,e[u]=n,r=u);else{if(!(s<a&&0>i(c,n)))break e;e[r]=c,e[s]=n,r=s}}}return t}function i(e,t){var n=e.sortIndex-t.sortIndex;return 0!==n?n:e.id-t.id}if(\"object\"==typeof performance&&\"function\"==typeof performance.now){var o=performance;t.unstable_now=function(){return o.now()}}else{var u=Date,l=u.now();t.unstable_now=function(){return u.now()-l}}var s=[],c=[],f=1,p=null,d=3,h=!1,v=!1,g=!1,y=\"function\"==typeof setTimeout?setTimeout:null,m=\"function\"==typeof clearTimeout?clearTimeout:null,b=\"undefined\"!=typeof setImmediate?setImmediate:null;function _(e){for(var t=r(c);null!==t;){if(null===t.callback)a(c);else{if(!(t.startTime<=e))break;a(c),t.sortIndex=t.expirationTime,n(s,t)}t=r(c)}}function w(e){if(g=!1,_(e),!v)if(null!==r(s))v=!0,O(x);else{var t=r(c);null!==t&&A(w,t.startTime-e)}}function x(e,n){v=!1,g&&(g=!1,m(C),C=-1),h=!0;var i=d;try{for(_(n),p=r(s);null!==p&&(!(p.expirationTime>n)||e&&!N());){var o=p.callback;if(\"function\"==typeof o){p.callback=null,d=p.priorityLevel;var u=o(p.expirationTime<=n);n=t.unstable_now(),\"function\"==typeof u?p.callback=u:p===r(s)&&a(s),_(n)}else a(s);p=r(s)}if(null!==p)var l=!0;else{var f=r(c);null!==f&&A(w,f.startTime-n),l=!1}return l}finally{p=null,d=i,h=!1}}\"undefined\"!=typeof navigator&&void 0!==navigator.scheduling&&void 0!==navigator.scheduling.isInputPending&&navigator.scheduling.isInputPending.bind(navigator.scheduling);var k,S=!1,E=null,C=-1,T=5,M=-1;function N(){return!(t.unstable_now()-M<T)}function P(){if(null!==E){var e=t.unstable_now();M=e;var n=!0;try{n=E(!0,e)}finally{n?k():(S=!1,E=null)}}else S=!1}if(\"function\"==typeof b)k=function(){b(P)};else if(\"undefined\"!=typeof MessageChannel){var z=new MessageChannel,L=z.port2;z.port1.onmessage=P,k=function(){L.postMessage(null)}}else k=function(){y(P,0)};function O(e){E=e,S||(S=!0,k())}function A(e,n){C=y((function(){e(t.unstable_now())}),n)}t.unstable_IdlePriority=5,t.unstable_ImmediatePriority=1,t.unstable_LowPriority=4,t.unstable_NormalPriority=3,t.unstable_Profiling=null,t.unstable_UserBlockingPriority=2,t.unstable_cancelCallback=function(e){e.callback=null},t.unstable_continueExecution=function(){v||h||(v=!0,O(x))},t.unstable_forceFrameRate=function(e){0>e||125<e?console.error(\"forceFrameRate takes a positive int between 0 and 125, forcing frame rates higher than 125 fps is not supported\"):T=0<e?Math.floor(1e3/e):5},t.unstable_getCurrentPriorityLevel=function(){return d},t.unstable_getFirstCallbackNode=function(){return r(s)},t.unstable_next=function(e){switch(d){case 1:case 2:case 3:var t=3;break;default:t=d}var n=d;d=t;try{return e()}finally{d=n}},t.unstable_pauseExecution=function(){},t.unstable_requestPaint=function(){},t.unstable_runWithPriority=function(e,t){switch(e){case 1:case 2:case 3:case 4:case 5:break;default:e=3}var n=d;d=e;try{return t()}finally{d=n}},t.unstable_scheduleCallback=function(e,a,i){var o=t.unstable_now();switch(i=\"object\"==typeof i&&null!==i&&\"number\"==typeof(i=i.delay)&&0<i?o+i:o,e){case 1:var u=-1;break;case 2:u=250;break;case 5:u=1073741823;break;case 4:u=1e4;break;default:u=5e3}return e={id:f++,callback:a,priorityLevel:e,startTime:i,expirationTime:u=i+u,sortIndex:-1},i>o?(e.sortIndex=i,n(c,e),null===r(s)&&e===r(c)&&(g?(m(C),C=-1):g=!0,A(w,i-o))):(e.sortIndex=u,n(s,e),v||h||(v=!0,O(x))),e},t.unstable_shouldYield=N,t.unstable_wrapCallback=function(e){var t=d;return function(){var n=d;d=t;try{return e.apply(this,arguments)}finally{d=n}}}},840:(e,t,n)=>{\"use strict\";e.exports=n(53)}},t={};function n(r){var a=t[r];if(void 0!==a)return a.exports;var i=t[r]={id:r,loaded:!1,exports:{}};return e[r].call(i.exports,i,i.exports,n),i.loaded=!0,i.exports}n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(e){if(\"object\"==typeof window)return window}}(),n.nmd=e=>(e.paths=[],e.children||(e.children=[]),e),(()=>{\"use strict\";var e=n(294),t=n(935);const r=Math.sqrt(50),a=Math.sqrt(10),i=Math.sqrt(2);function o(e,t,n){const u=(t-e)/Math.max(0,n),l=Math.floor(Math.log10(u)),s=u/Math.pow(10,l),c=s>=r?10:s>=a?5:s>=i?2:1;let f,p,d;return l<0?(d=Math.pow(10,-l)/c,f=Math.round(e*d),p=Math.round(t*d),f/d<e&&++f,p/d>t&&--p,d=-d):(d=Math.pow(10,l)*c,f=Math.round(e/d),p=Math.round(t/d),f*d<e&&++f,p*d>t&&--p),p<f&&.5<=n&&n<2?o(e,t,2*n):[f,p,d]}function u(e,t,n){return o(e=+e,t=+t,n=+n)[2]}function l(e,t,n){n=+n;const r=(t=+t)<(e=+e),a=r?u(t,e,n):u(e,t,n);return(r?-1:1)*(a<0?1/-a:a)}function s(e,t){return null==e||null==t?NaN:e<t?-1:e>t?1:e>=t?0:NaN}function c(e,t){return null==e||null==t?NaN:t<e?-1:t>e?1:t>=e?0:NaN}function f(e){let t,n,r;function a(e,r,a=0,i=e.length){if(a<i){if(0!==t(r,r))return i;do{const t=a+i>>>1;n(e[t],r)<0?a=t+1:i=t}while(a<i)}return a}return 2!==e.length?(t=s,n=(t,n)=>s(e(t),n),r=(t,n)=>e(t)-n):(t=e===s||e===c?e:p,n=e,r=e),{left:a,center:function(e,t,n=0,i=e.length){const o=a(e,t,n,i-1);return o>n&&r(e[o-1],t)>-r(e[o],t)?o-1:o},right:function(e,r,a=0,i=e.length){if(a<i){if(0!==t(r,r))return i;do{const t=a+i>>>1;n(e[t],r)<=0?a=t+1:i=t}while(a<i)}return a}}}function p(){return 0}const d=f(s),h=d.right,v=(d.left,f((function(e){return null===e?NaN:+e})).center,h);function g(e,t,n){e.prototype=t.prototype=n,n.constructor=e}function y(e,t){var n=Object.create(e.prototype);for(var r in t)n[r]=t[r];return n}function m(){}var b=.7,_=1/b,w=\"\\\\s*([+-]?\\\\d+)\\\\s*\",x=\"\\\\s*([+-]?(?:\\\\d*\\\\.)?\\\\d+(?:[eE][+-]?\\\\d+)?)\\\\s*\",k=\"\\\\s*([+-]?(?:\\\\d*\\\\.)?\\\\d+(?:[eE][+-]?\\\\d+)?)%\\\\s*\",S=/^#([0-9a-f]{3,8})$/,E=new RegExp(`^rgb\\\\(${w},${w},${w}\\\\)$`),C=new RegExp(`^rgb\\\\(${k},${k},${k}\\\\)$`),T=new RegExp(`^rgba\\\\(${w},${w},${w},${x}\\\\)$`),M=new RegExp(`^rgba\\\\(${k},${k},${k},${x}\\\\)$`),N=new RegExp(`^hsl\\\\(${x},${k},${k}\\\\)$`),P=new RegExp(`^hsla\\\\(${x},${k},${k},${x}\\\\)$`),z={aliceblue:15792383,antiquewhite:16444375,aqua:65535,aquamarine:8388564,azure:15794175,beige:16119260,bisque:16770244,black:0,blanchedalmond:16772045,blue:255,blueviolet:9055202,brown:10824234,burlywood:14596231,cadetblue:6266528,chartreuse:8388352,chocolate:13789470,coral:16744272,cornflowerblue:6591981,cornsilk:16775388,crimson:14423100,cyan:65535,darkblue:139,darkcyan:35723,darkgoldenrod:12092939,darkgray:11119017,darkgreen:25600,darkgrey:11119017,darkkhaki:12433259,darkmagenta:9109643,darkolivegreen:5597999,darkorange:16747520,darkorchid:10040012,darkred:9109504,darksalmon:15308410,darkseagreen:9419919,darkslateblue:4734347,darkslategray:3100495,darkslategrey:3100495,darkturquoise:52945,darkviolet:9699539,deeppink:16716947,deepskyblue:49151,dimgray:6908265,dimgrey:6908265,dodgerblue:2003199,firebrick:11674146,floralwhite:16775920,forestgreen:2263842,fuchsia:16711935,gainsboro:14474460,ghostwhite:16316671,gold:16766720,goldenrod:14329120,gray:8421504,green:32768,greenyellow:11403055,grey:8421504,honeydew:15794160,hotpink:16738740,indianred:13458524,indigo:4915330,ivory:16777200,khaki:15787660,lavender:15132410,lavenderblush:16773365,lawngreen:8190976,lemonchiffon:16775885,lightblue:11393254,lightcoral:15761536,lightcyan:14745599,lightgoldenrodyellow:16448210,lightgray:13882323,lightgreen:9498256,lightgrey:13882323,lightpink:16758465,lightsalmon:16752762,lightseagreen:2142890,lightskyblue:8900346,lightslategray:7833753,lightslategrey:7833753,lightsteelblue:11584734,lightyellow:16777184,lime:65280,limegreen:3329330,linen:16445670,magenta:16711935,maroon:8388608,mediumaquamarine:6737322,mediumblue:205,mediumorchid:12211667,mediumpurple:9662683,mediumseagreen:3978097,mediumslateblue:8087790,mediumspringgreen:64154,mediumturquoise:4772300,mediumvioletred:13047173,midnightblue:1644912,mintcream:16121850,mistyrose:16770273,moccasin:16770229,navajowhite:16768685,navy:128,oldlace:16643558,olive:8421376,olivedrab:7048739,orange:16753920,orangered:16729344,orchid:14315734,palegoldenrod:15657130,palegreen:10025880,paleturquoise:11529966,palevioletred:14381203,papayawhip:16773077,peachpuff:16767673,peru:13468991,pink:16761035,plum:14524637,powderblue:11591910,purple:8388736,rebeccapurple:6697881,red:16711680,rosybrown:12357519,royalblue:4286945,saddlebrown:9127187,salmon:16416882,sandybrown:16032864,seagreen:3050327,seashell:16774638,sienna:10506797,silver:12632256,skyblue:8900331,slateblue:6970061,slategray:7372944,slategrey:7372944,snow:16775930,springgreen:65407,steelblue:4620980,tan:13808780,teal:32896,thistle:14204888,tomato:16737095,turquoise:4251856,violet:15631086,wheat:16113331,white:16777215,whitesmoke:16119285,yellow:16776960,yellowgreen:10145074};function L(){return this.rgb().formatHex()}function O(){return this.rgb().formatRgb()}function A(e){var t,n;return e=(e+\"\").trim().toLowerCase(),(t=S.exec(e))?(n=t[1].length,t=parseInt(t[1],16),6===n?F(t):3===n?new j(t>>8&15|t>>4&240,t>>4&15|240&t,(15&t)<<4|15&t,1):8===n?D(t>>24&255,t>>16&255,t>>8&255,(255&t)/255):4===n?D(t>>12&15|t>>8&240,t>>8&15|t>>4&240,t>>4&15|240&t,((15&t)<<4|15&t)/255):null):(t=E.exec(e))?new j(t[1],t[2],t[3],1):(t=C.exec(e))?new j(255*t[1]/100,255*t[2]/100,255*t[3]/100,1):(t=T.exec(e))?D(t[1],t[2],t[3],t[4]):(t=M.exec(e))?D(255*t[1]/100,255*t[2]/100,255*t[3]/100,t[4]):(t=N.exec(e))?V(t[1],t[2]/100,t[3]/100,1):(t=P.exec(e))?V(t[1],t[2]/100,t[3]/100,t[4]):z.hasOwnProperty(e)?F(z[e]):\"transparent\"===e?new j(NaN,NaN,NaN,0):null}function F(e){return new j(e>>16&255,e>>8&255,255&e,1)}function D(e,t,n,r){return r<=0&&(e=t=n=NaN),new j(e,t,n,r)}function R(e,t,n,r){return 1===arguments.length?((a=e)instanceof m||(a=A(a)),a?new j((a=a.rgb()).r,a.g,a.b,a.opacity):new j):new j(e,t,n,null==r?1:r);var a}function j(e,t,n,r){this.r=+e,this.g=+t,this.b=+n,this.opacity=+r}function U(){return`#${W(this.r)}${W(this.g)}${W(this.b)}`}function I(){const e=$(this.opacity);return`${1===e?\"rgb(\":\"rgba(\"}${B(this.r)}, ${B(this.g)}, ${B(this.b)}${1===e?\")\":`, ${e})`}`}function $(e){return isNaN(e)?1:Math.max(0,Math.min(1,e))}function B(e){return Math.max(0,Math.min(255,Math.round(e)||0))}function W(e){return((e=B(e))<16?\"0\":\"\")+e.toString(16)}function V(e,t,n,r){return r<=0?e=t=n=NaN:n<=0||n>=1?e=t=NaN:t<=0&&(e=NaN),new Q(e,t,n,r)}function H(e){if(e instanceof Q)return new Q(e.h,e.s,e.l,e.opacity);if(e instanceof m||(e=A(e)),!e)return new Q;if(e instanceof Q)return e;var t=(e=e.rgb()).r/255,n=e.g/255,r=e.b/255,a=Math.min(t,n,r),i=Math.max(t,n,r),o=NaN,u=i-a,l=(i+a)/2;return u?(o=t===i?(n-r)/u+6*(n<r):n===i?(r-t)/u+2:(t-n)/u+4,u/=l<.5?i+a:2-i-a,o*=60):u=l>0&&l<1?0:o,new Q(o,u,l,e.opacity)}function q(e,t,n,r){return 1===arguments.length?H(e):new Q(e,t,n,null==r?1:r)}function Q(e,t,n,r){this.h=+e,this.s=+t,this.l=+n,this.opacity=+r}function Y(e){return(e=(e||0)%360)<0?e+360:e}function G(e){return Math.max(0,Math.min(1,e||0))}function K(e,t,n){return 255*(e<60?t+(n-t)*e/60:e<180?n:e<240?t+(n-t)*(240-e)/60:t)}function Z(e,t,n,r,a){var i=e*e,o=i*e;return((1-3*e+3*i-o)*t+(4-6*i+3*o)*n+(1+3*e+3*i-3*o)*r+o*a)/6}g(m,A,{copy(e){return Object.assign(new this.constructor,this,e)},displayable(){return this.rgb().displayable()},hex:L,formatHex:L,formatHex8:function(){return this.rgb().formatHex8()},formatHsl:function(){return H(this).formatHsl()},formatRgb:O,toString:O}),g(j,R,y(m,{brighter(e){return e=null==e?_:Math.pow(_,e),new j(this.r*e,this.g*e,this.b*e,this.opacity)},darker(e){return e=null==e?b:Math.pow(b,e),new j(this.r*e,this.g*e,this.b*e,this.opacity)},rgb(){return this},clamp(){return new j(B(this.r),B(this.g),B(this.b),$(this.opacity))},displayable(){return-.5<=this.r&&this.r<255.5&&-.5<=this.g&&this.g<255.5&&-.5<=this.b&&this.b<255.5&&0<=this.opacity&&this.opacity<=1},hex:U,formatHex:U,formatHex8:function(){return`#${W(this.r)}${W(this.g)}${W(this.b)}${W(255*(isNaN(this.opacity)?1:this.opacity))}`},formatRgb:I,toString:I})),g(Q,q,y(m,{brighter(e){return e=null==e?_:Math.pow(_,e),new Q(this.h,this.s,this.l*e,this.opacity)},darker(e){return e=null==e?b:Math.pow(b,e),new Q(this.h,this.s,this.l*e,this.opacity)},rgb(){var e=this.h%360+360*(this.h<0),t=isNaN(e)||isNaN(this.s)?0:this.s,n=this.l,r=n+(n<.5?n:1-n)*t,a=2*n-r;return new j(K(e>=240?e-240:e+120,a,r),K(e,a,r),K(e<120?e+240:e-120,a,r),this.opacity)},clamp(){return new Q(Y(this.h),G(this.s),G(this.l),$(this.opacity))},displayable(){return(0<=this.s&&this.s<=1||isNaN(this.s))&&0<=this.l&&this.l<=1&&0<=this.opacity&&this.opacity<=1},formatHsl(){const e=$(this.opacity);return`${1===e?\"hsl(\":\"hsla(\"}${Y(this.h)}, ${100*G(this.s)}%, ${100*G(this.l)}%${1===e?\")\":`, ${e})`}`}}));const X=e=>()=>e;function J(e,t){var n=t-e;return n?function(e,t){return function(n){return e+n*t}}(e,n):X(isNaN(e)?t:e)}const ee=function e(t){var n=function(e){return 1==(e=+e)?J:function(t,n){return n-t?function(e,t,n){return e=Math.pow(e,n),t=Math.pow(t,n)-e,n=1/n,function(r){return Math.pow(e+r*t,n)}}(t,n,e):X(isNaN(t)?n:t)}}(t);function r(e,t){var r=n((e=R(e)).r,(t=R(t)).r),a=n(e.g,t.g),i=n(e.b,t.b),o=J(e.opacity,t.opacity);return function(t){return e.r=r(t),e.g=a(t),e.b=i(t),e.opacity=o(t),e+\"\"}}return r.gamma=e,r}(1);function te(e){return function(t){var n,r,a=t.length,i=new Array(a),o=new Array(a),u=new Array(a);for(n=0;n<a;++n)r=R(t[n]),i[n]=r.r||0,o[n]=r.g||0,u[n]=r.b||0;return i=e(i),o=e(o),u=e(u),r.opacity=1,function(e){return r.r=i(e),r.g=o(e),r.b=u(e),r+\"\"}}}function ne(e,t){var n,r=t?t.length:0,a=e?Math.min(r,e.length):0,i=new Array(a),o=new Array(r);for(n=0;n<a;++n)i[n]=ce(e[n],t[n]);for(;n<r;++n)o[n]=t[n];return function(e){for(n=0;n<a;++n)o[n]=i[n](e);return o}}function re(e,t){var n=new Date;return e=+e,t=+t,function(r){return n.setTime(e*(1-r)+t*r),n}}function ae(e,t){return e=+e,t=+t,function(n){return e*(1-n)+t*n}}function ie(e,t){var n,r={},a={};for(n in null!==e&&\"object\"==typeof e||(e={}),null!==t&&\"object\"==typeof t||(t={}),t)n in e?r[n]=ce(e[n],t[n]):a[n]=t[n];return function(e){for(n in r)a[n]=r[n](e);return a}}te((function(e){var t=e.length-1;return function(n){var r=n<=0?n=0:n>=1?(n=1,t-1):Math.floor(n*t),a=e[r],i=e[r+1],o=r>0?e[r-1]:2*a-i,u=r<t-1?e[r+2]:2*i-a;return Z((n-r/t)*t,o,a,i,u)}})),te((function(e){var t=e.length;return function(n){var r=Math.floor(((n%=1)<0?++n:n)*t),a=e[(r+t-1)%t],i=e[r%t],o=e[(r+1)%t],u=e[(r+2)%t];return Z((n-r/t)*t,a,i,o,u)}}));var oe=/[-+]?(?:\\d+\\.?\\d*|\\.?\\d+)(?:[eE][-+]?\\d+)?/g,ue=new RegExp(oe.source,\"g\");function le(e,t){var n,r,a,i=oe.lastIndex=ue.lastIndex=0,o=-1,u=[],l=[];for(e+=\"\",t+=\"\";(n=oe.exec(e))&&(r=ue.exec(t));)(a=r.index)>i&&(a=t.slice(i,a),u[o]?u[o]+=a:u[++o]=a),(n=n[0])===(r=r[0])?u[o]?u[o]+=r:u[++o]=r:(u[++o]=null,l.push({i:o,x:ae(n,r)})),i=ue.lastIndex;return i<t.length&&(a=t.slice(i),u[o]?u[o]+=a:u[++o]=a),u.length<2?l[0]?function(e){return function(t){return e(t)+\"\"}}(l[0].x):function(e){return function(){return e}}(t):(t=l.length,function(e){for(var n,r=0;r<t;++r)u[(n=l[r]).i]=n.x(e);return u.join(\"\")})}function se(e,t){t||(t=[]);var n,r=e?Math.min(t.length,e.length):0,a=t.slice();return function(i){for(n=0;n<r;++n)a[n]=e[n]*(1-i)+t[n]*i;return a}}function ce(e,t){var n,r,a=typeof t;return null==t||\"boolean\"===a?X(t):(\"number\"===a?ae:\"string\"===a?(n=A(t))?(t=n,ee):le:t instanceof A?ee:t instanceof Date?re:(r=t,!ArrayBuffer.isView(r)||r instanceof DataView?Array.isArray(t)?ne:\"function\"!=typeof t.valueOf&&\"function\"!=typeof t.toString||isNaN(t)?ie:ae:se))(e,t)}function fe(e,t){return e=+e,t=+t,function(n){return Math.round(e*(1-n)+t*n)}}function pe(e){return+e}var de=[0,1];function he(e){return e}function ve(e,t){return(t-=e=+e)?function(n){return(n-e)/t}:(n=isNaN(t)?NaN:.5,function(){return n});var n}function ge(e,t,n){var r=e[0],a=e[1],i=t[0],o=t[1];return a<r?(r=ve(a,r),i=n(o,i)):(r=ve(r,a),i=n(i,o)),function(e){return i(r(e))}}function ye(e,t,n){var r=Math.min(e.length,t.length)-1,a=new Array(r),i=new Array(r),o=-1;for(e[r]<e[0]&&(e=e.slice().reverse(),t=t.slice().reverse());++o<r;)a[o]=ve(e[o],e[o+1]),i[o]=n(t[o],t[o+1]);return function(t){var n=v(e,t,1,r)-1;return i[n](a[n](t))}}function me(e,t){return t.domain(e.domain()).range(e.range()).interpolate(e.interpolate()).clamp(e.clamp()).unknown(e.unknown())}function be(){return function(){var e,t,n,r,a,i,o=de,u=de,l=ce,s=he;function c(){var e,t,n,l=Math.min(o.length,u.length);return s!==he&&(e=o[0],t=o[l-1],e>t&&(n=e,e=t,t=n),s=function(n){return Math.max(e,Math.min(t,n))}),r=l>2?ye:ge,a=i=null,f}function f(t){return null==t||isNaN(t=+t)?n:(a||(a=r(o.map(e),u,l)))(e(s(t)))}return f.invert=function(n){return s(t((i||(i=r(u,o.map(e),ae)))(n)))},f.domain=function(e){return arguments.length?(o=Array.from(e,pe),c()):o.slice()},f.range=function(e){return arguments.length?(u=Array.from(e),c()):u.slice()},f.rangeRound=function(e){return u=Array.from(e),l=fe,c()},f.clamp=function(e){return arguments.length?(s=!!e||he,c()):s!==he},f.interpolate=function(e){return arguments.length?(l=e,c()):l},f.unknown=function(e){return arguments.length?(n=e,f):n},function(n,r){return e=n,t=r,c()}}()(he,he)}function _e(e,t){switch(arguments.length){case 0:break;case 1:this.range(e);break;default:this.range(t).domain(e)}return this}var we,xe=/^(?:(.)?([<>=^]))?([+\\-( ])?([$#])?(0)?(\\d+)?(,)?(\\.\\d+)?(~)?([a-z%])?$/i;function ke(e){if(!(t=xe.exec(e)))throw new Error(\"invalid format: \"+e);var t;return new Se({fill:t[1],align:t[2],sign:t[3],symbol:t[4],zero:t[5],width:t[6],comma:t[7],precision:t[8]&&t[8].slice(1),trim:t[9],type:t[10]})}function Se(e){this.fill=void 0===e.fill?\" \":e.fill+\"\",this.align=void 0===e.align?\">\":e.align+\"\",this.sign=void 0===e.sign?\"-\":e.sign+\"\",this.symbol=void 0===e.symbol?\"\":e.symbol+\"\",this.zero=!!e.zero,this.width=void 0===e.width?void 0:+e.width,this.comma=!!e.comma,this.precision=void 0===e.precision?void 0:+e.precision,this.trim=!!e.trim,this.type=void 0===e.type?\"\":e.type+\"\"}function Ee(e,t){if((n=(e=t?e.toExponential(t-1):e.toExponential()).indexOf(\"e\"))<0)return null;var n,r=e.slice(0,n);return[r.length>1?r[0]+r.slice(2):r,+e.slice(n+1)]}function Ce(e){return(e=Ee(Math.abs(e)))?e[1]:NaN}function Te(e,t){var n=Ee(e,t);if(!n)return e+\"\";var r=n[0],a=n[1];return a<0?\"0.\"+new Array(-a).join(\"0\")+r:r.length>a+1?r.slice(0,a+1)+\".\"+r.slice(a+1):r+new Array(a-r.length+2).join(\"0\")}ke.prototype=Se.prototype,Se.prototype.toString=function(){return this.fill+this.align+this.sign+this.symbol+(this.zero?\"0\":\"\")+(void 0===this.width?\"\":Math.max(1,0|this.width))+(this.comma?\",\":\"\")+(void 0===this.precision?\"\":\".\"+Math.max(0,0|this.precision))+(this.trim?\"~\":\"\")+this.type};const Me={\"%\":(e,t)=>(100*e).toFixed(t),b:e=>Math.round(e).toString(2),c:e=>e+\"\",d:function(e){return Math.abs(e=Math.round(e))>=1e21?e.toLocaleString(\"en\").replace(/,/g,\"\"):e.toString(10)},e:(e,t)=>e.toExponential(t),f:(e,t)=>e.toFixed(t),g:(e,t)=>e.toPrecision(t),o:e=>Math.round(e).toString(8),p:(e,t)=>Te(100*e,t),r:Te,s:function(e,t){var n=Ee(e,t);if(!n)return e+\"\";var r=n[0],a=n[1],i=a-(we=3*Math.max(-8,Math.min(8,Math.floor(a/3))))+1,o=r.length;return i===o?r:i>o?r+new Array(i-o+1).join(\"0\"):i>0?r.slice(0,i)+\".\"+r.slice(i):\"0.\"+new Array(1-i).join(\"0\")+Ee(e,Math.max(0,t+i-1))[0]},X:e=>Math.round(e).toString(16).toUpperCase(),x:e=>Math.round(e).toString(16)};function Ne(e){return e}var Pe,ze,Le,Oe=Array.prototype.map,Ae=[\"y\",\"z\",\"a\",\"f\",\"p\",\"n\",\"µ\",\"m\",\"\",\"k\",\"M\",\"G\",\"T\",\"P\",\"E\",\"Z\",\"Y\"];function Fe(e){var t=e.domain;return e.ticks=function(e){var n=t();return function(e,t,n){if(!((n=+n)>0))return[];if((e=+e)==(t=+t))return[e];const r=t<e,[a,i,u]=r?o(t,e,n):o(e,t,n);if(!(i>=a))return[];const l=i-a+1,s=new Array(l);if(r)if(u<0)for(let e=0;e<l;++e)s[e]=(i-e)/-u;else for(let e=0;e<l;++e)s[e]=(i-e)*u;else if(u<0)for(let e=0;e<l;++e)s[e]=(a+e)/-u;else for(let e=0;e<l;++e)s[e]=(a+e)*u;return s}(n[0],n[n.length-1],null==e?10:e)},e.tickFormat=function(e,n){var r=t();return function(e,t,n,r){var a,i=l(e,t,n);switch((r=ke(null==r?\",f\":r)).type){case\"s\":var o=Math.max(Math.abs(e),Math.abs(t));return null!=r.precision||isNaN(a=function(e,t){return Math.max(0,3*Math.max(-8,Math.min(8,Math.floor(Ce(t)/3)))-Ce(Math.abs(e)))}(i,o))||(r.precision=a),Le(r,o);case\"\":case\"e\":case\"g\":case\"p\":case\"r\":null!=r.precision||isNaN(a=function(e,t){return e=Math.abs(e),t=Math.abs(t)-e,Math.max(0,Ce(t)-Ce(e))+1}(i,Math.max(Math.abs(e),Math.abs(t))))||(r.precision=a-(\"e\"===r.type));break;case\"f\":case\"%\":null!=r.precision||isNaN(a=function(e){return Math.max(0,-Ce(Math.abs(e)))}(i))||(r.precision=a-2*(\"%\"===r.type))}return ze(r)}(r[0],r[r.length-1],null==e?10:e,n)},e.nice=function(n){null==n&&(n=10);var r,a,i=t(),o=0,l=i.length-1,s=i[o],c=i[l],f=10;for(c<s&&(a=s,s=c,c=a,a=o,o=l,l=a);f-- >0;){if((a=u(s,c,n))===r)return i[o]=s,i[l]=c,t(i);if(a>0)s=Math.floor(s/a)*a,c=Math.ceil(c/a)*a;else{if(!(a<0))break;s=Math.ceil(s*a)/a,c=Math.floor(c*a)/a}r=a}return e},e}function De(){var e=be();return e.copy=function(){return me(e,De())},_e.apply(e,arguments),Fe(e)}Pe=function(e){var t,n,r=void 0===e.grouping||void 0===e.thousands?Ne:(t=Oe.call(e.grouping,Number),n=e.thousands+\"\",function(e,r){for(var a=e.length,i=[],o=0,u=t[0],l=0;a>0&&u>0&&(l+u+1>r&&(u=Math.max(1,r-l)),i.push(e.substring(a-=u,a+u)),!((l+=u+1)>r));)u=t[o=(o+1)%t.length];return i.reverse().join(n)}),a=void 0===e.currency?\"\":e.currency[0]+\"\",i=void 0===e.currency?\"\":e.currency[1]+\"\",o=void 0===e.decimal?\".\":e.decimal+\"\",u=void 0===e.numerals?Ne:function(e){return function(t){return t.replace(/[0-9]/g,(function(t){return e[+t]}))}}(Oe.call(e.numerals,String)),l=void 0===e.percent?\"%\":e.percent+\"\",s=void 0===e.minus?\"−\":e.minus+\"\",c=void 0===e.nan?\"NaN\":e.nan+\"\";function f(e){var t=(e=ke(e)).fill,n=e.align,f=e.sign,p=e.symbol,d=e.zero,h=e.width,v=e.comma,g=e.precision,y=e.trim,m=e.type;\"n\"===m?(v=!0,m=\"g\"):Me[m]||(void 0===g&&(g=12),y=!0,m=\"g\"),(d||\"0\"===t&&\"=\"===n)&&(d=!0,t=\"0\",n=\"=\");var b=\"$\"===p?a:\"#\"===p&&/[boxX]/.test(m)?\"0\"+m.toLowerCase():\"\",_=\"$\"===p?i:/[%p]/.test(m)?l:\"\",w=Me[m],x=/[defgprs%]/.test(m);function k(e){var a,i,l,p=b,k=_;if(\"c\"===m)k=w(e)+k,e=\"\";else{var S=(e=+e)<0||1/e<0;if(e=isNaN(e)?c:w(Math.abs(e),g),y&&(e=function(e){e:for(var t,n=e.length,r=1,a=-1;r<n;++r)switch(e[r]){case\".\":a=t=r;break;case\"0\":0===a&&(a=r),t=r;break;default:if(!+e[r])break e;a>0&&(a=0)}return a>0?e.slice(0,a)+e.slice(t+1):e}(e)),S&&0==+e&&\"+\"!==f&&(S=!1),p=(S?\"(\"===f?f:s:\"-\"===f||\"(\"===f?\"\":f)+p,k=(\"s\"===m?Ae[8+we/3]:\"\")+k+(S&&\"(\"===f?\")\":\"\"),x)for(a=-1,i=e.length;++a<i;)if(48>(l=e.charCodeAt(a))||l>57){k=(46===l?o+e.slice(a+1):e.slice(a))+k,e=e.slice(0,a);break}}v&&!d&&(e=r(e,1/0));var E=p.length+e.length+k.length,C=E<h?new Array(h-E+1).join(t):\"\";switch(v&&d&&(e=r(C+e,C.length?h-k.length:1/0),C=\"\"),n){case\"<\":e=p+e+k+C;break;case\"=\":e=p+C+e+k;break;case\"^\":e=C.slice(0,E=C.length>>1)+p+e+k+C.slice(E);break;default:e=C+p+e+k}return u(e)}return g=void 0===g?6:/[gprs]/.test(m)?Math.max(1,Math.min(21,g)):Math.max(0,Math.min(20,g)),k.toString=function(){return e+\"\"},k}return{format:f,formatPrefix:function(e,t){var n=f(((e=ke(e)).type=\"f\",e)),r=3*Math.max(-8,Math.min(8,Math.floor(Ce(t)/3))),a=Math.pow(10,-r),i=Ae[8+r/3];return function(e){return n(a*e)+i}}}}({thousands:\",\",grouping:[3],currency:[\"$\",\"\"]}),ze=Pe.format,Le=Pe.formatPrefix;var Re=n(486);const je={colors:{RdBu:[\"rgb(255, 13, 87)\",\"rgb(30, 136, 229)\"],GnPR:[\"rgb(24, 196, 93)\",\"rgb(124, 82, 255)\"],CyPU:[\"#0099C6\",\"#990099\"],PkYg:[\"#DD4477\",\"#66AA00\"],DrDb:[\"#B82E2E\",\"#316395\"],LpLb:[\"#994499\",\"#22AA99\"],YlDp:[\"#AAAA11\",\"#6633CC\"],OrId:[\"#E67300\",\"#3E0099\"]},gray:\"#777\"};function Ue(e){return Ue=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Ue(e)}function Ie(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Ue(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Ue(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Ue(a)?a:String(a)),r)}var a}function $e(e,t){return $e=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},$e(e,t)}function Be(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function We(e){return We=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},We(e)}var Ve=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&$e(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=We(a);if(i){var n=We(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Ue(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return Be(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),(e=o.call(this)).width=100,window.lastSimpleListInstance=Be(e),e.effectFormat=ze(\".2\"),e}return n=u,(r=[{key:\"render\",value:function(){var t=this,n=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?n=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),n=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(n=this.props.plot_cmap),console.log(this.props.features,this.props.features),this.scale=De().domain([0,(0,Re.max)((0,Re.map)(this.props.features,(function(e){return Math.abs(e.effect)})))]).range([0,this.width]);var r=(0,Re.reverse)((0,Re.sortBy)(Object.keys(this.props.features),(function(e){return Math.abs(t.props.features[e].effect)}))).map((function(r){var a,i,o=t.props.features[r],u=t.props.featureNames[r],l={width:t.scale(Math.abs(o.effect)),height:\"20px\",background:o.effect<0?n[0]:n[1],display:\"inline-block\"},s={lineHeight:\"20px\",display:\"inline-block\",width:t.width+40,verticalAlign:\"top\",marginRight:\"5px\",textAlign:\"right\"},c={lineHeight:\"20px\",display:\"inline-block\",width:t.width+40,verticalAlign:\"top\",marginLeft:\"5px\"};return o.effect<0?(i=e.createElement(\"span\",{style:c},u),s.width=40+t.width-t.scale(Math.abs(o.effect)),s.textAlign=\"right\",s.color=\"#999\",s.fontSize=\"13px\",a=e.createElement(\"span\",{style:s},t.effectFormat(o.effect))):(s.textAlign=\"right\",a=e.createElement(\"span\",{style:s},u),c.width=40,c.textAlign=\"left\",c.color=\"#999\",c.fontSize=\"13px\",i=e.createElement(\"span\",{style:c},t.effectFormat(o.effect))),e.createElement(\"div\",{key:r,style:{marginTop:\"2px\"}},a,e.createElement(\"div\",{style:l}),i)}));return e.createElement(\"span\",null,r)}}])&&Ie(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);Ve.defaultProps={plot_cmap:\"RdBu\"};const He=Ve;function qe(){}function Qe(e){return null==e?qe:function(){return this.querySelector(e)}}function Ye(){return[]}function Ge(e){return function(t){return t.matches(e)}}var Ke=Array.prototype.find;function Ze(){return this.firstElementChild}var Xe=Array.prototype.filter;function Je(){return Array.from(this.children)}function et(e){return new Array(e.length)}function tt(e,t){this.ownerDocument=e.ownerDocument,this.namespaceURI=e.namespaceURI,this._next=null,this._parent=e,this.__data__=t}function nt(e,t,n,r,a,i){for(var o,u=0,l=t.length,s=i.length;u<s;++u)(o=t[u])?(o.__data__=i[u],r[u]=o):n[u]=new tt(e,i[u]);for(;u<l;++u)(o=t[u])&&(a[u]=o)}function rt(e,t,n,r,a,i,o){var u,l,s,c=new Map,f=t.length,p=i.length,d=new Array(f);for(u=0;u<f;++u)(l=t[u])&&(d[u]=s=o.call(l,l.__data__,u,t)+\"\",c.has(s)?a[u]=l:c.set(s,l));for(u=0;u<p;++u)s=o.call(e,i[u],u,i)+\"\",(l=c.get(s))?(r[u]=l,l.__data__=i[u],c.delete(s)):n[u]=new tt(e,i[u]);for(u=0;u<f;++u)(l=t[u])&&c.get(d[u])===l&&(a[u]=l)}function at(e){return e.__data__}function it(e){return\"object\"==typeof e&&\"length\"in e?e:Array.from(e)}function ot(e,t){return e<t?-1:e>t?1:e>=t?0:NaN}tt.prototype={constructor:tt,appendChild:function(e){return this._parent.insertBefore(e,this._next)},insertBefore:function(e,t){return this._parent.insertBefore(e,t)},querySelector:function(e){return this._parent.querySelector(e)},querySelectorAll:function(e){return this._parent.querySelectorAll(e)}};var ut=\"http://www.w3.org/1999/xhtml\";const lt={svg:\"http://www.w3.org/2000/svg\",xhtml:ut,xlink:\"http://www.w3.org/1999/xlink\",xml:\"http://www.w3.org/XML/1998/namespace\",xmlns:\"http://www.w3.org/2000/xmlns/\"};function st(e){var t=e+=\"\",n=t.indexOf(\":\");return n>=0&&\"xmlns\"!==(t=e.slice(0,n))&&(e=e.slice(n+1)),lt.hasOwnProperty(t)?{space:lt[t],local:e}:e}function ct(e){return function(){this.removeAttribute(e)}}function ft(e){return function(){this.removeAttributeNS(e.space,e.local)}}function pt(e,t){return function(){this.setAttribute(e,t)}}function dt(e,t){return function(){this.setAttributeNS(e.space,e.local,t)}}function ht(e,t){return function(){var n=t.apply(this,arguments);null==n?this.removeAttribute(e):this.setAttribute(e,n)}}function vt(e,t){return function(){var n=t.apply(this,arguments);null==n?this.removeAttributeNS(e.space,e.local):this.setAttributeNS(e.space,e.local,n)}}function gt(e){return e.ownerDocument&&e.ownerDocument.defaultView||e.document&&e||e.defaultView}function yt(e){return function(){this.style.removeProperty(e)}}function mt(e,t,n){return function(){this.style.setProperty(e,t,n)}}function bt(e,t,n){return function(){var r=t.apply(this,arguments);null==r?this.style.removeProperty(e):this.style.setProperty(e,r,n)}}function _t(e){return function(){delete this[e]}}function wt(e,t){return function(){this[e]=t}}function xt(e,t){return function(){var n=t.apply(this,arguments);null==n?delete this[e]:this[e]=n}}function kt(e){return e.trim().split(/^|\\s+/)}function St(e){return e.classList||new Et(e)}function Et(e){this._node=e,this._names=kt(e.getAttribute(\"class\")||\"\")}function Ct(e,t){for(var n=St(e),r=-1,a=t.length;++r<a;)n.add(t[r])}function Tt(e,t){for(var n=St(e),r=-1,a=t.length;++r<a;)n.remove(t[r])}function Mt(e){return function(){Ct(this,e)}}function Nt(e){return function(){Tt(this,e)}}function Pt(e,t){return function(){(t.apply(this,arguments)?Ct:Tt)(this,e)}}function zt(){this.textContent=\"\"}function Lt(e){return function(){this.textContent=e}}function Ot(e){return function(){var t=e.apply(this,arguments);this.textContent=null==t?\"\":t}}function At(){this.innerHTML=\"\"}function Ft(e){return function(){this.innerHTML=e}}function Dt(e){return function(){var t=e.apply(this,arguments);this.innerHTML=null==t?\"\":t}}function Rt(){this.nextSibling&&this.parentNode.appendChild(this)}function jt(){this.previousSibling&&this.parentNode.insertBefore(this,this.parentNode.firstChild)}function Ut(e){return function(){var t=this.ownerDocument,n=this.namespaceURI;return n===ut&&t.documentElement.namespaceURI===ut?t.createElement(e):t.createElementNS(n,e)}}function It(e){return function(){return this.ownerDocument.createElementNS(e.space,e.local)}}function $t(e){var t=st(e);return(t.local?It:Ut)(t)}function Bt(){return null}function Wt(){var e=this.parentNode;e&&e.removeChild(this)}function Vt(){var e=this.cloneNode(!1),t=this.parentNode;return t?t.insertBefore(e,this.nextSibling):e}function Ht(){var e=this.cloneNode(!0),t=this.parentNode;return t?t.insertBefore(e,this.nextSibling):e}function qt(e){return function(){var t=this.__on;if(t){for(var n,r=0,a=-1,i=t.length;r<i;++r)n=t[r],e.type&&n.type!==e.type||n.name!==e.name?t[++a]=n:this.removeEventListener(n.type,n.listener,n.options);++a?t.length=a:delete this.__on}}}function Qt(e,t,n){return function(){var r,a=this.__on,i=function(e){return function(t){e.call(this,t,this.__data__)}}(t);if(a)for(var o=0,u=a.length;o<u;++o)if((r=a[o]).type===e.type&&r.name===e.name)return this.removeEventListener(r.type,r.listener,r.options),this.addEventListener(r.type,r.listener=i,r.options=n),void(r.value=t);this.addEventListener(e.type,i,n),r={type:e.type,name:e.name,value:t,listener:i,options:n},a?a.push(r):this.__on=[r]}}function Yt(e,t,n){var r=gt(e),a=r.CustomEvent;\"function\"==typeof a?a=new a(t,n):(a=r.document.createEvent(\"Event\"),n?(a.initEvent(t,n.bubbles,n.cancelable),a.detail=n.detail):a.initEvent(t,!1,!1)),e.dispatchEvent(a)}function Gt(e,t){return function(){return Yt(this,e,t)}}function Kt(e,t){return function(){return Yt(this,e,t.apply(this,arguments))}}Et.prototype={add:function(e){this._names.indexOf(e)<0&&(this._names.push(e),this._node.setAttribute(\"class\",this._names.join(\" \")))},remove:function(e){var t=this._names.indexOf(e);t>=0&&(this._names.splice(t,1),this._node.setAttribute(\"class\",this._names.join(\" \")))},contains:function(e){return this._names.indexOf(e)>=0}};var Zt=[null];function Xt(e,t){this._groups=e,this._parents=t}function Jt(e){return\"string\"==typeof e?new Xt([[document.querySelector(e)]],[document.documentElement]):new Xt([[e]],Zt)}function en(e){return e}Xt.prototype=function(){return new Xt([[document.documentElement]],Zt)}.prototype={constructor:Xt,select:function(e){\"function\"!=typeof e&&(e=Qe(e));for(var t=this._groups,n=t.length,r=new Array(n),a=0;a<n;++a)for(var i,o,u=t[a],l=u.length,s=r[a]=new Array(l),c=0;c<l;++c)(i=u[c])&&(o=e.call(i,i.__data__,c,u))&&(\"__data__\"in i&&(o.__data__=i.__data__),s[c]=o);return new Xt(r,this._parents)},selectAll:function(e){e=\"function\"==typeof e?function(e){return function(){return null==(t=e.apply(this,arguments))?[]:Array.isArray(t)?t:Array.from(t);var t}}(e):function(e){return null==e?Ye:function(){return this.querySelectorAll(e)}}(e);for(var t=this._groups,n=t.length,r=[],a=[],i=0;i<n;++i)for(var o,u=t[i],l=u.length,s=0;s<l;++s)(o=u[s])&&(r.push(e.call(o,o.__data__,s,u)),a.push(o));return new Xt(r,a)},selectChild:function(e){return this.select(null==e?Ze:function(e){return function(){return Ke.call(this.children,e)}}(\"function\"==typeof e?e:Ge(e)))},selectChildren:function(e){return this.selectAll(null==e?Je:function(e){return function(){return Xe.call(this.children,e)}}(\"function\"==typeof e?e:Ge(e)))},filter:function(e){\"function\"!=typeof e&&(e=function(e){return function(){return this.matches(e)}}(e));for(var t=this._groups,n=t.length,r=new Array(n),a=0;a<n;++a)for(var i,o=t[a],u=o.length,l=r[a]=[],s=0;s<u;++s)(i=o[s])&&e.call(i,i.__data__,s,o)&&l.push(i);return new Xt(r,this._parents)},data:function(e,t){if(!arguments.length)return Array.from(this,at);var n,r=t?rt:nt,a=this._parents,i=this._groups;\"function\"!=typeof e&&(n=e,e=function(){return n});for(var o=i.length,u=new Array(o),l=new Array(o),s=new Array(o),c=0;c<o;++c){var f=a[c],p=i[c],d=p.length,h=it(e.call(f,f&&f.__data__,c,a)),v=h.length,g=l[c]=new Array(v),y=u[c]=new Array(v);r(f,p,g,y,s[c]=new Array(d),h,t);for(var m,b,_=0,w=0;_<v;++_)if(m=g[_]){for(_>=w&&(w=_+1);!(b=y[w])&&++w<v;);m._next=b||null}}return(u=new Xt(u,a))._enter=l,u._exit=s,u},enter:function(){return new Xt(this._enter||this._groups.map(et),this._parents)},exit:function(){return new Xt(this._exit||this._groups.map(et),this._parents)},join:function(e,t,n){var r=this.enter(),a=this,i=this.exit();return\"function\"==typeof e?(r=e(r))&&(r=r.selection()):r=r.append(e+\"\"),null!=t&&(a=t(a))&&(a=a.selection()),null==n?i.remove():n(i),r&&a?r.merge(a).order():a},merge:function(e){for(var t=e.selection?e.selection():e,n=this._groups,r=t._groups,a=n.length,i=r.length,o=Math.min(a,i),u=new Array(a),l=0;l<o;++l)for(var s,c=n[l],f=r[l],p=c.length,d=u[l]=new Array(p),h=0;h<p;++h)(s=c[h]||f[h])&&(d[h]=s);for(;l<a;++l)u[l]=n[l];return new Xt(u,this._parents)},selection:function(){return this},order:function(){for(var e=this._groups,t=-1,n=e.length;++t<n;)for(var r,a=e[t],i=a.length-1,o=a[i];--i>=0;)(r=a[i])&&(o&&4^r.compareDocumentPosition(o)&&o.parentNode.insertBefore(r,o),o=r);return this},sort:function(e){function t(t,n){return t&&n?e(t.__data__,n.__data__):!t-!n}e||(e=ot);for(var n=this._groups,r=n.length,a=new Array(r),i=0;i<r;++i){for(var o,u=n[i],l=u.length,s=a[i]=new Array(l),c=0;c<l;++c)(o=u[c])&&(s[c]=o);s.sort(t)}return new Xt(a,this._parents).order()},call:function(){var e=arguments[0];return arguments[0]=this,e.apply(null,arguments),this},nodes:function(){return Array.from(this)},node:function(){for(var e=this._groups,t=0,n=e.length;t<n;++t)for(var r=e[t],a=0,i=r.length;a<i;++a){var o=r[a];if(o)return o}return null},size:function(){let e=0;for(const t of this)++e;return e},empty:function(){return!this.node()},each:function(e){for(var t=this._groups,n=0,r=t.length;n<r;++n)for(var a,i=t[n],o=0,u=i.length;o<u;++o)(a=i[o])&&e.call(a,a.__data__,o,i);return this},attr:function(e,t){var n=st(e);if(arguments.length<2){var r=this.node();return n.local?r.getAttributeNS(n.space,n.local):r.getAttribute(n)}return this.each((null==t?n.local?ft:ct:\"function\"==typeof t?n.local?vt:ht:n.local?dt:pt)(n,t))},style:function(e,t,n){return arguments.length>1?this.each((null==t?yt:\"function\"==typeof t?bt:mt)(e,t,null==n?\"\":n)):function(e,t){return e.style.getPropertyValue(t)||gt(e).getComputedStyle(e,null).getPropertyValue(t)}(this.node(),e)},property:function(e,t){return arguments.length>1?this.each((null==t?_t:\"function\"==typeof t?xt:wt)(e,t)):this.node()[e]},classed:function(e,t){var n=kt(e+\"\");if(arguments.length<2){for(var r=St(this.node()),a=-1,i=n.length;++a<i;)if(!r.contains(n[a]))return!1;return!0}return this.each((\"function\"==typeof t?Pt:t?Mt:Nt)(n,t))},text:function(e){return arguments.length?this.each(null==e?zt:(\"function\"==typeof e?Ot:Lt)(e)):this.node().textContent},html:function(e){return arguments.length?this.each(null==e?At:(\"function\"==typeof e?Dt:Ft)(e)):this.node().innerHTML},raise:function(){return this.each(Rt)},lower:function(){return this.each(jt)},append:function(e){var t=\"function\"==typeof e?e:$t(e);return this.select((function(){return this.appendChild(t.apply(this,arguments))}))},insert:function(e,t){var n=\"function\"==typeof e?e:$t(e),r=null==t?Bt:\"function\"==typeof t?t:Qe(t);return this.select((function(){return this.insertBefore(n.apply(this,arguments),r.apply(this,arguments)||null)}))},remove:function(){return this.each(Wt)},clone:function(e){return this.select(e?Ht:Vt)},datum:function(e){return arguments.length?this.property(\"__data__\",e):this.node().__data__},on:function(e,t,n){var r,a,i=function(e){return e.trim().split(/^|\\s+/).map((function(e){var t=\"\",n=e.indexOf(\".\");return n>=0&&(t=e.slice(n+1),e=e.slice(0,n)),{type:e,name:t}}))}(e+\"\"),o=i.length;if(!(arguments.length<2)){for(u=t?Qt:qt,r=0;r<o;++r)this.each(u(i[r],t,n));return this}var u=this.node().__on;if(u)for(var l,s=0,c=u.length;s<c;++s)for(r=0,l=u[s];r<o;++r)if((a=i[r]).type===l.type&&a.name===l.name)return l.value},dispatch:function(e,t){return this.each((\"function\"==typeof t?Kt:Gt)(e,t))},[Symbol.iterator]:function*(){for(var e=this._groups,t=0,n=e.length;t<n;++t)for(var r,a=e[t],i=0,o=a.length;i<o;++i)(r=a[i])&&(yield r)}};var tn=1,nn=2,rn=3,an=4,on=1e-6;function un(e){return\"translate(\"+e+\",0)\"}function ln(e){return\"translate(0,\"+e+\")\"}function sn(e){return t=>+e(t)}function cn(e,t){return t=Math.max(0,e.bandwidth()-2*t)/2,e.round()&&(t=Math.round(t)),n=>+e(n)+t}function fn(){return!this.__axis}function pn(e,t){var n=[],r=null,a=null,i=6,o=6,u=3,l=\"undefined\"!=typeof window&&window.devicePixelRatio>1?0:.5,s=e===tn||e===an?-1:1,c=e===an||e===nn?\"x\":\"y\",f=e===tn||e===rn?un:ln;function p(p){var d=null==r?t.ticks?t.ticks.apply(t,n):t.domain():r,h=null==a?t.tickFormat?t.tickFormat.apply(t,n):en:a,v=Math.max(i,0)+u,g=t.range(),y=+g[0]+l,m=+g[g.length-1]+l,b=(t.bandwidth?cn:sn)(t.copy(),l),_=p.selection?p.selection():p,w=_.selectAll(\".domain\").data([null]),x=_.selectAll(\".tick\").data(d,t).order(),k=x.exit(),S=x.enter().append(\"g\").attr(\"class\",\"tick\"),E=x.select(\"line\"),C=x.select(\"text\");w=w.merge(w.enter().insert(\"path\",\".tick\").attr(\"class\",\"domain\").attr(\"stroke\",\"currentColor\")),x=x.merge(S),E=E.merge(S.append(\"line\").attr(\"stroke\",\"currentColor\").attr(c+\"2\",s*i)),C=C.merge(S.append(\"text\").attr(\"fill\",\"currentColor\").attr(c,s*v).attr(\"dy\",e===tn?\"0em\":e===rn?\"0.71em\":\"0.32em\")),p!==_&&(w=w.transition(p),x=x.transition(p),E=E.transition(p),C=C.transition(p),k=k.transition(p).attr(\"opacity\",on).attr(\"transform\",(function(e){return isFinite(e=b(e))?f(e+l):this.getAttribute(\"transform\")})),S.attr(\"opacity\",on).attr(\"transform\",(function(e){var t=this.parentNode.__axis;return f((t&&isFinite(t=t(e))?t:b(e))+l)}))),k.remove(),w.attr(\"d\",e===an||e===nn?o?\"M\"+s*o+\",\"+y+\"H\"+l+\"V\"+m+\"H\"+s*o:\"M\"+l+\",\"+y+\"V\"+m:o?\"M\"+y+\",\"+s*o+\"V\"+l+\"H\"+m+\"V\"+s*o:\"M\"+y+\",\"+l+\"H\"+m),x.attr(\"opacity\",1).attr(\"transform\",(function(e){return f(b(e)+l)})),E.attr(c+\"2\",s*i),C.attr(c,s*v).text(h),_.filter(fn).attr(\"fill\",\"none\").attr(\"font-size\",10).attr(\"font-family\",\"sans-serif\").attr(\"text-anchor\",e===nn?\"start\":e===an?\"end\":\"middle\"),_.each((function(){this.__axis=b}))}return p.scale=function(e){return arguments.length?(t=e,p):t},p.ticks=function(){return n=Array.from(arguments),p},p.tickArguments=function(e){return arguments.length?(n=null==e?[]:Array.from(e),p):n.slice()},p.tickValues=function(e){return arguments.length?(r=null==e?null:Array.from(e),p):r&&r.slice()},p.tickFormat=function(e){return arguments.length?(a=e,p):a},p.tickSize=function(e){return arguments.length?(i=o=+e,p):i},p.tickSizeInner=function(e){return arguments.length?(i=+e,p):i},p.tickSizeOuter=function(e){return arguments.length?(o=+e,p):o},p.tickPadding=function(e){return arguments.length?(u=+e,p):u},p.offset=function(e){return arguments.length?(l=+e,p):l},p}function dn(e){return pn(rn,e)}function hn(e){return function(){return e}}function vn(e){this._context=e}function gn(e){return new vn(e)}Array.prototype.slice,vn.prototype={areaStart:function(){this._line=0},areaEnd:function(){this._line=NaN},lineStart:function(){this._point=0},lineEnd:function(){(this._line||0!==this._line&&1===this._point)&&this._context.closePath(),this._line=1-this._line},point:function(e,t){switch(e=+e,t=+t,this._point){case 0:this._point=1,this._line?this._context.lineTo(e,t):this._context.moveTo(e,t);break;case 1:this._point=2;default:this._context.lineTo(e,t)}}};const yn=Math.PI,mn=2*yn,bn=1e-6,_n=mn-bn;function wn(e){this._+=e[0];for(let t=1,n=e.length;t<n;++t)this._+=arguments[t]+e[t]}class xn{constructor(e){this._x0=this._y0=this._x1=this._y1=null,this._=\"\",this._append=null==e?wn:function(e){let t=Math.floor(e);if(!(t>=0))throw new Error(`invalid digits: ${e}`);if(t>15)return wn;const n=10**t;return function(e){this._+=e[0];for(let t=1,r=e.length;t<r;++t)this._+=Math.round(arguments[t]*n)/n+e[t]}}(e)}moveTo(e,t){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+t}`}closePath(){null!==this._x1&&(this._x1=this._x0,this._y1=this._y0,this._append`Z`)}lineTo(e,t){this._append`L${this._x1=+e},${this._y1=+t}`}quadraticCurveTo(e,t,n,r){this._append`Q${+e},${+t},${this._x1=+n},${this._y1=+r}`}bezierCurveTo(e,t,n,r,a,i){this._append`C${+e},${+t},${+n},${+r},${this._x1=+a},${this._y1=+i}`}arcTo(e,t,n,r,a){if(e=+e,t=+t,n=+n,r=+r,(a=+a)<0)throw new Error(`negative radius: ${a}`);let i=this._x1,o=this._y1,u=n-e,l=r-t,s=i-e,c=o-t,f=s*s+c*c;if(null===this._x1)this._append`M${this._x1=e},${this._y1=t}`;else if(f>bn)if(Math.abs(c*u-l*s)>bn&&a){let p=n-i,d=r-o,h=u*u+l*l,v=p*p+d*d,g=Math.sqrt(h),y=Math.sqrt(f),m=a*Math.tan((yn-Math.acos((h+f-v)/(2*g*y)))/2),b=m/y,_=m/g;Math.abs(b-1)>bn&&this._append`L${e+b*s},${t+b*c}`,this._append`A${a},${a},0,0,${+(c*p>s*d)},${this._x1=e+_*u},${this._y1=t+_*l}`}else this._append`L${this._x1=e},${this._y1=t}`}arc(e,t,n,r,a,i){if(e=+e,t=+t,i=!!i,(n=+n)<0)throw new Error(`negative radius: ${n}`);let o=n*Math.cos(r),u=n*Math.sin(r),l=e+o,s=t+u,c=1^i,f=i?r-a:a-r;null===this._x1?this._append`M${l},${s}`:(Math.abs(this._x1-l)>bn||Math.abs(this._y1-s)>bn)&&this._append`L${l},${s}`,n&&(f<0&&(f=f%mn+mn),f>_n?this._append`A${n},${n},0,1,${c},${e-o},${t-u}A${n},${n},0,1,${c},${this._x1=l},${this._y1=s}`:f>bn&&this._append`A${n},${n},0,${+(f>=yn)},${c},${this._x1=e+n*Math.cos(a)},${this._y1=t+n*Math.sin(a)}`)}rect(e,t,n,r){this._append`M${this._x0=this._x1=+e},${this._y0=this._y1=+t}h${n=+n}v${+r}h${-n}Z`}toString(){return this._}}function kn(e){return e[0]}function Sn(e){return e[1]}function En(e,t){var n=hn(!0),r=null,a=gn,i=null,o=function(e){let t=3;return e.digits=function(n){if(!arguments.length)return t;if(null==n)t=null;else{const e=Math.floor(n);if(!(e>=0))throw new RangeError(`invalid digits: ${n}`);t=e}return e},()=>new xn(t)}(u);function u(u){var l,s,c,f=(u=function(e){return\"object\"==typeof e&&\"length\"in e?e:Array.from(e)}(u)).length,p=!1;for(null==r&&(i=a(c=o())),l=0;l<=f;++l)!(l<f&&n(s=u[l],l,u))===p&&((p=!p)?i.lineStart():i.lineEnd()),p&&i.point(+e(s,l,u),+t(s,l,u));if(c)return i=null,c+\"\"||null}return e=\"function\"==typeof e?e:void 0===e?kn:hn(e),t=\"function\"==typeof t?t:void 0===t?Sn:hn(t),u.x=function(t){return arguments.length?(e=\"function\"==typeof t?t:hn(+t),u):e},u.y=function(e){return arguments.length?(t=\"function\"==typeof e?e:hn(+e),u):t},u.defined=function(e){return arguments.length?(n=\"function\"==typeof e?e:hn(!!e),u):n},u.curve=function(e){return arguments.length?(a=e,null!=r&&(i=a(r)),u):a},u.context=function(e){return arguments.length?(null==e?r=i=null:i=a(r=e),u):r},u}function Cn(e){return Cn=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Cn(e)}function Tn(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Cn(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Cn(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Cn(a)?a:String(a)),r)}var a}function Mn(e,t){return Mn=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},Mn(e,t)}function Nn(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function Pn(e){return Pn=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},Pn(e)}var zn=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&Mn(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=Pn(a);if(i){var n=Pn(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Cn(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return Nn(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),e=o.call(this),window.lastAdditiveForceVisualizer=Nn(e),e.effectFormat=ze(\".2\"),e.redraw=(0,Re.debounce)((function(){return e.draw()}),200),e}return n=u,(r=[{key:\"componentDidMount\",value:function(){var e=this;this.mainGroup=this.svg.append(\"g\"),this.axisElement=this.mainGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-axis\"),this.onTopGroup=this.svg.append(\"g\"),this.baseValueTitle=this.svg.append(\"text\"),this.joinPointLine=this.svg.append(\"line\"),this.joinPointLabelOutline=this.svg.append(\"text\"),this.joinPointLabel=this.svg.append(\"text\"),this.joinPointTitleLeft=this.svg.append(\"text\"),this.joinPointTitleLeftArrow=this.svg.append(\"text\"),this.joinPointTitle=this.svg.append(\"text\"),this.joinPointTitleRightArrow=this.svg.append(\"text\"),this.joinPointTitleRight=this.svg.append(\"text\"),this.hoverLabelBacking=this.svg.append(\"text\").attr(\"x\",10).attr(\"y\",20).attr(\"text-anchor\",\"middle\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").text(\"\").on(\"mouseover\",(function(){e.hoverLabel.attr(\"opacity\",1),e.hoverLabelBacking.attr(\"opacity\",1)})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)})),this.hoverLabel=this.svg.append(\"text\").attr(\"x\",10).attr(\"y\",20).attr(\"text-anchor\",\"middle\").attr(\"font-size\",12).attr(\"fill\",\"#0f0\").text(\"\").on(\"mouseover\",(function(){e.hoverLabel.attr(\"opacity\",1),e.hoverLabelBacking.attr(\"opacity\",1)})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)}));var t=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?t=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),t=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(t=this.props.plot_cmap),this.colors=t.map((function(e){return q(e)})),this.brighterColors=[1.45,1.6].map((function(t,n){return e.colors[n].brighter(t)})),this.colors.map((function(t,n){var r=e.svg.append(\"linearGradient\").attr(\"id\",\"linear-grad-\"+n).attr(\"x1\",\"0%\").attr(\"y1\",\"0%\").attr(\"x2\",\"0%\").attr(\"y2\",\"100%\");r.append(\"stop\").attr(\"offset\",\"0%\").attr(\"stop-color\",t).attr(\"stop-opacity\",.6),r.append(\"stop\").attr(\"offset\",\"100%\").attr(\"stop-color\",t).attr(\"stop-opacity\",0);var a=e.svg.append(\"linearGradient\").attr(\"id\",\"linear-backgrad-\"+n).attr(\"x1\",\"0%\").attr(\"y1\",\"0%\").attr(\"x2\",\"0%\").attr(\"y2\",\"100%\");a.append(\"stop\").attr(\"offset\",\"0%\").attr(\"stop-color\",t).attr(\"stop-opacity\",.5),a.append(\"stop\").attr(\"offset\",\"100%\").attr(\"stop-color\",t).attr(\"stop-opacity\",0)})),this.tickFormat=ze(\",.4\"),this.scaleCentered=De(),this.axis=dn().scale(this.scaleCentered).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.tickFormat(e.invLinkFunction(t))})).tickPadding(-18),window.addEventListener(\"resize\",this.redraw),window.setTimeout(this.redraw,50)}},{key:\"componentDidUpdate\",value:function(){this.draw()}},{key:\"draw\",value:function(){var e=this;(0,Re.each)(this.props.featureNames,(function(t,n){e.props.features[n]&&(e.props.features[n].name=t)})),\"identity\"===this.props.link?this.invLinkFunction=function(t){return e.props.baseValue+t}:\"logit\"===this.props.link?this.invLinkFunction=function(t){return 1/(1+Math.exp(-(e.props.baseValue+t)))}:console.log(\"ERROR: Unrecognized link function: \",this.props.link);var t=this.svg.node().parentNode.offsetWidth;if(0==t)return setTimeout((function(){return e.draw(e.props)}),500);this.svg.style(\"height\",\"150px\"),this.svg.style(\"width\",t+\"px\");var n=(0,Re.sortBy)(this.props.features,(function(e){return-1/(e.effect+1e-10)})),r=(0,Re.sum)((0,Re.map)(n,(function(e){return Math.abs(e.effect)}))),a=(0,Re.sum)((0,Re.map)((0,Re.filter)(n,(function(e){return e.effect>0})),(function(e){return e.effect})))||0,i=(0,Re.sum)((0,Re.map)((0,Re.filter)(n,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0;this.domainSize=3*Math.max(a,i);var o=De().domain([0,this.domainSize]).range([0,t]),u=t/2-o(i);this.scaleCentered.domain([-this.domainSize/2,this.domainSize/2]).range([0,t]).clamp(!0),this.axisElement.attr(\"transform\",\"translate(0,50)\").call(this.axis);var l,s,c,f=0;for(l=0;l<n.length;++l)n[l].x=f,n[l].effect<0&&void 0===s&&(s=f,c=l),f+=Math.abs(n[l].effect);void 0===s&&(s=f,c=l);var p=En().x((function(e){return e[0]})).y((function(e){return e[1]})),d=function(t){return void 0!==t.value&&null!==t.value&&\"\"!==t.value?t.name+\" = \"+(isNaN(t.value)?t.value:e.tickFormat(t.value)):t.name};n=this.props.hideBars?[]:n;var h=this.mainGroup.selectAll(\".force-bar-blocks\").data(n);h.enter().append(\"path\").attr(\"class\",\"force-bar-blocks\").merge(h).attr(\"d\",(function(e,t){var n=o(e.x)+u,r=o(Math.abs(e.effect)),a=e.effect<0?-4:4,i=a;return t===c&&(a=0),t===c-1&&(i=0),p([[n,56],[n+r,56],[n+r+i,64.5],[n+r,73],[n,73],[n+a,64.5]])})).attr(\"fill\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})).on(\"mouseover\",(function(t){if(o(Math.abs(t.effect))<o(r)/50||o(Math.abs(t.effect))<10){var n=o(t.x)+u,a=o(Math.abs(t.effect));e.hoverLabel.attr(\"opacity\",1).attr(\"x\",n+a/2).attr(\"y\",50.5).attr(\"fill\",t.effect>0?e.colors[0]:e.colors[1]).text(d(t)),e.hoverLabelBacking.attr(\"opacity\",1).attr(\"x\",n+a/2).attr(\"y\",50.5).text(d(t))}})).on(\"mouseout\",(function(){e.hoverLabel.attr(\"opacity\",0),e.hoverLabelBacking.attr(\"opacity\",0)})),h.exit().remove();var v=(0,Re.filter)(n,(function(e){return o(Math.abs(e.effect))>o(r)/50&&o(Math.abs(e.effect))>10})),g=this.onTopGroup.selectAll(\".force-bar-labels\").data(v);if(g.exit().remove(),g=g.enter().append(\"text\").attr(\"class\",\"force-bar-labels\").attr(\"font-size\",\"12px\").attr(\"y\",98).merge(g).text((function(t){return void 0!==t.value&&null!==t.value&&\"\"!==t.value?t.name+\" = \"+(isNaN(t.value)?t.value:e.tickFormat(t.value)):t.name})).attr(\"fill\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})).attr(\"stroke\",(function(e){return e.textWidth=Math.max(this.getComputedTextLength(),o(Math.abs(e.effect))-10),e.innerTextWidth=this.getComputedTextLength(),\"none\"})),this.filteredData=v,n.length>0){f=s+o.invert(5);for(var y=c;y<n.length;++y)n[y].textx=f,f+=o.invert(n[y].textWidth+10);f=s-o.invert(5);for(var m=c-1;m>=0;--m)n[m].textx=f,f-=o.invert(n[m].textWidth+10)}g.attr(\"x\",(function(e){return o(e.textx)+u+(e.effect>0?-e.textWidth/2:e.textWidth/2)})).attr(\"text-anchor\",\"middle\"),v=(0,Re.filter)(v,(function(n){return o(n.textx)+u>e.props.labelMargin&&o(n.textx)+u<t-e.props.labelMargin})),this.filteredData2=v;var b=v.slice(),_=(0,Re.findIndex)(n,v[0])-1;_>=0&&b.unshift(n[_]);var w=this.mainGroup.selectAll(\".force-bar-labelBacking\").data(v);w.enter().append(\"path\").attr(\"class\",\"force-bar-labelBacking\").attr(\"stroke\",\"none\").attr(\"opacity\",.2).merge(w).attr(\"d\",(function(e){return p([[o(e.x)+o(Math.abs(e.effect))+u,73],[(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5,83],[(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5,104],[(e.effect>0?o(e.textx)-e.textWidth:o(e.textx))+u-5,104],[(e.effect>0?o(e.textx)-e.textWidth:o(e.textx))+u-5,83],[o(e.x)+u,73]])})).attr(\"fill\",(function(e){return\"url(#linear-backgrad-\".concat(e.effect>0?0:1,\")\")})),w.exit().remove();var x=this.mainGroup.selectAll(\".force-bar-labelDividers\").data(v.slice(0,-1));x.enter().append(\"rect\").attr(\"class\",\"force-bar-labelDividers\").attr(\"height\",\"21px\").attr(\"width\",\"1px\").attr(\"y\",83).merge(x).attr(\"x\",(function(e){return(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+4.5})).attr(\"fill\",(function(e){return\"url(#linear-grad-\".concat(e.effect>0?0:1,\")\")})),x.exit().remove();var k=this.mainGroup.selectAll(\".force-bar-labelLinks\").data(v.slice(0,-1));k.enter().append(\"line\").attr(\"class\",\"force-bar-labelLinks\").attr(\"y1\",73).attr(\"y2\",83).attr(\"stroke-opacity\",.5).attr(\"stroke-width\",1).merge(k).attr(\"x1\",(function(e){return o(e.x)+o(Math.abs(e.effect))+u})).attr(\"x2\",(function(e){return(e.effect>0?o(e.textx):o(e.textx)+e.textWidth)+u+5})).attr(\"stroke\",(function(t){return t.effect>0?e.colors[0]:e.colors[1]})),k.exit().remove();var S=this.mainGroup.selectAll(\".force-bar-blockDividers\").data(n.slice(0,-1));S.enter().append(\"path\").attr(\"class\",\"force-bar-blockDividers\").attr(\"stroke-width\",2).attr(\"fill\",\"none\").merge(S).attr(\"d\",(function(e){var t=o(e.x)+o(Math.abs(e.effect))+u;return p([[t,56],[t+(e.effect<0?-4:4),64.5],[t,73]])})).attr(\"stroke\",(function(t,n){return c===n+1||Math.abs(t.effect)<1e-8?\"#rgba(0,0,0,0)\":t.effect>0?e.brighterColors[0]:e.brighterColors[1]})),S.exit().remove(),this.joinPointLine.attr(\"x1\",o(s)+u).attr(\"x2\",o(s)+u).attr(\"y1\",50).attr(\"y2\",56).attr(\"stroke\",\"#F2F2F2\").attr(\"stroke-width\",1).attr(\"opacity\",1),this.joinPointLabelOutline.attr(\"x\",o(s)+u).attr(\"y\",45).attr(\"color\",\"#fff\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",6).text(ze(\",.2f\")(this.invLinkFunction(s-i))).attr(\"opacity\",1),console.log(\"joinPoint\",s,u,50,i),this.joinPointLabel.attr(\"x\",o(s)+u).attr(\"y\",45).attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").text(ze(\",.2f\")(this.invLinkFunction(s-i))).attr(\"opacity\",1),this.joinPointTitle.attr(\"x\",o(s)+u).attr(\"y\",28).attr(\"text-anchor\",\"middle\").attr(\"font-size\",\"12\").attr(\"fill\",\"#000\").text(this.props.outNames[0]).attr(\"opacity\",.5),this.props.hideBars||(this.joinPointTitleLeft.attr(\"x\",o(s)+u-16).attr(\"y\",12).attr(\"text-anchor\",\"end\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[0]).text(\"higher\").attr(\"opacity\",1),this.joinPointTitleRight.attr(\"x\",o(s)+u+16).attr(\"y\",12).attr(\"text-anchor\",\"start\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[1]).text(\"lower\").attr(\"opacity\",1),this.joinPointTitleLeftArrow.attr(\"x\",o(s)+u+7).attr(\"y\",8).attr(\"text-anchor\",\"end\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[0]).text(\"→\").attr(\"opacity\",1),this.joinPointTitleRightArrow.attr(\"x\",o(s)+u-7).attr(\"y\",14).attr(\"text-anchor\",\"start\").attr(\"font-size\",\"13\").attr(\"fill\",this.colors[1]).text(\"←\").attr(\"opacity\",1)),this.props.hideBaseValueLabel||this.baseValueTitle.attr(\"x\",this.scaleCentered(0)).attr(\"y\",28).attr(\"text-anchor\",\"middle\").attr(\"font-size\",\"12\").attr(\"fill\",\"#000\").text(\"base value\").attr(\"opacity\",.5)}},{key:\"componentWillUnmount\",value:function(){window.removeEventListener(\"resize\",this.redraw)}},{key:\"render\",value:function(){var t=this;return e.createElement(\"svg\",{ref:function(e){return t.svg=Jt(e)},style:{userSelect:\"none\",display:\"block\",fontFamily:\"arial\",sansSerif:!0}},e.createElement(\"style\",{dangerouslySetInnerHTML:{__html:\"\\n          .force-bar-axis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-axis paths {\\n            display: none;\\n          }\\n          .tick line {\\n            stroke: #000;\\n            stroke-width: 1px;\\n            opacity: 0.4;\\n          }\\n          .tick text {\\n            fill: #000;\\n            opacity: 0.5;\\n            font-size: 12px;\\n            padding: 0px;\\n          }\"}}))}}])&&Tn(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);zn.defaultProps={plot_cmap:\"RdBu\"};const Ln=zn,On=1e3,An=6e4,Fn=36e5,Dn=864e5,Rn=6048e5,jn=31536e6,Un=new Date,In=new Date;function $n(e,t,n,r){function a(t){return e(t=0===arguments.length?new Date:new Date(+t)),t}return a.floor=t=>(e(t=new Date(+t)),t),a.ceil=n=>(e(n=new Date(n-1)),t(n,1),e(n),n),a.round=e=>{const t=a(e),n=a.ceil(e);return e-t<n-e?t:n},a.offset=(e,n)=>(t(e=new Date(+e),null==n?1:Math.floor(n)),e),a.range=(n,r,i)=>{const o=[];if(n=a.ceil(n),i=null==i?1:Math.floor(i),!(n<r&&i>0))return o;let u;do{o.push(u=new Date(+n)),t(n,i),e(n)}while(u<n&&n<r);return o},a.filter=n=>$n((t=>{if(t>=t)for(;e(t),!n(t);)t.setTime(t-1)}),((e,r)=>{if(e>=e)if(r<0)for(;++r<=0;)for(;t(e,-1),!n(e););else for(;--r>=0;)for(;t(e,1),!n(e););})),n&&(a.count=(t,r)=>(Un.setTime(+t),In.setTime(+r),e(Un),e(In),Math.floor(n(Un,In))),a.every=e=>(e=Math.floor(e),isFinite(e)&&e>0?e>1?a.filter(r?t=>r(t)%e==0:t=>a.count(0,t)%e==0):a:null)),a}const Bn=$n((()=>{}),((e,t)=>{e.setTime(+e+t)}),((e,t)=>t-e));Bn.every=e=>(e=Math.floor(e),isFinite(e)&&e>0?e>1?$n((t=>{t.setTime(Math.floor(t/e)*e)}),((t,n)=>{t.setTime(+t+n*e)}),((t,n)=>(n-t)/e)):Bn:null),Bn.range;const Wn=$n((e=>{e.setTime(e-e.getMilliseconds())}),((e,t)=>{e.setTime(+e+t*On)}),((e,t)=>(t-e)/On),(e=>e.getUTCSeconds())),Vn=(Wn.range,$n((e=>{e.setTime(e-e.getMilliseconds()-e.getSeconds()*On)}),((e,t)=>{e.setTime(+e+t*An)}),((e,t)=>(t-e)/An),(e=>e.getMinutes()))),Hn=(Vn.range,$n((e=>{e.setUTCSeconds(0,0)}),((e,t)=>{e.setTime(+e+t*An)}),((e,t)=>(t-e)/An),(e=>e.getUTCMinutes()))),qn=(Hn.range,$n((e=>{e.setTime(e-e.getMilliseconds()-e.getSeconds()*On-e.getMinutes()*An)}),((e,t)=>{e.setTime(+e+t*Fn)}),((e,t)=>(t-e)/Fn),(e=>e.getHours()))),Qn=(qn.range,$n((e=>{e.setUTCMinutes(0,0,0)}),((e,t)=>{e.setTime(+e+t*Fn)}),((e,t)=>(t-e)/Fn),(e=>e.getUTCHours()))),Yn=(Qn.range,$n((e=>e.setHours(0,0,0,0)),((e,t)=>e.setDate(e.getDate()+t)),((e,t)=>(t-e-(t.getTimezoneOffset()-e.getTimezoneOffset())*An)/Dn),(e=>e.getDate()-1))),Gn=(Yn.range,$n((e=>{e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+t)}),((e,t)=>(t-e)/Dn),(e=>e.getUTCDate()-1))),Kn=(Gn.range,$n((e=>{e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+t)}),((e,t)=>(t-e)/Dn),(e=>Math.floor(e/Dn))));function Zn(e){return $n((t=>{t.setDate(t.getDate()-(t.getDay()+7-e)%7),t.setHours(0,0,0,0)}),((e,t)=>{e.setDate(e.getDate()+7*t)}),((e,t)=>(t-e-(t.getTimezoneOffset()-e.getTimezoneOffset())*An)/Rn))}Kn.range;const Xn=Zn(0),Jn=Zn(1),er=Zn(2),tr=Zn(3),nr=Zn(4),rr=Zn(5),ar=Zn(6);function ir(e){return $n((t=>{t.setUTCDate(t.getUTCDate()-(t.getUTCDay()+7-e)%7),t.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCDate(e.getUTCDate()+7*t)}),((e,t)=>(t-e)/Rn))}Xn.range,Jn.range,er.range,tr.range,nr.range,rr.range,ar.range;const or=ir(0),ur=ir(1),lr=ir(2),sr=ir(3),cr=ir(4),fr=ir(5),pr=ir(6),dr=(or.range,ur.range,lr.range,sr.range,cr.range,fr.range,pr.range,$n((e=>{e.setDate(1),e.setHours(0,0,0,0)}),((e,t)=>{e.setMonth(e.getMonth()+t)}),((e,t)=>t.getMonth()-e.getMonth()+12*(t.getFullYear()-e.getFullYear())),(e=>e.getMonth()))),hr=(dr.range,$n((e=>{e.setUTCDate(1),e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCMonth(e.getUTCMonth()+t)}),((e,t)=>t.getUTCMonth()-e.getUTCMonth()+12*(t.getUTCFullYear()-e.getUTCFullYear())),(e=>e.getUTCMonth()))),vr=(hr.range,$n((e=>{e.setMonth(0,1),e.setHours(0,0,0,0)}),((e,t)=>{e.setFullYear(e.getFullYear()+t)}),((e,t)=>t.getFullYear()-e.getFullYear()),(e=>e.getFullYear())));vr.every=e=>isFinite(e=Math.floor(e))&&e>0?$n((t=>{t.setFullYear(Math.floor(t.getFullYear()/e)*e),t.setMonth(0,1),t.setHours(0,0,0,0)}),((t,n)=>{t.setFullYear(t.getFullYear()+n*e)})):null,vr.range;const gr=$n((e=>{e.setUTCMonth(0,1),e.setUTCHours(0,0,0,0)}),((e,t)=>{e.setUTCFullYear(e.getUTCFullYear()+t)}),((e,t)=>t.getUTCFullYear()-e.getUTCFullYear()),(e=>e.getUTCFullYear()));function yr(e,t,n,r,a,i){const o=[[Wn,1,On],[Wn,5,5e3],[Wn,15,15e3],[Wn,30,3e4],[i,1,An],[i,5,3e5],[i,15,9e5],[i,30,18e5],[a,1,Fn],[a,3,108e5],[a,6,216e5],[a,12,432e5],[r,1,Dn],[r,2,1728e5],[n,1,Rn],[t,1,2592e6],[t,3,7776e6],[e,1,jn]];function u(t,n,r){const a=Math.abs(n-t)/r,i=f((([,,e])=>e)).right(o,a);if(i===o.length)return e.every(l(t/jn,n/jn,r));if(0===i)return Bn.every(Math.max(l(t,n,r),1));const[u,s]=o[a/o[i-1][2]<o[i][2]/a?i-1:i];return u.every(s)}return[function(e,t,n){const r=t<e;r&&([e,t]=[t,e]);const a=n&&\"function\"==typeof n.range?n:u(e,t,n),i=a?a.range(e,+t+1):[];return r?i.reverse():i},u]}gr.every=e=>isFinite(e=Math.floor(e))&&e>0?$n((t=>{t.setUTCFullYear(Math.floor(t.getUTCFullYear()/e)*e),t.setUTCMonth(0,1),t.setUTCHours(0,0,0,0)}),((t,n)=>{t.setUTCFullYear(t.getUTCFullYear()+n*e)})):null,gr.range;const[mr,br]=yr(gr,hr,or,Kn,Qn,Hn),[_r,wr]=yr(vr,dr,Xn,Yn,qn,Vn);function xr(e){if(0<=e.y&&e.y<100){var t=new Date(-1,e.m,e.d,e.H,e.M,e.S,e.L);return t.setFullYear(e.y),t}return new Date(e.y,e.m,e.d,e.H,e.M,e.S,e.L)}function kr(e){if(0<=e.y&&e.y<100){var t=new Date(Date.UTC(-1,e.m,e.d,e.H,e.M,e.S,e.L));return t.setUTCFullYear(e.y),t}return new Date(Date.UTC(e.y,e.m,e.d,e.H,e.M,e.S,e.L))}function Sr(e,t,n){return{y:e,m:t,d:n,H:0,M:0,S:0,L:0}}var Er,Cr,Tr,Mr={\"-\":\"\",_:\" \",0:\"0\"},Nr=/^\\s*\\d+/,Pr=/^%/,zr=/[\\\\^$*+?|[\\]().{}]/g;function Lr(e,t,n){var r=e<0?\"-\":\"\",a=(r?-e:e)+\"\",i=a.length;return r+(i<n?new Array(n-i+1).join(t)+a:a)}function Or(e){return e.replace(zr,\"\\\\$&\")}function Ar(e){return new RegExp(\"^(?:\"+e.map(Or).join(\"|\")+\")\",\"i\")}function Fr(e){return new Map(e.map(((e,t)=>[e.toLowerCase(),t])))}function Dr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.w=+r[0],n+r[0].length):-1}function Rr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.u=+r[0],n+r[0].length):-1}function jr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.U=+r[0],n+r[0].length):-1}function Ur(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.V=+r[0],n+r[0].length):-1}function Ir(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.W=+r[0],n+r[0].length):-1}function $r(e,t,n){var r=Nr.exec(t.slice(n,n+4));return r?(e.y=+r[0],n+r[0].length):-1}function Br(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.y=+r[0]+(+r[0]>68?1900:2e3),n+r[0].length):-1}function Wr(e,t,n){var r=/^(Z)|([+-]\\d\\d)(?::?(\\d\\d))?/.exec(t.slice(n,n+6));return r?(e.Z=r[1]?0:-(r[2]+(r[3]||\"00\")),n+r[0].length):-1}function Vr(e,t,n){var r=Nr.exec(t.slice(n,n+1));return r?(e.q=3*r[0]-3,n+r[0].length):-1}function Hr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.m=r[0]-1,n+r[0].length):-1}function qr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.d=+r[0],n+r[0].length):-1}function Qr(e,t,n){var r=Nr.exec(t.slice(n,n+3));return r?(e.m=0,e.d=+r[0],n+r[0].length):-1}function Yr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.H=+r[0],n+r[0].length):-1}function Gr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.M=+r[0],n+r[0].length):-1}function Kr(e,t,n){var r=Nr.exec(t.slice(n,n+2));return r?(e.S=+r[0],n+r[0].length):-1}function Zr(e,t,n){var r=Nr.exec(t.slice(n,n+3));return r?(e.L=+r[0],n+r[0].length):-1}function Xr(e,t,n){var r=Nr.exec(t.slice(n,n+6));return r?(e.L=Math.floor(r[0]/1e3),n+r[0].length):-1}function Jr(e,t,n){var r=Pr.exec(t.slice(n,n+1));return r?n+r[0].length:-1}function ea(e,t,n){var r=Nr.exec(t.slice(n));return r?(e.Q=+r[0],n+r[0].length):-1}function ta(e,t,n){var r=Nr.exec(t.slice(n));return r?(e.s=+r[0],n+r[0].length):-1}function na(e,t){return Lr(e.getDate(),t,2)}function ra(e,t){return Lr(e.getHours(),t,2)}function aa(e,t){return Lr(e.getHours()%12||12,t,2)}function ia(e,t){return Lr(1+Yn.count(vr(e),e),t,3)}function oa(e,t){return Lr(e.getMilliseconds(),t,3)}function ua(e,t){return oa(e,t)+\"000\"}function la(e,t){return Lr(e.getMonth()+1,t,2)}function sa(e,t){return Lr(e.getMinutes(),t,2)}function ca(e,t){return Lr(e.getSeconds(),t,2)}function fa(e){var t=e.getDay();return 0===t?7:t}function pa(e,t){return Lr(Xn.count(vr(e)-1,e),t,2)}function da(e){var t=e.getDay();return t>=4||0===t?nr(e):nr.ceil(e)}function ha(e,t){return e=da(e),Lr(nr.count(vr(e),e)+(4===vr(e).getDay()),t,2)}function va(e){return e.getDay()}function ga(e,t){return Lr(Jn.count(vr(e)-1,e),t,2)}function ya(e,t){return Lr(e.getFullYear()%100,t,2)}function ma(e,t){return Lr((e=da(e)).getFullYear()%100,t,2)}function ba(e,t){return Lr(e.getFullYear()%1e4,t,4)}function _a(e,t){var n=e.getDay();return Lr((e=n>=4||0===n?nr(e):nr.ceil(e)).getFullYear()%1e4,t,4)}function wa(e){var t=e.getTimezoneOffset();return(t>0?\"-\":(t*=-1,\"+\"))+Lr(t/60|0,\"0\",2)+Lr(t%60,\"0\",2)}function xa(e,t){return Lr(e.getUTCDate(),t,2)}function ka(e,t){return Lr(e.getUTCHours(),t,2)}function Sa(e,t){return Lr(e.getUTCHours()%12||12,t,2)}function Ea(e,t){return Lr(1+Gn.count(gr(e),e),t,3)}function Ca(e,t){return Lr(e.getUTCMilliseconds(),t,3)}function Ta(e,t){return Ca(e,t)+\"000\"}function Ma(e,t){return Lr(e.getUTCMonth()+1,t,2)}function Na(e,t){return Lr(e.getUTCMinutes(),t,2)}function Pa(e,t){return Lr(e.getUTCSeconds(),t,2)}function za(e){var t=e.getUTCDay();return 0===t?7:t}function La(e,t){return Lr(or.count(gr(e)-1,e),t,2)}function Oa(e){var t=e.getUTCDay();return t>=4||0===t?cr(e):cr.ceil(e)}function Aa(e,t){return e=Oa(e),Lr(cr.count(gr(e),e)+(4===gr(e).getUTCDay()),t,2)}function Fa(e){return e.getUTCDay()}function Da(e,t){return Lr(ur.count(gr(e)-1,e),t,2)}function Ra(e,t){return Lr(e.getUTCFullYear()%100,t,2)}function ja(e,t){return Lr((e=Oa(e)).getUTCFullYear()%100,t,2)}function Ua(e,t){return Lr(e.getUTCFullYear()%1e4,t,4)}function Ia(e,t){var n=e.getUTCDay();return Lr((e=n>=4||0===n?cr(e):cr.ceil(e)).getUTCFullYear()%1e4,t,4)}function $a(){return\"+0000\"}function Ba(){return\"%\"}function Wa(e){return+e}function Va(e){return Math.floor(+e/1e3)}function Ha(e){return new Date(e)}function qa(e){return e instanceof Date?+e:+new Date(+e)}function Qa(e,t,n,r,a,i,o,u,l,s){var c=be(),f=c.invert,p=c.domain,d=s(\".%L\"),h=s(\":%S\"),v=s(\"%I:%M\"),g=s(\"%I %p\"),y=s(\"%a %d\"),m=s(\"%b %d\"),b=s(\"%B\"),_=s(\"%Y\");function w(e){return(l(e)<e?d:u(e)<e?h:o(e)<e?v:i(e)<e?g:r(e)<e?a(e)<e?y:m:n(e)<e?b:_)(e)}return c.invert=function(e){return new Date(f(e))},c.domain=function(e){return arguments.length?p(Array.from(e,qa)):p().map(Ha)},c.ticks=function(t){var n=p();return e(n[0],n[n.length-1],null==t?10:t)},c.tickFormat=function(e,t){return null==t?w:s(t)},c.nice=function(e){var n=p();return e&&\"function\"==typeof e.range||(e=t(n[0],n[n.length-1],null==e?10:e)),e?p(function(e,t){var n,r=0,a=(e=e.slice()).length-1,i=e[r],o=e[a];return o<i&&(n=r,r=a,a=n,n=i,i=o,o=n),e[r]=t.floor(i),e[a]=t.ceil(o),e}(n,e)):c},c.copy=function(){return me(c,Qa(e,t,n,r,a,i,o,u,l,s))},c}function Ya(){return _e.apply(Qa(_r,wr,vr,dr,Xn,Yn,qn,Vn,Wn,Cr).domain([new Date(2e3,0,1),new Date(2e3,0,2)]),arguments)}function Ga(e,t){var n=\"undefined\"!=typeof Symbol&&e[Symbol.iterator]||e[\"@@iterator\"];if(!n){if(Array.isArray(e)||(n=function(e,t){if(e){if(\"string\"==typeof e)return Ka(e,t);var n=Object.prototype.toString.call(e).slice(8,-1);return\"Object\"===n&&e.constructor&&(n=e.constructor.name),\"Map\"===n||\"Set\"===n?Array.from(e):\"Arguments\"===n||/^(?:Ui|I)nt(?:8|16|32)(?:Clamped)?Array$/.test(n)?Ka(e,t):void 0}}(e))||t&&e&&\"number\"==typeof e.length){n&&(e=n);var r=0,a=function(){};return{s:a,n:function(){return r>=e.length?{done:!0}:{done:!1,value:e[r++]}},e:function(e){throw e},f:a}}throw new TypeError(\"Invalid attempt to iterate non-iterable instance.\\nIn order to be iterable, non-array objects must have a [Symbol.iterator]() method.\")}var i,o=!0,u=!1;return{s:function(){n=n.call(e)},n:function(){var e=n.next();return o=e.done,e},e:function(e){u=!0,i=e},f:function(){try{o||null==n.return||n.return()}finally{if(u)throw i}}}}function Ka(e,t){(null==t||t>e.length)&&(t=e.length);for(var n=0,r=new Array(t);n<t;n++)r[n]=e[n];return r}function Za(e){return Za=\"function\"==typeof Symbol&&\"symbol\"==typeof Symbol.iterator?function(e){return typeof e}:function(e){return e&&\"function\"==typeof Symbol&&e.constructor===Symbol&&e!==Symbol.prototype?\"symbol\":typeof e},Za(e)}function Xa(e,t){for(var n=0;n<t.length;n++){var r=t[n];r.enumerable=r.enumerable||!1,r.configurable=!0,\"value\"in r&&(r.writable=!0),Object.defineProperty(e,(void 0,a=function(e,t){if(\"object\"!==Za(e)||null===e)return e;var n=e[Symbol.toPrimitive];if(void 0!==n){var r=n.call(e,\"string\");if(\"object\"!==Za(r))return r;throw new TypeError(\"@@toPrimitive must return a primitive value.\")}return String(e)}(r.key),\"symbol\"===Za(a)?a:String(a)),r)}var a}function Ja(e,t){return Ja=Object.setPrototypeOf?Object.setPrototypeOf.bind():function(e,t){return e.__proto__=t,e},Ja(e,t)}function ei(e){if(void 0===e)throw new ReferenceError(\"this hasn't been initialised - super() hasn't been called\");return e}function ti(e){return ti=Object.setPrototypeOf?Object.getPrototypeOf.bind():function(e){return e.__proto__||Object.getPrototypeOf(e)},ti(e)}Er=function(e){var t=e.dateTime,n=e.date,r=e.time,a=e.periods,i=e.days,o=e.shortDays,u=e.months,l=e.shortMonths,s=Ar(a),c=Fr(a),f=Ar(i),p=Fr(i),d=Ar(o),h=Fr(o),v=Ar(u),g=Fr(u),y=Ar(l),m=Fr(l),b={a:function(e){return o[e.getDay()]},A:function(e){return i[e.getDay()]},b:function(e){return l[e.getMonth()]},B:function(e){return u[e.getMonth()]},c:null,d:na,e:na,f:ua,g:ma,G:_a,H:ra,I:aa,j:ia,L:oa,m:la,M:sa,p:function(e){return a[+(e.getHours()>=12)]},q:function(e){return 1+~~(e.getMonth()/3)},Q:Wa,s:Va,S:ca,u:fa,U:pa,V:ha,w:va,W:ga,x:null,X:null,y:ya,Y:ba,Z:wa,\"%\":Ba},_={a:function(e){return o[e.getUTCDay()]},A:function(e){return i[e.getUTCDay()]},b:function(e){return l[e.getUTCMonth()]},B:function(e){return u[e.getUTCMonth()]},c:null,d:xa,e:xa,f:Ta,g:ja,G:Ia,H:ka,I:Sa,j:Ea,L:Ca,m:Ma,M:Na,p:function(e){return a[+(e.getUTCHours()>=12)]},q:function(e){return 1+~~(e.getUTCMonth()/3)},Q:Wa,s:Va,S:Pa,u:za,U:La,V:Aa,w:Fa,W:Da,x:null,X:null,y:Ra,Y:Ua,Z:$a,\"%\":Ba},w={a:function(e,t,n){var r=d.exec(t.slice(n));return r?(e.w=h.get(r[0].toLowerCase()),n+r[0].length):-1},A:function(e,t,n){var r=f.exec(t.slice(n));return r?(e.w=p.get(r[0].toLowerCase()),n+r[0].length):-1},b:function(e,t,n){var r=y.exec(t.slice(n));return r?(e.m=m.get(r[0].toLowerCase()),n+r[0].length):-1},B:function(e,t,n){var r=v.exec(t.slice(n));return r?(e.m=g.get(r[0].toLowerCase()),n+r[0].length):-1},c:function(e,n,r){return S(e,t,n,r)},d:qr,e:qr,f:Xr,g:Br,G:$r,H:Yr,I:Yr,j:Qr,L:Zr,m:Hr,M:Gr,p:function(e,t,n){var r=s.exec(t.slice(n));return r?(e.p=c.get(r[0].toLowerCase()),n+r[0].length):-1},q:Vr,Q:ea,s:ta,S:Kr,u:Rr,U:jr,V:Ur,w:Dr,W:Ir,x:function(e,t,r){return S(e,n,t,r)},X:function(e,t,n){return S(e,r,t,n)},y:Br,Y:$r,Z:Wr,\"%\":Jr};function x(e,t){return function(n){var r,a,i,o=[],u=-1,l=0,s=e.length;for(n instanceof Date||(n=new Date(+n));++u<s;)37===e.charCodeAt(u)&&(o.push(e.slice(l,u)),null!=(a=Mr[r=e.charAt(++u)])?r=e.charAt(++u):a=\"e\"===r?\" \":\"0\",(i=t[r])&&(r=i(n,a)),o.push(r),l=u+1);return o.push(e.slice(l,u)),o.join(\"\")}}function k(e,t){return function(n){var r,a,i=Sr(1900,void 0,1);if(S(i,e,n+=\"\",0)!=n.length)return null;if(\"Q\"in i)return new Date(i.Q);if(\"s\"in i)return new Date(1e3*i.s+(\"L\"in i?i.L:0));if(t&&!(\"Z\"in i)&&(i.Z=0),\"p\"in i&&(i.H=i.H%12+12*i.p),void 0===i.m&&(i.m=\"q\"in i?i.q:0),\"V\"in i){if(i.V<1||i.V>53)return null;\"w\"in i||(i.w=1),\"Z\"in i?(a=(r=kr(Sr(i.y,0,1))).getUTCDay(),r=a>4||0===a?ur.ceil(r):ur(r),r=Gn.offset(r,7*(i.V-1)),i.y=r.getUTCFullYear(),i.m=r.getUTCMonth(),i.d=r.getUTCDate()+(i.w+6)%7):(a=(r=xr(Sr(i.y,0,1))).getDay(),r=a>4||0===a?Jn.ceil(r):Jn(r),r=Yn.offset(r,7*(i.V-1)),i.y=r.getFullYear(),i.m=r.getMonth(),i.d=r.getDate()+(i.w+6)%7)}else(\"W\"in i||\"U\"in i)&&(\"w\"in i||(i.w=\"u\"in i?i.u%7:\"W\"in i?1:0),a=\"Z\"in i?kr(Sr(i.y,0,1)).getUTCDay():xr(Sr(i.y,0,1)).getDay(),i.m=0,i.d=\"W\"in i?(i.w+6)%7+7*i.W-(a+5)%7:i.w+7*i.U-(a+6)%7);return\"Z\"in i?(i.H+=i.Z/100|0,i.M+=i.Z%100,kr(i)):xr(i)}}function S(e,t,n,r){for(var a,i,o=0,u=t.length,l=n.length;o<u;){if(r>=l)return-1;if(37===(a=t.charCodeAt(o++))){if(a=t.charAt(o++),!(i=w[a in Mr?t.charAt(o++):a])||(r=i(e,n,r))<0)return-1}else if(a!=n.charCodeAt(r++))return-1}return r}return b.x=x(n,b),b.X=x(r,b),b.c=x(t,b),_.x=x(n,_),_.X=x(r,_),_.c=x(t,_),{format:function(e){var t=x(e+=\"\",b);return t.toString=function(){return e},t},parse:function(e){var t=k(e+=\"\",!1);return t.toString=function(){return e},t},utcFormat:function(e){var t=x(e+=\"\",_);return t.toString=function(){return e},t},utcParse:function(e){var t=k(e+=\"\",!0);return t.toString=function(){return e},t}}}({dateTime:\"%x, %X\",date:\"%-m/%-d/%Y\",time:\"%-I:%M:%S %p\",periods:[\"AM\",\"PM\"],days:[\"Sunday\",\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\",\"Friday\",\"Saturday\"],shortDays:[\"Sun\",\"Mon\",\"Tue\",\"Wed\",\"Thu\",\"Fri\",\"Sat\"],months:[\"January\",\"February\",\"March\",\"April\",\"May\",\"June\",\"July\",\"August\",\"September\",\"October\",\"November\",\"December\"],shortMonths:[\"Jan\",\"Feb\",\"Mar\",\"Apr\",\"May\",\"Jun\",\"Jul\",\"Aug\",\"Sep\",\"Oct\",\"Nov\",\"Dec\"]}),Cr=Er.format,Tr=Er.parse,Er.utcFormat,Er.utcParse;var ni=function(t){!function(e,t){if(\"function\"!=typeof t&&null!==t)throw new TypeError(\"Super expression must either be null or a function\");e.prototype=Object.create(t&&t.prototype,{constructor:{value:e,writable:!0,configurable:!0}}),Object.defineProperty(e,\"prototype\",{writable:!1}),t&&Ja(e,t)}(u,t);var n,r,a,i,o=(a=u,i=function(){if(\"undefined\"==typeof Reflect||!Reflect.construct)return!1;if(Reflect.construct.sham)return!1;if(\"function\"==typeof Proxy)return!0;try{return Boolean.prototype.valueOf.call(Reflect.construct(Boolean,[],(function(){}))),!0}catch(e){return!1}}(),function(){var e,t=ti(a);if(i){var n=ti(this).constructor;e=Reflect.construct(t,arguments,n)}else e=t.apply(this,arguments);return function(e,t){if(t&&(\"object\"===Za(t)||\"function\"==typeof t))return t;if(void 0!==t)throw new TypeError(\"Derived constructors may only return object or undefined\");return ei(e)}(this,e)});function u(){var e;return function(e,t){if(!(e instanceof t))throw new TypeError(\"Cannot call a class as a function\")}(this,u),e=o.call(this),window.lastAdditiveForceArrayVisualizer=ei(e),e.topOffset=28,e.leftOffset=80,e.height=350,e.effectFormat=ze(\".2\"),e.redraw=(0,Re.debounce)((function(){return e.draw()}),200),e}return n=u,(r=[{key:\"componentDidMount\",value:function(){var e=this;this.mainGroup=this.svg.append(\"g\"),this.onTopGroup=this.svg.append(\"g\"),this.xaxisElement=this.onTopGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-array-xaxis\"),this.yaxisElement=this.onTopGroup.append(\"g\").attr(\"transform\",\"translate(0,35)\").attr(\"class\",\"force-bar-array-yaxis\"),this.hoverGroup1=this.svg.append(\"g\"),this.hoverGroup2=this.svg.append(\"g\"),this.baseValueTitle=this.svg.append(\"text\"),this.hoverLine=this.svg.append(\"line\"),this.hoverxOutline=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#fff\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",\"6\").attr(\"font-size\",\"12px\"),this.hoverx=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").attr(\"font-size\",\"12px\"),this.hoverxTitle=this.svg.append(\"text\").attr(\"text-anchor\",\"middle\").attr(\"opacity\",.6).attr(\"font-size\",\"12px\"),this.hoveryOutline=this.svg.append(\"text\").attr(\"text-anchor\",\"end\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#fff\").attr(\"stroke\",\"#fff\").attr(\"stroke-width\",\"6\").attr(\"font-size\",\"12px\"),this.hovery=this.svg.append(\"text\").attr(\"text-anchor\",\"end\").attr(\"font-weight\",\"bold\").attr(\"fill\",\"#000\").attr(\"font-size\",\"12px\"),this.xlabel=this.wrapper.select(\".additive-force-array-xlabel\"),this.ylabel=this.wrapper.select(\".additive-force-array-ylabel\");var t=void 0;\"string\"==typeof this.props.plot_cmap?this.props.plot_cmap in je.colors?t=je.colors[this.props.plot_cmap]:(console.log(\"Invalid color map name, reverting to default.\"),t=je.colors.RdBu):Array.isArray(this.props.plot_cmap)&&(t=this.props.plot_cmap),this.colors=t.map((function(e){return q(e)})),this.brighterColors=[1.45,1.6].map((function(t,n){return e.colors[n].brighter(t)}));var n=ze(\",.4\");null!=this.props.ordering_keys&&null!=this.props.ordering_keys_time_format?(this.parseTime=Tr(this.props.ordering_keys_time_format),this.formatTime=Cr(this.props.ordering_keys_time_format),this.xtickFormat=function(e){return\"object\"==Za(e)?this.formatTime(e):n(e)}):(this.parseTime=null,this.formatTime=null,this.xtickFormat=n),this.xscale=De(),this.xaxis=dn().scale(this.xscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.xtickFormat(t)})).tickPadding(-18),this.ytickFormat=n,this.yscale=De(),this.yaxis=pn(an,undefined).scale(this.yscale).tickSizeInner(4).tickSizeOuter(0).tickFormat((function(t){return e.ytickFormat(e.invLinkFunction(t))})).tickPadding(2),this.xlabel.node().onchange=function(){return e.internalDraw()},this.ylabel.node().onchange=function(){return e.internalDraw()},this.svg.on(\"mousemove\",(function(t){return e.mouseMoved(t)})),this.svg.on(\"click\",(function(){return alert(\"This original index of the sample you clicked is \"+e.nearestExpIndex)})),this.svg.on(\"mouseout\",(function(t){return e.mouseOut(t)})),window.addEventListener(\"resize\",this.redraw),window.setTimeout(this.redraw,50)}},{key:\"componentDidUpdate\",value:function(){this.draw()}},{key:\"mouseOut\",value:function(){this.hoverLine.attr(\"display\",\"none\"),this.hoverx.attr(\"display\",\"none\"),this.hoverxOutline.attr(\"display\",\"none\"),this.hoverxTitle.attr(\"display\",\"none\"),this.hovery.attr(\"display\",\"none\"),this.hoveryOutline.attr(\"display\",\"none\"),this.hoverGroup1.attr(\"display\",\"none\"),this.hoverGroup2.attr(\"display\",\"none\")}},{key:\"mouseMoved\",value:function(e){var t,n,r=this;this.hoverLine.attr(\"display\",\"\"),this.hoverx.attr(\"display\",\"\"),this.hoverxOutline.attr(\"display\",\"\"),this.hoverxTitle.attr(\"display\",\"\"),this.hovery.attr(\"display\",\"\"),this.hoveryOutline.attr(\"display\",\"\"),this.hoverGroup1.attr(\"display\",\"\"),this.hoverGroup2.attr(\"display\",\"\");var a=function(e,t){if(e=function(e){let t;for(;t=e.sourceEvent;)e=t;return e}(e),void 0===t&&(t=e.currentTarget),t){var n=t.ownerSVGElement||t;if(n.createSVGPoint){var r=n.createSVGPoint();return r.x=e.clientX,r.y=e.clientY,[(r=r.matrixTransform(t.getScreenCTM().inverse())).x,r.y]}if(t.getBoundingClientRect){var a=t.getBoundingClientRect();return[e.clientX-a.left-t.clientLeft,e.clientY-a.top-t.clientTop]}}return[e.pageX,e.pageY]}(e,this.svg.node())[0];if(this.props.explanations){for(t=0;t<this.currExplanations.length;++t)(!n||Math.abs(n.xmapScaled-a)>Math.abs(this.currExplanations[t].xmapScaled-a))&&(n=this.currExplanations[t]);this.nearestExpIndex=n.origInd,this.hoverLine.attr(\"x1\",n.xmapScaled).attr(\"x2\",n.xmapScaled).attr(\"y1\",0+this.topOffset).attr(\"y2\",this.height),this.hoverx.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-5).text(this.xtickFormat(n.xmap)),this.hoverxOutline.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-5).text(this.xtickFormat(n.xmap)),this.hoverxTitle.attr(\"x\",n.xmapScaled).attr(\"y\",this.topOffset-18).text(n.count>1?n.count+\" averaged samples\":\"\"),this.hovery.attr(\"x\",this.leftOffset-6).attr(\"y\",n.joinPointy).text(this.ytickFormat(this.invLinkFunction(n.joinPoint))),this.hoveryOutline.attr(\"x\",this.leftOffset-6).attr(\"y\",n.joinPointy).text(this.ytickFormat(this.invLinkFunction(n.joinPoint)));for(var i,o,u=[],l=this.currPosOrderedFeatures.length-1;l>=0;--l){var s=this.currPosOrderedFeatures[l],c=n.features[s];o=5+(c.posyTop+c.posyBottom)/2,(!i||o-i>=15)&&c.posyTop-c.posyBottom>=6&&(u.push(c),i=o)}var f=[];i=void 0;var p,d=Ga(this.currNegOrderedFeatures);try{for(d.s();!(p=d.n()).done;){var h=p.value,v=n.features[h];o=5+(v.negyTop+v.negyBottom)/2,(!i||i-o>=15)&&v.negyTop-v.negyBottom>=6&&(f.push(v),i=o)}}catch(e){d.e(e)}finally{d.f()}var g=function(e){var t=\"\";return null!==e.value&&void 0!==e.value&&(t=\" = \"+(isNaN(e.value)?e.value:r.ytickFormat(e.value))),n.count>1?\"mean(\"+r.props.featureNames[e.ind]+\")\"+t:r.props.featureNames[e.ind]+t},y=this.hoverGroup1.selectAll(\".pos-values\").data(u);y.enter().append(\"text\").attr(\"class\",\"pos-values\").merge(y).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.posyTop+e.posyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").attr(\"opacity\",1).text(g),y.exit().remove();var m=this.hoverGroup2.selectAll(\".pos-values\").data(u);m.enter().append(\"text\").attr(\"class\",\"pos-values\").merge(m).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.posyTop+e.posyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"fill\",this.colors[0]).text(g),m.exit().remove();var b=this.hoverGroup1.selectAll(\".neg-values\").data(f);b.enter().append(\"text\").attr(\"class\",\"neg-values\").merge(b).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.negyTop+e.negyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"stroke\",\"#fff\").attr(\"fill\",\"#fff\").attr(\"stroke-width\",\"4\").attr(\"stroke-linejoin\",\"round\").attr(\"opacity\",1).text(g),b.exit().remove();var _=this.hoverGroup2.selectAll(\".neg-values\").data(f);_.enter().append(\"text\").attr(\"class\",\"neg-values\").merge(_).attr(\"x\",n.xmapScaled+5).attr(\"y\",(function(e){return 4+(e.negyTop+e.negyBottom)/2})).attr(\"text-anchor\",\"start\").attr(\"font-size\",12).attr(\"fill\",this.colors[1]).text(g),_.exit().remove()}}},{key:\"draw\",value:function(){var e=this;if(this.props.explanations&&0!==this.props.explanations.length){(0,Re.each)(this.props.explanations,(function(e,t){return e.origInd=t}));var t,n={},r={},a={},i=Ga(this.props.explanations);try{for(i.s();!(t=i.n()).done;){var o=t.value;for(var u in o.features)void 0===n[u]&&(n[u]=0,r[u]=0,a[u]=0),o.features[u].effect>0?n[u]+=o.features[u].effect:r[u]-=o.features[u].effect,null!==o.features[u].value&&void 0!==o.features[u].value&&(a[u]+=1)}}catch(e){i.e(e)}finally{i.f()}this.usedFeatures=(0,Re.sortBy)((0,Re.keys)(n),(function(e){return-(n[e]+r[e])})),console.log(\"found \",this.usedFeatures.length,\" used features\"),this.posOrderedFeatures=(0,Re.sortBy)(this.usedFeatures,(function(e){return n[e]})),this.negOrderedFeatures=(0,Re.sortBy)(this.usedFeatures,(function(e){return-r[e]})),this.singleValueFeatures=(0,Re.filter)(this.usedFeatures,(function(e){return a[e]>0}));var l=[\"sample order by similarity\",\"sample order by output value\",\"original sample ordering\"].concat(this.singleValueFeatures.map((function(t){return e.props.featureNames[t]})));null!=this.props.ordering_keys&&l.unshift(\"sample order by key\");var s=this.xlabel.selectAll(\"option\").data(l);s.enter().append(\"option\").merge(s).attr(\"value\",(function(e){return e})).text((function(e){return e})),s.exit().remove();var c=this.props.outNames[0]?this.props.outNames[0]:\"model output value\";(l=(0,Re.map)(this.usedFeatures,(function(t){return[e.props.featureNames[t],e.props.featureNames[t]+\" effects\"]}))).unshift([\"model output value\",c]);var f=this.ylabel.selectAll(\"option\").data(l);f.enter().append(\"option\").merge(f).attr(\"value\",(function(e){return e[0]})).text((function(e){return e[1]})),f.exit().remove(),this.ylabel.style(\"top\",(this.height-10-this.topOffset)/2+this.topOffset+\"px\").style(\"left\",10-this.ylabel.node().offsetWidth/2+\"px\"),this.internalDraw()}}},{key:\"internalDraw\",value:function(){var e,t,n=this,r=Ga(this.props.explanations);try{for(r.s();!(e=r.n()).done;){var a,i=e.value,o=Ga(this.usedFeatures);try{for(o.s();!(a=o.n()).done;){var u=a.value;i.features.hasOwnProperty(u)||(i.features[u]={effect:0,value:0}),i.features[u].ind=u}}catch(e){o.e(e)}finally{o.f()}}}catch(e){r.e(e)}finally{r.f()}var l=this.xlabel.node().value,s=\"sample order by key\"===l&&null!=this.props.ordering_keys_time_format;if(this.xscale=s?Ya():De(),this.xaxis.scale(this.xscale),\"sample order by similarity\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return e.simIndex})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"sample order by output value\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return-e.outValue})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"original sample ordering\"===l)t=(0,Re.sortBy)(this.props.explanations,(function(e){return e.origInd})),(0,Re.each)(t,(function(e,t){return e.xmap=t}));else if(\"sample order by key\"===l)t=this.props.explanations,s?(0,Re.each)(t,(function(e,t){return e.xmap=n.parseTime(n.props.ordering_keys[t])})):(0,Re.each)(t,(function(e,t){return e.xmap=n.props.ordering_keys[t]})),t=(0,Re.sortBy)(t,(function(e){return e.xmap}));else{var c=(0,Re.findKey)(this.props.featureNames,(function(e){return e===l}));(0,Re.each)(this.props.explanations,(function(e,t){return e.xmap=e.features[c].value}));var f=(0,Re.sortBy)(this.props.explanations,(function(e){return e.xmap})),p=(0,Re.map)(f,(function(e){return e.xmap}));if(\"string\"==typeof p[0])return void alert(\"Ordering by category names is not yet supported.\");var d,h,v=(0,Re.min)(p),g=((0,Re.max)(p)-v)/100;t=[];for(var y=0;y<f.length;++y){var m=f[y];if(d&&!h&&m.xmap-d.xmap<=g||h&&m.xmap-h.xmap<=g){h||((h=(0,Re.cloneDeep)(d)).count=1);var b,_=Ga(this.usedFeatures);try{for(_.s();!(b=_.n()).done;){var w=b.value;h.features[w].effect+=m.features[w].effect,h.features[w].value+=m.features[w].value}}catch(e){_.e(e)}finally{_.f()}h.count+=1}else if(d)if(h){var x,k=Ga(this.usedFeatures);try{for(k.s();!(x=k.n()).done;){var S=x.value;h.features[S].effect/=h.count,h.features[S].value/=h.count}}catch(e){k.e(e)}finally{k.f()}t.push(h),h=void 0}else t.push(d);d=m}d.xmap-t[t.length-1].xmap>g&&t.push(d)}this.currUsedFeatures=this.usedFeatures,this.currPosOrderedFeatures=this.posOrderedFeatures,this.currNegOrderedFeatures=this.negOrderedFeatures;var E=this.ylabel.node().value;if(\"model output value\"!==E){var C=t;t=(0,Re.cloneDeep)(t);for(var T=(0,Re.findKey)(this.props.featureNames,(function(e){return e===E})),M=0;M<t.length;++M){var N=t[M].features[T];t[M].features={},t[M].features[T]=N,C[M].remapped_version=t[M]}this.currUsedFeatures=[T],this.currPosOrderedFeatures=[T],this.currNegOrderedFeatures=[T]}this.currExplanations=t,\"identity\"===this.props.link?this.invLinkFunction=function(e){return n.props.baseValue+e}:\"logit\"===this.props.link?this.invLinkFunction=function(e){return 1/(1+Math.exp(-(n.props.baseValue+e)))}:console.log(\"ERROR: Unrecognized link function: \",this.props.link),this.predValues=(0,Re.map)(t,(function(e){return(0,Re.sum)((0,Re.map)(e.features,(function(e){return e.effect})))}));var P=this.wrapper.node().offsetWidth;if(0==P)return setTimeout((function(){return n.draw(t)}),500);this.svg.style(\"height\",this.height+\"px\"),this.svg.style(\"width\",P+\"px\");var z=(0,Re.map)(t,(function(e){return e.xmap}));this.xscale.domain([(0,Re.min)(z),(0,Re.max)(z)]).range([this.leftOffset,P]).clamp(!0),this.xaxisElement.attr(\"transform\",\"translate(0,\"+this.topOffset+\")\").call(this.xaxis);for(var L=0;L<this.currExplanations.length;++L)this.currExplanations[L].xmapScaled=this.xscale(this.currExplanations[L].xmap);for(var O=t.length,A=0,F=0;F<O;++F){var D=t[F].features,R=(0,Re.sum)((0,Re.map)((0,Re.filter)(D,(function(e){return e.effect>0})),(function(e){return e.effect})))||0,j=(0,Re.sum)((0,Re.map)((0,Re.filter)(D,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0;A=Math.max(A,2.2*Math.max(R,j))}this.yscale.domain([-A/2,A/2]).range([this.height-10,this.topOffset]),this.yaxisElement.attr(\"transform\",\"translate(\"+this.leftOffset+\",0)\").call(this.yaxis);for(var U=0;U<O;++U){var I,$=t[U].features,B=-((0,Re.sum)((0,Re.map)((0,Re.filter)($,(function(e){return e.effect<0})),(function(e){return-e.effect})))||0),W=void 0,V=Ga(this.currPosOrderedFeatures);try{for(V.s();!(I=V.n()).done;)$[W=I.value].posyTop=this.yscale(B),$[W].effect>0&&(B+=$[W].effect),$[W].posyBottom=this.yscale(B),$[W].ind=W}catch(e){V.e(e)}finally{V.f()}var H,q=B,Q=Ga(this.currNegOrderedFeatures);try{for(Q.s();!(H=Q.n()).done;)$[W=H.value].negyTop=this.yscale(B),$[W].effect<0&&(B-=$[W].effect),$[W].negyBottom=this.yscale(B)}catch(e){Q.e(e)}finally{Q.f()}t[U].joinPoint=q,t[U].joinPointy=this.yscale(q)}var Y=En().x((function(e){return e[0]})).y((function(e){return e[1]})),G=this.mainGroup.selectAll(\".force-bar-array-area-pos\").data(this.currUsedFeatures);G.enter().append(\"path\").attr(\"class\",\"force-bar-array-area-pos\").merge(G).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyTop]})),r=(0,Re.map)((0,Re.rangeRight)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyBottom]}));return Y(n.concat(r))})).attr(\"fill\",this.colors[0]),G.exit().remove();var K=this.mainGroup.selectAll(\".force-bar-array-area-neg\").data(this.currUsedFeatures);K.enter().append(\"path\").attr(\"class\",\"force-bar-array-area-neg\").merge(K).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyTop]})),r=(0,Re.map)((0,Re.rangeRight)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyBottom]}));return Y(n.concat(r))})).attr(\"fill\",this.colors[1]),K.exit().remove();var Z=this.mainGroup.selectAll(\".force-bar-array-divider-pos\").data(this.currUsedFeatures);Z.enter().append(\"path\").attr(\"class\",\"force-bar-array-divider-pos\").merge(Z).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].posyBottom]}));return Y(n)})).attr(\"fill\",\"none\").attr(\"stroke-width\",1).attr(\"stroke\",(function(){return n.colors[0].brighter(1.2)})),Z.exit().remove();var X=this.mainGroup.selectAll(\".force-bar-array-divider-neg\").data(this.currUsedFeatures);X.enter().append(\"path\").attr(\"class\",\"force-bar-array-divider-neg\").merge(X).attr(\"d\",(function(e){var n=(0,Re.map)((0,Re.range)(O),(function(n){return[t[n].xmapScaled,t[n].features[e].negyTop]}));return Y(n)})).attr(\"fill\",\"none\").attr(\"stroke-width\",1).attr(\"stroke\",(function(){return n.colors[1].brighter(1.5)})),X.exit().remove();for(var J=function(e,t,n,r,a){var i,o,u,l;\"pos\"===a?(i=e[n].features[t].posyBottom,o=e[n].features[t].posyTop):(i=e[n].features[t].negyBottom,o=e[n].features[t].negyTop);for(var s=n+1;s<=r;++s)\"pos\"===a?(u=e[s].features[t].posyBottom,l=e[s].features[t].posyTop):(u=e[s].features[t].negyBottom,l=e[s].features[t].negyTop),u>i&&(i=u),l<o&&(o=l);return{top:i,bottom:o}},ee=[],te=0,ne=[\"pos\",\"neg\"];te<ne.length;te++){var re,ae=ne[te],ie=Ga(this.currUsedFeatures);try{for(ie.s();!(re=ie.n()).done;)for(var oe=re.value,ue=0,le=0,se=0,ce={top:0,bottom:0},fe=void 0;le<O-1;){for(;se<100&&le<O-1;)++le,se=t[le].xmapScaled-t[ue].xmapScaled;for(ce=J(t,oe,ue,le,ae);ce.bottom-ce.top<20&&ue<le;)++ue,ce=J(t,oe,ue,le,ae);if(se=t[le].xmapScaled-t[ue].xmapScaled,ce.bottom-ce.top>=20&&se>=100){for(;le<O-1;){if(++le,!((fe=J(t,oe,ue,le,ae)).bottom-fe.top>20)){--le;break}ce=fe}se=t[le].xmapScaled-t[ue].xmapScaled,ee.push([(t[le].xmapScaled+t[ue].xmapScaled)/2,(ce.top+ce.bottom)/2,this.props.featureNames[oe]]);var pe=t[le].xmapScaled;for(ue=le;pe+100>t[ue].xmapScaled&&ue<O-1;)++ue;le=ue}}}catch(e){ie.e(e)}finally{ie.f()}}var de=this.onTopGroup.selectAll(\".force-bar-array-flabels\").data(ee);de.enter().append(\"text\").attr(\"class\",\"force-bar-array-flabels\").merge(de).attr(\"x\",(function(e){return e[0]})).attr(\"y\",(function(e){return e[1]+4})).text((function(e){return e[2]})),de.exit().remove()}},{key:\"componentWillUnmount\",value:function(){window.removeEventListener(\"resize\",this.redraw)}},{key:\"render\",value:function(){var t=this;return e.createElement(\"div\",{ref:function(e){return t.wrapper=Jt(e)},style:{textAlign:\"center\"}},e.createElement(\"style\",{dangerouslySetInnerHTML:{__html:\"\\n          .force-bar-array-wrapper {\\n            text-align: center;\\n          }\\n          .force-bar-array-xaxis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-array-xaxis .domain {\\n            opacity: 0;\\n          }\\n          .force-bar-array-xaxis paths {\\n            display: none;\\n          }\\n          .force-bar-array-yaxis path {\\n            fill: none;\\n            opacity: 0.4;\\n          }\\n          .force-bar-array-yaxis paths {\\n            display: none;\\n          }\\n          .tick line {\\n            stroke: #000;\\n            stroke-width: 1px;\\n            opacity: 0.4;\\n          }\\n          .tick text {\\n            fill: #000;\\n            opacity: 0.5;\\n            font-size: 12px;\\n            padding: 0px;\\n          }\\n          .force-bar-array-flabels {\\n            font-size: 12px;\\n            fill: #fff;\\n            text-anchor: middle;\\n          }\\n          .additive-force-array-xlabel {\\n            background: none;\\n            border: 1px solid #ccc;\\n            opacity: 0.5;\\n            margin-bottom: 0px;\\n            font-size: 12px;\\n            font-family: arial;\\n            margin-left: 80px;\\n            max-width: 300px;\\n          }\\n          .additive-force-array-xlabel:focus {\\n            outline: none;\\n          }\\n          .additive-force-array-ylabel {\\n            position: relative;\\n            top: 0px;\\n            left: 0px;\\n            transform: rotate(-90deg);\\n            background: none;\\n            border: 1px solid #ccc;\\n            opacity: 0.5;\\n            margin-bottom: 0px;\\n            font-size: 12px;\\n            font-family: arial;\\n            max-width: 150px;\\n          }\\n          .additive-force-array-ylabel:focus {\\n            outline: none;\\n          }\\n          .additive-force-array-hoverLine {\\n            stroke-width: 1px;\\n            stroke: #fff;\\n            opacity: 1;\\n          }\"}}),e.createElement(\"select\",{className:\"additive-force-array-xlabel\"}),e.createElement(\"div\",{style:{height:\"0px\",textAlign:\"left\"}},e.createElement(\"select\",{className:\"additive-force-array-ylabel\"})),e.createElement(\"svg\",{ref:function(e){return t.svg=Jt(e)},style:{userSelect:\"none\",display:\"block\",fontFamily:\"arial\",sansSerif:!0}}))}}])&&Xa(n.prototype,r),Object.defineProperty(n,\"prototype\",{writable:!1}),u}(e.Component);ni.defaultProps={plot_cmap:\"RdBu\",ordering_keys:null,ordering_keys_time_format:null};const ri=ni;window.SHAP={SimpleListVisualizer:He,AdditiveForceVisualizer:Ln,AdditiveForceArrayVisualizer:ri,React:e,ReactDom:t}})()})();\n",
              "</script>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "<div id='iEV49GW1UN9427QD9AFZA'>\n",
              "<div style='color: #900; text-align: center;'>\n",
              "  <b>Visualization omitted, Javascript library not loaded!</b><br>\n",
              "  Have you run `initjs()` in this notebook? If this notebook was from another\n",
              "  user you must also trust this notebook (File -> Trust notebook). If you are viewing\n",
              "  this notebook on github the Javascript has been stripped for security. If you are using\n",
              "  JupyterLab this error is because a JupyterLab extension has not yet been written.\n",
              "</div></div>\n",
              " <script>\n",
              "   if (window.SHAP) SHAP.ReactDom.render(\n",
              "    SHAP.React.createElement(SHAP.AdditiveForceVisualizer, {\"outNames\": [\"f(x)\"], \"baseValue\": 0.5164396574068814, \"outValue\": 0.061530523728985376, \"link\": \"identity\", \"featureNames\": [\"PC1\", \"PC2\", \"PC3\", \"PC4\", \"PC5\", \"PC6\", \"PC7\", \"exclamation_flag\", \"Sentiment_VADER\", \"Drama\", \"Comedy\", \"ActionAdventure\", \"Crime\", \"ThrillerHorror\", \"Fantasy\", \"Sci-Fi\", \"Animation\", \"Romance\"], \"features\": {\"0\": {\"effect\": -0.03947188100080736, \"value\": -0.0027680590781728816}, \"1\": {\"effect\": -0.004178884477815596, \"value\": 0.9679527881005678}, \"2\": {\"effect\": -0.006008492670109443, \"value\": -0.1282599052749971}, \"4\": {\"effect\": 0.02148345229377723, \"value\": 1.1699708940818712}, \"5\": {\"effect\": -0.04482911834039942, \"value\": -1.6807885628184003}, \"6\": {\"effect\": -0.11053166560029797, \"value\": 1.5548236116472036}, \"7\": {\"effect\": 0.02802403657706089, \"value\": 1.1264178639834663}, \"8\": {\"effect\": -0.019498867434207125, \"value\": -0.662296280856509}, \"9\": {\"effect\": -0.04996143130016973, \"value\": 1.1085455513584543}, \"10\": {\"effect\": -0.1695951902450662, \"value\": 2.3086792761230397}, \"12\": {\"effect\": 0.008354995782138037, \"value\": -0.43314808182421005}, \"13\": {\"effect\": 0.004909038582047645, \"value\": -0.40711897076111686}, \"14\": {\"effect\": 0.010769248772767187, \"value\": -0.3235924008452868}, \"15\": {\"effect\": -0.014760667328747146, \"value\": -0.33537744652025486}, \"16\": {\"effect\": -0.035597604735653036, \"value\": -0.5132962632751338}, \"17\": {\"effect\": -0.03401610255241394, \"value\": 0.7157282312689407}}, \"plot_cmap\": \"RdBu\", \"labelMargin\": 20}),\n",
              "    document.getElementById('iEV49GW1UN9427QD9AFZA')\n",
              "  );\n",
              "</script>"
            ],
            "text/plain": [
              "<shap.plots._force.AdditiveForceVisualizer at 0x7bf482a45570>"
            ]
          },
          "execution_count": 639,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Look at the decision plot of the FN quote that had a probability closest to 20%\n",
        "shap.initjs()\n",
        "\n",
        "max_prob_index = FN_indices[np.argmax(y_test_pred_prob[FN_indices, 1])] # get index of FN that had highest problematic prob\n",
        "\n",
        "shap_value = explainer.shap_values(test_SHAP[max_prob_index])[ :, 1]\n",
        "\n",
        "shap.force_plot(explainer.expected_value[1], features = test_SHAP[max_prob_index, ], shap_values = shap_value, feature_names = feature_names)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gcE1qDzm3ZyQ",
      "metadata": {
        "id": "gcE1qDzm3ZyQ"
      },
      "source": [
        "The FN quote with the highest predicted probability was: \"Better dead and cool then alive and uncool.\". We see that its value for Comedy and PC7 contributed most to the prediction."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WhZEdqyg_gXZ",
      "metadata": {
        "id": "WhZEdqyg_gXZ"
      },
      "source": [
        "####**Overall summary**  \n",
        "\n",
        "We evaluated our final neural network model for identifying problematic quotes in movie scripts.  Here are the key findings:\n",
        "\n",
        "- **Moderate Recall:** The model achieved a recall of 50%. This is likely too low for our context where the cost of false negatives is high.\n",
        "- **Similar performance compared to simpler models:** A rule-based classifier based on profanity search achieved similar results but with significantly higher precision (100%). Our model did outperform a GBM however, indicating that the true decision boundary is complex.\n",
        "- **Model struggled with nuance**: The model may be less useful at identifying quotes that are problematic due to nuanced themes such as politics and religion.\n",
        "\n",
        "**Recommendations:**\n",
        "\n",
        "Consider using a rule-based approach together with a neural network to better identify problematic quotes, especially for those containing profanity.\n",
        "Explore deeper neural networks or other complex models if capturing more nuanced categories is crucial (e.g. religion, sex, politics).\n",
        "Improve data quality and quantity for better modelling results.\n",
        "\n",
        "Given the poor recall of our model on the test set it is *not* recommended that the current neural network be deployed without further exploration of alternative models or data improvements. Discussion of this is left to Q4."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "toc-autonumbering": false,
    "toc-showmarkdowntxt": false,
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "094c2072533c40fbb34946335b6ce067": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0aa437253e9d4b72b329daa1025ac32c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13e0bd3c56ee4909a2a827f965f3d1a8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_094c2072533c40fbb34946335b6ce067",
            "placeholder": "​",
            "style": "IPY_MODEL_2ce2a19413674a5dbb6ac4b9f49306b2",
            "value": " 20/20 [00:22&lt;00:00,  2.12it/s]"
          }
        },
        "172c86288ee64c77b750b26c0abe3f15": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1d4cd5b95d4245d2a5ecd54bcbec526c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f8e6684340334b3f83274abc2bde73fc",
              "IPY_MODEL_9be42af1ec70421c96a6d07f94fe4309",
              "IPY_MODEL_57d619e4da874bb4b351e26e53021625"
            ],
            "layout": "IPY_MODEL_172c86288ee64c77b750b26c0abe3f15"
          }
        },
        "218cfa52b5e247f38a1a4512f70b6d23": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "29d708f2da85420b9324b4fb84a2216a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2ce2a19413674a5dbb6ac4b9f49306b2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2e12e9e8989245d49fdfd876ab26902d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_afa5e07d340e481a862dd688661eb528",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_29d708f2da85420b9324b4fb84a2216a",
            "value": 3
          }
        },
        "332321a2620e440fbd1b22a99f812397": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "371bc66b36204a50af02f67ef34d4601": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3886cd94444c4779abd58d6ac64c27af": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "499f1ae29f25402fb098629dde767fd7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "536283bf66984e71a756ce6085f9f416": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "54b7dcc8fe5a4ad2b122ff88b2b08e38": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "57d619e4da874bb4b351e26e53021625": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1d813e6f89548769657995e998d4220",
            "placeholder": "​",
            "style": "IPY_MODEL_332321a2620e440fbd1b22a99f812397",
            "value": " 80/80 [01:21&lt;00:00,  1.09it/s]"
          }
        },
        "5cd4d7e96b5c46e7b5ff4c65e1ef5b6c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a4400eb9ff1e4366a9b99f6d60149bbe",
            "placeholder": "​",
            "style": "IPY_MODEL_9b68ac5354ab4087adba39f584d85a50",
            "value": "Batches: 100%"
          }
        },
        "69a687b6617447df9ba34917429ac22d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e0073a2535c041568c4b312bba3cbb5e",
              "IPY_MODEL_f8f5ba9019ed41b09c30ab64fbece01f",
              "IPY_MODEL_e3b0d0bb057549719d12c1cf6c5c8264"
            ],
            "layout": "IPY_MODEL_9d09cb0d16084877b5ebf4f585a9e3cd"
          }
        },
        "766cdbc2a9bc4143a8f210e939c4fb7f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8767387f3f99442b895f0b4c13085cd9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "88a991949e6f4f58b00dca1afc9c2452": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "90397a505a1d4853b96742e74e60c6bc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9b68ac5354ab4087adba39f584d85a50": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9be42af1ec70421c96a6d07f94fe4309": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_371bc66b36204a50af02f67ef34d4601",
            "max": 80,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8767387f3f99442b895f0b4c13085cd9",
            "value": 80
          }
        },
        "9d09cb0d16084877b5ebf4f585a9e3cd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9fe2b848f3ad47368718b5accc11e128": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e82a4acb617f4b6ea90cb82c4fa3a0e9",
            "placeholder": "​",
            "style": "IPY_MODEL_aeee4a5796154124a3b517dc7a99017f",
            "value": " 3/3 [00:03&lt;00:00,  1.21it/s]"
          }
        },
        "a177f37cef344140be45f6d0657e8fe4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a4400eb9ff1e4366a9b99f6d60149bbe": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae21a1422fe447f18fececf73a745673": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a177f37cef344140be45f6d0657e8fe4",
            "placeholder": "​",
            "style": "IPY_MODEL_cd3a0ef9887a4aaf8a23e38b7e9eb058",
            "value": "Batches: 100%"
          }
        },
        "aeee4a5796154124a3b517dc7a99017f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "afa5e07d340e481a862dd688661eb528": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c1d813e6f89548769657995e998d4220": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c5dacd0307b04c77908d0171fbae6413": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ae21a1422fe447f18fececf73a745673",
              "IPY_MODEL_2e12e9e8989245d49fdfd876ab26902d",
              "IPY_MODEL_9fe2b848f3ad47368718b5accc11e128"
            ],
            "layout": "IPY_MODEL_cc838166f5af415fa3bbcee7591f7a41"
          }
        },
        "c80ec3a9b8a24707951a77ba917fed01": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5cd4d7e96b5c46e7b5ff4c65e1ef5b6c",
              "IPY_MODEL_e8b415f8e1ed4d34852942164fe77e4c",
              "IPY_MODEL_13e0bd3c56ee4909a2a827f965f3d1a8"
            ],
            "layout": "IPY_MODEL_54b7dcc8fe5a4ad2b122ff88b2b08e38"
          }
        },
        "cc838166f5af415fa3bbcee7591f7a41": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd3a0ef9887a4aaf8a23e38b7e9eb058": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d5123101af044321b6cc92793aeecc94": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dff23367b0e44e09b9555ed731cb5c5b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e0073a2535c041568c4b312bba3cbb5e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_766cdbc2a9bc4143a8f210e939c4fb7f",
            "placeholder": "​",
            "style": "IPY_MODEL_dff23367b0e44e09b9555ed731cb5c5b",
            "value": "Batches: 100%"
          }
        },
        "e3b0d0bb057549719d12c1cf6c5c8264": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0aa437253e9d4b72b329daa1025ac32c",
            "placeholder": "​",
            "style": "IPY_MODEL_218cfa52b5e247f38a1a4512f70b6d23",
            "value": " 3/3 [00:03&lt;00:00,  1.10it/s]"
          }
        },
        "e82a4acb617f4b6ea90cb82c4fa3a0e9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8b415f8e1ed4d34852942164fe77e4c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_90397a505a1d4853b96742e74e60c6bc",
            "max": 20,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_499f1ae29f25402fb098629dde767fd7",
            "value": 20
          }
        },
        "f8e6684340334b3f83274abc2bde73fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d5123101af044321b6cc92793aeecc94",
            "placeholder": "​",
            "style": "IPY_MODEL_3886cd94444c4779abd58d6ac64c27af",
            "value": "100%"
          }
        },
        "f8f5ba9019ed41b09c30ab64fbece01f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_536283bf66984e71a756ce6085f9f416",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_88a991949e6f4f58b00dca1afc9c2452",
            "value": 3
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
