{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "VG9PY05cFJ_R",
      "metadata": {
        "id": "VG9PY05cFJ_R"
      },
      "source": [
        "# DSA 2024 S2 Assignment\n",
        "\n",
        "Member number: 16764\n",
        "\n",
        "## Purpose:\n",
        "The purpose of this assignment is to help the Actuaries Institute improve member engagement with their CPD Hub. We have observed a continuous decline in member engagement from monthly download since 2022, and our focus is to address this on the follwoing two main area:\n",
        "\n",
        "1. Proposing a new grouping system: The current tagging system is confusing and overlapping, making it difficult for members to find relevant content. We aim to replace this system with a more effective and streamlined clustering model.\n",
        "2. Enhancing engagement through asset visibility: We aim to identify high-popularity assets and display them more prominently on the CPD Hub's front page. This will involve creating a classification model to predict the popularity level of assets (high, medium, low).\n",
        "Both the clustering and classification models will serve as decision-supporting tools to improve user experience and engagement on the CPD Hub.\n",
        "\n",
        "Outline:\n",
        "- Q1: Focuses on exploratory data analysis (EDA) to understand ongoing patterns and prepare data for modeling. We will also clean the data if necessary.\n",
        "- Q2: Involves constructing the clustering model to replace the current tagging system.\n",
        "- Q3: Develops the predictive model to determine the popularity of assets and enhance visibility for high-value content.\n",
        "\n",
        "Additionally, we will conduct a separate discussion (Q4) on how AI tools can be used to generate more engaging asset descriptions. This section will be included in a separate PDF.\n",
        "\n",
        "Finally, we will provide a 5-minute presentation to share the findings of Q2, Q3, and Q4.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XtZXmepZWbBz",
      "metadata": {
        "id": "XtZXmepZWbBz"
      },
      "source": [
        "## Packages\n",
        "This section imports the packages that are used in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "KQVVeMMzHYK2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KQVVeMMzHYK2",
        "outputId": "07823002-a94a-43f1-8de7-38c2051dc868"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting fuzzywuzzy\n",
            "  Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl.metadata (4.9 kB)\n",
            "Downloading fuzzywuzzy-0.18.0-py2.py3-none-any.whl (18 kB)\n",
            "Installing collected packages: fuzzywuzzy\n",
            "Successfully installed fuzzywuzzy-0.18.0\n",
            "Collecting python-Levenshtein\n",
            "  Downloading python_Levenshtein-0.26.0-py3-none-any.whl.metadata (3.7 kB)\n",
            "Collecting Levenshtein==0.26.0 (from python-Levenshtein)\n",
            "  Downloading levenshtein-0.26.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)\n",
            "Collecting rapidfuzz<4.0.0,>=3.9.0 (from Levenshtein==0.26.0->python-Levenshtein)\n",
            "  Downloading rapidfuzz-3.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\n",
            "Downloading python_Levenshtein-0.26.0-py3-none-any.whl (9.4 kB)\n",
            "Downloading levenshtein-0.26.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (162 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m162.6/162.6 kB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading rapidfuzz-3.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m49.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: rapidfuzz, Levenshtein, python-Levenshtein\n",
            "Successfully installed Levenshtein-0.26.0 python-Levenshtein-0.26.0 rapidfuzz-3.10.0\n"
          ]
        }
      ],
      "source": [
        "# Package 1\n",
        "import pandas as pd\n",
        "\n",
        "# Package 2/3/4 etc.\n",
        "from google.colab import drive\n",
        "\n",
        "# for doing charts#\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "# for the text in file name\n",
        "!pip install fuzzywuzzy\n",
        "!pip install python-Levenshtein\n",
        "import pandas as pd\n",
        "from fuzzywuzzy import fuzz, process\n",
        "\n",
        "# Function 12\n",
        "import seaborn as sns\n",
        "\n",
        "# for data clenaing\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "# Used to build the neural network and evaluate it.\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense\n",
        "\n",
        "\n",
        "# for data split\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "# Question 3 NN\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense, Dropout\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xoME0nrg1r9X",
      "metadata": {
        "id": "xoME0nrg1r9X"
      },
      "source": [
        "## Functions\n",
        "This section sets out user defined functions that are used in this notebook.\n",
        "\n",
        "If you prefer, you may instead choose to define these under the relevant assignment question section below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bdb58870-1f66-4302-b7d5-1576d58e3dbf",
      "metadata": {
        "id": "bdb58870-1f66-4302-b7d5-1576d58e3dbf"
      },
      "outputs": [],
      "source": [
        "# Function 1\n",
        "# from Tutorial 2 Helper functions\n",
        "\n",
        "def eda_column_dtypes(df, column_name):\n",
        "    \"\"\"\n",
        "    Counts the different data types within a specified column of a pandas DataFrame\n",
        "    and displays 3 sample values for each data type.\n",
        "\n",
        "    Args:\n",
        "        df: The pandas DataFrame.\n",
        "        column_name: The name of the column to analyze.\n",
        "\n",
        "    Returns:\n",
        "        None. Prints the dtype counts and sample values.\n",
        "    \"\"\"\n",
        "    if column_name not in df.columns:\n",
        "        raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "    # Determine the dtype of each row in the specified column\n",
        "    dtype_series = df[column_name].apply(type)\n",
        "\n",
        "    # Count the occurrences of each dtype\n",
        "    dtype_counts = dtype_series.value_counts()\n",
        "\n",
        "    # Print the data types, their counts, and sample values\n",
        "    for dtype, count in dtype_counts.items():\n",
        "        print(f\"Data type: {dtype}, Count: {count}\")\n",
        "        sample_values = df[column_name][dtype_series == dtype].sample(min(count, 3)).tolist()\n",
        "        print(f\"Sample values: {sample_values}\\n\")\n",
        "\n",
        "\n",
        "#--------------------------------------#\n",
        "# Function 2\n",
        "# prompt: write a function that takes a pandas dataset and a column name as parameters then prints\n",
        "# * the count of unique values\n",
        "# * the count of missing values\n",
        "# * the count of invalid and error values e.g. nan\n",
        "# * a list of the 5 most common values\n",
        "\n",
        "def eda_describe_column(df, column_name):\n",
        "  \"\"\"\n",
        "  Prints descriptive statistics for a specified column in a pandas DataFrame.\n",
        "\n",
        "  Args:\n",
        "      df: The pandas DataFrame.\n",
        "      column_name: The name of the column to analyze.\n",
        "\n",
        "  Returns:\n",
        "      None. Prints the descriptive statistics.\n",
        "  \"\"\"\n",
        "  if column_name not in df.columns:\n",
        "    raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "  # Count unique values\n",
        "  unique_values = df[column_name].nunique()\n",
        "  print(f\"Unique Values: {unique_values}\")\n",
        "\n",
        "  # Count missing values\n",
        "  missing_values = df[column_name].isnull().sum()\n",
        "  print(f\"Missing Values: {missing_values}\")\n",
        "\n",
        "  # Count invalid and error values\n",
        "  invalid_values = df[column_name].isin([np.nan, np.inf, -np.inf]).sum()\n",
        "  print(f\"Invalid and Error Values: {invalid_values}\")\n",
        "\n",
        "  # List the 5 most common values\n",
        "  most_common_values = df[column_name].value_counts().head(5)\n",
        "  print(f\"Most Common Values: {most_common_values}\")\n",
        "\n",
        "    # Find the number of rows with values that appear more than once\n",
        "  duplicate_rows = df[df.duplicated(column_name, keep=False)]\n",
        "  num_duplicate_rows = duplicate_rows.shape[0]\n",
        "  print(f\"Number of rows with {column_name} values that appear more than once: {num_duplicate_rows}\")\n",
        "\n",
        "\n",
        "\n",
        "#-----------------------------------------------#\n",
        "#Function 3 - Text variable\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "def analyze_words(df, column_name):\n",
        "    \"\"\"\n",
        "    Analyzes the words in a specified column of a pandas DataFrame.\n",
        "\n",
        "    Args:\n",
        "        df: The pandas DataFrame.\n",
        "        column_name: The name of the column to analyze.\n",
        "\n",
        "    Returns:\n",
        "        None. Prints the number of empty strings, most common words, least common words, and displays a wordcloud.\n",
        "    \"\"\"\n",
        "    if column_name not in df.columns:\n",
        "        raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "    # Count empty strings\n",
        "    empty_strings = df[column_name].isnull().sum()\n",
        "    print(f\"Number of empty strings: {empty_strings}\")\n",
        "\n",
        "    # Parse words and count occurrences\n",
        "    words = []\n",
        "    word_counts_per_row = []\n",
        "    for text in df[column_name]:\n",
        "        if isinstance(text, str):\n",
        "            word_list = text.split()\n",
        "            words.extend(word_list)\n",
        "            word_counts_per_row.append(len(word_list))\n",
        "        else:\n",
        "            word_counts_per_row.append(0)\n",
        "\n",
        "    word_counts = pd.Series(words).value_counts()\n",
        "\n",
        "    # Most common words\n",
        "    most_common = word_counts.head(5)\n",
        "    print(\"Most common words:\")\n",
        "    for word, count in most_common.items():\n",
        "        print(f\"\\t{word}: {count}\")\n",
        "\n",
        "    # Least common words\n",
        "    least_common = word_counts.tail(5)\n",
        "    print(\"Least common words:\")\n",
        "    for word, count in least_common.items():\n",
        "        print(f\"\\t{word}: {count}\")\n",
        "\n",
        "    # Wordcloud\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color=\"white\").generate(\" \".join(words))\n",
        "    plt.figure(figsize=(10, 5))\n",
        "    plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
        "    plt.axis(\"off\")\n",
        "    plt.show()\n",
        "\n",
        "    # Histogram of number of words per row\n",
        "    plt.figure(figsize=(10, 5))\n",
        "    plt.hist(word_counts_per_row, bins=30, edgecolor='black')\n",
        "    plt.xlabel(\"Number of Words per Row\")\n",
        "    plt.ylabel(\"Frequency\")\n",
        "    plt.title(\"Histogram of Number of Words per Row\")\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "#Function 4 - numerical\n",
        "# prompt: This function tells me whether a pandas column only contains numeric values\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "def is_numeric_column(df, column_name):\n",
        "    \"\"\"\n",
        "    Checks if a specified column in a pandas DataFrame contains only numeric values.\n",
        "\n",
        "    Args:\n",
        "        df: The pandas DataFrame.\n",
        "        column_name: The name of the column to analyze.\n",
        "\n",
        "    Returns:\n",
        "        True if the column contains only numeric values, False otherwise.\n",
        "    \"\"\"\n",
        "    if column_name not in df.columns:\n",
        "        raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "    # Check if the column data type is numeric\n",
        "    is_numeric = pd.api.types.is_numeric_dtype(df[column_name])\n",
        "\n",
        "    # Check for missing values\n",
        "    if is_numeric:\n",
        "        missing_values = df[column_name].isnull().sum()\n",
        "        if missing_values > 0:\n",
        "            print(f\"Warning: Column contains {missing_values} missing values.\")\n",
        "\n",
        "    return is_numeric\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "#function 5 - String\n",
        "#  function that tells me whether a pandas column only contains string values\n",
        "# This function was based on helper function in Tutorial 2 with improved with Chat-gpt\n",
        "\n",
        "# prompt: write a function that tells me whether a pandas column only contains string values\n",
        "\n",
        "def is_string_column(df, column_name):\n",
        "  \"\"\"\n",
        "  Checks if a specified column in a pandas DataFrame contains only string values.\n",
        "\n",
        "  Args:\n",
        "      df: The pandas DataFrame.\n",
        "      column_name: The name of the column to analyze.\n",
        "\n",
        "  Returns:\n",
        "      True if the column contains only string values, False otherwise.\n",
        "  \"\"\"\n",
        "  if column_name not in df.columns:\n",
        "    raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "  # Check if the column data type is string\n",
        "  if pd.api.types.is_string_dtype(df[column_name]):\n",
        "    return True\n",
        "  else:\n",
        "    return False\n",
        "\n",
        "\n",
        "#function 6\n",
        "#  function check whether a pandas column only contains date values\n",
        "# prompt: write a function that tells me whether a pandas column only contains date values\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "def is_date_column(df, column_name):\n",
        "  \"\"\"\n",
        "  Checks if a specified column in a pandas DataFrame contains only date values.\n",
        "\n",
        "  Args:\n",
        "      df: The pandas DataFrame.\n",
        "      column_name: The name of the column to analyze.\n",
        "\n",
        "  Returns:\n",
        "      True if the column contains only date values, False otherwise.\n",
        "  \"\"\"\n",
        "  if column_name not in df.columns:\n",
        "    raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "  # Check if the column data type is datetime\n",
        "  if pd.api.types.is_datetime64_any_dtype(df[column_name]):\n",
        "    return True\n",
        "  else:\n",
        "    return False\n",
        "\n",
        "\n",
        "#-------------------------------#\n",
        "# function 7\n",
        "\n",
        "# EDA displays of a variable including :\n",
        "# * mean\n",
        "# * standard deviation\n",
        "# * skewness\n",
        "# * maximum number of decimal places\n",
        "# * histogram of values\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "def eda_numeric_column(df, column_name):\n",
        "  \"\"\"\n",
        "  Prints descriptive statistics for a specified column in a pandas DataFrame.\n",
        "\n",
        "  Args:\n",
        "      df: The pandas DataFrame.\n",
        "      column_name: The name of the column to analyze.\n",
        "\n",
        "  Returns:\n",
        "      None. Prints the descriptive statistics.\n",
        "  \"\"\"\n",
        "  if column_name not in df.columns:\n",
        "    raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "  # Calculate statistics\n",
        "  mean = df[column_name].mean()\n",
        "  std_dev = df[column_name].std()\n",
        "  skewness = df[column_name].skew()\n",
        "  min_value = df[column_name].min()\n",
        "  max_value = df[column_name].max()\n",
        "  decimal_places = df[column_name].apply(lambda x: len(str(x).split('.')[1]) if '.' in str(x) else 0).max()\n",
        "\n",
        "  # Print statistics\n",
        "  print(f\"Mean: {mean}\")\n",
        "  print(f\"Standard Deviation: {std_dev}\")\n",
        "  print(f\"Skewness: {skewness}\")\n",
        "  print(f\"Minimum: {min_value}\")\n",
        "  print(f\"Maximum: {max_value}\")\n",
        "  print(f\"Maximum Number of Decimal Places: {decimal_places}\")\n",
        "\n",
        "  # Plot histogram\n",
        "  plt.figure(figsize=(8, 4))\n",
        "  sns.histplot(data=df, x=column_name, bins=10)\n",
        "  plt.title(f'{column_name} Histogram')\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "#function 8  - EDA - display for 2 numerical variables\n",
        "# This function display the following:\n",
        "# * a scatterplot of the values from the two numeric columns\n",
        "# * a red trendline\n",
        "# * the correlation coefficient\n",
        "\n",
        "def plot_correlation(df, col1, col2):\n",
        "  \"\"\"\n",
        "  Plots a scatterplot of two columns with a trend line and displays the correlation coefficient.\n",
        "\n",
        "  Args:\n",
        "      df: The pandas DataFrame.\n",
        "      col1: The name of the first column.\n",
        "      col2: The name of the second column.\n",
        "\n",
        "  Returns:\n",
        "      None. Displays the scatterplot and prints the correlation coefficient.\n",
        "  \"\"\"\n",
        "  if col1 not in df.columns or col2 not in df.columns:\n",
        "    raise ValueError(\"One or both columns not found in DataFrame.\")\n",
        "\n",
        "  # Check if both columns are numeric\n",
        "  if not (pd.api.types.is_numeric_dtype(df[col1]) and pd.api.types.is_numeric_dtype(df[col2])):\n",
        "    raise TypeError(\"Both columns must be numeric.\")\n",
        "\n",
        "  # Plot scatterplot with trend line\n",
        "  plt.figure(figsize=(8, 4))\n",
        "  sns.regplot(data=df, x=col1, y=col2, scatter_kws={'alpha':0.5}, line_kws={'color': 'red'})\n",
        "\n",
        "  # Calculate and display correlation coefficient\n",
        "  correlation = df[col1].corr(df[col2])\n",
        "  plt.text(0.1, 0.9, f\"Correlation: {correlation:.2f}\", transform=plt.gca().transAxes)\n",
        "\n",
        "  # Display plot\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "\n",
        "# function 9 -- Cleaning function\n",
        "#Function clean_to_integer is designed to convert a given value to an integer if possible, #\n",
        "#and return NaN if the conversion fails.#\n",
        "\n",
        "def clean_to_integer(value):\n",
        "    if isinstance(value, int):\n",
        "        return value  # Keep original value if it's already an integer\n",
        "    try:\n",
        "        return int(value)  # Try converting to integer\n",
        "    except:\n",
        "        return np.nan  # Return NaN if conversion fails\n",
        "\n",
        "\n",
        "\n",
        "#function 10\n",
        "\n",
        "# prompt: write a function that accepts a pandas table and two column names as parameters, then plots a line for the average values from the first column (vertical axis) grouped by each value of the second column (horizontal axis), label the axes with the column names\n",
        "\n",
        "def plot_average_by_group(df, col1, col2):\n",
        "    \"\"\"\n",
        "    Plots a line for the average values from the first column (vertical axis) grouped by each value of the second column (horizontal axis).\n",
        "\n",
        "    Args:\n",
        "        df: The pandas DataFrame.\n",
        "        col1: The name of the column to plot values from.\n",
        "        col2: The name of the column to group by.\n",
        "\n",
        "    Returns:\n",
        "        None. Displays the line plot.\n",
        "    \"\"\"\n",
        "    if col1 not in df.columns or col2 not in df.columns:\n",
        "        raise ValueError(\"One or both columns not found in DataFrame.\")\n",
        "\n",
        "    # Group by the second column and calculate the mean of the first column\n",
        "    grouped_data = df.groupby(col2)[col1].mean().reset_index()\n",
        "\n",
        "    # Plot the line chart\n",
        "    plt.figure(figsize=(8, 4))\n",
        "    sns.lineplot(data=grouped_data, x=col2, y=col1)\n",
        "\n",
        "    # Label the axes\n",
        "    plt.xlabel(col2)\n",
        "    plt.ylabel(col1)\n",
        "\n",
        "    # Display the plot\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "\n",
        "# function 11 - non numerical EDA\n",
        "\n",
        "# prompt: write a function that accepts a pandas table and a column name as input parameters and displays:\n",
        "# * a list of the 5 least common values in that column, with a count of how often they occur\n",
        "# * a bar chart of the counts of each unique value in the column, order from the highest count to the lowest count\n",
        "# * a pie chart of the proportion of rows for each value (show top 10 then all others together)\n",
        "\n",
        "def eda_categorical_column(df, column_name):\n",
        "    \"\"\"\n",
        "    Displays information about the least common values in a column of a pandas DataFrame.\n",
        "\n",
        "    Args:\n",
        "        df: The pandas DataFrame.\n",
        "        column_name: The name of the column to analyze.\n",
        "\n",
        "    Returns:\n",
        "        None. Prints the least common values, and displays the bar and pie charts.\n",
        "    \"\"\"\n",
        "    if column_name not in df.columns:\n",
        "        raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "    # Get the value counts\n",
        "    value_counts = df[column_name].value_counts()\n",
        "\n",
        "    # Print the 5 least common values with counts\n",
        "    least_common = value_counts.tail(5)\n",
        "    print(\"Least Common Values:\")\n",
        "    for value, count in least_common.items():\n",
        "        print(f\"\\t{value}: {count}\")\n",
        "\n",
        "    # Plot bar chart of value counts\n",
        "    plt.figure(figsize=(8, 4))\n",
        "    value_counts.plot(kind='bar')\n",
        "    plt.xlabel(column_name)\n",
        "    plt.ylabel(\"Count\")\n",
        "    plt.title(\"Value Counts - \" + column_name)\n",
        "\n",
        "    # Hide x-axis labels if there are more than 20 unique values\n",
        "    if len(value_counts) > 20:\n",
        "        plt.xticks([])\n",
        "\n",
        "    plt.show()\n",
        "\n",
        "    # Plot pie chart of top 10 values and others\n",
        "    others = value_counts[10:].sum()\n",
        "    top_10 = value_counts.head(10)\n",
        "    top_10['Others'] = others\n",
        "    plt.figure(figsize=(8, 4))\n",
        "    plt.pie(top_10, autopct=\"%1.1f%%\", labels=top_10.index)\n",
        "    plt.title(\"Proportion of Values - \" + column_name)\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "# Function 12\n",
        "# This function is generated using ChatGPT4 with the comment to generate me a bar chart, table to illustrate the Date data\n",
        "# a specific clause is added to pick up any assets before Aug2018 (the starting date of DMS)\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "def examine_start_publishing(df, column_name='StartPublishing'):\n",
        "    if column_name not in df.columns:\n",
        "        raise ValueError(f\"Column '{column_name}' not found in DataFrame.\")\n",
        "\n",
        "    if not pd.api.types.is_datetime64_any_dtype(df[column_name]):\n",
        "        try:\n",
        "            df[column_name] = pd.to_datetime(df[column_name])\n",
        "        except Exception as e:\n",
        "            raise TypeError(f\"Column '{column_name}' could not be converted to datetime. Error: {e}\")\n",
        "\n",
        "    df['YearMonth'] = df[column_name].dt.to_period('M')\n",
        "\n",
        "    month_counts = df['YearMonth'].value_counts().sort_index()\n",
        "    month_counts_df = month_counts.reset_index()\n",
        "    month_counts_df.columns = ['Month-Year', 'Number of Assets']\n",
        "\n",
        "\n",
        "    plt.figure(figsize=(16, 8))\n",
        "    sns.barplot(x=month_counts_df['Month-Year'].astype(str), y=month_counts_df['Number of Assets'], palette='Blues_d')\n",
        "    plt.title(f'Number of Assets Published Per Month-Year')\n",
        "    plt.xlabel('Month-Year')\n",
        "    plt.ylabel('Number of Assets')\n",
        "    plt.xticks(rotation=90, ha='center', fontsize=8)\n",
        "    plt.grid(axis='y')\n",
        "    plt.tight_layout()\n",
        "    plt.show()\n",
        "\n",
        "    return month_counts_df\n",
        "\n",
        "\n",
        "\n",
        "# Define a function to print and plot a confusion matrix.\n",
        "def plot_confusion_matrix(cm, classes,\n",
        "                          normalise=False,\n",
        "                          title='Confusion matrix',\n",
        "                          cmap=plt.cm.Blues):\n",
        "    '''\n",
        "    This function prints and plots a confusion matrix.\n",
        "    Normalisation of the matrix can be applied by setting `normalise=True`.\n",
        "    Normalsiation ensures that the sum of each row in the confusion matrix is 1.\n",
        "    '''\n",
        "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    plt.title(title)\n",
        "    plt.colorbar()\n",
        "    tick_marks = np.arange(len(classes))\n",
        "    plt.xticks(tick_marks, classes, rotation=45)\n",
        "    plt.yticks(tick_marks, classes)\n",
        "\n",
        "    if normalise:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "\n",
        "    thresh = cm.max() / 2.\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i, cm[i, j],\n",
        "                 horizontalalignment='center',\n",
        "                 color='white' if cm[i, j] > thresh else 'black')\n",
        "\n",
        "    plt.tight_layout()\n",
        "    plt.ylabel('True response')\n",
        "    plt.xlabel('Predicted response')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1LOeKQASWwat",
      "metadata": {
        "id": "1LOeKQASWwat"
      },
      "source": [
        "## Import data\n",
        "This section imports the data that is used in this notebook.\n",
        "\n",
        "*You should include a reference to the dataset you have used here.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "PIuA9lJII86e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PIuA9lJII86e",
        "outputId": "cc62dde3-58ca-4fb0-82f9-e7a493fc2531"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/gdrive/\n"
          ]
        }
      ],
      "source": [
        "# Code for importing data\n",
        "# Mount google drive to import the required data\n",
        "from google.colab import drive\n",
        "# Mount google drive to import the required data\n",
        "drive.mount('/content/gdrive/', force_remount=True)\n",
        "\n",
        "# Read in dataset\n",
        "infolder = '/content/gdrive/My Drive/DSA datasets/'\n",
        "file_name = 'DSA 2024 S2 assignment dataset.xlsx'\n",
        "\n",
        "docs_data = pd.read_excel(infolder + file_name, sheet_name='Docs')\n",
        "downloads_data = pd.read_excel(infolder + file_name, sheet_name='Downloads')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5jbUFlfRUCmH",
      "metadata": {
        "id": "5jbUFlfRUCmH"
      },
      "source": [
        "## Q1 - Explore and Prepare the data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7640511f-7f89-41bf-99b8-fd27f1ff4048",
      "metadata": {
        "id": "7640511f-7f89-41bf-99b8-fd27f1ff4048"
      },
      "source": [
        "### Q1a - Examine and the dataset\n",
        "#### Purpose:\n",
        "The purpose of this section it to gain an understanding of the dataset and its features. This will involve summarising the key chararacteristics of the dataset, assessing data and concept drift (including download and publishing patterns), and addressing some initial data cleaning needs. Additionally, the dataset will be split into training, validation and test sets for later model purpose.\n",
        "\n",
        "#### We have done the following Cleaning:\n",
        "1. Duplication Removal - Applied to the entire doc_data dataset and DMSId in the download data.\n",
        "2. Inconsistent Data Correction - Applied to StartPublishing date to ensure consistency with download occurrences.\n",
        "3. Date Format Standardization - Applied to StopPublishing .\n",
        "4. Addressing Missing Values for 4 assets - Applied to columns including Title, FileType, StartPublishing Date, Format, Level, Region, and Copyright.\n",
        "5. Outlier Removal - Applied to an asset with anormal FileSize.\n",
        "6. Outlier Removal - Applied to an asset with unusually high Cpd value.\n",
        "7. File Type Correction - Applied to an asset with potentailly incorrect FileType label.\n",
        "8. Handling Missing Values in EventAgeGroup - Applied to the EventAgeGroup column.\n",
        "8. Handling Missing Values in EventType - Applied to the EventType column.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ze6Gv4kr2yom",
      "metadata": {
        "id": "ze6Gv4kr2yom"
      },
      "source": [
        "#### Q1a Step 1 Examine entire dataset\n",
        "**Outline**\n",
        "\n",
        "Firstly, we examine the dataset as a whole to understand the following:\n",
        "\n",
        "*   What variables we have and their data type\n",
        "*   Summary statistics\n",
        "*   Missing values\n",
        "\n",
        "This is to get an initial idea of the data and to highlight any potential issues that may exist.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7rY-kX09kRdV",
      "metadata": {
        "id": "7rY-kX09kRdV"
      },
      "source": [
        "We are starting with an initial exploration of the dataset using docs_data.info(). This step is essential because it gives us a quick overview of the dataset's structure, including the number of entries, columns, data types, and any missing values. This insight helps identify potential issues and informs us about the cleaning steps we'll need to take next."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ff1278c5-cb17-4542-ab1c-631366898f27",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ff1278c5-cb17-4542-ab1c-631366898f27",
        "outputId": "1cf7ee8f-65aa-4723-8060-63dc714428fb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7570 entries, 0 to 7569\n",
            "Data columns (total 17 columns):\n",
            " #   Column           Non-Null Count  Dtype         \n",
            "---  ------           --------------  -----         \n",
            " 0   DMSId            7570 non-null   int64         \n",
            " 1   FileName         7570 non-null   object        \n",
            " 2   FileType         7570 non-null   object        \n",
            " 3   FileSize         7570 non-null   int64         \n",
            " 4   Title            7566 non-null   object        \n",
            " 5   StartPublishing  7570 non-null   datetime64[ns]\n",
            " 6   StopPublishing   7570 non-null   object        \n",
            " 7   Cpd              7570 non-null   int64         \n",
            " 8   Format           7566 non-null   object        \n",
            " 9   Level            7566 non-null   object        \n",
            " 10  Description      6807 non-null   object        \n",
            " 11  Region           7566 non-null   object        \n",
            " 12  Copyright        7566 non-null   object        \n",
            " 13  EventAgeGroup    4715 non-null   object        \n",
            " 14  EventType        5019 non-null   object        \n",
            " 15  Link             7570 non-null   object        \n",
            " 16  Tags             6987 non-null   object        \n",
            "dtypes: datetime64[ns](1), int64(3), object(13)\n",
            "memory usage: 1005.5+ KB\n"
          ]
        }
      ],
      "source": [
        "# Q1a Step 1\n",
        "# Get initial summary of the dataset\n",
        "docs_data.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8BUzU3OnqKA-",
      "metadata": {
        "id": "8BUzU3OnqKA-"
      },
      "source": [
        "**Findings:**\n",
        "\n",
        "The dataset contains 17 columns with a total of 7,570 entries.\n",
        "\n",
        "Most columns are complete, but a few have missing values:\n",
        "Title, Format, Level, Region, and Copyright have 4 missing entries each.\n",
        "\n",
        "StopPublishing has inconsistent data type suggested by the Data dictionary.\n",
        "\n",
        "The Description column is missing values in over 760 entries.\n",
        "EventAgeGroup and EventType columns have the most missing values, suggesting that a large portion of the data might not be event-related.\n",
        "\n",
        "**So what does it mean?**\n",
        "\n",
        "The presence of these missing values indicates that we’ll need to address them in our cleaning steps to ensure data quality. Our next step will involve handling these gaps to maintain consistency, as missing data could impact any future analyses or modeling tasks."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OSFvF_x8ljps",
      "metadata": {
        "id": "OSFvF_x8ljps"
      },
      "source": [
        "Next, we will use docs_data.head(5) to inspect the first 5 rows of our dataset. This step helps us verify the dataset’s content and structure, ensuring that the data appears as expected and that the columns contain the kind of information we anticipate. It also offers an initial sense of any inconsistencies or potential cleaning needs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0SzR_06wpoSj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0SzR_06wpoSj",
        "outputId": "ea6fda27-b6bb-49cd-8def-667f09b6d2b1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     DMSId                                           FileName FileType  \\\n",
              "0  4194310               international-cpd-opportunities.html     html   \n",
              "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
              "2  4194592                           1997 - March Edition.pdf      pdf   \n",
              "3  4194593                       1997 - September Edition.pdf      pdf   \n",
              "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
              "\n",
              "   FileSize                                              Title  \\\n",
              "0      8892                    International CPD opportunities   \n",
              "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
              "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
              "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
              "4    209993  Disability Committee - The 1997 Report into mo...   \n",
              "\n",
              "  StartPublishing StopPublishing  Cpd       Format         Level  \\\n",
              "0      2019-01-07              0    1      Article  Introductory   \n",
              "1      2018-08-20              0    2  Publication      Advanced   \n",
              "2      2018-08-20              0    2  Publication           All   \n",
              "3      2018-08-20              0    1  Publication           All   \n",
              "4      2018-08-20              0    2  Publication      Advanced   \n",
              "\n",
              "                                         Description  \\\n",
              "0  There are over 30 committees, subcommittees, t...   \n",
              "1  The report is based on data in respect of indi...   \n",
              "2  Inadequacy of LIASB Mortality Reserves Allen T...   \n",
              "3  Analysis of Surplus: Mixing Realism with Conse...   \n",
              "4  The 1997 Report into morbidity for insured and...   \n",
              "\n",
              "                       Region  Copyright EventAgeGroup EventType  \\\n",
              "0               International  Institute           All     Other   \n",
              "1  Domestic and International  Institute           NaN       NaN   \n",
              "2  Domestic and International  Institute           NaN       NaN   \n",
              "3  Domestic and International  Institute           NaN       NaN   \n",
              "4  Domestic and International  Institute           NaN       NaN   \n",
              "\n",
              "                                                Link  \\\n",
              "0  https://www.actuaries.digital/2018/08/21/inter...   \n",
              "1  https://actuaries.logicaldoc.cloud/download-ti...   \n",
              "2  https://actuaries.logicaldoc.cloud/download-ti...   \n",
              "3  https://actuaries.logicaldoc.cloud/download-ti...   \n",
              "4  https://actuaries.logicaldoc.cloud/download-ti...   \n",
              "\n",
              "                                                Tags  \n",
              "0  Banking|Business Acumen|Data Science|General I...  \n",
              "1  Life Insurance|Life Insurance - Practice Updat...  \n",
              "2  Life Insurance|Life Insurance - Valuation|Weal...  \n",
              "3  Life Insurance|Life Insurance - Pricing|Life I...  \n",
              "4  Life Insurance|Life Insurance - Practice Updat...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8d0a0def-4921-49c0-827c-003821d432e7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DMSId</th>\n",
              "      <th>FileName</th>\n",
              "      <th>FileType</th>\n",
              "      <th>FileSize</th>\n",
              "      <th>Title</th>\n",
              "      <th>StartPublishing</th>\n",
              "      <th>StopPublishing</th>\n",
              "      <th>Cpd</th>\n",
              "      <th>Format</th>\n",
              "      <th>Level</th>\n",
              "      <th>Description</th>\n",
              "      <th>Region</th>\n",
              "      <th>Copyright</th>\n",
              "      <th>EventAgeGroup</th>\n",
              "      <th>EventType</th>\n",
              "      <th>Link</th>\n",
              "      <th>Tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4194310</td>\n",
              "      <td>international-cpd-opportunities.html</td>\n",
              "      <td>html</td>\n",
              "      <td>8892</td>\n",
              "      <td>International CPD opportunities</td>\n",
              "      <td>2019-01-07</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>Article</td>\n",
              "      <td>Introductory</td>\n",
              "      <td>There are over 30 committees, subcommittees, t...</td>\n",
              "      <td>International</td>\n",
              "      <td>Institute</td>\n",
              "      <td>All</td>\n",
              "      <td>Other</td>\n",
              "      <td>https://www.actuaries.digital/2018/08/21/inter...</td>\n",
              "      <td>Banking|Business Acumen|Data Science|General I...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4194590</td>\n",
              "      <td>1995 Report of the Disability Committee.pdf</td>\n",
              "      <td>pdf</td>\n",
              "      <td>1674088</td>\n",
              "      <td>Disability Committee - 1995 Report of the Disa...</td>\n",
              "      <td>2018-08-20</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>Publication</td>\n",
              "      <td>Advanced</td>\n",
              "      <td>The report is based on data in respect of indi...</td>\n",
              "      <td>Domestic and International</td>\n",
              "      <td>Institute</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>https://actuaries.logicaldoc.cloud/download-ti...</td>\n",
              "      <td>Life Insurance|Life Insurance - Practice Updat...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4194592</td>\n",
              "      <td>1997 - March Edition.pdf</td>\n",
              "      <td>pdf</td>\n",
              "      <td>300105</td>\n",
              "      <td>Australian Actuarial Journal 1997 (March Edition)</td>\n",
              "      <td>2018-08-20</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>Publication</td>\n",
              "      <td>All</td>\n",
              "      <td>Inadequacy of LIASB Mortality Reserves Allen T...</td>\n",
              "      <td>Domestic and International</td>\n",
              "      <td>Institute</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>https://actuaries.logicaldoc.cloud/download-ti...</td>\n",
              "      <td>Life Insurance|Life Insurance - Valuation|Weal...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4194593</td>\n",
              "      <td>1997 - September Edition.pdf</td>\n",
              "      <td>pdf</td>\n",
              "      <td>166894</td>\n",
              "      <td>Australian Actuarial Journal 1997 (September E...</td>\n",
              "      <td>2018-08-20</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>Publication</td>\n",
              "      <td>All</td>\n",
              "      <td>Analysis of Surplus: Mixing Realism with Conse...</td>\n",
              "      <td>Domestic and International</td>\n",
              "      <td>Institute</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>https://actuaries.logicaldoc.cloud/download-ti...</td>\n",
              "      <td>Life Insurance|Life Insurance - Pricing|Life I...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4194594</td>\n",
              "      <td>1997 Report of the Disability Committee-The Da...</td>\n",
              "      <td>pdf</td>\n",
              "      <td>209993</td>\n",
              "      <td>Disability Committee - The 1997 Report into mo...</td>\n",
              "      <td>2018-08-20</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "      <td>Publication</td>\n",
              "      <td>Advanced</td>\n",
              "      <td>The 1997 Report into morbidity for insured and...</td>\n",
              "      <td>Domestic and International</td>\n",
              "      <td>Institute</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>https://actuaries.logicaldoc.cloud/download-ti...</td>\n",
              "      <td>Life Insurance|Life Insurance - Practice Updat...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8d0a0def-4921-49c0-827c-003821d432e7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8d0a0def-4921-49c0-827c-003821d432e7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8d0a0def-4921-49c0-827c-003821d432e7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-dc9204a7-e551-4832-941b-fe8e253e854b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-dc9204a7-e551-4832-941b-fe8e253e854b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-dc9204a7-e551-4832-941b-fe8e253e854b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "docs_data",
              "summary": "{\n  \"name\": \"docs_data\",\n  \"rows\": 7570,\n  \"fields\": [\n    {\n      \"column\": \"DMSId\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 74278,\n        \"min\": 4194310,\n        \"max\": 4370237,\n        \"num_unique_values\": 7570,\n        \"samples\": [\n          4202113,\n          4361355,\n          4365706\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"FileName\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7485,\n        \"samples\": [\n          \"October2017.pdf\",\n          \"Peter Leonard Rights wrongs and regulation Using the term 'discrimination' with discrimination 271122.pdf\",\n          \"LeasMartinERMPaper.pdf\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"FileType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 12,\n        \"samples\": [\n          \"txt\",\n          \"xlsx\",\n          \"html\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"FileSize\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6819969,\n        \"min\": 553,\n        \"max\": 142884489,\n        \"num_unique_values\": 7210,\n        \"samples\": [\n          1462627,\n          5772060,\n          41052\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Title\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 6785,\n        \"samples\": [\n          \"GIG15 - P1 - Big Data - Presentation\",\n          \"Mortality Committee - Mortality Investigation 1A 1995-1997 M & F: Graduated Mortality Tables (April 2001)\",\n          \"Actuaries Summit 2020 - Asset Liability Management Across Multiple Goals in Personal Financial Planning\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"StartPublishing\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2001-12-30 00:00:00\",\n        \"max\": \"2024-03-24 00:00:00\",\n        \"num_unique_values\": 661,\n        \"samples\": [\n          \"2023-05-04 00:00:00\",\n          \"2020-06-15 00:00:00\",\n          \"2019-05-29 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"StopPublishing\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 138,\n        \"samples\": [\n          \"2040-07-11 00:00:00\",\n          \"2033-02-22 00:00:00\",\n          \"3019-03-12 00:00:00\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Cpd\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 10,\n        \"num_unique_values\": 11,\n        \"samples\": [\n          5,\n          1,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Format\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 11,\n        \"samples\": [\n          \"Video\",\n          \"Article\",\n          \"Webinar\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Level\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"Introductory\",\n          \"Advanced\",\n          \"All\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Description\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5248,\n        \"samples\": [\n          \"General Insurance Seminar 2010. Concurrent 10C. Making assumptions about future inflation rates (CPI, AWE or LPI) is an important part of many actuarial tasks, and is particularly relevant for long-tailed insurance valuations. Despite this, these estimates sometimes receive only cursory attention and choosing a good estimate is fraught with difficulties. This paper discusses desirable characteristics of an inflation forecast and reports some recent research into possible ways of choosing inflation assumptions. This includes discussion of available independent forecasts, market based methods and historic estimators.\",\n          \"IInjury Disability Scheme Seminar - C2E - Comcare Scheme Update\",\n          \"Demographic changes demand financial literacy or retirement defaults. Do you have the knowledge to decide? Anthony Saliba gives us a glimpse of his session at the upcoming Actuaries Summit.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Region\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"International\",\n          \"Domestic and International\",\n          \"Domestic\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Copyright\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"Other\",\n          \"Speaker/Author\",\n          \"Institute\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EventAgeGroup\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"All\",\n          \"Retired Actuaries\",\n          \"YAP\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EventType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"Other\",\n          \"Major\",\n          \"Insights/Conferences\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Link\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7551,\n        \"samples\": [\n          \"https://actuaries.logicaldoc.cloud/download-ticket?ticketId=cd57e688-9336-4210-9538-46753faddcc0\",\n          \"https://actuaries.logicaldoc.cloud/download-ticket?ticketId=a2fec9e3-3d05-4fad-9fdc-cbe4d53fa2f7\",\n          \"https://actuaries.logicaldoc.cloud/download-ticket?ticketId=9dbeef6a-addb-4d03-b2ed-d4a52a5d7513\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Tags\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1658,\n        \"samples\": [\n          \"Banking|Banking - Risk Management|Banking - Thought Leadership|Life Insurance|Life Insurance - Thought Leadership|Risk Management|Risk Management - Frameworks, Regulation &amp; Standards|Risk Management - Thought Leadership\",\n          \"Data Science|Data Science - Data Analysis|General Insurance|General Insurance - Pricing\",\n          \"Banking - Economy Updates|Personal Effectiveness &amp; Leadership|Personal Effectiveness &amp; Leadership - Diversity &amp; Inclusion|Professional Updates|Professional Updates - Industry &amp; Economic\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "docs_data.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_1J6tkOg4GS-",
      "metadata": {
        "id": "_1J6tkOg4GS-"
      },
      "source": [
        "**Findings and Implications**:\n",
        "\n",
        "The first 5 rows confirm that the dataset includes a variety of columns, covering different aspects such as Title, Description, FileType, Format, EventType, and Tags. However, it’s evident that several columns (e.g., EventAgeGroup and EventType) contain NaN values even in this small sample, reaffirming that missing data is a notable issue.\n",
        "\n",
        "Additionally, columns like StopPublishing seem to have non-standard values (e.g., '0'), suggesting further investigation is needed to understand if these are placeholders or incorrect entries.\n",
        "\n",
        "**What does it mean?**\n",
        "This preliminary check reinforces our need for data cleaning and standardization as the next step, particularly focusing on missing values and inconsistencies to ensure our dataset is ready for deeper analysis.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "swU-TIyUmYl8",
      "metadata": {
        "id": "swU-TIyUmYl8"
      },
      "source": [
        "We will examine the **downloads_data dataset** using downloads_data.info(). This step will provide an overview of the dataset’s structure, including the number of rows, columns, data types, and any missing values.\n",
        "\n",
        "Understanding this is crucial because it helps us identify if there are any data quality issues that might affect how we analyze download patterns in Q3 later on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "C-UCqigbMgPk",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C-UCqigbMgPk",
        "outputId": "18df3972-d1c5-4dd3-a4d9-8e2a91aa9258"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 118300 entries, 0 to 118299\n",
            "Data columns (total 3 columns):\n",
            " #   Column        Non-Null Count   Dtype         \n",
            "---  ------        --------------   -----         \n",
            " 0   DMSId         118300 non-null  int64         \n",
            " 1   DownloadDate  118300 non-null  datetime64[ns]\n",
            " 2   Month         118300 non-null  int32         \n",
            "dtypes: datetime64[ns](1), int32(1), int64(1)\n",
            "memory usage: 2.3 MB\n"
          ]
        }
      ],
      "source": [
        "downloads_data.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zONZd7R3nS1q",
      "metadata": {
        "id": "zONZd7R3nS1q"
      },
      "source": [
        "**Findings**:\n",
        "The downloads_data dataset contains DMSId and DownloadDate, with no missing values. The DownloadDate is according to the data dictionary.\n",
        "\n",
        "**Implication:**\n",
        "This dataset is well-structured with no obvious error."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "31KmDfCLn2Nr",
      "metadata": {
        "id": "31KmDfCLn2Nr"
      },
      "source": [
        "We then take a look at the first few entries of the downloads_data dataset using downloads_data.head().\n",
        "\n",
        "This step helps us confirm the structure and content, ensuring that the DMSId and DownloadDate columns are populated correctly and align with our expectations based on the data dictionary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "l64A8El4MtBc",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l64A8El4MtBc",
        "outputId": "cbe66e5b-2098-48fe-bde8-310757710607"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     DMSId DownloadDate  Month\n",
              "0  4194310   2024-04-02      4\n",
              "1  4194310   2023-11-14     11\n",
              "2  4194310   2019-05-02      5\n",
              "3  4194310   2019-01-09      1\n",
              "4  4194310   2018-09-21      9"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fa8d974d-2df7-490f-b495-226748f91955\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DMSId</th>\n",
              "      <th>DownloadDate</th>\n",
              "      <th>Month</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>4194310</td>\n",
              "      <td>2024-04-02</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4194310</td>\n",
              "      <td>2023-11-14</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4194310</td>\n",
              "      <td>2019-05-02</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4194310</td>\n",
              "      <td>2019-01-09</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4194310</td>\n",
              "      <td>2018-09-21</td>\n",
              "      <td>9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fa8d974d-2df7-490f-b495-226748f91955')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-fa8d974d-2df7-490f-b495-226748f91955 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-fa8d974d-2df7-490f-b495-226748f91955');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3dabbee8-6b53-4a84-b61e-392f1f3bc59d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3dabbee8-6b53-4a84-b61e-392f1f3bc59d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3dabbee8-6b53-4a84-b61e-392f1f3bc59d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "downloads_data"
            }
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "downloads_data.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dhmoYz4VNS1O",
      "metadata": {
        "id": "dhmoYz4VNS1O"
      },
      "source": [
        "**Findings and Implications**\n",
        "The downloads_data preview shows that multiple entries share the same DMSId (e.g., 4194310), indicating that this particular asset was downloaded multiple times at different dates. This aligns with the data dictionary, which confirms DMSId represents a unique asset identifier, while DownloadDate tracks each download event.\n",
        "\n",
        "\n",
        "**Implications:**\n",
        "This confirms that download data is structured in a way that allows us to analyze download patterns over time for each asset.\n",
        "\n",
        "The next logical step will be to merge this downloads_data with the docs_data dataset using DMSId as the key. This will enrich our dataset, enabling us to explore how various factors (like FileType, Region, or CPD points) might influence download behavior over time.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ySLmm_pI3n7W",
      "metadata": {
        "id": "ySLmm_pI3n7W"
      },
      "source": [
        "**Summary**  \n",
        "*Provide a **clear** and **concise** summary here of your conclusions based on the EDA and cleaning conducted above.*\n",
        "\n",
        "\n",
        "**Summary**\n",
        "The goal of this EDA was to understand our datasets and get them ready for further analyzing in Q3.\n",
        "\n",
        "Data Structure: We found that docs_data has 17 columns, and downloads_data has 2 columns that record when each asset was downloaded. The DMSId column is the link between the two, which means we can combine them for our analysis.\n",
        "\n",
        "Missing Values: There were a lot of missing entries in the EventAgeGroup and EventType columns, meaning many assets aren’t linked to specific events. We will need to fix the gap using our domaint knowledge.\n",
        "\n",
        "Data Consistency: We spotted StopPublishing column is not matching the expected data type from Data dictionary. Data cleaning will be needed to fix this column correctly.\n",
        "\n",
        "Download Trends: The repeated download entries in downloads_data confirmed that we can examinate download popularity pattern across different time horizon."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "milVPFlGh1mY",
      "metadata": {
        "id": "milVPFlGh1mY"
      },
      "source": [
        "####Q1a Step 2 EDA on Downloads\n",
        "\n",
        "Purpose:\n",
        "The aim here is to confirm that DMSId acts as a unique identifier for assets in the docs_data dataset, ensuring it's ready to join with downloads_data. This check is crucial because any discrepancies in DMSId would prevent an accurate analysis of how different assets are being downloaded.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "My Expectation:\n",
        "- Based on the data dictionary, we expect to see 7570 distinct integers in the DMSId column because it is a unique identifier for each asset in the DMS system.\n",
        "- DMSID is a table joint key used for joining with the second table, Downloads Data.\n",
        "\n",
        "**Approach:**\n",
        "-  Verify that there are 7570 unique values in the doc dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xS7V6fwwrwPk",
      "metadata": {
        "id": "xS7V6fwwrwPk"
      },
      "source": [
        "We use tutorial provided EDA formula to do the inital examiniation.\n",
        "\n",
        "My Expectation:\n",
        "\n",
        "Based on the data dictionary, we expect to see 7570 distinct integers in the DMSId column because it is a unique identifier for each asset in the DMS system.\n",
        "DMSID is a table joint key used for joining with the second table, Downloads Data.\n",
        "Approach:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "MYwzAmngf7WO",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MYwzAmngf7WO",
        "outputId": "cee9373f-496a-4098-86fb-88237d471eb7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 7570\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: DMSId\n",
            "4194310    1\n",
            "4362263    1\n",
            "4362275    1\n",
            "4362274    1\n",
            "4362273    1\n",
            "Name: count, dtype: int64\n",
            "Number of rows with DMSId values that appear more than once: 0\n",
            "---------------\n",
            "Data type: <class 'int'>, Count: 7570\n",
            "Sample values: [4363028, 4361133, 4365241]\n",
            "\n",
            "---------------\n"
          ]
        }
      ],
      "source": [
        "# EDA of DMSId #\n",
        "# Check for unique values in Doc dataset#\n",
        "df = pd.DataFrame(docs_data)\n",
        "eda_describe_column(df,'DMSId')\n",
        "\n",
        "# Check for dtype\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'DMSId')\n",
        "print(\"---------------\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iYjOUaGfiACr",
      "metadata": {
        "id": "iYjOUaGfiACr"
      },
      "source": [
        "**Summary of DMSId EDA**:\n",
        "\n",
        "- The Doc table contains 7570 unique DMSIds with no missing or invalid entries.\n",
        "\n",
        "**Implication:**\n",
        "This confirms that DMSId is a reliable unique key for identifying assets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SJIEWY6Y-9sz",
      "metadata": {
        "id": "SJIEWY6Y-9sz"
      },
      "source": [
        "**Examinate data consistency between docs_data and downloads_data**\n",
        "\n",
        "**Purpose:**\n",
        "We’re comparing DMSId values between docs_data and downloads_data to ensure **consistency**. This step is crucial because it confirms that every asset with download data has corresponding details in docs_data. Any discrepancies here could signal significant data inconsistencies that might affect our analysis.\n",
        "\n",
        "**Expectation:**\n",
        "All assets recorded in downloads_data should have a matching entry in docs_data. If any DMSId from downloads_data is missing in docs_data, it indicates a significant data inconsistency that needs to be addressed.\n",
        "\n",
        "Reference: The following code was generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "GoUbkHdy_V9v",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GoUbkHdy_V9v",
        "outputId": "9f8bb75d-225d-47d8-8188-bd33c0d18e67"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of DMSIDs missing in document dataset: 0\n",
            "Number of DMSIDs missing in downloads dataset: 3094\n",
            "DMSIDs missing in downloads dataset: {4227073, 4358145, 4227075, 4227076, 4227077, 4227078, 4358146, 4358147, 4227081, 4227082, 4325385, 4358148, 4227085, 4227086, 4227087, 4358151, 4227089, 4227090, 4227091, 4259859, 4227093, 4227094, 4292628, 4358160, 4358161, 4325402, 4358162, 4358163, 4227101, 4325405, 4227103, 4227104, 4325406, 4325407, 4227107, 4227108, 4325408, 4227110, 4227111, 4325409, 4325410, 4366369, 4227115, 4227116, 4227117, 4227119, 4227120, 4227121, 4227123, 4227124, 4227125, 4227126, 4227127, 4227128, 4227129, 4227130, 4227131, 4227133, 4227134, 4227136, 4227138, 4227139, 4227140, 4227141, 4227143, 4227144, 4227145, 4227146, 4227147, 4227148, 4227149, 4227151, 4227154, 4227155, 4366418, 4366420, 4227162, 4227163, 4366426, 4227165, 4227166, 4366428, 4227170, 4325475, 4325476, 4366436, 4325478, 4325479, 4227176, 4227177, 4227178, 4227179, 4227180, 4227181, 4227182, 4227183, 4325487, 4325488, 4325489, 4325490, 4325491, 4325492, 4227190, 4325493, 4325494, 4325495, 4325496, 4325497, 4325500, 4325501, 4325502, 4325503, 4325504, 4325505, 4325506, 4325507, 4325508, 4325509, 4325510, 4325511, 4325512, 4325513, 4325514, 4325515, 4325516, 4325517, 4325518, 4325519, 4325520, 4366474, 4366475, 4366477, 4325524, 4325525, 4325526, 4325527, 4325528, 4325529, 4366484, 4325531, 4325532, 4325533, 4325534, 4366487, 4325536, 4366489, 4366491, 4366494, 4366497, 4366499, 4366501, 4366503, 4366505, 4366508, 4366510, 4325551, 4325552, 4325553, 4366511, 4325555, 4325556, 4325557, 4325558, 4325559, 4325560, 4325561, 4325562, 4325563, 4325564, 4325565, 4325566, 4325567, 4325568, 4325569, 4325570, 4325571, 4325572, 4325573, 4366525, 4366527, 4366529, 4366532, 4366534, 4366537, 4366542, 4202707, 4202708, 4202713, 4202714, 4202716, 4202717, 4202718, 4366558, 4202720, 4202721, 4202722, 4202723, 4202724, 4202725, 4202726, 4366560, 4202728, 4366562, 4202730, 4366565, 4366567, 4366570, 4202735, 4202736, 4202738, 4202739, 4202741, 4202742, 4366582, 4202744, 4366585, 4202746, 4202747, 4202750, 4366592, 4202754, 4202755, 4366595, 4202758, 4202769, 4366609, 4202772, 4366612, 4366615, 4202776, 4202777, 4366616, 4202779, 4366619, 4366622, 4202783, 4202784, 4366623, 4202786, 4202787, 4202788, 4202789, 4202791, 4202792, 4202793, 4202794, 4202795, 4366636, 4202797, 4202798, 4202800, 4366641, 4202802, 4202803, 4366643, 4202805, 4202806, 4366648, 4202809, 4202810, 4366653, 4358472, 4358473, 4202827, 4202828, 4366669, 4202831, 4202833, 4366677, 4358492, 4358494, 4358495, 4358496, 4358498, 4358499, 4358500, 4358501, 4358502, 4358503, 4366693, 4358505, 4366697, 4358507, 4358509, 4358511, 4358512, 4358513, 4358514, 4366708, 4366709, 4366713, 4366715, 4366716, 4358528, 4358529, 4358530, 4358531, 4358532, 4358533, 4358534, 4358535, 4366721, 4366727, 4358538, 4366731, 4366750, 4366752, 4358566, 4358567, 4358568, 4358569, 4358570, 4358571, 4358572, 4358573, 4358574, 4358575, 4358576, 4358577, 4202922, 4358579, 4358580, 4202924, 4358582, 4358583, 4358584, 4358585, 4202929, 4358587, 4358588, 4358589, 4358590, 4358591, 4358592, 4358593, 4358594, 4202940, 4202941, 4202942, 4202944, 4202946, 4366787, 4202953, 4202958, 4366799, 4202961, 4202962, 4202963, 4366818, 4358628, 4358629, 4358630, 4358631, 4358632, 4358633, 4358634, 4358635, 4358636, 4358637, 4358638, 4358639, 4358640, 4358641, 4358642, 4358643, 4358644, 4366839, 4366842, 4366844, 4366847, 4363742, 4363743, 4366871, 4366875, 4366882, 4366899, 4366912, 4366913, 4366929, 4366932, 4366939, 4366940, 4366961, 4366969, 4366970, 4366993, 4358804, 4358815, 4367008, 4358825, 4367022, 4367024, 4367025, 4367026, 4367029, 4367031, 4367039, 4367046, 4358859, 4358860, 4367051, 4358862, 4358864, 4358865, 4358866, 4358867, 4358868, 4367065, 4367070, 4367072, 4358883, 4367079, 4367080, 4358889, 4358890, 4358891, 4358892, 4358893, 4358894, 4367082, 4358896, 4358898, 4358904, 4358905, 4367098, 4367099, 4358908, 4367100, 4358910, 4358911, 4358912, 4367102, 4367105, 4367107, 4367108, 4367110, 4367111, 4367112, 4358921, 4358926, 4358927, 4358928, 4358932, 4358934, 4358935, 4358937, 4358938, 4358939, 4358942, 4358943, 4358944, 4358945, 4358946, 4358949, 4358950, 4358952, 4358953, 4358956, 4358957, 4358958, 4358959, 4358960, 4358961, 4358962, 4358963, 4358964, 4358965, 4358966, 4358967, 4367151, 4358969, 4367154, 4358971, 4367155, 4358973, 4358974, 4358975, 4367161, 4358978, 4367170, 4358980, 4359002, 4359027, 4359029, 4367254, 4367258, 4367278, 4367281, 4367298, 4367301, 4367302, 4367304, 4367317, 4359126, 4359128, 4367322, 4367324, 4359134, 4359136, 4359137, 4359138, 4359139, 4359141, 4359143, 4359144, 4367339, 4359148, 4359150, 4359159, 4359161, 4367353, 4359163, 4367354, 4367357, 4359166, 4359167, 4367359, 4359169, 4359170, 4359171, 4359173, 4359174, 4359175, 4367365, 4367369, 4359178, 4359179, 4359180, 4359181, 4359182, 4367376, 4359185, 4359186, 4359187, 4359188, 4359189, 4367379, 4367383, 4359193, 4359194, 4359195, 4359197, 4359198, 4359199, 4359205, 4359206, 4359207, 4359208, 4359209, 4367399, 4359211, 4359213, 4359214, 4359215, 4359216, 4359218, 4359219, 4367410, 4367411, 4367414, 4367417, 4367424, 4367425, 4359240, 4359241, 4359242, 4359243, 4367438, 4359247, 4359248, 4359249, 4367440, 4359251, 4359252, 4359253, 4359254, 4359255, 4367448, 4359257, 4359258, 4359259, 4367451, 4367453, 4359265, 4359266, 4359267, 4359268, 4359269, 4359270, 4359271, 4359273, 4359274, 4367467, 4359276, 4359277, 4367470, 4359280, 4359281, 4359282, 4359283, 4359284, 4367472, 4367474, 4367477, 4367483, 4367484, 4367486, 4359295, 4359296, 4359297, 4359298, 4367489, 4359301, 4359302, 4359303, 4359304, 4359305, 4359306, 4359307, 4359308, 4359309, 4359310, 4359311, 4359312, 4367498, 4359314, 4359315, 4359316, 4367500, 4367507, 4367509, 4359321, 4359323, 4367521, 4367525, 4359335, 4367527, 4367530, 4367532, 4359350, 4367545, 4367548, 4367549, 4367557, 4367559, 4367560, 4367572, 4367573, 4367578, 4367597, 4367598, 4367614, 4367619, 4367623, 4367152, 4367645, 4367648, 4367158, 4367658, 4367671, 4367675, 4367677, 4359488, 4359489, 4359491, 4359493, 4359496, 4359497, 4359498, 4359499, 4359500, 4359506, 4359507, 4359512, 4359515, 4359516, 4359517, 4359518, 4359519, 4359521, 4367714, 4359524, 4359525, 4367717, 4359527, 4359528, 4367718, 4359530, 4359531, 4367720, 4367721, 4367723, 4367725, 4359536, 4359542, 4359543, 4359544, 4359545, 4359546, 4359547, 4359548, 4359549, 4367739, 4359551, 4367741, 4367743, 4367745, 4359555, 4359556, 4359557, 4359558, 4359559, 4359560, 4367746, 4367748, 4359563, 4359564, 4359565, 4367750, 4359567, 4367753, 4359569, 4359570, 4359571, 4367755, 4367759, 4359574, 4359575, 4359576, 4359577, 4367770, 4359580, 4367772, 4367774, 4359583, 4359584, 4367776, 4359586, 4359587, 4367779, 4367781, 4367782, 4359593, 4359594, 4359595, 4359596, 4359597, 4359598, 4359599, 4359600, 4367786, 4359602, 4359603, 4367787, 4359605, 4359606, 4359607, 4359608, 4359609, 4359610, 4359611, 4367809, 4359618, 4359619, 4359620, 4367812, 4367815, 4359626, 4367819, 4367822, 4359631, 4359632, 4359635, 4367828, 4367829, 4359638, 4359639, 4359643, 4359644, 4359647, 4359648, 4359649, 4359651, 4359652, 4359653, 4367844, 4367846, 4359656, 4359657, 4367847, 4359659, 4367850, 4359661, 4367851, 4359664, 4367856, 4359666, 4359668, 4359670, 4359671, 4359674, 4359677, 4359680, 4359681, 4359682, 4359684, 4359685, 4359687, 4359688, 4359689, 4359690, 4359691, 4359692, 4367880, 4367883, 4367884, 4367886, 4367887, 4367889, 4359703, 4359705, 4367901, 4367903, 4367904, 4367906, 4359723, 4359724, 4359726, 4359729, 4359730, 4359731, 4359734, 4359735, 4359736, 4359737, 4359738, 4359739, 4359740, 4359741, 4367929, 4367931, 4359744, 4359745, 4367932, 4367936, 4367948, 4367951, 4367956, 4367957, 4367962, 4367964, 4367966, 4367974, 4367975, 4367978, 4367980, 4367981, 4367993, 4368001, 4368010, 4368011, 4368017, 4368036, 4368040, 4368051, 4368056, 4368058, 4359880, 4368074, 4359884, 4359885, 4368078, 4359887, 4368080, 4359889, 4359891, 4359893, 4359894, 4359897, 4359898, 4359900, 4359901, 4368095, 4368097, 4359909, 4359910, 4359911, 4359913, 4359915, 4359916, 4359919, 4359920, 4359922, 4359923, 4359924, 4359925, 4359927, 4359928, 4368125, 4368129, 4359943, 4359946, 4359950, 4368142, 4359952, 4359953, 4359954, 4368144, 4359956, 4368146, 4359961, 4359963, 4359964, 4359965, 4359968, 4368181, 4368184, 4368187, 4368220, 4368225, 4368226, 4368239, 4368243, 4368251, 4368253, 4368255, 4368269, 4360079, 4360080, 4360081, 4368273, 4368274, 4360085, 4360087, 4360088, 4360090, 4360092, 4360093, 4360094, 4368286, 4368288, 4360097, 4360098, 4360099, 4360102, 4360104, 4360107, 4360110, 4360111, 4360115, 4360116, 4360117, 4368310, 4360119, 4360123, 4360124, 4360125, 4360126, 4360127, 4360130, 4360131, 4360132, 4360133, 4360134, 4360135, 4360136, 4360137, 4360138, 4360140, 4360142, 4360146, 4360149, 4360150, 4360152, 4360154, 4360155, 4360159, 4360160, 4360161, 4368354, 4360163, 4360164, 4360165, 4360167, 4360168, 4368360, 4360170, 4360173, 4360174, 4368377, 4360186, 4360189, 4360191, 4360192, 4360193, 4360194, 4360195, 4360202, 4360203, 4360205, 4360206, 4360209, 4360210, 4360211, 4360213, 4360214, 4360215, 4360216, 4360217, 4360218, 4360219, 4360220, 4360221, 4360222, 4360223, 4360224, 4360225, 4360226, 4360227, 4360228, 4360237, 4360238, 4360239, 4360240, 4360241, 4360242, 4360243, 4360244, 4360245, 4360246, 4360247, 4360248, 4360250, 4360251, 4360253, 4360258, 4206125, 4360297, 4360302, 4360311, 4360313, 4360314, 4360317, 4360318, 4360320, 4360321, 4360322, 4360323, 4360324, 4360325, 4360335, 4360336, 4360337, 4196529, 4196530, 4196531, 4196535, 4196538, 4196540, 4196541, 4196542, 4196543, 4206163, 4206164, 4360481, 4360482, 4360483, 4360484, 4360485, 4206166, 4360488, 4360490, 4360491, 4206168, 4360494, 4360497, 4360500, 4360505, 4360506, 4360509, 4360510, 4360513, 4360516, 4360517, 4360518, 4360520, 4202925, 4202926, 4202930, 4202932, 4202933, 4202934, 4202935, 4202936, 4202937, 4360696, 4360710, 4360711, 4360712, 4360713, 4360714, 4360716, 4360722, 4360723, 4360724, 4360727, 4360729, 4360730, 4360734, 4360737, 4360738, 4360741, 4360743, 4360745, 4360747, 4360749, 4360750, 4360753, 4360755, 4360757, 4360760, 4360761, 4360764, 4360765, 4360768, 4360769, 4360771, 4360778, 4360781, 4360783, 4360784, 4360785, 4360786, 4202983, 4360928, 4360930, 4360931, 4360932, 4360933, 4360934, 4360939, 4360940, 4360941, 4360942, 4360943, 4360944, 4360946, 4360947, 4360948, 4360950, 4360951, 4360952, 4360954, 4360955, 4360956, 4360957, 4360959, 4360960, 4360964, 4360965, 4360966, 4360967, 4360968, 4369168, 4369170, 4369172, 4360983, 4360984, 4369176, 4369178, 4360988, 4369180, 4360990, 4369182, 4360992, 4369184, 4369186, 4360995, 4369188, 4369190, 4369192, 4361001, 4369194, 4361003, 4361004, 4369196, 4361006, 4369198, 4361008, 4369203, 4369205, 4361015, 4361016, 4361017, 4369207, 4361019, 4361020, 4361021, 4361022, 4369208, 4361024, 4361025, 4361026, 4361027, 4361028, 4361029, 4369214, 4361031, 4361032, 4369216, 4361034, 4361035, 4369220, 4361037, 4369222, 4361039, 4361040, 4369224, 4369228, 4361043, 4369230, 4361045, 4369232, 4369234, 4361048, 4361049, 4369236, 4361051, 4369238, 4361053, 4361054, 4369240, 4369242, 4361057, 4361058, 4361059, 4369244, 4369246, 4361062, 4361063, 4369248, 4361065, 4361066, 4361067, 4361068, 4361069, 4361070, 4361071, 4369256, 4361073, 4369258, 4361075, 4369260, 4369262, 4369264, 4369268, 4369272, 4369274, 4361083, 4361084, 4361085, 4361086, 4361087, 4369276, 4361089, 4361090, 4361091, 4361092, 4361093, 4369278, 4361095, 4369280, 4361097, 4361098, 4361099, 4361100, 4369284, 4361102, 4361103, 4369288, 4361105, 4361106, 4369290, 4361108, 4361109, 4369294, 4361111, 4361112, 4369296, 4361114, 4361115, 4361116, 4361117, 4361118, 4369302, 4361120, 4361121, 4361122, 4369306, 4369308, 4369310, 4361126, 4361127, 4369312, 4361129, 4361130, 4369314, 4361132, 4361133, 4361134, 4361135, 4369319, 4369321, 4369323, 4369325, 4369327, 4369329, 4361142, 4369331, 4361144, 4361145, 4361146, 4361147, 4361148, 4369333, 4361150, 4361151, 4361152, 4369337, 4369339, 4369341, 4369343, 4369345, 4369347, 4369349, 4369351, 4369353, 4369355, 4369357, 4369359, 4369361, 4369363, 4361172, 4361174, 4361175, 4361176, 4369366, 4369368, 4361179, 4361180, 4361181, 4361182, 4361183, 4369370, 4361185, 4361186, 4361187, 4361188, 4361189, 4369373, 4369377, 4369379, 4369381, 4369383, 4361195, 4369385, 4369387, 4369389, 4369391, 4369393, 4361202, 4361203, 4361204, 4369394, 4361206, 4361207, 4361208, 4369396, 4369398, 4369400, 4369402, 4369404, 4369406, 4369408, 4369409, 4369411, 4369413, 4369416, 4369418, 4369420, 4369422, 4369424, 4369426, 4369428, 4369430, 4369432, 4369434, 4369436, 4369438, 4369440, 4369442, 4369444, 4369446, 4369448, 4369450, 4369452, 4369454, 4369456, 4369457, 4369459, 4369461, 4369463, 4369465, 4361277, 4361278, 4361279, 4361280, 4361281, 4361282, 4361283, 4361284, 4361285, 4369469, 4361287, 4361288, 4361289, 4361290, 4361291, 4361292, 4361293, 4369479, 4361295, 4369481, 4361297, 4361298, 4361299, 4361300, 4361301, 4361302, 4361303, 4369487, 4361305, 4361306, 4369491, 4361308, 4369493, 4369495, 4361311, 4361312, 4361313, 4361314, 4361315, 4361316, 4369501, 4361318, 4361319, 4361320, 4361321, 4361322, 4361323, 4361324, 4361325, 4361326, 4361327, 4361328, 4361329, 4361330, 4369514, 4369516, 4361333, 4369518, 4361335, 4361336, 4361337, 4369521, 4369523, 4361340, 4361341, 4361342, 4361343, 4369528, 4361345, 4369530, 4361347, 4361348, 4369532, 4361350, 4369536, 4361352, 4369538, 4369540, 4361355, 4361356, 4361357, 4361358, 4369542, 4369544, 4361361, 4361362, 4361363, 4369547, 4369549, 4361366, 4361367, 4369551, 4361369, 4369553, 4361371, 4369555, 4369558, 4361374, 4361375, 4361376, 4361373, 4361378, 4361379, 4361380, 4361377, 4361385, 4361386, 4361388, 4361389, 4361418, 4361442, 4361559, 4361561, 4361562, 4361590, 4361591, 4369210, 4361600, 4369212, 4361609, 4361614, 4361634, 4361635, 4369218, 4197806, 4197808, 4197809, 4197814, 4197815, 4197816, 4197817, 4197818, 4197819, 4197821, 4197822, 4197824, 4197825, 4197826, 4197827, 4197828, 4197831, 4197832, 4197835, 4197843, 4206035, 4206036, 4197846, 4197847, 4197848, 4206037, 4206039, 4206040, 4206041, 4206042, 4206043, 4206045, 4206046, 4206047, 4206049, 4206050, 4206051, 4206052, 4206053, 4206054, 4206055, 4206056, 4361706, 4206057, 4206058, 4206059, 4206060, 4206061, 4206062, 4206063, 4206064, 4206065, 4206066, 4361717, 4361718, 4206067, 4206068, 4206069, 4206070, 4206071, 4206072, 4197885, 4206073, 4206075, 4197888, 4206076, 4206077, 4206078, 4206079, 4197893, 4197894, 4197895, 4206080, 4206081, 4206082, 4206083, 4206084, 4197901, 4206085, 4206086, 4206087, 4206088, 4206089, 4206090, 4361741, 4197912, 4197913, 4361755, 4206109, 4206110, 4206112, 4206113, 4197922, 4206114, 4206115, 4197925, 4206116, 4206117, 4197928, 4206118, 4197930, 4206119, 4197932, 4197933, 4197934, 4206120, 4206121, 4206122, 4197938, 4206123, 4206124, 4197941, 4197942, 4206127, 4206128, 4206129, 4206130, 4206131, 4206138, 4361780, 4197958, 4197959, 4369250, 4197961, 4206151, 4206152, 4206154, 4197965, 4206155, 4197967, 4369252, 4206156, 4197970, 4206157, 4206158, 4206159, 4197974, 4197975, 4206160, 4206161, 4369254, 4206162, 4197980, 4197981, 4206165, 4197983, 4206167, 4197985, 4206169, 4206170, 4206171, 4206172, 4197990, 4206176, 4197993, 4197995, 4197996, 4197998, 4206230, 4206233, 4206236, 4206243, 4206246, 4206249, 4206256, 4206258, 4206260, 4206264, 4206265, 4361922, 4206275, 4206281, 4361931, 4206286, 4370129, 4206292, 4361945, 4361946, 4206297, 4206300, 4361954, 4361957, 4369282, 4361971, 4369286, 4361980, 4369292, 4362039, 4369298, 4362042, 4362045, 4362046, 4369300, 4362062, 4362064, 4369304, 4362080, 4362091, 4362110, 4362125, 4362128, 4362129, 4369316, 4362137, 4362138, 4362139, 4362140, 4362141, 4362142, 4362147, 4362150, 4362152, 4362155, 4362157, 4362159, 4362161, 4362162, 4362163, 4362164, 4362165, 4362166, 4362167, 4362169, 4362171, 4362173, 4362175, 4362176, 4362177, 4362178, 4362180, 4362183, 4362184, 4362185, 4362187, 4362188, 4362189, 4362190, 4362191, 4362194, 4362198, 4362199, 4362200, 4362201, 4362202, 4362203, 4362204, 4362207, 4362211, 4362213, 4362214, 4362215, 4362217, 4362218, 4362219, 4362220, 4362221, 4362222, 4362225, 4369335, 4362228, 4362229, 4362230, 4362231, 4362232, 4362235, 4362236, 4362237, 4362239, 4362240, 4362241, 4362242, 4362243, 4362244, 4362245, 4362247, 4362251, 4362252, 4362253, 4362254, 4362255, 4362256, 4362258, 4362262, 4362263, 4362265, 4362266, 4362267, 4362269, 4362270, 4362271, 4362274, 4362275, 4362276, 4362277, 4362297, 4362298, 4362301, 4362302, 4362303, 4362304, 4362305, 4362306, 4362308, 4362310, 4362311, 4362312, 4362324, 4362329, 4362331, 4362332, 4362333, 4362334, 4362336, 4362339, 4362340, 4362341, 4362342, 4362343, 4362344, 4362345, 4362346, 4362347, 4362349, 4362350, 4362351, 4362378, 4362381, 4362384, 4362387, 4362388, 4362394, 4362396, 4362397, 4362398, 4362399, 4362401, 4362402, 4362407, 4362410, 4362425, 4362426, 4362427, 4362428, 4362434, 4362435, 4362437, 4206898, 4206900, 4206903, 4362568, 4362572, 4362575, 4362584, 4362606, 4362609, 4198775, 4198776, 4362616, 4362617, 4362618, 4362619, 4198783, 4362623, 4198789, 4362631, 4198793, 4362634, 4367788, 4362645, 4362660, 4362661, 4362668, 4362674, 4362678, 4362698, 4362705, 4362716, 4362725, 4362732, 4362733, 4362734, 4362737, 4362738, 4362739, 4362740, 4362741, 4362742, 4362750, 4362763, 4325480, 4362765, 4325481, 4325482, 4325483, 4325484, 4362787, 4325485, 4362803, 4207156, 4362807, 4362808, 4362809, 4362814, 4362840, 4362841, 4362842, 4362843, 4199012, 4199014, 4199015, 4199016, 4199018, 4199021, 4199022, 4199024, 4199025, 4199026, 4199029, 4199030, 4362872, 4199033, 4199034, 4199035, 4199038, 4199039, 4199040, 4362878, 4199042, 4199043, 4199046, 4199047, 4199049, 4199053, 4199054, 4199056, 4199058, 4199060, 4199062, 4199064, 4199065, 4369470, 4199067, 4199068, 4199071, 4207263, 4207264, 4207265, 4369472, 4207268, 4207276, 4369476, 4199110, 4199112, 4362952, 4199114, 4199119, 4199120, 4199122, 4199123, 4199124, 4369482, 4199129, 4369483, 4199132, 4199133, 4199134, 4199135, 4199137, 4199138, 4199140, 4199141, 4199142, 4362980, 4369485, 4362984, 4362988, 4199150, 4199151, 4361346, 4362993, 4362990, 4199155, 4199156, 4362994, 4199158, 4199159, 4369489, 4199161, 4199163, 4199164, 4199165, 4199166, 4199169, 4199170, 4199171, 4199172, 4199174, 4199177, 4199178, 4199179, 4199180, 4199181, 4363023, 4363039, 4369497, 4363043, 4363044, 4369499, 4363066, 4369503, 4207431, 4207432, 4369505, 4207434, 4207435, 4363086, 4207438, 4207439, 4207440, 4207442, 4369507, 4363092, 4207443, 4207446, 4207448, 4207451, 4207452, 4369509, 4207454, 4207455, 4207456, 4369511, 4369513, 4207498, 4207499, 4207505, 4207506, 4369520, 4207515, 4207518, 4207519, 4207527, 4369524, 4207529, 4207531, 4207532, 4369525, 4207537, 4369526, 4207539, 4207540, 4207543, 4207545, 4207563, 4207564, 4207565, 4207566, 4207567, 4207568, 4207569, 4207570, 4207571, 4207572, 4207573, 4207575, 4199386, 4207581, 4199390, 4199391, 4207582, 4207584, 4207585, 4207587, 4207589, 4207590, 4199399, 4199400, 4199401, 4199402, 4199403, 4199404, 4207592, 4207594, 4207595, 4207596, 4207597, 4207599, 4207602, 4207603, 4207604, 4207605, 4207606, 4207607, 4207609, 4207610, 4207611, 4207612, 4207613, 4207615, 4207616, 4207618, 4207621, 4207623, 4207629, 4207631, 4207632, 4207633, 4369545, 4207635, 4207636, 4207637, 4207638, 4207639, 4207640, 4358149, 4358150, 4199456, 4207647, 4199458, 4199459, 4207648, 4207650, 4199462, 4199463, 4207651, 4358152, 4207652, 4207653, 4207654, 4207657, 4358153, 4207660, 4207661, 4207663, 4207664, 4358154, 4207665, 4207666, 4207667, 4207668, 4207670, 4358156, 4358157, 4358158, 4363344, 4363346, 4358164, 4363394, 4363398, 4363399, 4363400, 4363404, 4363422, 4363438, 4363445, 4363450, 4363451, 4363452, 4363453, 4363459, 4363478, 4363479, 4363489, 4363494, 4363499, 4363500, 4366341, 4199715, 4199717, 4199718, 4199719, 4199720, 4199721, 4199723, 4199726, 4199728, 4199729, 4199731, 4199732, 4199735, 4199736, 4199737, 4366347, 4199740, 4366348, 4199743, 4199744, 4199745, 4199746, 4199747, 4199748, 4199749, 4199753, 4199754, 4199755, 4199757, 4199758, 4366358, 4199796, 4199797, 4199798, 4199800, 4199802, 4199803, 4199804, 4199805, 4199806, 4199807, 4199808, 4199809, 4199810, 4366362, 4199812, 4199813, 4199815, 4199816, 4199817, 4199818, 4199819, 4199820, 4199822, 4199823, 4199824, 4199825, 4363664, 4199827, 4199828, 4199831, 4199832, 4199833, 4199836, 4199837, 4199838, 4199839, 4199843, 4199846, 4199847, 4199848, 4199849, 4199852, 4199857, 4199858, 4199859, 4199860, 4199861, 4199862, 4199864, 4199866, 4199868, 4199871, 4199872, 4199874, 4199875, 4199876, 4199878, 4199879, 4199882, 4199884, 4199885, 4199886, 4199887, 4199888, 4199889, 4199892, 4199894, 4199895, 4199897, 4363738, 4199899, 4199900, 4363741, 4363739, 4363740, 4199904, 4363745, 4199905, 4199906, 4199907, 4363749, 4363750, 4363751, 4363752, 4199908, 4363754, 4363755, 4363756, 4363757, 4199911, 4363759, 4199916, 4363761, 4363762, 4199917, 4363764, 4363765, 4199919, 4199920, 4199921, 4199924, 4199925, 4199927, 4199928, 4199929, 4199930, 4199931, 4363776, 4199932, 4199933, 4363779, 4363780, 4199934, 4199938, 4199939, 4199940, 4363781, 4363785, 4363787, 4363788, 4363791, 4363795, 4363797, 4363798, 4363809, 4363813, 4363815, 4363816, 4363817, 4363818, 4363820, 4363821, 4363824, 4363825, 4363827, 4363828, 4363833, 4363834, 4363835, 4363839, 4363840, 4363841, 4363842, 4363843, 4363845, 4363846, 4363849, 4363851, 4363852, 4363853, 4363855, 4363856, 4363858, 4363860, 4363862, 4363863, 4363864, 4366438, 4366441, 4364051, 4366443, 4366444, 4208423, 4208424, 4208425, 4366446, 4208426, 4208427, 4366448, 4364090, 4366451, 4366453, 4366455, 4366457, 4364132, 4200298, 4200299, 4200300, 4200301, 4366459, 4200303, 4200305, 4200307, 4200308, 4200310, 4200312, 4200313, 4200314, 4200316, 4200318, 4200321, 4200323, 4200324, 4200325, 4200326, 4366464, 4200329, 4200331, 4200333, 4200334, 4200336, 4200338, 4200339, 4200340, 4200342, 4200345, 4366468, 4200350, 4200351, 4200352, 4200353, 4200354, 4366470, 4200356, 4200357, 4200359, 4200360, 4200361, 4200362, 4200363, 4200364, 4200365, 4366471, 4200367, 4364209, 4200370, 4200372, 4200373, 4200374, 4200375, 4208568, 4200377, 4200380, 4200381, 4200382, 4200383, 4200384, 4200385, 4200386, 4200390, 4200392, 4200393, 4200394, 4200395, 4200396, 4200397, 4200398, 4200405, 4366480, 4366485, 4364287, 4200476, 4364329, 4364335, 4200498, 4364340, 4200505, 4364345, 4200509, 4364350, 4200511, 4200512, 4200515, 4364358, 4200519, 4200520, 4364360, 4364363, 4200523, 4200524, 4364370, 4364371, 4364374, 4364376, 4200537, 4200538, 4200539, 4364377, 4200541, 4200544, 4364385, 4200548, 4200550, 4200551, 4364392, 4200552, 4200553, 4200554, 4200556, 4200557, 4364398, 4200558, 4200560, 4200565, 4200568, 4200570, 4200571, 4200572, 4200573, 4366513, 4200576, 4200579, 4366515, 4364422, 4200583, 4200584, 4200587, 4200588, 4366517, 4364435, 4200597, 4200599, 4364441, 4366520, 4364451, 4366522, 4364460, 4364469, 4364472, 4364482, 4364483, 4364493, 4200661, 4200662, 4200663, 4200665, 4200668, 4364509, 4200670, 4200672, 4200674, 4200675, 4364514, 4200677, 4200678, 4200680, 4364520, 4200686, 4200688, 4200689, 4200690, 4200691, 4200692, 4200693, 4364535, 4364541, 4364554, 4364570, 4364590, 4364592, 4200766, 4364619, 4200827, 4364745, 4210278, 4210279, 4210292, 4210293, 4210298, 4210304, 4364931, 4210307, 4210308, 4210309, 4210310, 4210312, 4210316, 4210319, 4210320, 4210325, 4210327, 4210328, 4361776, 4361778, 4361779, 4365230, 4365232, 4365233, 4365234, 4365235, 4365236, 4365237, 4365238, 4365239, 4365240, 4365241, 4365242, 4365243, 4365244, 4365245, 4365246, 4365247, 4365248, 4365249, 4365250, 4365251, 4365252, 4365253, 4365255, 4365256, 4365257, 4365258, 4365259, 4365260, 4365262, 4365263, 4365264, 4365265, 4365266, 4365267, 4365268, 4365269, 4365270, 4365271, 4365272, 4365273, 4365274, 4365275, 4365276, 4365277, 4365283, 4365289, 4365316, 4365319, 4365327, 4365330, 4365344, 4365351, 4365372, 4365383, 4365384, 4365386, 4365389, 4365407, 4365413, 4365419, 4365426, 4365427, 4365429, 4365433, 4365434, 4365435, 4365443, 4365445, 4201627, 4201630, 4201631, 4201637, 4201638, 4201642, 4201644, 4201647, 4201648, 4201649, 4365489, 4201651, 4201652, 4365492, 4365494, 4201657, 4201663, 4365505, 4365507, 4365508, 4201669, 4201670, 4201671, 4365512, 4201673, 4201676, 4201677, 4201679, 4201682, 4201683, 4201685, 4201687, 4365528, 4201689, 4201693, 4365534, 4201696, 4201698, 4201700, 4201701, 4365541, 4201706, 4201713, 4201715, 4201718, 4201719, 4365558, 4201721, 4201726, 4201729, 4201732, 4201734, 4365575, 4201736, 4201737, 4365579, 4201747, 4365593, 4365596, 4201759, 4365613, 4365615, 4201776, 4201777, 4201782, 4209979, 4201792, 4201793, 4201794, 4201795, 4201796, 4201797, 4201799, 4201800, 4365641, 4201802, 4365643, 4201804, 4201805, 4201806, 4201808, 4201810, 4201811, 4201812, 4366760, 4366762, 4365672, 4365673, 4366764, 4365677, 4210037, 4210039, 4210040, 4210044, 4210049, 4210050, 4210051, 4210053, 4210054, 4210055, 4210056, 4365705, 4365706, 4210059, 4365708, 4210061, 4210062, 4210063, 4210068, 4210069, 4210072, 4210073, 4210074, 4365725, 4210078, 4210079, 4365728, 4210080, 4365730, 4210081, 4210085, 4210086, 4365736, 4365737, 4210088, 4210090, 4210093, 4210094, 4210095, 4210096, 4366779, 4210103, 4210104, 4210105, 4210106, 4210107, 4210108, 4210110, 4210111, 4210112, 4365761, 4365762, 4210113, 4210114, 4210115, 4365766, 4210117, 4210118, 4210119, 4210123, 4210125, 4210126, 4210127, 4365776, 4210128, 4210129, 4210130, 4365780, 4210131, 4210136, 4210137, 4210138, 4210139, 4210140, 4210141, 4210143, 4210144, 4210145, 4365796, 4210148, 4210149, 4210151, 4210152, 4210153, 4365802, 4210154, 4210155, 4210156, 4210157, 4210159, 4365808, 4210160, 4210161, 4365811, 4210163, 4210164, 4210165, 4210166, 4210167, 4210171, 4210172, 4210173, 4365822, 4210174, 4210175, 4210176, 4210178, 4210179, 4210180, 4210181, 4365830, 4201991, 4365832, 4210183, 4210184, 4210185, 4210186, 4210188, 4210189, 4201999, 4210190, 4202001, 4365841, 4202003, 4210191, 4210193, 4210195, 4210196, 4210200, 4210201, 4210202, 4210204, 4210205, 4210206, 4210207, 4210208, 4210209, 4210210, 4210211, 4210212, 4210213, 4365862, 4365863, 4210214, 4210216, 4210218, 4365867, 4210222, 4365872, 4210226, 4210228, 4210229, 4210230, 4210231, 4210233, 4210235, 4210238, 4210239, 4210240, 4210241, 4210242, 4210244, 4210245, 4210246, 4365896, 4210248, 4210250, 4210252, 4210253, 4365902, 4210254, 4210257, 4210258, 4210259, 4202068, 4210260, 4365910, 4365911, 4210261, 4210262, 4210263, 4202075, 4202076, 4365915, 4365916, 4210264, 4210267, 4210270, 4210271, 4210272, 4202084, 4210273, 4202086, 4210274, 4202088, 4210275, 4365930, 4210276, 4202092, 4365933, 4365934, 4202095, 4365935, 4202097, 4202098, 4210282, 4210283, 4365941, 4210288, 4202103, 4210290, 4365945, 4202106, 4210291, 4365948, 4202109, 4202110, 4210296, 4210297, 4202113, 4202114, 4365954, 4366820, 4210301, 4210302, 4202119, 4202120, 4202121, 4202122, 4202123, 4202124, 4202125, 4202126, 4366822, 4210311, 4202129, 4365969, 4202131, 4202132, 4365971, 4202134, 4365974, 4366823, 4365977, 4365978, 4366824, 4365980, 4210324, 4365982, 4210326, 4365984, 4365985, 4210329, 4210330, 4210331, 4365989, 4210333, 4210334, 4365992, 4210336, 4210337, 4210338, 4210339, 4210340, 4210343, 4210345, 4210347, 4210348, 4210349, 4366003, 4366005, 4366010, 4366014, 4366018, 4366021, 4366022, 4366023, 4366025, 4366030, 4366034, 4366041, 4366042, 4366046, 4366050, 4366055, 4366057, 4366059, 4366062, 4366065, 4366081, 4366087, 4366089, 4366097, 4366098, 4366126, 4366129, 4366136, 4366155, 4366160, 4366165, 4366167, 4366180, 4366182, 4366184, 4366189, 4366192, 4366194, 4366206, 4366219, 4366221, 4366222, 4366223, 4366235, 4366253, 4366268, 4366273, 4366274, 4366275, 4366277, 4366295, 4366298, 4366299, 4366303, 4366310, 4366335}\n"
          ]
        }
      ],
      "source": [
        "## Get the distinct DMSIDs from both datasets\n",
        "docs_dmsids = set(docs_data['DMSId'].unique())\n",
        "downloads_dmsids = set(downloads_data['DMSId'].unique())\n",
        "\n",
        "# Find DMSIDs present in downloads_data but missing in docs_data_df\n",
        "missing_in_docs = downloads_dmsids - docs_dmsids\n",
        "\n",
        "# Find DMSIDs present in docs_data_df but missing in downloads_data\n",
        "missing_in_downloads = docs_dmsids - downloads_dmsids\n",
        "\n",
        "# Output the results\n",
        "print(f\"Number of DMSIDs missing in document dataset: {len(missing_in_docs)}\")\n",
        "if len(missing_in_docs) > 0:\n",
        "    print(\"DMSIDs missing in document dataset:\", missing_in_docs)\n",
        "\n",
        "print(f\"Number of DMSIDs missing in downloads dataset: {len(missing_in_downloads)}\")\n",
        "if len(missing_in_downloads) > 0:\n",
        "    print(\"DMSIDs missing in downloads dataset:\", missing_in_downloads)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pc2X2riB_9FM",
      "metadata": {
        "id": "pc2X2riB_9FM"
      },
      "source": [
        "**Findings**\n",
        "\n",
        "- All DMSIds in the Downloads dataset are appearing on the Doc dataset, confirming the dataset's completeness.\n",
        "\n",
        "- 3094 assets have never been downloaded and this aligns with our expecation that a large proportion of assets have zero downlaods."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PNJDYP9TtmyC",
      "metadata": {
        "id": "PNJDYP9TtmyC"
      },
      "source": [
        "**EDA DMSID in Downloads Table**\n",
        "\n",
        "We want to know what the cumulative downloads look like for each asset.\n",
        "We perform two checks:\n",
        "\n",
        "1. Is there any asset that appears in the download set but not in the document set (consistency check)?\n",
        "2. How many downloads each asset has (frequency study, which could be 0, 1, or many)?\n",
        "\n",
        "**Our expectations are:**\n",
        "- There should be no missing values in the DMSId column, as confirmed in Step 1.\n",
        "- Some DMSId may appear multiple times, indicating the number of downloads per asset.\n",
        "- Assets with more DMSId appearances are more popular.\n",
        "\n",
        "The approach involves counting the frequency of each DMSId to gauge popularity and visualizing the top 20 most downloaded assets using a bar chart for clearer insights.\n",
        "\n",
        "\n",
        "**Reference:**\n",
        "While we use some helper function from tutorial the rest of the code was generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4MO_-ceQtO3g",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4MO_-ceQtO3g",
        "outputId": "d3d8a87c-1794-4edc-c998-5b206b150645"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 4476\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: DMSId\n",
            "4364319    2135\n",
            "4364315    1973\n",
            "4367625    1646\n",
            "4364487    1554\n",
            "4364453    1399\n",
            "Name: count, dtype: int64\n",
            "Number of rows with DMSId values that appear more than once: 116423\n",
            "---------------\n",
            "Data type: <class 'int'>, Count: 118300\n",
            "Sample values: [4369557, 4364453, 4325376]\n",
            "\n",
            "---------------\n",
            "Total number of rows in the Downloads table: 118300\n",
            "Number of missing values in the DMSID column: 0\n",
            "\n",
            "Most Downloaded Assets:\n",
            "        DMSId  Number of Downloads\n",
            "0     4364319                 2135\n",
            "1     4364315                 1973\n",
            "2     4367625                 1646\n",
            "3     4364487                 1554\n",
            "4     4364453                 1399\n",
            "...       ...                  ...\n",
            "3218  4200900                    1\n",
            "3217  4200896                    1\n",
            "3216  4200888                    1\n",
            "3215  4200887                    1\n",
            "4475  4360758                    1\n",
            "\n",
            "[4476 rows x 2 columns]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 63728 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# using default function as a starting point\n",
        "df = pd.DataFrame(downloads_data)\n",
        "eda_describe_column(df,'DMSId')\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'DMSId')\n",
        "print(\"---------------\")\n",
        "\n",
        "# Total number of rows in the Downloads table\n",
        "total_rows = downloads_data['DMSId'].shape[0]\n",
        "print(f\"Total number of rows in the Downloads table: {total_rows}\")\n",
        "\n",
        "# Check for any missing values in the DMSID column\n",
        "missing_values_count = downloads_data['DMSId'].isnull().sum()\n",
        "print(f\"Number of missing values in the DMSID column: {missing_values_count}\")\n",
        "\n",
        "\n",
        "# Cross-check distinct DMSIDs in Downloads vs Documents dataset\n",
        "docs_dmsid_set = set(docs_data['DMSId'])\n",
        "downloads_dmsid_set = set(downloads_data['DMSId'])\n",
        "\n",
        "\n",
        "# Table to visualize popularity (number of downloads per DMSID)\n",
        "popularity_table = downloads_data['DMSId'].value_counts().reset_index()\n",
        "popularity_table.columns = ['DMSId', 'Number of Downloads']\n",
        "\n",
        "\n",
        "# Display the sorted popularity table\n",
        "popularity_table_sorted = popularity_table.sort_values(by='Number of Downloads', ascending=False)\n",
        "print(\"\\nMost Downloaded Assets:\")\n",
        "print(popularity_table_sorted)\n",
        "\n",
        "# Select the top 20 most downloaded assets\n",
        "top_20_popularity = popularity_table_sorted.head(20)\n",
        "\n",
        "# Plot the horizontal bar chart\n",
        "plt.figure(figsize=(10, 8))\n",
        "plt.barh(top_20_popularity['DMSId'].astype(str), top_20_popularity['Number of Downloads'], color='skyblue')\n",
        "plt.xlabel('Number of Downloads')\n",
        "plt.ylabel('DMSId')\n",
        "plt.title('Top 20 Most Downloaded Assets')\n",
        "plt.gca().invert_yaxis()  # Invert y-axis to show the most popular asset at the top\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eiBCvQsZx627",
      "metadata": {
        "id": "eiBCvQsZx627"
      },
      "source": [
        "Findings:\n",
        "- Unique Values: There are 4,476 unique DMSId values in the downloads_data table, meaning that out of all assets, these have been downloaded at least once.\n",
        "- Missing Values: As expected, there are no missing values in the DMSId column.\n",
        "- Frequency: The DMSId 4364319 was downloaded the most (2,135 times), making it the most popular asset. Overall, 116,423 rows had DMSId values that appeared more than once, confirming that multiple downloads are common.\n",
        "\n",
        "- The bar chart clearly shows the top 20 most downloaded assets, and each has more than 500 cummulative downloads.\n",
        "\n",
        "\n",
        "**Implications**:\n",
        "\n",
        "The analysis reveals that asset downloads are **unevenly distributed**, with a few assets dominating user engagement.\n",
        "\n",
        "This suggests that if the Institute aims **to improve member engagement, it’s crucial to identify which assets are likely to become popular and feature them prominently on high-visibility pages**, such as the front page. By doing so, the Institute can better align its offerings with member interests and increase overall engagement.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fLAfnEGvbkKU",
      "metadata": {
        "id": "fLAfnEGvbkKU"
      },
      "source": [
        "**Download distribution**\n",
        "\n",
        "The purpose of examining the DownloadDate column is to understand download behavior over time and identify trends in member engagement. While the previous analysis focused on cumulative downloads, our focus is now on how monthly downloads change over time.\n",
        "\n",
        "Expectation Based on Data Dictionary:\n",
        "- Data Type: Should be datetime, confirmed in Q1a Step 1.\n",
        "- Expected Date Range: Between August 20, 2018 (when the DMS system started) and April 2024 , suggested by Data Dictionary. Any downloads before this date would be an inconsistency.\n",
        "\n",
        "**Visualisation:**\n",
        "-  a histrogam to display the frequency of downloads over time\n",
        "- Identify any downloads dated before the assets's  StartPublishing, as these would be suspect if the download time for an assets is before it's startpublsiing date\n",
        "\n",
        "**Reference:**\n",
        "While we use some helper functions from tutorial the rest of the code was generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wh3KZ5Gv2Bgh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wh3KZ5Gv2Bgh",
        "outputId": "bd5ae31a-8505-45ed-edf9-f8cf073f9b09"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 1632\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: DownloadDate\n",
            "2024-03-15    2253\n",
            "2023-06-08    1036\n",
            "2020-04-17     766\n",
            "2020-04-20     677\n",
            "2023-08-01     661\n",
            "Name: count, dtype: int64\n",
            "Number of rows with DownloadDate values that appear more than once: 118248\n",
            "---------------\n",
            "Data type: <class 'pandas._libs.tslibs.timestamps.Timestamp'>, Count: 118300\n",
            "Sample values: [Timestamp('2021-05-14 00:00:00'), Timestamp('2023-11-22 00:00:00'), Timestamp('2021-07-27 00:00:00')]\n",
            "\n",
            "---------------\n",
            "Number of missing DownloadDate values: 0\n",
            "Number of downloads before 20th August 2018: 0\n",
            "Earliest DownloadDate: 2018-08-24 00:00:00\n",
            "Latest DownloadDate: 2024-04-09 00:00:00\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 27972 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(downloads_data)\n",
        "eda_describe_column(df,'DownloadDate')\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'DownloadDate')\n",
        "print(\"---------------\")\n",
        "\n",
        "# Assuming downloads_data is your DataFrame for the Downloads table\n",
        "downloads_df = pd.DataFrame(downloads_data)\n",
        "\n",
        "# Convert DownloadDate to datetime format\n",
        "downloads_df['DownloadDate'] = pd.to_datetime(downloads_df['DownloadDate'], errors='coerce')\n",
        "\n",
        "# Check for any missing values in DownloadDate\n",
        "missing_download_dates = downloads_df['DownloadDate'].isnull().sum()\n",
        "print(f\"Number of missing DownloadDate values: {missing_download_dates}\")\n",
        "\n",
        "# Find any downloads before the DMS system start date (20th August 2018)\n",
        "invalid_downloads = downloads_df[downloads_df['DownloadDate'] < '2018-08-20']\n",
        "print(f\"Number of downloads before 20th August 2018: {len(invalid_downloads)}\")\n",
        "\n",
        "if len(invalid_downloads) > 0:\n",
        "    print(\"Invalid Download Dates:\")\n",
        "    print(invalid_downloads[['DMSId', 'DownloadDate']])\n",
        "\n",
        "# Check the range of the DownloadDate\n",
        "earliest_download = downloads_df['DownloadDate'].min()\n",
        "latest_download = downloads_df['DownloadDate'].max()\n",
        "print(f\"Earliest DownloadDate: {earliest_download}\")\n",
        "print(f\"Latest DownloadDate: {latest_download}\")\n",
        "\n",
        "# Plot a histogram of DownloadDate\n",
        "plt.figure(figsize=(10, 6))\n",
        "downloads_df['DownloadDate'].hist(bins=50, color='skyblue')\n",
        "plt.title('Distribution of Download Dates')\n",
        "plt.xlabel('Download Date')\n",
        "plt.ylabel('Frequency')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dH8szdc-BF1O",
      "metadata": {
        "id": "dH8szdc-BF1O"
      },
      "source": [
        "Findings:\n",
        "- Data Type: The DownloadDate is correctly stored as datetime, matching our expectations.\n",
        "- Date Range: The earliest download was on August 24, 2018, and the latest on April 9, 2024, which aligns with the data dictionary.\n",
        "\n",
        "\n",
        "**Insights:**\n",
        "The histogram shows fluctuating download activity over time:\n",
        "- There were limited downloads before 2020,\n",
        "- download increased and stabilization from mid-2020 onward.\n",
        "- After the peak in year 2020, there appears to be a gradual decline in overall download volume.\n",
        "\n",
        "\n",
        "**Implications**:\n",
        "1. This trend indicates that download data before mid-2020 is sparse and may not be representative for popularity analysis in Q3. Therefore, only the period from mid-2020 onwards should be considered for analyzing asset popularity.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "B_2BEI-52hwm",
      "metadata": {
        "id": "B_2BEI-52hwm"
      },
      "source": [
        "**Seasonality analysis:**\n",
        "\n",
        "We are investigating the monthly seasonality by examining how many downloads occur in each calendar month.\n",
        "\n",
        "Why: This analysis aims to identify trends in member behavior throughout the year, which can be valuable feature for later classification model.\n",
        "\n",
        "Reference: follwoing code was generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "71gIWOohbuA9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "71gIWOohbuA9",
        "outputId": "1eca48fa-2e44-4b3d-ab34-a94911c99a77"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 34236 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a plot , i want to see monthly seasonality of dowload data. so 12 bar, average downlaod of each month, Jan ,Feb, Mar ...\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "# Convert 'DownloadDate' to datetime objects if not already\n",
        "downloads_data['DownloadDate'] = pd.to_datetime(downloads_data['DownloadDate'])\n",
        "\n",
        "# Extract the month from the 'DownloadDate' column\n",
        "downloads_data['Month'] = downloads_data['DownloadDate'].dt.month\n",
        "\n",
        "# Group the data by month and calculate the average downloads for each month\n",
        "monthly_downloads = downloads_data.groupby('Month')['DMSId'].count().reset_index()\n",
        "\n",
        "# Rename the column for clarity\n",
        "monthly_downloads = monthly_downloads.rename(columns={'DMSId': 'Average Downloads'})\n",
        "\n",
        "# Create a bar plot\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.bar(monthly_downloads['Month'], monthly_downloads['Average Downloads'])\n",
        "\n",
        "# Customize the plot\n",
        "plt.xlabel('Month')\n",
        "plt.ylabel('Average Downloads')\n",
        "plt.title('Monthly Seasonality of Downloads')\n",
        "plt.xticks(monthly_downloads['Month'], ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'])\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8F5Reytt2s9n",
      "metadata": {
        "id": "8F5Reytt2s9n"
      },
      "source": [
        "**Findings:**\n",
        "\n",
        "Seasonal Peaks:\n",
        "- The highest average downloads occur in September and August, followed by December, indicating peak engagement periods.\n",
        "- The months with the lowest average downloads are February and November\n",
        "\n",
        "\n",
        "**Implications:**\n",
        "\n",
        "- We need to ensure each data partition set contains at least 12 months of data so that we can capture monthly seasonality in the later model learning process.\n",
        "\n",
        "- We should consider month seasonality as a feature in the later Q2 and Q3 feature engineering, as there is a clear pattern.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "enuYZWqF2Xgv",
      "metadata": {
        "id": "enuYZWqF2Xgv"
      },
      "source": [
        "**Examining Inconsistent StartPublishing Dates**\n",
        "\n",
        "Purpose: Before analyzing data or concept drift, we need to ensure the StartPublishing date is accurate. Incorrect StartPublishing dates (e.g., set after the first download) could lead to misleading insights, such as incorrectly suggesting negative recency for assets.\n",
        "\n",
        "How:\n",
        "We cross-examine the DownloadDate against the StartPublishing date to check for logical inconsistencies. This step ensures that assets were not downloaded before they were officially published, which would indicate data quality issues.\n",
        "\n",
        "Approach:\n",
        "Merging: Merge downloads_data with docs_data using DMSId to bring in the corresponding StartPublishing dates.\n",
        "Comparison: Compare each DownloadDate with its StartPublishing date to identify any records where downloads occurred before the asset was published.\n",
        "\n",
        "Reference: follwoing code was generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "N5hVfu2S4gXK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N5hVfu2S4gXK",
        "outputId": "66ad52eb-a06e-4e6a-e4ba-0a9ad1d8b52a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of downloads before the asset's StartPublishing date: 49\n",
            "Downloads before StartPublishing Date:\n",
            "          DMSId DownloadDate StartPublishing\n",
            "4       4194310   2018-09-21      2019-01-07\n",
            "5       4194310   2018-09-07      2019-01-07\n",
            "6       4194310   2018-09-04      2019-01-07\n",
            "7       4194310   2018-09-04      2019-01-07\n",
            "307     4197902   2018-09-21      2018-09-22\n",
            "2086    4201738   2018-09-21      2018-09-22\n",
            "2091    4201740   2018-09-21      2018-09-22\n",
            "2092    4201740   2018-09-21      2018-09-22\n",
            "2097    4201743   2018-09-21      2018-09-22\n",
            "2098    4201743   2018-09-21      2018-09-22\n",
            "2100    4201744   2018-09-21      2018-09-22\n",
            "2101    4201745   2018-09-21      2018-09-22\n",
            "2120    4201749   2018-09-21      2018-09-22\n",
            "2128    4201751   2018-09-21      2018-09-22\n",
            "2146    4201753   2018-09-21      2018-09-22\n",
            "2150    4201755   2018-09-21      2018-09-22\n",
            "2152    4201757   2018-09-21      2018-09-22\n",
            "2162    4201767   2018-09-21      2018-09-22\n",
            "2285    4202115   2018-09-21      2018-09-23\n",
            "2286    4202115   2018-09-21      2018-09-23\n",
            "3163    4207831   2018-09-09      2018-11-17\n",
            "3164    4207831   2018-09-07      2018-11-17\n",
            "3730    4210036   2018-09-06      2018-11-19\n",
            "3736    4210046   2018-09-06      2018-11-17\n",
            "3741    4210064   2018-09-07      2018-11-17\n",
            "3780    4210168   2019-01-21      2019-01-29\n",
            "3807    4210243   2020-03-02      2021-08-10\n",
            "3809    4210249   2019-01-17      2019-01-29\n",
            "3819    4210281   2019-01-16      2019-01-29\n",
            "3831    4210315   2020-03-02      2021-08-10\n",
            "3842    4210350   2020-03-02      2021-08-10\n",
            "3908    4227079   2020-03-02      2021-08-10\n",
            "3913    4227092   2019-01-17      2019-01-18\n",
            "3914    4227095   2019-01-17      2019-01-18\n",
            "6786    4325432   2018-11-23      2019-03-16\n",
            "6812    4325435   2018-11-13      2019-03-16\n",
            "7003    4325462   2018-11-13      2019-01-22\n",
            "9378    4358675   2019-01-24      2019-03-16\n",
            "31628   4363506   2020-10-26      2021-08-10\n",
            "32750   4363628   2020-10-26      2023-08-01\n",
            "32751   4363629   2020-10-26      2021-12-05\n",
            "33360   4364268   2020-10-26      2021-02-22\n",
            "42615   4364468   2020-10-26      2022-01-20\n",
            "70110   4365763   2021-10-04      2023-08-01\n",
            "73274   4365949   2021-10-04      2024-01-10\n",
            "73968   4366052   2023-11-14      2024-01-10\n",
            "73969   4366052   2021-10-04      2024-01-10\n",
            "74003   4366066   2021-10-04      2024-01-10\n",
            "108609  4369372   2023-07-29      2023-07-31\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Cross examination of StartPublishing  with DownloadDate\n",
        "\n",
        "# Merge the downloads_df with the docs_data_df to get the corresponding StartPublishing dates\n",
        "merged_df = downloads_df.merge(docs_data[['DMSId', 'StartPublishing']], on='DMSId', how='left')\n",
        "\n",
        "# Identify any downloads before the asset's StartPublishing date\n",
        "downloads_before_start = merged_df[merged_df['DownloadDate'] < merged_df['StartPublishing']]\n",
        "\n",
        "# Display the results\n",
        "print(f\"Number of downloads before the asset's StartPublishing date: {len(downloads_before_start)}\")\n",
        "if len(downloads_before_start) > 0:\n",
        "    print(\"Downloads before StartPublishing Date:\")\n",
        "    print(downloads_before_start[['DMSId', 'DownloadDate', 'StartPublishing']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PtR6S2tc5z_i",
      "metadata": {
        "id": "PtR6S2tc5z_i"
      },
      "source": [
        "**Findings:**\n",
        "There were 49 instances where the **DownloadDate occurred before the asset's StartPublishing date**, suggesting data inconsistencies.\n",
        "\n",
        "**We believe:**\n",
        "DownloadDate is system-generated and generally more reliable, while the StartPublishing date is usually manually input, so it might contain human errors.\n",
        "\n",
        "**Implications:**\n",
        "- Data Cleaning: We recommend adjusting the StartPublishing dates to align with the DownloadDate for these 49 instances. DownloadDate is likely to be more accurate as it is usually auto-generated from computer system whille StartPublishing dates are more subject to humman manual error.\n",
        "\n",
        "-\n",
        "This step should be done before splitting the data into training, validation, and testing sets to ensure consistency throughout the dataset.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aodEgZ5KXjtz",
      "metadata": {
        "id": "aodEgZ5KXjtz"
      },
      "source": [
        "Summary\n",
        "\n",
        "- DMSId values in downloads matched with docs_data, confirming accurate links between datasets. However, the discovery of 49 instances where downloads occurred before the StartPublishing date highlighted inconsistencies that must be corrected to maintain data integrity.\n",
        "\n",
        "-  We observed clear peaks in download activity, especially starting in 2020, which then stabilized. This suggests shifts in member engagement that could be tied to significant events, content releases, or external factors, giving us a strong foundation for understanding popularity patterns.\n",
        "\n",
        "- Monthly seasonality was evident, with peak downloads in August, September, and December, while February and November showed lower engagement. This insight is essential for planning future content releases to align with periods of high member activity.\n",
        "\n",
        "- The DownloadDate column was complete, covering the expected range from August 2018 to April 2024, ensuring we have a full and accurate timeline for analysis.\n",
        "\n",
        "\n",
        "Implications: This EDA lays the groundwork for precise data partitioning and feature engineering in Q3. By understanding trends, seasonality, and data inconsistencies, we’re better equipped to model asset popularity and member engagement accurately, ensuring the Institute's strategies are data-driven and effective."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2e1QAAj0_MWa",
      "metadata": {
        "id": "2e1QAAj0_MWa"
      },
      "source": [
        "####Q1a Step3 Data Cleaning Before splitting data\n",
        "\n",
        "Purpose: We are addressing data issues at this early stage to ensure consistency in our analysis of concept drift and data drift. By performing these cleaning steps before splitting the data, we create a more reliable foundation for subsequent modeling and analysis.\n",
        "\n",
        "Data Cleaning Steps Taken:\n",
        "1. Duplication Removal\n",
        "2. StartPublishing Date Consistency:"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "op_FCAg-n6Nb",
      "metadata": {
        "id": "op_FCAg-n6Nb"
      },
      "source": [
        "##### Q1a Step3.1 - Duplicates cleaning\n",
        "\n",
        "**Outline**\n",
        "\n",
        "Purpose: Duplicates can give a false impression of model performance. For example, if a duplicate gets split between training and validation sets, we won't be able to identify it later, leading to misleading performance results.\n",
        "\n",
        "Our logic:\n",
        "- The risk of duplicates causing major issues is high, so we chose to remove all duplicates now to ensure accurate and unbiased analysis and model training before partitioning the data.\n",
        "\n",
        "Reference:\n",
        "All codes were generated using ChatGPT-4, where we provided the AI with our purpose and the comparison we wanted to perform.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FVmyRyEgYsZ7",
      "metadata": {
        "id": "FVmyRyEgYsZ7"
      },
      "source": [
        "**Examining Duplicates Based on Links**\n",
        "\n",
        "Purpose: We will use the Link column to check whether an asset is duplicated.\n",
        "\n",
        "\n",
        "**Rationale for Using Only the Link Column:**\n",
        "We use only Link column as the  indicator of duplication for the following reasons:\n",
        "\n",
        "- Links are unique, and each identical link refers to the same asset. It is a logical and reliable way to identify duplicates.\n",
        "- Unlike columns like Title or Description, which may have slight variations, the Link is an absolute reference to the asset’s location.\n",
        "\n",
        "**Why not add the Link with other variables?**\n",
        "Using only the Link avoids confusion, as two assets might have slightly different file names but still point to the same asset. Adding other variables could mistakenly consider these as separate assets, while the Link ensures accurate identification.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zTGLAKoEWXxY",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 720
        },
        "id": "zTGLAKoEWXxY",
        "outputId": "daa7d1d8-74f3-49f9-c58d-03a261877676"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "------------------\n",
            "\n",
            "Summary of Link EDA:\n",
            "Total links: 7570\n",
            "Unique links: 7551\n",
            "Missing links: 0\n",
            "Number of duplicate pairs: 13\n",
            "Number of duplicate links: 32\n",
            "Number of invalid links: 0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 105360 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "def link_eda(df):\n",
        "    # 1. Check for missing values in the Link column\n",
        "    missing_links = df['Link'].isnull().sum()\n",
        "\n",
        "    # 2. Check for unique links\n",
        "    unique_links = df['Link'].nunique()\n",
        "    total_links = df['Link'].count()\n",
        "\n",
        "    # 3. Identify duplicate links\n",
        "    duplicate_links = df[df.duplicated(subset=['Link'], keep=False)]\n",
        "\n",
        "    # 4. Check for valid links (basic validation for 'http://' or 'https://')\n",
        "    valid_links = df['Link'].apply(lambda x: x.startswith(('http://', 'https://')) if pd.notnull(x) else False)\n",
        "    invalid_links = df[~valid_links]\n",
        "\n",
        "    # 5. Count the number of unique links that are duplicated\n",
        "    duplicated_pairs_count = duplicate_links['Link'].nunique()\n",
        "\n",
        "    # Display the result\n",
        "    print(\"------------------\")\n",
        "    # Summary\n",
        "    print(\"\\nSummary of Link EDA:\")\n",
        "    print(f\"Total links: {total_links}\")\n",
        "    print(f\"Unique links: {unique_links}\")\n",
        "    print(f\"Missing links: {missing_links}\")\n",
        "    print(f\"Number of duplicate pairs: {duplicated_pairs_count}\")\n",
        "    print(f\"Number of duplicate links: {len(duplicate_links)}\")\n",
        "    print(f\"Number of invalid links: {len(invalid_links)}\")\n",
        "\n",
        "    # Bar chart to show the frequency of duplication\n",
        "    # Group by 'Link' and count the occurrences\n",
        "    duplicate_links_grouped = df.groupby('Link').filter(lambda x: len(x) > 1)\n",
        "\n",
        "    # Count the frequency of each duplicated link\n",
        "    link_counts = duplicate_links_grouped.groupby('Link').size().reset_index(name='Frequency')\n",
        "\n",
        "    # Sort the links by frequency for better visualization\n",
        "    link_counts = link_counts.sort_values(by='Frequency', ascending=False)\n",
        "\n",
        "    # Plotting the bar chart\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    plt.barh(link_counts['Link'], link_counts['Frequency'], color='skyblue')\n",
        "    plt.xlabel('Frequency')\n",
        "    plt.ylabel('Link')\n",
        "    plt.title('Frequency of Duplicated Links')\n",
        "    plt.gca().invert_yaxis()  # Invert y-axis to have the most frequent on top\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "df = pd.DataFrame(docs_data)\n",
        "link_eda(df)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yLydlslXXe-M",
      "metadata": {
        "id": "yLydlslXXe-M"
      },
      "source": [
        "**Finding:**\n",
        "\n",
        "Out of a total of 7,570 links, 7,551 are unique, with no missing or invalid entries. We identified 32 duplicate entries, originating from 13 pairs of duplicated links.\n",
        "\n",
        "The use of links as an indicator of duplication is reliable, as identical links point to the exact same asset, regardless of any differences in other columns, such as titles or descriptions. This confirms that our approach to cleaning duplicates is accurate and ensures that only truly distinct assets are retained for analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fbFjXiGPXAhQ",
      "metadata": {
        "id": "fbFjXiGPXAhQ"
      },
      "source": [
        "**Next Steps:**  Removing Duplicates Based on Links\n",
        "\n",
        "Purpose: The goal here is to create a mapping to identify which DMSId to retain and which to drop among the identified duplicate pairs. This step ensures that only one version of each duplicated asset remains, thereby maintaining data integrity and preventing redundancy.\n",
        "\n",
        "For example, in the pair [4210076, 4362059], 4210076 is retained while 4362059 will be dropped."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "vXJoUb1A7wk_",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vXJoUb1A7wk_",
        "outputId": "4260a2f8-a64e-4738-9869-ee0c6dbc1d0b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of Duplicated Pairs: 13\n",
            "Duplicated Pairs:\n",
            "                                   DMSId\n",
            "0                     [4210076, 4362059]\n",
            "1                     [4362808, 4362809]\n",
            "2                     [4367717, 4367718]\n",
            "3   [4367480, 4367481, 4367482, 4367483]\n",
            "4                     [4366480, 4366481]\n",
            "5                     [4369493, 4369495]\n",
            "6   [4367407, 4367408, 4367409, 4367410]\n",
            "7                     [4366527, 4366528]\n",
            "8                     [4366494, 4366495]\n",
            "9                     [4366513, 4366515]\n",
            "10                    [4366432, 4366433]\n",
            "11  [4367403, 4367404, 4367405, 4367406]\n",
            "12                    [4365953, 4366474]\n",
            "-----------------------\n",
            "Table of DMSIDs to Keep and Drop:\n",
            "                                                 Link     Keep     Drop\n",
            "0   https://www.actuaries.digital/2018/06/28/fast-...  4210076  4362059\n",
            "1   https://www.actuaries.digital/2019/10/31/the-c...  4362808  4362809\n",
            "2   https://www.actuaries.digital/2022/04/30/actua...  4367717  4367718\n",
            "3                        https://youtu.be/3bULhUCO6-k  4367480  4367481\n",
            "4                        https://youtu.be/3bULhUCO6-k  4367480  4367482\n",
            "5                        https://youtu.be/3bULhUCO6-k  4367480  4367483\n",
            "6                        https://youtu.be/5U3D8V_Khn8  4366480  4366481\n",
            "7                        https://youtu.be/EXk1S-gn3dA  4369493  4369495\n",
            "8                        https://youtu.be/X9__Ra9qEHI  4367407  4367408\n",
            "9                        https://youtu.be/X9__Ra9qEHI  4367407  4367409\n",
            "10                       https://youtu.be/X9__Ra9qEHI  4367407  4367410\n",
            "11                       https://youtu.be/d3x0WdwBVXo  4366527  4366528\n",
            "12                       https://youtu.be/d8pStyun300  4366494  4366495\n",
            "13                       https://youtu.be/edMcftRCRsU  4366513  4366515\n",
            "14                       https://youtu.be/lX-pP742M0Q  4366432  4366433\n",
            "15                       https://youtu.be/pMIktM8h1LA  4367403  4367404\n",
            "16                       https://youtu.be/pMIktM8h1LA  4367403  4367405\n",
            "17                       https://youtu.be/pMIktM8h1LA  4367403  4367406\n",
            "18                       https://youtu.be/qN-Uo0D5ki0  4365953  4366474\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_data)\n",
        "\n",
        "# Identify duplicate links\n",
        "duplicate_links = df[df.duplicated(subset=['Link'], keep=False)]\n",
        "\n",
        "# Create a table of matching duplicates\n",
        "duplicated_pairs = duplicate_links.sort_values('DMSId').groupby(['Link']).agg({\n",
        "    'DMSId': lambda x: list(x)\n",
        "}).reset_index()\n",
        "\n",
        "# Display the duplicated pairs\n",
        "print(\"Number of Duplicated Pairs:\", len(duplicated_pairs))\n",
        "print(\"Duplicated Pairs:\")\n",
        "print(duplicated_pairs[['DMSId']])\n",
        "\n",
        "print('-----------------------')\n",
        "# Create the retain_to_drop_mapping\n",
        "retain_to_drop_mapping = {}\n",
        "rows = []\n",
        "for index, row in duplicated_pairs.iterrows():\n",
        "    retain_dmsid = row['DMSId'][0]\n",
        "    drop_dmsids = row['DMSId'][1:]\n",
        "    for drop_dmsid in drop_dmsids:\n",
        "        retain_to_drop_mapping[drop_dmsid] = retain_dmsid\n",
        "        rows.append({'Link': row['Link'], 'Keep': retain_dmsid, 'Drop': drop_dmsid})\n",
        "\n",
        "# Convert the list of rows to a DataFrame\n",
        "retain_drop_df = pd.DataFrame(rows)\n",
        "\n",
        "# Display the table showing which DMSIDs to keep and which to drop\n",
        "print(\"Table of DMSIDs to Keep and Drop:\")\n",
        "print(retain_drop_df)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ekgTrc5MLVEf",
      "metadata": {
        "id": "ekgTrc5MLVEf"
      },
      "source": [
        "**Remove of duplication**:\n",
        "The purpose of the following step is to remove rows from the docs_data table that contain DMSId entries identified for removal in the previous step. This ensures that our dataset is free from duplicate assets, based on the mapping created earlier.\n",
        "\n",
        "\n",
        "Below codes are doing the following:\n",
        "- Filtering the Data: We filtered out the rows where DMSId matches any entry in our retain_to_drop_mapping dictionary.\n",
        "This effectively drops the duplicated entries, retaining only the first instance of each unique link.\n",
        "\n",
        "- Verification Using link_eda:We re-ran the link_eda function to ensure all duplicates were removed.\n",
        "The summary is expected to confirm that there are now 7,551 total links, all of which are unique, with no missing or invalid entries.\n",
        "\n",
        "- Final Check for Duplicates: We performed a final check to ensure no remaining duplicates based on Link and Tags columns.\n",
        "The output confirms that there are no duplicates left in the cleaned documents table.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "V_-3QtZrV1Jz",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 738
        },
        "id": "V_-3QtZrV1Jz",
        "outputId": "1698808c-8e02-45e7-a9d2-12fe65c413ec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "------------------\n",
            "\n",
            "Summary of Link EDA:\n",
            "Total links: 7551\n",
            "Unique links: 7551\n",
            "Missing links: 0\n",
            "Number of duplicate pairs: 0\n",
            "Number of duplicate links: 0\n",
            "Number of invalid links: 0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 21752 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "No duplicates found in the cleaned documents table.\n"
          ]
        }
      ],
      "source": [
        "# Step 3: Adjust the Documents Table\n",
        "# Drop the rows that have DMSIds in the 'drop_dmsids' list\n",
        "docs_data_cleaned = docs_data[~docs_data['DMSId'].isin(retain_to_drop_mapping.keys())]\n",
        "\n",
        "# Check\n",
        "df = pd.DataFrame(docs_data_cleaned)\n",
        "link_eda(df)\n",
        "\n",
        "# Check - Ensure no more duplicates in the cleaned documents table\n",
        "final_duplicates = docs_data_cleaned.duplicated(subset=['Link', 'Tags'], keep=False)\n",
        "\n",
        "if final_duplicates.any():\n",
        "    print(\"Warning: There are still duplicates in the cleaned documents table.\")\n",
        "    print(docs_data_cleaned[final_duplicates][['DMSId', 'Title', 'FileSize', 'Link', 'Tags']])\n",
        "else:\n",
        "    print(\"No duplicates found in the cleaned documents table.\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "33_4TlGAZjHf",
      "metadata": {
        "id": "33_4TlGAZjHf"
      },
      "source": [
        "Check :\n",
        "The cleaning process is appear to align with our expecation.The cleaned docs_data_cleaned dataset is now free from duplicate entries, ensuring data consistency and accuracy for future analysis or modeling steps. This adjustment helps prevent data leakage and ensures that our insights and models are based on unique and distinct assets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "GKYNZjP4ZCmL",
      "metadata": {
        "id": "GKYNZjP4ZCmL"
      },
      "source": [
        "**Adjust downloads table for deleted DMSID**\n",
        "\n",
        "**Purpose:**\n",
        "The purpose of this step is to adjust the downloads table to reflect changes made in the docs_data_cleaned table. Since some DMSId entries were removed from the documents table, we need to replace those deleted DMSId values with their retained counterparts. This ensures that the downloads data remains consistent and can be correctly linked to the cleaned docs_data.\n",
        "\n",
        "What the following codes do:\n",
        "- Replacing DMSIds: We made a copy of the downloads_df to create downloads_clean.\n",
        "Using the retain_to_drop_mapping, we replaced any DMSId values that were marked for deletion with their corresponding retained DMSId.\n",
        "\n",
        "- Checks: We confirm that all DMSId entries in the downloads_clean table are now present in docs_data_cleaned.  All downloads should be linking to valid documents.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "MHPAvgdQZAHv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MHPAvgdQZAHv",
        "outputId": "804fb158-a3f3-4e8e-e842-f39a1ac5a326"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-------------------\n",
            "Check if any DMSId in the Downloads table does not exist in the cleaned documents table:\n",
            "All DMSIds in the Downloads table are present in the cleaned documents table.\n",
            "-------------------\n",
            "-------------------\n",
            "Count how many DMSIds were adjusted in the Downloads table:\n",
            "Number of DMSIds adjusted in the Downloads table: 412\n"
          ]
        }
      ],
      "source": [
        "# Step 1: Adjust the Downloads Table\n",
        "# Replace dropped DMSIds with the retained DMSIds\n",
        "downloads_clean = downloads_df.copy()\n",
        "downloads_clean['DMSId'] = downloads_clean['DMSId'].replace(retain_to_drop_mapping)\n",
        "\n",
        "# Check\n",
        "# Ensure that no DMSId in the Downloads table is missing in the Documents table\n",
        "downloads_dmsids_cleaned = set(downloads_clean['DMSId'])\n",
        "docs_dmsids_cleaned = set(docs_data_cleaned['DMSId'])\n",
        "\n",
        "# Check if any DMSId in the Downloads table does not exist in the cleaned documents table\n",
        "print('-------------------')\n",
        "print('Check if any DMSId in the Downloads table does not exist in the cleaned documents table:')\n",
        "missing_dmsids_cleaned = downloads_dmsids_cleaned - docs_dmsids_cleaned\n",
        "\n",
        "if missing_dmsids_cleaned:\n",
        "    print(f\"Missing DMSIds in the cleaned documents table: {missing_dmsids_cleaned}\")\n",
        "else:\n",
        "    print(\"All DMSIds in the Downloads table are present in the cleaned documents table.\")\n",
        "print('-------------------')\n",
        "\n",
        "# Count how many DMSIds were adjusted in the Downloads table\n",
        "adjusted_dmsids = downloads_clean['DMSId'].isin(retain_to_drop_mapping.values()).sum()\n",
        "print('-------------------')\n",
        "print('Count how many DMSIds were adjusted in the Downloads table:')\n",
        "print(f\"Number of DMSIds adjusted in the Downloads table: {adjusted_dmsids}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xO7HDAWdfi9g",
      "metadata": {
        "id": "xO7HDAWdfi9g"
      },
      "source": [
        "The downloads_clean table is now fully aligned with the docs_data_cleaned table, ensuring that every download record has a corresponding asset entry in the documents dataset. This step is crucial for maintaining data integrity and ensuring the accuracy of any subsequent analysis or modeling efforts that rely on both the documents and downloads data."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "erXty42h1Z6G",
      "metadata": {
        "id": "erXty42h1Z6G"
      },
      "source": [
        "##### Q1a Step 3.2 - Inconsistency cleaning -  StartPublishing Date After Download Date\n",
        "**Reason of cleaning inconsistent dates before splitting:**\n",
        "\n",
        "Purpose of Cleaning Inconsistent Dates Before Splitting:\n",
        "\n",
        "- We want to ensure there is no noise that could distort the analysis of how assets were published over time. Accurate download data is essential for conducting a reliable data drift analysis.\n",
        "\n",
        "- We plan to partition the data using the publication date due to observed data drift. It's crucial to have the publication dates correct before this step to avoid errors in the analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TAkAndkZgRzj",
      "metadata": {
        "id": "TAkAndkZgRzj"
      },
      "source": [
        "The following codes are doing:\n",
        "\n",
        "1. Adjusting Inconsistent StartPublishing Dates: We created a dictionary, earliest_download_dates, to capture the earliest DownloadDate for each DMSId. Using the apply function, we adjusted the StartPublishing date to be the earlier of either the original StartPublishing date or the earliest DownloadDate.\n",
        "\n",
        "2. Handling the DataFrame Without Warnings: The code utilized .loc to avoid the common SettingWithCopyWarning, ensuring that changes were made directly and accurately.\n",
        "\n",
        "3. Dropping Temporary Columns:\n",
        "We dropped the temporary AdjustedStartPublishing column once the adjustments were complete, maintaining a clean dataset.\n",
        "\n",
        "Reference: the following codes are generated by ChatGPT4, with the instruction of correct Startpublishing dates based on the earliest download dates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "KpDMa3u02uKs",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "KpDMa3u02uKs",
        "outputId": "8ffe6567-54f6-4f47-c64d-25526365c3e1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Adjusted StartPublishing dates where necessary:\n",
            "     DMSId                                              Title StartPublishing\n",
            "0  4194310                    International CPD opportunities      2018-09-04\n",
            "1  4194590  Disability Committee - 1995 Report of the Disa...      2018-08-20\n",
            "2  4194592  Australian Actuarial Journal 1997 (March Edition)      2018-08-20\n",
            "3  4194593  Australian Actuarial Journal 1997 (September E...      2018-08-20\n",
            "4  4194594  Disability Committee - The 1997 Report into mo...      2018-08-20\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-42-7c6211df80b6>:5: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_data_cleaned.loc[:, 'AdjustedStartPublishing'] = docs_data_cleaned.apply(\n",
            "<ipython-input-42-7c6211df80b6>:15: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_data_cleaned.drop(columns=['AdjustedStartPublishing'], inplace=True)\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Create a dictionary with the earliest DownloadDate for each DMSId\n",
        "earliest_download_dates = downloads_clean.groupby('DMSId')['DownloadDate'].min().to_dict()\n",
        "\n",
        "# Adjust 'StartPublishing' dates using .loc to avoid SettingWithCopyWarning\n",
        "docs_data_cleaned.loc[:, 'AdjustedStartPublishing'] = docs_data_cleaned.apply(\n",
        "    lambda row: min(row['StartPublishing'], earliest_download_dates[row['DMSId']])\n",
        "    if row['DMSId'] in earliest_download_dates else row['StartPublishing'],\n",
        "    axis=1\n",
        ")\n",
        "\n",
        "# Replace the original 'StartPublishing' with the adjusted dates\n",
        "docs_data_cleaned.loc[:, 'StartPublishing'] = docs_data_cleaned['AdjustedStartPublishing']\n",
        "\n",
        "# Drop the temporary adjustment column\n",
        "docs_data_cleaned.drop(columns=['AdjustedStartPublishing'], inplace=True)\n",
        "\n",
        "# Display a summary to verify changes\n",
        "print(\"Adjusted StartPublishing dates where necessary:\")\n",
        "print(docs_data_cleaned[['DMSId', 'Title', 'StartPublishing']].head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "NW1DxcZo5Bbi",
      "metadata": {
        "id": "NW1DxcZo5Bbi"
      },
      "source": [
        "**Check to ensure no date inconsistency**\n",
        "\n",
        "The purpose of the following check is to validate that there are no remaining inconsistencies between the StartPublishing date and DownloadDate. This ensures that no asset is marked as being downloaded before its official publication date, thereby confirming the integrity of our cleaned dataset.\n",
        "\n",
        "We expect zero assets to have publishing date before its earliest donwload after the above cleaning step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2AJdLHx431Ss",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2AJdLHx431Ss",
        "outputId": "c9074e2a-50cc-4ff9-8733-54ec10dc48f8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of downloads before the asset's StartPublishing date: 0\n"
          ]
        }
      ],
      "source": [
        "# Cross examination of StartPublishing  with DownloadDate\n",
        "\n",
        "# Merge the downloads_df with the docs_data_df to get the corresponding StartPublishing dates\n",
        "merged_df = downloads_df.merge(docs_data_cleaned[['DMSId', 'StartPublishing']], on='DMSId', how='left')\n",
        "\n",
        "# Identify any downloads before the asset's StartPublishing date\n",
        "downloads_before_start = merged_df[merged_df['DownloadDate'] < merged_df['StartPublishing']]\n",
        "\n",
        "# Display the results\n",
        "print(f\"Number of downloads before the asset's StartPublishing date: {len(downloads_before_start)}\")\n",
        "if len(downloads_before_start) > 0:\n",
        "    print(\"Downloads before StartPublishing Date:\")\n",
        "    print(downloads_before_start[['DMSId', 'DownloadDate', 'StartPublishing']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MsNxhNv1hdrv",
      "metadata": {
        "id": "MsNxhNv1hdrv"
      },
      "source": [
        "Findings:\n",
        "The output shows that there are 0 downloads before the asset's StartPublishing date.\n",
        "This confirms that all inconsistencies have been successfully corrected, and there are no remaining logical errors in terms of publication and download timelines."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "q_i3FT5Ahe6S",
      "metadata": {
        "id": "q_i3FT5Ahe6S"
      },
      "source": [
        "**Summary of Cleaning Section**\n",
        "\n",
        "We tackled two key issues: duplicate entries and date inconsistencies.\n",
        "\n",
        "Duplicate Removal: We removed 32 duplicate entries based on the Link column, ensuring each asset appears only once, which prevents data leakage and keeps our analysis accurate.\n",
        "\n",
        "Date Consistency: Adjusted 49 assets where downloads occurred before their StartPublishing date, aligning dates to ensure logical consistency.\n",
        "\n",
        "Impact: These steps clean and streamline our dataset, setting the stage for accurate trend analysis and preventing bias in future modeling."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zmZIyPwBzMTW",
      "metadata": {
        "id": "zmZIyPwBzMTW"
      },
      "source": [
        "#### Q1a Step4 - EDA and Cleaning of StartPublishing Dates\n",
        "\n",
        " Inconsistency Cleaning:  StartPublishing Date Before 2018\n",
        "\n",
        "**What are we looking:**\n",
        "We examinate the StartPublishing dates to identify any assets published before 2018, which is the assumed earliest possible date for DMS entries. This step ensures our dataset remains accurate and aligns with the introduction of the DMS system in August 2018.\n",
        "\n",
        "Expectation: Based on the data dictionary, we do not expect to see any data published before 2018."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "SlWPSbVO1xKr",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 602
        },
        "id": "SlWPSbVO1xKr",
        "outputId": "e3dc1f5e-acab-4bd5-9d65-076eb0f757be"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of assets with StartPublishing date before DMS start date before fix: 1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 32652 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Define the DMS start date\n",
        "dms_start_date = pd.to_datetime(\"2018-08-01\")\n",
        "# Check if the correction was successful\n",
        "before_fix = sum(docs_data['StartPublishing'] < dms_start_date)\n",
        "print(f\"Number of assets with StartPublishing date before DMS start date before fix: {before_fix}\")\n",
        "\n",
        "# Visualization: Distribution of StartPublishing dates\n",
        "plt.figure(figsize=(10, 6))\n",
        "docs_data['StartPublishing'].dt.year.value_counts().sort_index().plot(kind='bar', color='skyblue')\n",
        "plt.title('Distribution of StartPublishing Dates (Pre-Fix)')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Number of Assets')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3ZibHkNNj0N1",
      "metadata": {
        "id": "3ZibHkNNj0N1"
      },
      "source": [
        "The key observation here is the presence of assets published in 2001, which is inconsistent with the expected earliest start date of August 2018. This confirms the need to adjust those dates to ensure our data accurately reflects the DMS system's timeline."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4RZVwTGzj4Ls",
      "metadata": {
        "id": "4RZVwTGzj4Ls"
      },
      "source": [
        "**Cleaning Assets with StartPublishing Dates Before 2018**\n",
        "\n",
        "**Key Assumption:**\n",
        "We set August 1, 2018, as the earliest possible publishing date, based on our domaint knowledge that the DMS syste was introduced in August 2018.\n",
        "\n",
        " This assumption allows us to align all assets with the correct start timeline, ensuring consistency and accuracy for subsequent analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "K9eQeNoI21qH",
      "metadata": {
        "id": "K9eQeNoI21qH"
      },
      "source": [
        "We now perfrom the data clearning and also provide a plot to check the data cleaning step is done correctly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0RWnC0O62wcb",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 811
        },
        "id": "0RWnC0O62wcb",
        "outputId": "032c26fe-afc8-4e5a-e000-75213d90797e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of assets with StartPublishing date before DMS start date after fix: 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-45-366d5d7ae5b5>:6: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_data_cleaned['StartPublishing'] = pd.to_datetime(docs_data_cleaned['StartPublishing'])\n",
            "<ipython-input-45-366d5d7ae5b5>:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_data_cleaned['StartPublishing'] = docs_data_cleaned['StartPublishing'].apply(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 32328 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Assuming docs_data_cleaned is your DataFrame\n",
        "# Define the DMS start date\n",
        "dms_start_date = pd.to_datetime(\"2018-08-01\")\n",
        "\n",
        "# Fix StartPublishing dates that are before the DMS start date\n",
        "docs_data_cleaned['StartPublishing'] = pd.to_datetime(docs_data_cleaned['StartPublishing'])\n",
        "docs_data_cleaned['StartPublishing'] = docs_data_cleaned['StartPublishing'].apply(\n",
        "    lambda x: dms_start_date if x < dms_start_date else x\n",
        ")\n",
        "\n",
        "# Check if the correction was successful\n",
        "before_fix = sum(docs_data_cleaned['StartPublishing'] < dms_start_date)\n",
        "print(f\"Number of assets with StartPublishing date before DMS start date after fix: {before_fix}\")\n",
        "\n",
        "# Visualization: Distribution of StartPublishing dates\n",
        "plt.figure(figsize=(10, 6))\n",
        "docs_data_cleaned['StartPublishing'].dt.year.value_counts().sort_index().plot(kind='bar', color='skyblue')\n",
        "plt.title('Distribution of StartPublishing Dates (Post-Fix)')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Number of Assets')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZkTB5R8HBecu",
      "metadata": {
        "id": "ZkTB5R8HBecu"
      },
      "source": [
        "**Summary of Cleaning and EDA on StartPublishing Dates**\n",
        "\n",
        "We corrected StartPublishing dates to ensure consistency with the DMS introduction in August 2018. This adjustment resolved unrealistic entries (e.g., assets from 2001), which is essential for accurate data drift analysis and effective stratified sampling. By doing so, we set a solid foundation for reliable and meaningful insights in the next stages of analysis.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eTbnoU47aciK",
      "metadata": {
        "id": "eTbnoU47aciK"
      },
      "source": [
        "####Q1 Step 5: Data and concept drift\n",
        "\n",
        "**Outline:**\n",
        "We will examine potential data drift and concept drift across the entire dataset to understand how asset characteristics and preferences may have changed over time.\n",
        "\n",
        "**Purpose:**\n",
        "We want to understand how download patterns and publication trends change over time so we can provide a suitable model to address member engagement issues and offer better explanations for these shifts.\n",
        "\n",
        "\n",
        "- Data Drift: We aim to identify changes in the statistical properties of asset features over time, which may affect how actuaries engage with CPD content.\n",
        "\n",
        "- Concept Drift: Though concept drift typically examines changes in the relationship between features and a response variable, our analysis is limited due to the absence of a cleaned response variable. We could consider using download counts as a proxy, but this will be explored in more depth once the response variable is defined.\n",
        "\n",
        "Reference:\n",
        "Following Codes are generated using ChatGPT4"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VdMjzMMr5bKM",
      "metadata": {
        "id": "VdMjzMMr5bKM"
      },
      "source": [
        "**Examination of Popularity Decay**\n",
        "\n",
        "**Purpose:**\n",
        "We examined how the average downloads per asset change over time since publication.\n",
        "\n",
        "This helps in understanding how member engagement with assets age.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ESW_g__l6cE4",
      "metadata": {
        "id": "ESW_g__l6cE4"
      },
      "source": [
        "**What are the following codes doing**\n",
        "\n",
        "We group by MonthsSincePublication to compute the total downloads and unique assets available for each month.\n",
        "Calculate the average downloads per asset for each month since publication.\n",
        "\n",
        "Plot this trend to visualize how asset popularity changes over time after publication. This plot helps identify general patterns, such as whether assets receive most downloads shortly after publication and how interest declines or increases over time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YIyr8Eva57sS",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YIyr8Eva57sS",
        "outputId": "d6baefba-fdf7-453d-e8b8-5c06e5213632"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId DownloadDate  Month StartPublishing  MonthsSincePublication\n",
            "0  4194310   2024-04-02      4      2018-09-04                      67\n",
            "1  4194310   2023-11-14     11      2018-09-04                      62\n",
            "2  4194310   2019-05-02      5      2018-09-04                       8\n",
            "3  4194310   2019-01-09      1      2018-09-04                       4\n",
            "4  4194310   2018-09-21      9      2018-09-04                       0\n"
          ]
        }
      ],
      "source": [
        "# Merge downloads with document data on 'DMSId'\n",
        "merged_df = pd.merge(downloads_data, docs_data_cleaned[['DMSId', 'StartPublishing']], on='DMSId', how='inner')\n",
        "\n",
        "# Calculate 'MonthsSincePublication'\n",
        "merged_df['MonthsSincePublication'] = (merged_df['DownloadDate'].dt.to_period('M') - merged_df['StartPublishing'].dt.to_period('M')).apply(lambda x: x.n)\n",
        "\n",
        "# Filter out downloads that occurred before publication date\n",
        "merged_df = merged_df[merged_df['MonthsSincePublication'] >= 0]\n",
        "\n",
        "# Preview merged data\n",
        "print(merged_df.head())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oJ9J-Ydc6Buq",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "oJ9J-Ydc6Buq",
        "outputId": "7563ec51-7424-49d5-aaff-b3d2421c5c38"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 58040 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Calculate average downloads per month since publication across all assets\n",
        "monthly_trends = merged_df.groupby('MonthsSincePublication').size().reset_index(name='DownloadCount')\n",
        "\n",
        "# Calculate total number of assets available at each month since publication\n",
        "assets_count = merged_df.groupby('MonthsSincePublication')['DMSId'].nunique().reset_index(name='UniqueAssets')\n",
        "\n",
        "# Merge to get downloads per asset per month\n",
        "monthly_trends = pd.merge(monthly_trends, assets_count, on='MonthsSincePublication')\n",
        "monthly_trends['AvgDownloadsPerAsset'] = monthly_trends['DownloadCount'] / monthly_trends['UniqueAssets']\n",
        "\n",
        "# Plotting the trend\n",
        "plt.figure(figsize=(12, 6))\n",
        "sns.lineplot(data=monthly_trends, x='MonthsSincePublication', y='AvgDownloadsPerAsset', marker='o')\n",
        "plt.title('Average Downloads per Asset Over Time Since Publication')\n",
        "plt.xlabel('Months Since Publication')\n",
        "plt.ylabel('Average Downloads per Asset')\n",
        "plt.xticks(range(0, monthly_trends['MonthsSincePublication'].max()+1, 6))  # Adjust x-ticks as needed\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kkYhFZun63md",
      "metadata": {
        "id": "kkYhFZun63md"
      },
      "source": [
        "**Findings:**\n",
        "\n",
        "- Assets usually have high downloads within the first few months of publication, averaging over 20 downloads per asset initially.\n",
        "\n",
        "- After the initial spike, there's a rapid decline in average downloads, stabilizing around 2-3 downloads per asset from around 12 months onward.\n",
        "\n",
        "\n",
        "**Implications:**\n",
        "- This trend suggests that newly published assets receive the most attention, and interest quickly wanes over time.\n",
        "\n",
        "- This suggest Recency (assets Age) is a valid feature and we should consider it in later Q2 and Q3.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EvlPh4URtJqU",
      "metadata": {
        "id": "EvlPh4URtJqU"
      },
      "source": [
        "Examinate Concept drift on year of publication(Year Cohort)\n",
        "\n",
        "**Purpose:**\n",
        "We aim to investigate whether there is a cohort effect based on the publishing year. This means that the popularity of assets may develop differently depending on the year they were published.\n",
        "\n",
        "**Approach:**\n",
        "We'll display the average downloads per asset over time (in months) since publication, grouped by different publication years.\n",
        "\n",
        "\n",
        "**Expected Outcome:**\n",
        "If concept drift (cohort effect) exists, we expect to see different popularity decay patterns for assets published in different years, suggesting that factors driving asset popularity change over time.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VRbrMkWl7mUK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "VRbrMkWl7mUK",
        "outputId": "806d6a9e-689a-4325-a125-bf817a38d4b4"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 93220 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Assuming 'downloads_clean' is the cleaned version of your downloads data and 'docs_data_cleaned' is the adjusted documents data.\n",
        "\n",
        "# Merge the cleaned downloads data with the cleaned documents data to get the corresponding StartPublishing dates\n",
        "merged_df = downloads_clean.merge(docs_data_cleaned[['DMSId', 'StartPublishing']], on='DMSId', how='left')\n",
        "\n",
        "# Convert 'StartPublishing' to datetime format and extract the year (if not already done)\n",
        "merged_df['StartPublishing'] = pd.to_datetime(merged_df['StartPublishing'])\n",
        "merged_df['PublicationYear'] = merged_df['StartPublishing'].dt.year\n",
        "\n",
        "# Calculate the months since publication for each download\n",
        "merged_df['MonthsSincePublication'] = ((merged_df['DownloadDate'].dt.year - merged_df['StartPublishing'].dt.year) * 12 +\n",
        "                                       (merged_df['DownloadDate'].dt.month - merged_df['StartPublishing'].dt.month))\n",
        "\n",
        "# Remove rows where 'MonthsSincePublication' is negative\n",
        "merged_df = merged_df[merged_df['MonthsSincePublication'] >= 0]\n",
        "\n",
        "# Group by Publication Year and Months Since Publication to calculate the average downloads per asset\n",
        "average_downloads = merged_df.groupby(['PublicationYear', 'MonthsSincePublication']).size().reset_index(name='DownloadCount')\n",
        "\n",
        "# Merge back with the docs_data_cleaned to get the count of assets published in each year\n",
        "assets_per_year = docs_data_cleaned.groupby(docs_data_cleaned['StartPublishing'].dt.year).size().reset_index(name='AssetCount')\n",
        "average_downloads = average_downloads.merge(assets_per_year, left_on='PublicationYear', right_on='StartPublishing')\n",
        "\n",
        "# Calculate the average downloads per asset\n",
        "average_downloads['AvgDownloadsPerAsset'] = average_downloads['DownloadCount'] / average_downloads['AssetCount']\n",
        "\n",
        "# Limit the x-axis to the first 50 months\n",
        "average_downloads = average_downloads[average_downloads['MonthsSincePublication'] <= 50]\n",
        "\n",
        "# Smooth the lines using a rolling average (optional)\n",
        "average_downloads['AvgDownloadsPerAsset'] = average_downloads.groupby('PublicationYear')['AvgDownloadsPerAsset'].transform(lambda x: x.rolling(3, 1).mean())\n",
        "\n",
        "# Plotting the download trends for different publication years (using linear scale)\n",
        "plt.figure(figsize=(12, 6))\n",
        "for year in sorted(average_downloads['PublicationYear'].unique()):\n",
        "    year_data = average_downloads[average_downloads['PublicationYear'] == year]\n",
        "    plt.plot(year_data['MonthsSincePublication'], year_data['AvgDownloadsPerAsset'], label=f'Published in {year}')\n",
        "\n",
        "plt.xlabel('Months Since Publication')\n",
        "plt.ylabel('Average Downloads per Asset')\n",
        "plt.title('Download Trends for Assets Published in Different Years')\n",
        "plt.legend()\n",
        "plt.grid(True)\n",
        "plt.show()\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "h3tEM1U3_Qem",
      "metadata": {
        "id": "h3tEM1U3_Qem"
      },
      "source": [
        "**What the Chart Shows:**\n",
        "\n",
        "- Across all years, assets experience the highest download rates immediately after publication, followed by a rapid decline. This trend reflects the typical lifecycle of new content, where it garners immediate attention but gradually loses interest over time.\n",
        "\n",
        "-  More recent assets (published from 2020 onward) tend to have higher initial download rates compared to older assets (published in 2018-2019). This suggests that the publishing cohort does impact download patterns.\n",
        "\n",
        "**Implications to data partition:**\n",
        "The clear presence of data and concept drift means that it's crucial to implement a temporal split for your training and validation datasets.\n",
        "\n",
        "Instead of random splits, consider splitting based on publication periods. For example:\n",
        "Training: Use assets published from 2018 to 2021.\n",
        "Validation/Testing: Use assets from 2022 onward.\n",
        "\n",
        "This can allow model can learn from recency effect and cohort effect.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yBmeHS5PA1hd",
      "metadata": {
        "id": "yBmeHS5PA1hd"
      },
      "source": [
        "**Normalised download count by duration**\n",
        "\n",
        "**Purpose:**\n",
        "The primary goal is to examine how download counts normalize over an asset's life since publication.\n",
        "\n",
        "This helps us understand the average download trend over time, determining whether older assets are disadvantaged due to being available longer or if their popularity has genuinely decreased.\n",
        "\n",
        "\n",
        " Reference: The following Codes were generated by ChatGPT4\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "qa0vaWfPA2Wx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "id": "qa0vaWfPA2Wx",
        "outputId": "8466dd5b-8045-4adb-af30-4b31e77833d6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 37768 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "# Assuming today's date is 30 April 2024\n",
        "today = datetime(2024, 4, 30)\n",
        "\n",
        "# Calculate the total number of downloads per asset\n",
        "download_counts = downloads_df.groupby('DMSId')['DownloadDate'].count().reset_index(name='DownloadCount')\n",
        "\n",
        "# Merge the download counts back into the main documents table\n",
        "docs_data_cleaned = docs_data_cleaned.merge(download_counts, on='DMSId', how='left')\n",
        "\n",
        "# Fill any NaN values in the DownloadCount column with 0 (indicating no downloads)\n",
        "docs_data_cleaned['DownloadCount'] = docs_data_cleaned['DownloadCount'].fillna(0).astype(int)\n",
        "\n",
        "# Calculate the duration (in years) each asset has been on the DMS platform\n",
        "docs_data_cleaned['DurationOnDMS'] = round((today - docs_data_cleaned['StartPublishing']).dt.days / 365.25, 1)\n",
        "\n",
        "# Calculate the annualized download count (downloads per year)\n",
        "docs_data_cleaned['AnnualizedDownloadCount'] = round(docs_data_cleaned['DownloadCount'] / docs_data_cleaned['DurationOnDMS'], 2)\n",
        "\n",
        "# Extract the year of StartPublishing\n",
        "docs_data_cleaned['YearOfStartPublishing'] = pd.DatetimeIndex(docs_data_cleaned['StartPublishing']).year\n",
        "\n",
        "# Calculate the average annualized downloads by year. Use the correct column name 'AnnualizedDownloadCount'\n",
        "avg_annualized_downloads_by_year = docs_data_cleaned.groupby('YearOfStartPublishing')['AnnualizedDownloadCount'].mean().reset_index()\n",
        "\n",
        "# Step 3: Plot the relationship between the year of release and the average annualized download count\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.bar(avg_annualized_downloads_by_year['YearOfStartPublishing'], avg_annualized_downloads_by_year['AnnualizedDownloadCount'], color='skyblue') # Use the correct column name here as well\n",
        "plt.xlabel('Year of Start Publishing')\n",
        "plt.ylabel('Average Annualized Downloads Per Year')\n",
        "plt.title('Average Annualized Downloads Per Year by Year of Start Publishing')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hk6iJSevoKjf",
      "metadata": {
        "id": "hk6iJSevoKjf"
      },
      "source": [
        "Key Observations:\n",
        "- Increasing Trend in Annualized Downloads: Assets published in more recent years (especially from 2020 onward) have significantly higher average annualized download counts compared to older assets. This indicates a growing level of engagement with newer assets, suggesting that the popularity of assets on the CPD Hub has increased over time.\n",
        "\n",
        "- Steep Increase in Recent Years: The sharp rise in 2023 and 2024 suggests that recent publications are gaining attention much faster, pointing to changes in user engagement behavior.\n",
        "\n",
        "\n",
        "**Caution**\n",
        "\n",
        "Please note that The annualized download measure indeed blends the cohort and recency effects, making it less reliable for capturing true popularity trends. This metric can underrepresent older assets and fail to account for the fact that assets published before the DMS Hub launch have truncated download data, skewing the comparison.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ct6-4FRQB9gN",
      "metadata": {
        "id": "Ct6-4FRQB9gN"
      },
      "source": [
        "**Assets Published by Format and Year**\n",
        "\n",
        "**Purpose of This Analysis**\n",
        "This data drift analysis is critical for identifying shifts in format type across different published year. This allow us to have better understanding how the mix of format changes over time.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "o3KTlln8vdtL",
      "metadata": {
        "id": "o3KTlln8vdtL"
      },
      "source": [
        "Rationale: The release of different content formats (e.g., PDFs, videos, presentations) can change over time, reflecting shifts in user and author preferences. By understanding how the distribution of file types changes, you can gain insights into evolving trends and adjust your content strategy or model features accordingly. This variable is directly related to how users interact with and consume the content, making it a key factor in detecting data drift."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "TSRe-ONxkcm0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "TSRe-ONxkcm0",
        "outputId": "fc2c6074-fddb-4ac5-d045-720df17669e0"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 61516 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# First, we create a pivot table to count the number of assets by year and format\n",
        "format_by_year = docs_data_cleaned.pivot_table(\n",
        "    index=docs_data_cleaned['StartPublishing'].dt.year,  # Group by the year of publication\n",
        "    columns='Format',  # The format types will form the columns\n",
        "    aggfunc='size',  # Count the occurrences\n",
        "    fill_value=0  # Fill missing values with 0\n",
        ")\n",
        "\n",
        "# Plot the stacked bar chart\n",
        "format_by_year.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Assets Published by Format and Year')\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('Number of Assets')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='Format', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5uOQAoLwue5O",
      "metadata": {
        "id": "5uOQAoLwue5O"
      },
      "source": [
        "The above plot is a bit hard to see because the number of asset publsihed in each year are quite different. We should use a proportion stacked bar chart to get a better comparision."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5tH9wydxk8y7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "5tH9wydxk8y7",
        "outputId": "39c81fde-e406-467d-e287-e2e82595965d"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 61520 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "\n",
        "# First, we create a pivot table to count the number of assets by year and format\n",
        "format_by_year = docs_data_cleaned.pivot_table(\n",
        "    index=docs_data_cleaned['StartPublishing'].dt.year,  # Group by the year of publication\n",
        "    columns='Format',  # The format types will form the columns\n",
        "    aggfunc='size',  # Count the occurrences\n",
        "    fill_value=0  # Fill missing values with 0\n",
        ")\n",
        "\n",
        "# Normalize the data to get proportions\n",
        "format_by_year_proportions = format_by_year.div(format_by_year.sum(axis=1), axis=0)\n",
        "\n",
        "# Plot the proportion stacked bar chart\n",
        "format_by_year_proportions.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Proportion of Assets Published by Format and Year')\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('Proportion of Assets')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='Format', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "naYMwLQVvkGq",
      "metadata": {
        "id": "naYMwLQVvkGq"
      },
      "source": [
        "Findings:\n",
        "\n",
        "- Change in Asset Composition: The proportion of different formats such as \"Article,\" \"Presentation Slides,\" \"Publication,\" and others has shifted from year to year. For example, articles became more prominent in recent years, while formats like \"Presentation Slides\" and \"Publications\" have seen varying levels of representation.\n",
        "\n",
        "- By observing this data drift, we recognize that the mix of content types available to users has evolved, and any predictive model should account for this changing distribution to maintain accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-2p6w6TwvZQb",
      "metadata": {
        "id": "-2p6w6TwvZQb"
      },
      "source": [
        "**Examinate concept drift on format to donwlaod**\n",
        "\n",
        "Rationale for Examining Concept Drift in Format:\n",
        "-  As user preferences shift over time, certain formats become more or less popular. By examining concept drift, we can identify which formats are gaining or losing traction among members, helping the Institute to adapt content strategies accordingly.\n",
        "\n",
        "- Strategy Optimization: Identifying concept drift enables the institute to tailor content offerings that align with current trends, ensuring that the CPD Hub features formats that members are more likely to engage with. This can improve member satisfaction and engagement rates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zYIiZxnPc6jy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "zYIiZxnPc6jy",
        "outputId": "1fa01665-9a46-40e0-f2fc-789297ae5624"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 61032 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a proportion of assets donwlaod by year and by format\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Calculate total downloads for each year\n",
        "total_downloads_by_year = downloads_clean.groupby(downloads_clean['DownloadDate'].dt.year)['DMSId'].count()\n",
        "\n",
        "# Merge downloads data with document data to get the format\n",
        "merged_df = downloads_clean.merge(docs_data_cleaned[['DMSId', 'Format']], on='DMSId', how='left')\n",
        "\n",
        "# Group by year and format and count downloads\n",
        "downloads_by_year_format = merged_df.groupby([merged_df['DownloadDate'].dt.year, 'Format'])['DMSId'].count().unstack()\n",
        "\n",
        "# Calculate the proportion of downloads for each format within each year\n",
        "proportion_downloads_by_year_format = downloads_by_year_format.div(downloads_by_year_format.sum(axis=1), axis=0)\n",
        "\n",
        "# Plot the stacked bar chart for proportions\n",
        "proportion_downloads_by_year_format.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Proportion of Downloads by Format and Year')\n",
        "plt.xlabel('Year of Download')\n",
        "plt.ylabel('Proportion of Downloads')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='Format', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "o_msd15Nyc_l",
      "metadata": {
        "id": "o_msd15Nyc_l"
      },
      "source": [
        "**Key Observations:**\n",
        "- Changing Popularity: The proportion of downloads for each format has changed significantly over time. For instance, \"Publication\" was a dominant format in earlier years (e.g., 2018, 2019), but its dominance has decreased over the years. Meanwhile, \"Presentation Slides\" have become more prominent in recent years.\n",
        "\n",
        "- Emerging Formats: Formats like \"Article\" and \"Video\" have become more relevant, indicating that user preferences for content types are evolving.\n",
        "\n",
        "- Declining Formats: Some formats, such as \"Podcast/Vodcast/Audio,\" have seen a decrease in download share over the years, suggesting they might be becoming less attractive to the audience.\n",
        "\n",
        "**Implications to Q3**\n",
        "- Concept Drift: The changes in format preference imply concept drift, as the relationship between format types and download behavior has shifted. This means that models predicting asset popularity or format effectiveness might need to adjust over time to maintain accuracy.\n",
        "\n",
        "- Model Training: Any predictive model should be aware of these trends and incorporate time as a feature to account for evolving preferences. This ensures the model remains relevant and adapts to the changing dynamics of content consumption."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AVkPyCyT2jM3",
      "metadata": {
        "id": "AVkPyCyT2jM3"
      },
      "source": [
        "**Data and Concept Drift Analysis Across File Type and Event Type**\n",
        "\n",
        "Similarly, we examined data drift in file types and concept drift in event types. The rationale for this analysis aligns with our previous discussion on format changes, so we won't repeat the details here. A summarized insight is provided at the end of this section."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "i0hsw5pVxoeZ",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "i0hsw5pVxoeZ",
        "outputId": "70b9dbb6-1ba6-4e3c-89da-fee8b1e74061"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 48264 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# First, we create a pivot table to count the number of assets by year and FileType\n",
        "filetype_by_year = docs_data_cleaned.pivot_table(\n",
        "    index=docs_data_cleaned['StartPublishing'].dt.year,  # Group by the year of publication\n",
        "    columns='FileType',  # The file types will form the columns\n",
        "    aggfunc='size',  # Count the occurrences\n",
        "    fill_value=0  # Fill missing values with 0\n",
        ")\n",
        "\n",
        "# Normalize the data to get proportions\n",
        "filetype_by_year_proportions = filetype_by_year.div(filetype_by_year.sum(axis=1), axis=0)\n",
        "\n",
        "# Plot the proportion stacked bar chart\n",
        "filetype_by_year_proportions.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Proportion of Assets Published by FileType and Year')\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('Proportion of Assets')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='FileType', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ksjaxgo_dP4n",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "ksjaxgo_dP4n",
        "outputId": "9076ea75-3d43-454b-f7f0-a18e83fc0a37"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 47556 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a polt , proportion of assets donwlaod by fileType and year\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Merge downloads data with document data to get the FileType\n",
        "merged_df = downloads_clean.merge(docs_data_cleaned[['DMSId', 'FileType']], on='DMSId', how='left')\n",
        "\n",
        "# Group by year and FileType and count downloads\n",
        "downloads_by_year_filetype = merged_df.groupby([merged_df['DownloadDate'].dt.year, 'FileType'])['DMSId'].count().unstack()\n",
        "\n",
        "# Calculate the proportion of downloads for each FileType within each year\n",
        "proportion_downloads_by_year_filetype = downloads_by_year_filetype.div(downloads_by_year_filetype.sum(axis=1), axis=0)\n",
        "\n",
        "# Plot the stacked bar chart for proportions\n",
        "proportion_downloads_by_year_filetype.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Proportion of Downloads by FileType and Year')\n",
        "plt.xlabel('Year of Download')\n",
        "plt.ylabel('Proportion of Downloads')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='FileType', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Tg9aCMDBdqNH",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "id": "Tg9aCMDBdqNH",
        "outputId": "0bf2a7c2-5ef4-4f59-fc25-8f610cb25952"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 54212 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a plot, porotion of downlaod by eventype and year. if the eventype is missing, assignt it to a new bucket of non event related\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Merge downloads data with document data to get the EventType\n",
        "merged_df = downloads_clean.merge(docs_data_cleaned[['DMSId', 'EventType']], on='DMSId', how='left')\n",
        "\n",
        "# Fill missing EventType with 'Non Event Related'\n",
        "merged_df['EventType'] = merged_df['EventType'].fillna('Non Event Related')\n",
        "\n",
        "# Group by year and EventType and count downloads\n",
        "downloads_by_year_eventtype = merged_df.groupby([merged_df['DownloadDate'].dt.year, 'EventType'])['DMSId'].count().unstack()\n",
        "\n",
        "# Calculate the proportion of downloads for each EventType within each year\n",
        "proportion_downloads_by_year_eventtype = downloads_by_year_eventtype.div(downloads_by_year_eventtype.sum(axis=1), axis=0)\n",
        "\n",
        "# Plot the stacked bar chart for proportions\n",
        "proportion_downloads_by_year_eventtype.plot(kind='bar', stacked=True, figsize=(12, 8), colormap='tab20')\n",
        "\n",
        "# Customize the plot\n",
        "plt.title('Proportion of Downloads by EventType and Year')\n",
        "plt.xlabel('Year of Download')\n",
        "plt.ylabel('Proportion of Downloads')\n",
        "plt.xticks(rotation=45)\n",
        "plt.legend(title='EventType', bbox_to_anchor=(1.05, 1), loc='upper left')\n",
        "plt.tight_layout()\n",
        "\n",
        "# Show the plot\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pB1AqCeUzZ_f",
      "metadata": {
        "id": "pB1AqCeUzZ_f"
      },
      "source": [
        "**Format Analysis:**\n",
        "- Data Drift: There has been a significant change in the types of assets published over the years. While traditional formats like presentations and publications were prevalent initially, more recent years have seen an increase in articles, workshops, and webinars.\n",
        "- Concept Drift: The download trends mirror this shift, with older formats still maintaining popularity but newer formats gaining traction, highlighting evolving user preferences for content types.\n",
        "\n",
        "**Event Type Analysis:**\n",
        "- Concept Drift: A clear shift is seen in the download patterns across event types over the years. Initially, major events attracted more downloads, but there's a growing trend towards more diverse events such as insights/conferences and other non-major events, indicating changing interests among members over time.\n",
        "\n",
        "\n",
        "Overall, these charts demonstrate clear evidence of both data drift (changes in the distribution of published assets) and concept drift (changes in user engagement and preferences over time). This insight is crucial for understanding evolving user behaviors and adjusting future strategies for asset publication and predicting asset popularity effectively."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ke2Vl-Y0W50",
      "metadata": {
        "id": "5ke2Vl-Y0W50"
      },
      "source": [
        "Examine Cpd point change over time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uXibNI1nPqLi",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "uXibNI1nPqLi",
        "outputId": "625abc4f-3a54-476c-8cc1-432f683ee480"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 46856 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Convert StartPublishing to datetime and extract the year\n",
        "docs_data_cleaned['Year'] = pd.to_datetime(docs_data_cleaned['StartPublishing']).dt.year\n",
        "\n",
        "# Calculate the average CPD points by the StartPublishing year\n",
        "average_cpd = docs_data_cleaned.groupby('Year')['Cpd'].mean()\n",
        "\n",
        "# Plot the average CPD points against the publication year\n",
        "plt.figure(figsize=(10, 6))\n",
        "average_cpd.plot(kind='line', marker='o', color='b')\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('Average CPD Points')\n",
        "plt.title('Average CPD Points Over Time')\n",
        "plt.grid(True)\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HQ76e1M0FjRo",
      "metadata": {
        "id": "HQ76e1M0FjRo"
      },
      "source": [
        "CPD points change over time\n",
        "\n",
        "The purpose of examining the CPD points over time is to understand how the credit allocation per asset changes across different publication years. This helps identify trends in the perceived value or learning outcomes provided by these assets over time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "193gHRz6QyKP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "id": "193gHRz6QyKP",
        "outputId": "86944229-4747-4f26-c510-7e694c2e44b8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 0 Axes>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "[encoded data, 35956 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Convert StartPublishing to datetime and extract the year\n",
        "docs_data_cleaned['Year'] = pd.to_datetime(docs_data_cleaned['StartPublishing']).dt.year\n",
        "\n",
        "# Calculate the number of assets published each year\n",
        "year_counts = docs_data_cleaned['Year'].value_counts().sort_index()\n",
        "\n",
        "# Create a box plot to show the distribution of CPD points for each year\n",
        "plt.figure(figsize=(12, 8))\n",
        "boxplot = docs_data_cleaned.boxplot(column='Cpd', by='Year', grid=False, vert=True)\n",
        "\n",
        "# Customize x-axis labels with the number of assets\n",
        "xticks_labels = [f'{int(year)}\\n(N={int(count)})' for year, count in year_counts.items()]\n",
        "plt.xticks(ticks=range(1, len(xticks_labels) + 1), labels=xticks_labels)\n",
        "\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('CPD Points')\n",
        "plt.title('Distribution of CPD Points by Year of Publication')\n",
        "plt.suptitle('')  # Suppress the automatic title\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Y9qdC4mP0s4j",
      "metadata": {
        "id": "Y9qdC4mP0s4j"
      },
      "source": [
        "Insights from the Charts:\n",
        "\n",
        "- Average CPD Points Over Time:\n",
        " The line chart shows a peak in average CPD points in 2020, followed by a steady decline from 2021 onwards. This suggests that newer assets are receiving fewer CPD points on average, which might indicate changes in the criteria for CPD point allocation or shifts in the complexity/value of assets.\n",
        "\n",
        "\n",
        "- Distribution of CPD Points by Year: The boxplot reveals that although the median CPD points remain low and consistent across years, there are more outliers with high CPD points in earlier years (e.g., 2020). The distribution indicates that while most assets provide a similar level of CPD credit, a few high-value assets exist, particularly in older publications.\n",
        "\n",
        "\n",
        "These findings highlight a potential data drift, as the nature of CPD point allocation appears to change over time, impacting the evaluation of asset quality and relevance."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mp9vf6Zt05Zy",
      "metadata": {
        "id": "mp9vf6Zt05Zy"
      },
      "source": [
        "**Summary of the Data and Concept Drift analysis**\n",
        "\n",
        "**Data Drift:**\n",
        "- Download Patterns: More recent assets show higher download rates initially, suggesting increased engagement with newer content, but overall popularity decays rapidly over time.\n",
        "\n",
        "- Format and File Type: The proportion of assets published in different formats and file types has shifted, with newer formats (e.g., videos, e-learning) becoming more prevalent. This indicates evolving user preferences and content delivery methods.\n",
        "\n",
        "- CPD Points: There’s a noticeable decline in average CPD points for assets published in recent years, indicating changing criteria for point allocation or a shift in content complexity/value.\n",
        "\n",
        "**Concept Drift:**\n",
        "\n",
        "- Event, Format and FileType: The relevance of different types for downloads has also changed, highlighting evolving interests or changes in how events contribute to asset engagement.\n",
        "\n",
        "Overall, these insights emphasize the importance of adapting strategies for predicting asset popularity and grouping based on shifting user behaviors and institutional practices."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BByZQQoXlWeP",
      "metadata": {
        "id": "BByZQQoXlWeP"
      },
      "source": [
        "#### Q1a Step 6 EDA on download pattern\n",
        "**Outline**\n",
        "\n",
        "The section we want to exmainate the implication of the way of our data split on published date. this analysis is mainly for Q3 when we do our NN mdoel . however, do it in the early stage allow us to better understand.\n",
        "\n",
        "Key Consideration:\n",
        "If we split assets by their publish date, assets in the training dataset will not appear in the validation or testing sets. This means that, in terms of download patterns, the validation set will never see the full picture since all assets from the training set are excluded from it.\n",
        "\n",
        "**We want to assess the impact on download analysis if we partition the data by publish date and understand how this approach might affect our results.**\n",
        "\n",
        "\n",
        "\n",
        "We have conducted the following anaysis and checks:\n",
        "\n",
        "1. Monthly Download Analysis:\n",
        "2. Distribution of StartPublishing Dates:\n",
        "3. Download Level Analysis:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nP9VecOtlWeV",
      "metadata": {
        "id": "nP9VecOtlWeV"
      },
      "source": [
        "#####Monthly Download Analysis:\n",
        "We look at the download for each month since CPD hub is established.\n",
        "\n",
        "**Why:** This step helps identify the level of download of each month since CPD hub is established. Understanding this pattern allows us to determine the relevant period to focus on in model training, ensuring we are capturing valid download behavior.\n",
        "\n",
        "**Expectation:**\n",
        "Since the CPD Hub was new in 2018, we expect low initial download activity, which gradually stabilizes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fO9qTUgOlWeV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "fO9qTUgOlWeV",
        "outputId": "0fc5cea8-99ac-4636-9c2c-332f7b81ca49"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 78652 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Ensure 'DownloadDate' is in datetime format\n",
        "downloads_clean['DownloadDate'] = pd.to_datetime(downloads_clean['DownloadDate'])\n",
        "\n",
        "# Create a new column that extracts the year-month from the 'DownloadDate'\n",
        "downloads_clean['YearMonth'] = downloads_clean['DownloadDate'].dt.to_period('M')\n",
        "\n",
        "# Group the data by YearMonth to count the number of downloads for each month\n",
        "monthly_downloads = downloads_clean.groupby('YearMonth').size()\n",
        "\n",
        "# Plot the monthly download counts\n",
        "plt.figure(figsize=(10, 6))\n",
        "monthly_downloads.plot(kind='line', marker='o', color='b')\n",
        "plt.title('Monthly Download Counts (Dec 2018 to Apr 2024)')\n",
        "plt.xlabel('Year-Month')\n",
        "plt.ylabel('Number of Downloads')\n",
        "plt.grid(True)\n",
        "plt.xticks(rotation=45)\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7l45ClMXlWeV",
      "metadata": {
        "id": "7l45ClMXlWeV"
      },
      "source": [
        "**What does this check tell us:**\n",
        "Pre-mid-2019 shows minimal downloads, likely due to the CPD Hub's newness and members' limited usage until mid-2020.\n",
        "\n",
        "**So what:**\n",
        "We will need to examinate the download popularity in Q3, we need to ensure there are enough reliable donwload pattern in the training set so that NN can learn the pattern.\n",
        "\n",
        "So, we need to ensure tranining set end at 31/Dec/2021, so that training set has around 1.5 year (from mid2020 to end2021)  of donwlaod pattern to learn from."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "IGtTUkjUlWeV",
      "metadata": {
        "id": "IGtTUkjUlWeV"
      },
      "source": [
        "**What does this check tell us:**\n",
        "The training set includes assets published between August 2018 and August 2020, with a large upload in 2018 due to the initial CPD Hub setup. We confirm there is no overlap in published dates between the training, validation, and test datasets, ensuring proper partitioning.\n",
        "\n",
        "**So What:**\n",
        "To avoid noise from truncated data, we exclude the initial download activity from 2018 and 2019 when dealing with the training set. This helps focus the model on a more stable download pattern starting from mid-2020.\n",
        "\n",
        "The validation set only includes assets published between 27 August 2020 and 24 August 2022. While this might exclude older assets, potentially underestimating their relevance, our decision to partition by publication date avoids data leakage and is considered sensible. This ensures that older assets from the training set do not bias the validation process.\n",
        "\n",
        "**Next Steps:**\n",
        "We will further examine our partitioning approach to ensure that excluding older assets does not introduce significant issues, especially in terms of capturing trends or patterns in the validation phase."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CnJ36KlZlWeV",
      "metadata": {
        "id": "CnJ36KlZlWeV"
      },
      "source": [
        "#####Download Level Analysis:\n",
        "**What:** We analyzed the volume of downloads between July 2021 and December 2022, focusing on two aspects:\n",
        "- total monthly downloads and\n",
        "- the number of those downloads that came from assets published before August 2020 (i.e., assets that would have been excluded from the validation and testing sets).\n",
        "\n",
        "**Why:** Since we partitioned the data based on the asset publication date, older assets (published before August 2020) do not appear in the validation or test sets. However, in reality, users can still download older assets. The goal of this analysis is to determine whether excluding these older assets from the validation and testing sets could bias the model. By checking how frequently older assets are downloaded, we can assess the impact of their exclusion."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "PzUC_vSSlWeW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "PzUC_vSSlWeW",
        "outputId": "3e898923-287f-422f-99e7-cdc9abeae8ab"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 84116 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Filter the downloads_clean dataset for the range July 2021 to December 2022\n",
        "start_date = '2022-07-01'\n",
        "end_date = '2023-06-30'\n",
        "downloads_filtered = downloads_clean[(downloads_clean['DownloadDate'] >= start_date) &\n",
        "                                     (downloads_clean['DownloadDate'] <= end_date)]\n",
        "\n",
        "# Filter for assets published before 31 Dec 2021\n",
        "docs_pre_dec2021 = docs_data_cleaned[docs_data_cleaned['StartPublishing'] < '2021-12-31']\n",
        "downloads_pre_dec2021 = downloads_filtered[downloads_filtered['DMSId'].isin(docs_pre_dec2021['DMSId'])]\n",
        "\n",
        "# Group the downloads by month for total downloads and downloads for pre-Dec 2022 assets\n",
        "downloads_by_month = downloads_filtered.groupby(downloads_filtered['DownloadDate'].dt.to_period('M')).size()\n",
        "downloads_pre_dec2021_by_month = downloads_pre_dec2021.groupby(downloads_pre_dec2021['DownloadDate'].dt.to_period('M')).size()\n",
        "\n",
        "# Convert to datetime index for plotting\n",
        "downloads_by_month.index = downloads_by_month.index.to_timestamp()\n",
        "downloads_pre_dec2021_by_month.index = downloads_pre_dec2021_by_month.index.to_timestamp()\n",
        "\n",
        "# Plot the data\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(downloads_by_month.index, downloads_by_month, marker='o', label='Total Downloads', color='blue')\n",
        "plt.plot(downloads_pre_dec2021_by_month.index, downloads_pre_dec2021_by_month, marker='o', label='Downloads from Pre-dec 2021 Assets', color='orange')\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('Month')\n",
        "plt.ylabel('Number of Downloads')\n",
        "plt.title('Monthly Downloads from July 2022 to Jun 2023')\n",
        "plt.xticks(rotation=45)\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "\n",
        "# Show the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ugn8-GPLlWeW",
      "metadata": {
        "id": "ugn8-GPLlWeW"
      },
      "source": [
        "**What do we see:**\n",
        "The plot shows that although there are downloads from older assets (published before August 2020), their volume relative to the total monthly downloads has been gradually decreasing. This suggests that the influence of earlier assets on current downloading behavior is diminishing. Consequently, this supports the decision to partition by publication date, as the impact of excluding older assets from the validation and testing sets will likely not materially affect the model's ability to learn and predict asset popularity."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "76Q73dtglWeW",
      "metadata": {
        "id": "76Q73dtglWeW"
      },
      "source": [
        "We will now examine the monthly downloads for assets published before August 27, 2020, specifically focusing on those with more than 2 downloads per month , and a plot for more than 5 downloads per month.\n",
        "\n",
        "Rationale: The primary goal is to identify the top 20 most popular assets. To achieve this, we exclude low-volume downloads to concentrate on assets with significant engagement, ensuring that we focus on the most relevant and potentially popular items."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "qDDf6fl4lWeW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "id": "qDDf6fl4lWeW",
        "outputId": "1e514f77-fec0-4591-ea1d-935f2db870f6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 94508 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Filter the downloads_clean dataset for the validation date range (July 2021 to December 2022)\n",
        "start_date = '2021-07-01'\n",
        "end_date = '2022-12-31'\n",
        "downloads_filtered = downloads_clean[(downloads_clean['DownloadDate'] >= start_date) &\n",
        "                                     (downloads_clean['DownloadDate'] <= end_date)]\n",
        "\n",
        "# Filter for assets published before December 31, 2021 (assets that were in training)\n",
        "docs_pre_dec2021 = docs_data_cleaned[docs_data_cleaned['StartPublishing'] < '2021-12-31']\n",
        "\n",
        "# Filter the downloads in validation set to those associated with assets published before Dec 2021\n",
        "downloads_pre_dec2021 = downloads_filtered[downloads_filtered['DMSId'].isin(docs_pre_dec2021['DMSId'])]\n",
        "\n",
        "# Group the downloads by month for total downloads and downloads for pre-Dec 2021 assets\n",
        "downloads_by_month = downloads_filtered.groupby(downloads_filtered['DownloadDate'].dt.to_period('M')).size()\n",
        "downloads_pre_dec2021_by_month = downloads_pre_dec2021.groupby(downloads_pre_dec2021['DownloadDate'].dt.to_period('M')).size()\n",
        "\n",
        "# Convert to datetime index for plotting\n",
        "downloads_by_month.index = downloads_by_month.index.to_timestamp()\n",
        "downloads_pre_dec2021_by_month.index = downloads_pre_dec2021_by_month.index.to_timestamp()\n",
        "\n",
        "# Plot the data\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(downloads_by_month.index, downloads_by_month, marker='o', label='Total Downloads', color='blue')\n",
        "plt.plot(downloads_pre_dec2021_by_month.index, downloads_pre_dec2021_by_month, marker='o', label='Downloads from Pre-Dec 2021 Assets', color='orange')\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('Month')\n",
        "plt.ylabel('Number of Downloads')\n",
        "plt.title('Monthly Downloads from July 2021 to December 2022')\n",
        "plt.xticks(rotation=45)\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "\n",
        "# Show the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LCj1k_4ylWeW",
      "metadata": {
        "id": "LCj1k_4ylWeW"
      },
      "source": [
        "what does this graph tell us ?\n",
        "\n",
        "when we train our validation data, we should ignore the inital few months after the threshold date between training and validation.\n",
        "\n",
        "why ?  \n",
        "1. trainin assets do appear in validation set, so intial few month would only contain assets recently published assets after the validaiton date. So it gives a biased"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3HlltamglWeW",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "3HlltamglWeW",
        "outputId": "bb9827a4-a877-4584-ad33-5820d211be84"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 78476 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 80008 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Define the correct time window for validation (1 July 2022 to 30 June 2023)\n",
        "start_date = '2022-07-01'\n",
        "end_date = '2023-06-30'\n",
        "downloads_filtered = downloads_clean[(downloads_clean['DownloadDate'] >= start_date) &\n",
        "                                     (downloads_clean['DownloadDate'] <= end_date)]\n",
        "\n",
        "# Filter for assets published before 31 December 2021 (new cutoff date for training)\n",
        "docs_pre_dec2021 = docs_data_cleaned[docs_data_cleaned['StartPublishing'] < '2021-12-31']\n",
        "downloads_pre_dec2021 = downloads_filtered[downloads_filtered['DMSId'].isin(docs_pre_dec2021['DMSId'])]\n",
        "\n",
        "# Group the downloads by DMSId and month\n",
        "downloads_pre_dec2021_grouped = downloads_pre_dec2021.groupby([downloads_pre_dec2021['DownloadDate'].dt.to_period('M'), 'DMSId']).size()\n",
        "\n",
        "# Filter for assets with more than 2 downloads per month\n",
        "downloads_pre_dec2021_gt2 = downloads_pre_dec2021_grouped[downloads_pre_dec2021_grouped > 2].reset_index()\n",
        "\n",
        "# Count the number of unique assets with more than 2 downloads per month\n",
        "assets_gt2_by_month = downloads_pre_dec2021_gt2.groupby('DownloadDate')['DMSId'].nunique()\n",
        "\n",
        "# Convert to datetime index for plotting\n",
        "assets_gt2_by_month.index = assets_gt2_by_month.index.to_timestamp()\n",
        "\n",
        "# Plot the data for assets with more than 2 downloads per month\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(assets_gt2_by_month.index, assets_gt2_by_month, marker='o', color='purple', label='Assets with > 2 Monthly Downloads (Pre-Dec 2021)')\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('Month')\n",
        "plt.ylabel('Number of Assets')\n",
        "plt.title('Number of Pre-Dec 2021 Assets with >2 Monthly Downloads (July 2022 - June 2023)')\n",
        "plt.xticks(rotation=45)\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "\n",
        "# Show the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Now, filter for assets with more than 5 downloads per month\n",
        "downloads_pre_dec2021_gt5 = downloads_pre_dec2021_grouped[downloads_pre_dec2021_grouped > 5].reset_index()\n",
        "\n",
        "# Count the number of unique assets with more than 5 downloads per month\n",
        "assets_gt5_by_month = downloads_pre_dec2021_gt5.groupby('DownloadDate')['DMSId'].nunique()\n",
        "\n",
        "# Convert to datetime index for plotting\n",
        "assets_gt5_by_month.index = assets_gt5_by_month.index.to_timestamp()\n",
        "\n",
        "# Plot the data for assets with more than 5 downloads per month\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(assets_gt5_by_month.index, assets_gt5_by_month, marker='o', color='green', label='Assets with > 5 Monthly Downloads (Pre-Dec 2021)')\n",
        "\n",
        "# Add labels and title\n",
        "plt.xlabel('Month')\n",
        "plt.ylabel('Number of Assets')\n",
        "plt.title('Number of Pre-Dec 2021 Assets with >5 Monthly Downloads (July 2022 - June 2023)')\n",
        "plt.xticks(rotation=45)\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "\n",
        "# Show the plot\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c7qm_VYUlWeX",
      "metadata": {
        "id": "c7qm_VYUlWeX"
      },
      "source": [
        "**Key highlight:**\n",
        "While older assets (published pre-August 2020) are still being downloaded, the volume is low, and these assets generally do not contribute significantly to the download activity. Downloads for these older assets tend to average less than one per month. Therefore, excluding them from the validation and test sets does not significantly hinder our ability to identify popular assets. The model can focus on more recent assets, which are more relevant to current user engagement patterns."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "n_IODjcA84QJ",
      "metadata": {
        "id": "n_IODjcA84QJ"
      },
      "source": [
        "####Q1a Step 7 Splitting data\n",
        "\n",
        "Purpose: In this section, we split the dataset into training, validation, and test sets. The training set will fit our models, the validation set will be for tuning, and the test set will serve as unseen data for evaluation.\n",
        "\n",
        "**Importance of early Data Split:** Performing this split before in-depth EDA prevents data leakage and biased modeling decisions influenced by unseen data.\n",
        "\n",
        "**Data Drift Consideration:**\n",
        "- By splitting based on published date, we account for data drift, ensuring that the model is trained on older data and tested on more recent data, which simulates a real-world scenario where new data may have different patterns.\n",
        "\n",
        "**Why we split using publsihing date rather than downlaod date**\n",
        "1. Avoiding Data Leakage: Splitting by publishing date ensures that each DMSID asset appears in only one dataset (training, validation, or test), preventing overlap that could lead to data leakage.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dgBWi6Cc24fz",
      "metadata": {
        "id": "dgBWi6Cc24fz"
      },
      "source": [
        "**Unit of analysis and why does it affect our data split ?**\n",
        "\n",
        "The unit of analysis is defined (in Q3a) as the monthly download count for each asset, summarized in a download table for each DMSID at various \"as-at\" dates.\n",
        "\n",
        "The volume of data in the training, validation, and test sets depends on the number of rows in the download summary (including zero download), rather than just the total number of download transactions. To achieve a reasonable split, we must carefully examine the number of rows in this summary, which is influenced by the published date cut-offs for the training, validation, and testing sets.\n",
        "\n",
        "**Trial & Error approach to archieve reasoanle training/validation/testing split**:\n",
        "\n",
        "We have not yet found an elegant formula to determine direct cut-offs, as changes in the training and validation thresholds affect the number of DMSIDs and the corresponding row counts in each set. Due to technical difficulties, I haven't developed a refined code for this. However, after several trial-and-error approaches, we have established cut-off methods that yield a reasonable distribution of training, validation, and testing sets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aIq7X6J7Ea4M",
      "metadata": {
        "id": "aIq7X6J7Ea4M"
      },
      "source": [
        "The following codes to calucualte volumne of data within each dataset (training, validation, test)\n",
        "\n",
        "**Here is our selected published date cut off.**\n",
        "- Training set: Up to January 31, 2021\n",
        "- Validation set: From February 1, 2021, to August 31, 2022\n",
        "- Test set: From September 1, 2022, to April 30, 2024\n",
        "\n",
        "**Key consideration on downlaod windows**:\n",
        "- We applied the same cut-off dates to the monthly download windows because assets do not overlap between training, validation, and test sets. Using download windows beyond the publication date cut-offs would exclude the latest published assets, leading to misleading popularity patterns, as newer assets tend to be more popular.\n",
        "\n",
        "- The only exception is for the training set, where we consider downloads from 30 May 2020 onwards. This is because download volumes were unreliable before mid-2020, and including data before this date would distort the download patterns.\n",
        "\n",
        "**What the following Code does**:\n",
        "The code calculates the data volume for each dataset, aiming for around 80% in the training set and 10% each in the validation and test sets. This split ensures sufficient data for the model to learn effectively while maintaining enough data for reliable validation and testing.\n",
        "\n",
        "Reference : below codes are generated using ChatGPT4 with my word description"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "LgvrrXhFHstx",
      "metadata": {
        "id": "LgvrrXhFHstx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3fa1772d-8d23-4ad2-9448-dca90861417e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of rows in each dataset:\n",
            "Documents - Training: 6069\n",
            "Documents - Validation: 757\n",
            "Documents - Test: 725\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "\n",
        "# Convert StartPublishing to datetime format\n",
        "docs_data_cleaned['StartPublishing'] = pd.to_datetime(docs_data_cleaned['StartPublishing'])\n",
        "\n",
        "# Define the cut-off dates for each partition\n",
        "train_end_date = datetime(2021, 1, 31)\n",
        "val_start_date = datetime(2021, 2, 1)\n",
        "val_end_date = datetime(2022, 8, 31)\n",
        "test_start_date = datetime(2022,9, 1)\n",
        "test_end_date = datetime(2024, 4, 30)\n",
        "\n",
        "# Partition the documents based on these cut-off dates\n",
        "docs_train = docs_data_cleaned[docs_data_cleaned['StartPublishing'] <= train_end_date]\n",
        "docs_val = docs_data_cleaned[(docs_data_cleaned['StartPublishing'] >= val_start_date) &\n",
        "                             (docs_data_cleaned['StartPublishing'] <= val_end_date)]\n",
        "docs_test = docs_data_cleaned[(docs_data_cleaned['StartPublishing'] >= test_start_date) &\n",
        "                              (docs_data_cleaned['StartPublishing'] <= test_end_date)]\n",
        "\n",
        "# Define download cut-off periods for each partition\n",
        "train_download_start_date = datetime(2020, 5, 30)\n",
        "train_download_end_date = datetime(2021, 1, 31)\n",
        "val_download_start_date = datetime(2021, 2, 1)\n",
        "val_download_end_date = datetime(2022, 8, 31)\n",
        "test_download_start_date = datetime(2022, 9, 1)\n",
        "test_download_end_date = datetime(2024, 3, 30)\n",
        "\n",
        "# Function to filter downloads based on the DMSId and download date range\n",
        "def filter_downloads(docs_split, downloads_df, start_date, end_date):\n",
        "    filtered_downloads = downloads_df[\n",
        "        (downloads_df['DMSId'].isin(docs_split['DMSId'])) &\n",
        "        (downloads_df['DownloadDate'] >= start_date) &\n",
        "        (downloads_df['DownloadDate'] <= end_date)\n",
        "    ]\n",
        "    return filtered_downloads\n",
        "\n",
        "# Apply the time-based filtering to the downloads table\n",
        "downloads_train = filter_downloads(docs_train, downloads_clean, train_download_start_date, train_download_end_date)\n",
        "downloads_val = filter_downloads(docs_val, downloads_clean, val_download_start_date, val_download_end_date)\n",
        "downloads_test = filter_downloads(docs_test, downloads_clean, test_download_start_date, test_download_end_date)\n",
        "\n",
        "# Calculate total downloads and percentages\n",
        "total_downloads = downloads_train.shape[0]+downloads_val.shape[0]+downloads_test.shape[0]\n",
        "train_percentage = (downloads_train.shape[0] / total_downloads) * 100\n",
        "val_percentage = (downloads_val.shape[0] / total_downloads) * 100\n",
        "test_percentage = (downloads_test.shape[0] / total_downloads) * 100\n",
        "\n",
        "# Check that splits have been done correctly and show percentages\n",
        "print('Number of rows in each dataset:')\n",
        "print(f'Documents - Training: {docs_train.shape[0]}')\n",
        "print(f'Documents - Validation: {docs_val.shape[0]}')\n",
        "print(f'Documents - Test: {docs_test.shape[0]}')\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J6JIsc58Epxb",
      "metadata": {
        "id": "J6JIsc58Epxb"
      },
      "source": [
        "Successful Partitioning: The training set contains 6,069 documents, while the validation and test sets have 757 and 725 documents, respectively. This ensures ample historical data for training, with enough data for effective validation and testing."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "R8YSDXHiDd0R",
      "metadata": {
        "id": "R8YSDXHiDd0R"
      },
      "source": [
        "**Additional Check:**\n",
        "\n",
        "We'll examine the data split by calculating the percentage of data volume in each dataset and using a plot to visually illustrate how assets are distributed over the publication timeline. This ensures our split is proportionate and chronologically accurate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-_Xf01Ds4zJb",
      "metadata": {
        "id": "-_Xf01Ds4zJb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "06bdf6cb-1e66-4cbf-d8f0-922534848d53"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-63-48e5288dea4d>:7: FutureWarning: 'M' is deprecated and will be removed in a future version, please use 'ME' instead.\n",
            "  as_at_dates = pd.date_range(start='2020-05-01', end='2024-03-01', freq='M')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 305496 entries, 0 to 347299\n",
            "Data columns (total 3 columns):\n",
            " #   Column           Non-Null Count   Dtype         \n",
            "---  ------           --------------   -----         \n",
            " 0   DMSId            305496 non-null  int64         \n",
            " 1   AsAtDate         305496 non-null  datetime64[ns]\n",
            " 2   StartPublishing  305496 non-null  datetime64[ns]\n",
            "dtypes: datetime64[ns](2), int64(1)\n",
            "memory usage: 9.3 MB\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Example structure of your dataset\n",
        "# docs_data_cleaned: ['DMSId', 'StartPublishing']\n",
        "\n",
        "# Step 1: Create a list of all possible \"as-at\" dates (46 months from May 2020 to March 2024)\n",
        "as_at_dates = pd.date_range(start='2020-05-01', end='2024-03-01', freq='M')\n",
        "\n",
        "# Step 2: Expand each DMSId to have all 46 as-at dates (default 46 rows per DMSId)\n",
        "docs_expanded = docs_data_cleaned.assign(key=1).merge(pd.DataFrame({'AsAtDate': as_at_dates, 'key': 1}), on='key').drop('key', axis=1)\n",
        "\n",
        "# Step 3: Remove rows where StartPublishing date is later than AsAtDate\n",
        "docs_expanded = docs_expanded[docs_expanded['StartPublishing'] <= docs_expanded['AsAtDate']]\n",
        "\n",
        "# Step 4: Select only the necessary columns: DMSId, AsAtDate, and StartPublishing\n",
        "docs_expanded = docs_expanded[['DMSId', 'AsAtDate', 'StartPublishing']]\n",
        "\n",
        "# Step 5: Sort by DMSId and AsAtDate for clarity\n",
        "docs_expanded.sort_values(by=['DMSId', 'AsAtDate'], inplace=True)\n",
        "\n",
        "# Step 6: View the final table\n",
        "docs_expanded.head()\n",
        "docs_expanded.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Qb8UZrld4tX4",
      "metadata": {
        "id": "Qb8UZrld4tX4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2bc83bbb-9ff2-49e2-c4fe-ba22957d7df7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Count the number of rows for later NN\n",
            "Training set (expanded): 53401 rows\n",
            "Validation set (expanded): 7107 rows\n",
            "Testing set (expanded): 6809 rows\n",
            "------------\n",
            "Original Partition Results: in %\n",
            "79.32765868948408\n",
            "10.557511475555952\n",
            "10.114829834959965\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "\n",
        "# Assuming docs_expanded has been prepared and contains multiple rows per asset (for each download window)\n",
        "\n",
        "# Convert StartPublishing to datetime format (if not already done)\n",
        "docs_expanded['StartPublishing'] = pd.to_datetime(docs_expanded['StartPublishing'])\n",
        "docs_expanded['AsAtDate'] = pd.to_datetime(docs_expanded['AsAtDate'])\n",
        "\n",
        "# Define the cut-off dates for each partition\n",
        "train_end_date = datetime(2021, 1, 31)  # Training set cutoff\n",
        "val_start_date = datetime(2021, 2, 1)   # Validation start date\n",
        "val_end_date = datetime(2022, 8, 31)    # Validation end date\n",
        "test_start_date = datetime(2022, 9, 1)  # Testing start date\n",
        "test_end_date = datetime(2024, 4, 30)   # Testing end date\n",
        "\n",
        "# Partition docs_expanded based on both StartPublishing and AsAtDate\n",
        "# Training set: StartPublishing <= train_end_date, AsAtDate <= train_end_date\n",
        "docs_train_expanded = docs_expanded[(docs_expanded['StartPublishing'] <= train_end_date) &\n",
        "                                    (docs_expanded['AsAtDate'] <= train_end_date)]\n",
        "\n",
        "# Validation set: StartPublishing between val_start_date and val_end_date, AsAtDate <= val_end_date\n",
        "docs_val_expanded = docs_expanded[(docs_expanded['StartPublishing'] >= val_start_date) &\n",
        "                                  (docs_expanded['StartPublishing'] <= val_end_date) &\n",
        "                                  (docs_expanded['AsAtDate'] <= val_end_date)]\n",
        "\n",
        "# Testing set: StartPublishing >= test_start_date, AsAtDate <= test_end_date\n",
        "docs_test_expanded = docs_expanded[(docs_expanded['StartPublishing'] >= test_start_date) &\n",
        "                                   (docs_expanded['AsAtDate'] <= test_end_date)]\n",
        "\n",
        "# Count the number of rows in docs_expanded (i.e., download possibilities)\n",
        "print(\"Count the number of rows for later NN\")\n",
        "print(f\"Training set (expanded): {docs_train_expanded.shape[0]} rows\")\n",
        "print(f\"Validation set (expanded): {docs_val_expanded.shape[0]} rows\")\n",
        "print(f\"Testing set (expanded): {docs_test_expanded.shape[0]} rows\")\n",
        "\n",
        "\n",
        "# Calculate total rows and proportions for the original partition\n",
        "total_rows = docs_train_expanded.shape[0] + docs_val_expanded.shape[0] + docs_test_expanded.shape[0]\n",
        "train_percentage = (docs_train_expanded.shape[0] / total_rows) * 100\n",
        "val_percentage = (docs_val_expanded.shape[0] / total_rows) * 100\n",
        "test_percentage = (docs_test_expanded.shape[0] / total_rows) * 100\n",
        "\n",
        "# Display the original results\n",
        "print('------------')\n",
        "print(\"Original Partition Results: in %\")\n",
        "print(train_percentage)\n",
        "print(val_percentage)\n",
        "print(test_percentage)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7XnRRfnsJb_v",
      "metadata": {
        "id": "7XnRRfnsJb_v"
      },
      "source": [
        "Training Set: 53,401 rows (79.3%),  Validation Set: 7,107 rows (10.6%) and Testing Set: 6,809 rows (10.1%)\n",
        "\n",
        "These percentages confirm that  data split aligns well with our target distribution, ensuring the training set is comprehensive enough for learning while maintaining sufficient data for validation and testing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "P3nC5tQpDdSV",
      "metadata": {
        "id": "P3nC5tQpDdSV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 616
        },
        "outputId": "43dae971-79cc-4fbc-bfd2-656a6ff120c7"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 45536 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training set start: 2018-08-01 00:00:00, end: 2021-01-28 00:00:00\n",
            "Validation set start: 2021-02-03 00:00:00, end: 2022-08-31 00:00:00\n",
            "Test set start: 2022-09-04 00:00:00, end: 2024-03-24 00:00:00\n"
          ]
        }
      ],
      "source": [
        "# Plot the distribution of StartPublishing dates for each set\n",
        "plt.figure(figsize=(10, 6))\n",
        "\n",
        "plt.hist(docs_train['StartPublishing'], bins=20, alpha=0.7, label='Training Set')\n",
        "plt.hist(docs_val['StartPublishing'], bins=20, alpha=0.7, label='Validation Set')\n",
        "plt.hist(docs_test['StartPublishing'], bins=20, alpha=0.7, label='Test Set')\n",
        "\n",
        "plt.xlabel('StartPublishing Date')\n",
        "plt.ylabel('Number of Documents')\n",
        "plt.title('Distribution of StartPublishing Dates Across Training, Validation, and Test Sets')\n",
        "plt.legend(loc='upper left')\n",
        "\n",
        "plt.show()\n",
        "\n",
        "# Checking the start and end dates for the training, validation, and test sets\n",
        "train_start = docs_train['StartPublishing'].min()\n",
        "train_end = docs_train['StartPublishing'].max()\n",
        "\n",
        "val_start = docs_val['StartPublishing'].min()\n",
        "val_end = docs_val['StartPublishing'].max()\n",
        "\n",
        "test_start = docs_test['StartPublishing'].min()\n",
        "test_end = docs_test['StartPublishing'].max()\n",
        "\n",
        "print(f'Training set start: {train_start}, end: {train_end}')\n",
        "print(f'Validation set start: {val_start}, end: {val_end}')\n",
        "print(f'Test set start: {test_start}, end: {test_end}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3DSQ3DrfD_gD",
      "metadata": {
        "id": "3DSQ3DrfD_gD"
      },
      "source": [
        "The plot visually confirms the chronological partitioning of the data:\n",
        "\n",
        "- Training Set (Blue): Contains older assets, predominantly those published before 2021, providing a strong foundation for learning historical patterns.\n",
        "- Validation Set (Orange): Represents assets published between early 2021 and mid-2022, allowing the model to fine-tune against more recent trends.\n",
        "- Test Set (Green): Includes the most recent assets, primarily from late 2022 to early 2024, ensuring that the model is evaluated on the latest data patterns."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f01l2r5HGfDy",
      "metadata": {
        "id": "f01l2r5HGfDy"
      },
      "source": [
        "**Check data format after splitting**\n",
        "\n",
        "The purpose is to ensure consistency, check for missing values or anomalies, and confirm that the training, validation, and test sets are correctly formatted. This ensures smooth model training and evaluation, reducing the risk of errors later in the process.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "17-u4q7AGgXn",
      "metadata": {
        "id": "17-u4q7AGgXn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "64d34089-a8e4-4ceb-adff-815e06699202"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "data info for validation set\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 757 entries, 2078 to 6866\n",
            "Data columns (total 22 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    757 non-null    int64         \n",
            " 1   FileName                 757 non-null    object        \n",
            " 2   FileType                 757 non-null    object        \n",
            " 3   FileSize                 757 non-null    int64         \n",
            " 4   Title                    757 non-null    object        \n",
            " 5   StartPublishing          757 non-null    datetime64[ns]\n",
            " 6   StopPublishing           757 non-null    object        \n",
            " 7   Cpd                      757 non-null    int64         \n",
            " 8   Format                   757 non-null    object        \n",
            " 9   Level                    757 non-null    object        \n",
            " 10  Description              554 non-null    object        \n",
            " 11  Region                   757 non-null    object        \n",
            " 12  Copyright                757 non-null    object        \n",
            " 13  EventAgeGroup            212 non-null    object        \n",
            " 14  EventType                320 non-null    object        \n",
            " 15  Link                     757 non-null    object        \n",
            " 16  Tags                     518 non-null    object        \n",
            " 17  DownloadCount            757 non-null    int64         \n",
            " 18  DurationOnDMS            757 non-null    float64       \n",
            " 19  AnnualizedDownloadCount  757 non-null    float64       \n",
            " 20  YearOfStartPublishing    757 non-null    int32         \n",
            " 21  Year                     757 non-null    int32         \n",
            "dtypes: datetime64[ns](1), float64(2), int32(2), int64(4), object(13)\n",
            "memory usage: 130.1+ KB\n",
            "None\n",
            "------------------\n",
            "data info for testing set\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 725 entries, 2174 to 7550\n",
            "Data columns (total 22 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    725 non-null    int64         \n",
            " 1   FileName                 725 non-null    object        \n",
            " 2   FileType                 725 non-null    object        \n",
            " 3   FileSize                 725 non-null    int64         \n",
            " 4   Title                    725 non-null    object        \n",
            " 5   StartPublishing          725 non-null    datetime64[ns]\n",
            " 6   StopPublishing           725 non-null    object        \n",
            " 7   Cpd                      725 non-null    int64         \n",
            " 8   Format                   725 non-null    object        \n",
            " 9   Level                    725 non-null    object        \n",
            " 10  Description              598 non-null    object        \n",
            " 11  Region                   725 non-null    object        \n",
            " 12  Copyright                725 non-null    object        \n",
            " 13  EventAgeGroup            560 non-null    object        \n",
            " 14  EventType                559 non-null    object        \n",
            " 15  Link                     725 non-null    object        \n",
            " 16  Tags                     567 non-null    object        \n",
            " 17  DownloadCount            725 non-null    int64         \n",
            " 18  DurationOnDMS            725 non-null    float64       \n",
            " 19  AnnualizedDownloadCount  725 non-null    float64       \n",
            " 20  YearOfStartPublishing    725 non-null    int32         \n",
            " 21  Year                     725 non-null    int32         \n",
            "dtypes: datetime64[ns](1), float64(2), int32(2), int64(4), object(13)\n",
            "memory usage: 124.6+ KB\n",
            "None\n",
            "------------------\n",
            "data info for training set\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Index: 6069 entries, 0 to 6137\n",
            "Data columns (total 22 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    6069 non-null   int64         \n",
            " 1   FileName                 6069 non-null   object        \n",
            " 2   FileType                 6069 non-null   object        \n",
            " 3   FileSize                 6069 non-null   int64         \n",
            " 4   Title                    6065 non-null   object        \n",
            " 5   StartPublishing          6069 non-null   datetime64[ns]\n",
            " 6   StopPublishing           6069 non-null   object        \n",
            " 7   Cpd                      6069 non-null   int64         \n",
            " 8   Format                   6065 non-null   object        \n",
            " 9   Level                    6065 non-null   object        \n",
            " 10  Description              5636 non-null   object        \n",
            " 11  Region                   6065 non-null   object        \n",
            " 12  Copyright                6065 non-null   object        \n",
            " 13  EventAgeGroup            3929 non-null   object        \n",
            " 14  EventType                4124 non-null   object        \n",
            " 15  Link                     6069 non-null   object        \n",
            " 16  Tags                     5885 non-null   object        \n",
            " 17  DownloadCount            6069 non-null   int64         \n",
            " 18  DurationOnDMS            6069 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  6069 non-null   float64       \n",
            " 20  YearOfStartPublishing    6069 non-null   int32         \n",
            " 21  Year                     6069 non-null   int32         \n",
            "dtypes: datetime64[ns](1), float64(2), int32(2), int64(4), object(13)\n",
            "memory usage: 1.0+ MB\n",
            "None\n"
          ]
        }
      ],
      "source": [
        "print(\"data info for validation set\")\n",
        "print(docs_val.info())\n",
        "print('------------------')\n",
        "print(\"data info for testing set\")\n",
        "print(docs_test.info())\n",
        "print('------------------')\n",
        "print(\"data info for training set\")\n",
        "print(docs_train.info())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fnn8cW5DLlIq",
      "metadata": {
        "id": "fnn8cW5DLlIq"
      },
      "source": [
        "The number of rows for each dataset appear consistent with our expecation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Er-ErGmLLx_S",
      "metadata": {
        "id": "Er-ErGmLLx_S"
      },
      "source": [
        "**Data Splitting Summary**\n",
        "\n",
        "In this section, we successfully split the dataset into training, validation, and test sets to prepare for modeling. The training set contains 6,069 documents, the validation set has 757, and the test set holds 725 documents, achieving the desired 80/10/10 split ratio.\n",
        "\n",
        "We verified that the split respects chronological order, with the training set covering older assets up to January 2021, the validation set spanning February 2021 to August 2022, and the test set from September 2022 to April 2024, ensuring alignment with observed data drift.\n",
        "\n",
        "Further checks confirmed that each set is correctly formatted, free of missing values, and has the expected proportion of rows for later neural network training.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QjtAIQZwwQDr",
      "metadata": {
        "id": "QjtAIQZwwQDr"
      },
      "source": [
        "#### **Q1a Step 8 EDA FileName**\n",
        "\n",
        "**Outline**\n",
        "- As a starting point, we apply tutorial function to perform EDA for FileName visualizations for each variable based on its type.\n",
        "- Examine the missing or invalid values.\n",
        "- Bar chart to visulaise most filenames that have appear more than once.*italicized text*\n",
        "\n",
        "\n",
        "My expecation:\n",
        "- A perfect FileName column should have distinct filenames for each asset.\n",
        "- Filenames should be meaningful and provide users with some information about the content of the file.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "tepZd0vVgR2z",
      "metadata": {
        "id": "tepZd0vVgR2z",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "cdf0951b-6f28-4807-f4a7-04ec8f5c5d8d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 5986\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: FileName\n",
            "130705ApraCps220.pdf                                                                                          3\n",
            "Concurrent 4 A.mp3                                                                                            2\n",
            "Actuaries Summit 2020 - Developments in Multi-factor Continuous Time Mortality Modelling.pdf                  2\n",
            "Actuaries Summit 2020 - Actuaries and Discrimination - Report of the Anti-Discrimination Working Group.pdf    2\n",
            "Tony Coleman.pdf                                                                                              2\n",
            "Name: count, dtype: int64\n",
            "Number of rows with FileName values that appear more than once: 165\n",
            "---------------\n",
            "Data type: <class 'str'>, Count: 6069\n",
            "Sample values: ['C8B_ManWhoAteHimself-Presentation.pdf', 'INS160405Audio.mp3', '2c-conv03presgregor.pdf']\n",
            "\n",
            "---------------\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 128820 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# EDA for FileName\n",
        "df = pd.DataFrame(docs_train)\n",
        "eda_describe_column(df, 'FileName')\n",
        "\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'FileName')\n",
        "print(\"---------------\")\n",
        "\n",
        "\n",
        "# Bar chart of the most common filenames\n",
        "top_filenames = df['FileName'].value_counts().head(10)\n",
        "plt.figure(figsize=(10, 6))\n",
        "top_filenames.plot(kind='bar')\n",
        "plt.title('Top 10 Most Common Filenames')\n",
        "plt.xlabel('FileName')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=90)\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eIzCLfXzgaC1",
      "metadata": {
        "id": "eIzCLfXzgaC1"
      },
      "source": [
        "Summary of EDA on File Names:\n",
        "Key Findings:\n",
        "- Uniqueness: Out of 5,956 filenames, 5,878 are unique, indicating that most assets have distinct filenames, but 165 rows have filenames that appear more than once.\n",
        "- No Missing/Invalid Values: All entries in the FileName column are complete, with no missing or invalid values.\n",
        "- Common Filenames: The top 10 most common filenames include a few that are repeated up to 3 times.\n",
        "\n",
        "**Insights:**\n",
        "- The occurrence of common file names may indicate that different assets share the same file name. Drawing from my domain knowledge, this is quite typical for the Actuaries Summit, where multiple assets, such as PPT slides, recordings, and associated publications, are often generated from a single discussion.\n",
        "\n",
        "- Since there are no missing or invalid values, the data quality for file names is generally good."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "n9U9x-ytgXb1",
      "metadata": {
        "id": "n9U9x-ytgXb1"
      },
      "source": [
        "#### **Q1a Step 9 EDA FileType**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function.\n",
        "- Check for any missing or invalid values.\n",
        "- Since FileType is a categorical variable, visualize the frequency of each file type using a bar chart.\n",
        "\n",
        "My expecation:\n",
        "\n",
        "- According to the data dictionary, we expect up to 13 distinct file types, such as PDF, PPT, MP3, etc.\n",
        "- PDF, PPT, Word documents, and videos are expected to be the most common file types.\n",
        "- There should be no missing or invalid data, as each asset is required to have a file type.\n",
        "- All entries in this column should represent actual file types and not contain irrelevant or erroneous values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wQpWSbTH1mU-",
      "metadata": {
        "id": "wQpWSbTH1mU-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 862
        },
        "outputId": "986c98dc-e170-4683-a892-aaaa4660644a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 12\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: FileType\n",
            "pdf     3559\n",
            "mp3     1594\n",
            "html     735\n",
            "docx     144\n",
            "wma       16\n",
            "Name: count, dtype: int64\n",
            "Number of rows with FileType values that appear more than once: 6066\n",
            "---------------\n",
            "Data type: <class 'str'>, Count: 6069\n",
            "Sample values: ['mp3', 'html', 'html']\n",
            "\n",
            "---------------\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 36196 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# EDA for FileType\n",
        "\n",
        "df = pd.DataFrame(docs_train)\n",
        "eda_describe_column(df, 'FileType')\n",
        "\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'FileType')\n",
        "print(\"---------------\")\n",
        "\n",
        "\n",
        "\n",
        "# Bar chart of the most common filenames\n",
        "top_filenames = df['FileType'].value_counts().head(12)\n",
        "plt.figure(figsize=(12, 6))\n",
        "top_filenames.plot(kind='bar')\n",
        "plt.title('Top 12 Most Common FileType')\n",
        "plt.xlabel('FileType')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gcJKXMnd0eXn",
      "metadata": {
        "id": "gcJKXMnd0eXn"
      },
      "source": [
        "Obeservation of FileType EDA:\n",
        "- There are 12 unique file types, with a few rarer types like WMA, PPT, and ICS. which is slightly fewer than the 13 expected from data dictionary.\n",
        "- No Missing or Invalid Values: All entries have valid file types, with no missing or erroneous data.\n",
        "- The most common file types are 'pdf' (3,559), followed by 'mp3' (1,594) and 'html' (735). Less common file types include 'docx' and 'wma'.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Cac7b6E4BPry",
      "metadata": {
        "id": "Cac7b6E4BPry"
      },
      "source": [
        "We've observed 12 unique file types, but the data dictionary suggests there should be 13. To verify if the missing file type is very rare and only appears in a split that wasn’t analyzed, we should check the entire dataset, including the validation and test sets, to ensure that no file types are missing due to the allocation of assets across different splits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9JxXqRo9BIa_",
      "metadata": {
        "id": "9JxXqRo9BIa_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fbe7f7f9-3a19-418e-dbfa-2b680a4d94b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique File Types in Combined Data: 12\n"
          ]
        }
      ],
      "source": [
        "# Combine all data splits\n",
        "combined_data = pd.concat([docs_train, docs_val, docs_test])\n",
        "\n",
        "# Perform the same EDA on the combined data\n",
        "unique_file_types = combined_data['FileType'].nunique()\n",
        "print(f\"Unique File Types in Combined Data: {unique_file_types}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rK_ldSZ8BlBa",
      "metadata": {
        "id": "rK_ldSZ8BlBa"
      },
      "source": [
        "we have confirmed the whole dataset have only 12 types"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ggMK9j0CCVxm",
      "metadata": {
        "id": "ggMK9j0CCVxm"
      },
      "source": [
        "**Abnormalities:**\n",
        "-  ICS files are calendar files saved in a calendar format, typically used for meeting requests or schedules. It is unlikely that ICS files should be included in a DMS unless they are related to meetings or schedules.\n",
        "- Investigate rows containing ICS files, as these may indicate inconsistencies or errors. This will be addressed during the data cleaning step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "z74SUivVAIJV",
      "metadata": {
        "id": "z74SUivVAIJV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "755191c1-de5d-4c16-b3dd-dd5156049e12"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Rows with ICS FileType:\n",
            "        DMSId                             FileName FileType  FileSize  \\\n",
            "5120  4362377  ProgressingYourInvestmentCareer.ics      ics      9418   \n",
            "\n",
            "                                           Title StartPublishing  \\\n",
            "5120  INS11 - Progressing Your Investment Career      2019-08-01   \n",
            "\n",
            "     StopPublishing  Cpd                 Format Level  ...  Copyright  \\\n",
            "5120              0    2  Podcast/Vodcast/Audio   All  ...  Institute   \n",
            "\n",
            "     EventAgeGroup       EventType  \\\n",
            "5120           All  Insights/Other   \n",
            "\n",
            "                                                   Link  \\\n",
            "5120  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "\n",
            "                                                   Tags DownloadCount  \\\n",
            "5120  Personal Effectiveness &amp; Leadership|Person...             1   \n",
            "\n",
            "     DurationOnDMS  AnnualizedDownloadCount  YearOfStartPublishing  Year  \n",
            "5120           4.7                     0.21                   2019  2019  \n",
            "\n",
            "[1 rows x 22 columns]\n"
          ]
        }
      ],
      "source": [
        "# Identify rows with ICS files\n",
        "ics_files = df[df['FileType'] == 'ics']\n",
        "\n",
        "# Display rows with ICS files\n",
        "print(\"Rows with ICS FileType:\")\n",
        "print(ics_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "33esobcNAgXz",
      "metadata": {
        "id": "33esobcNAgXz"
      },
      "source": [
        "**Data Incorrect value of ICS and Data Cleaning**\n",
        "\n",
        "*Issue Identified*:\n",
        "The asset titled \"Progressing Your Investment Career\" was initially classified as an ICS file type. However, it is actually a PDF file that contains a link to an ICS schedule.\n",
        "\n",
        "\n",
        "**Data Cleaning**:\n",
        "\n",
        "Manually correct the file type from ICS to PDF in the dataset.\n",
        "\n",
        "**Rationale:**\n",
        "\n",
        "Addressing this inconsistency ensures the dataset remains clean, accurate, and reliable, which is essential for effective analysis and decision-making. By correcting this classification, we maintain the integrity of our data, enabling more accurate insights in subsequent analysis.\n",
        "\n",
        "**We will perform this cleaning step during the data cleaning phase, once all EDA activities are completed.**"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "w4Qx4yRHPGf8",
      "metadata": {
        "id": "w4Qx4yRHPGf8"
      },
      "source": [
        "**Summary of FileType EDA:**\n",
        "\n",
        "The FileType column revealed 12 unique types, whcih is less data dictionary suggestion. PDFs and MP3s are the most common file type.\n",
        "\n",
        "An inconsistency with an ICS file (actually a PDF) should be corrected to ensure data integrity.\n",
        "\n",
        "Given the high cardinality, in later part of quetion, similar file types will be grouped into broader categories like \"Document,\" \"Audio,\" and \"Presentation\" to simplify analysis.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HWBWqhc0QMFZ",
      "metadata": {
        "id": "HWBWqhc0QMFZ"
      },
      "source": [
        "#### **Q1a Step 9  EDA FileSize**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function.\n",
        "- Check for any missing or invalid values.\n",
        "- Since FileType is a numerical variable, visualize the distribution using a histogram and boxplot chart.\n",
        "\n",
        "My Expectation:\n",
        "- We expect the FileSize column to be an integer.\n",
        "- Each asset should have a distinct file size because it is very unlikely for two distinct files to have an identical size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WL6B4IwiQK__",
      "metadata": {
        "id": "WL6B4IwiQK__",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d29eacc4-92c6-47ff-ec3b-499d0b0e2e8c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean: 4015981.5068380293\n",
            "Standard Deviation: 7419580.796027278\n",
            "Skewness: 3.9758724392450557\n",
            "Minimum: 1054\n",
            "Maximum: 142884489\n",
            "Maximum Number of Decimal Places: 0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 20680 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------------\n",
            "Data type: <class 'int'>, Count: 6069\n",
            "Sample values: [9123644, 6315, 450099]\n",
            "\n",
            "---------------\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 16964 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "eda_numeric_column(df,'FileSize')\n",
        "\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'FileSize')\n",
        "print(\"---------------\")\n",
        "\n",
        "# Box Plot\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.boxplot(x=docs_data['FileSize'])\n",
        "plt.title('Box Plot of FileSize')\n",
        "plt.xlabel('FileSize')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "RquPm1k_PSZ5",
      "metadata": {
        "id": "RquPm1k_PSZ5"
      },
      "source": [
        "**Key findings of FileSize Analysis**:\n",
        "- The FileSize column is correctly identified as an integer with no missing or invalid values, covering 6,069 assets. The mean file size is approximately 4MB, but the data is highly skewed (skewness of 3.98), indicating the presence of very large files. The minimum file size is 1KB, and the maximum is around 143MB.\n",
        "- This skewness suggests potential outliers, which might need further examination to ensure they don't distort the analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "767rZfn_VM3w",
      "metadata": {
        "id": "767rZfn_VM3w"
      },
      "source": [
        "#####**Outlier Analysis**\n",
        "Purpose of Outlier Analysis:\n",
        "\n",
        "The purpose of this outlier analysis is to investigate whether large file sizes are justifiable based on the file type. For instance, larger sizes are expected for MP3 or video files, while unusually large sizes for file types like PDF or PPTX might indicate an issue or a special case. This analysis helps ensure data integrity and identify potential anomalies that could affect the model's performance.\n",
        "\n",
        "\n",
        "Reference: Below codes were generated using ChatGPT4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Z8xgaGjxB9la",
      "metadata": {
        "id": "Z8xgaGjxB9la"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "def find_outliers(df, column='FileSize'):\n",
        "  \"\"\"\n",
        "  Identifies outliers in a specified column of a DataFrame using the IQR method.\n",
        "\n",
        "  Args:\n",
        "    df: The pandas DataFrame.\n",
        "    column: The name of the column to analyze.\n",
        "\n",
        "  Returns:\n",
        "    A DataFrame containing only the outlier rows.\n",
        "  \"\"\"\n",
        "  Q1 = df[column].quantile(0.25)\n",
        "  Q3 = df[column].quantile(0.75)\n",
        "  IQR = Q3 - Q1\n",
        "  lower_bound = Q1 - 1.5 * IQR\n",
        "  upper_bound = Q3 + 1.5 * IQR\n",
        "  return df[(df[column] < lower_bound) | (df[column] > upper_bound)]\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "lpqlzmP5GpyU",
      "metadata": {
        "id": "lpqlzmP5GpyU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 807
        },
        "outputId": "a950d465-21b2-4b50-8992-bbe50870ac49"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 67432 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Assuming 'find_outliers' is a function that returns outlier data points\n",
        "outliers = find_outliers(docs_train)\n",
        "\n",
        "# Scatter plot of file size vs file type, with file sizes represented by point size\n",
        "plt.figure(figsize=(12, 8))\n",
        "sns.scatterplot(data=outliers, x='FileType', y='FileSize', hue='FileType', size='FileSize', sizes=(20, 200), alpha=0.7)\n",
        "plt.yscale('log')  # Log scale to manage the wide range of file sizes\n",
        "plt.xlabel('File Type')\n",
        "plt.ylabel('File Size (Log Scale)')\n",
        "plt.title('Scatter Plot of Outlier File Sizes by File Type')\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dSsiQiVwRoZR",
      "metadata": {
        "id": "dSsiQiVwRoZR"
      },
      "source": [
        "It is uncommon to see large pptx file hence we do a spot check on the outlier assets for justification."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "I0N4iiVFfkUs",
      "metadata": {
        "id": "I0N4iiVFfkUs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9b223b3-893e-46a2-ae26-785c447c92ec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Outlier pptx files:\n",
            "        DMSId                  FileName FileType   FileSize  \\\n",
            "4686  4361700  Plenary 5 Pete Goss.pptx     pptx  142884489   \n",
            "\n",
            "                      Title StartPublishing StopPublishing  Cpd  \\\n",
            "4686  Actuaries Summit 2019      2019-06-04              0    1   \n",
            "\n",
            "                   Format Level  ...  Copyright EventAgeGroup EventType  \\\n",
            "4686  Presentation Slides   All  ...  Institute           NaN     Major   \n",
            "\n",
            "                                                   Link  \\\n",
            "4686  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "\n",
            "                                                   Tags DownloadCount  \\\n",
            "4686  Banking|Business Acumen|Data Science|General I...             3   \n",
            "\n",
            "     DurationOnDMS  AnnualizedDownloadCount  YearOfStartPublishing  Year  \n",
            "4686           4.9                     0.61                   2019  2019  \n",
            "\n",
            "[1 rows x 22 columns]\n",
            "\n",
            "Outlier pptx files with download counts:\n",
            "     DMSId                  FileName   FileSize  Number of Downloads\n",
            "0  4361700  Plenary 5 Pete Goss.pptx  142884489                    2\n"
          ]
        }
      ],
      "source": [
        "# Filter the outliers for pptx file type\n",
        "pptx_outliers = outliers[outliers['FileType'] == 'pptx']\n",
        "\n",
        "# Display the outlier pptx files\n",
        "print(\"Outlier pptx files:\")\n",
        "print(pptx_outliers)\n",
        "\n",
        "# Assuming downloads_train contains the download information for the training set\n",
        "pptx_outliers_with_downloads = pptx_outliers.merge(downloads_train, on='DMSId', how='left')\n",
        "\n",
        "# Count the number of downloads for each pptx outlier\n",
        "download_counts_pptx = pptx_outliers_with_downloads.groupby('DMSId').size().reset_index(name='Number of Downloads')\n",
        "\n",
        "# Merge the download counts back to the pptx_outliers data to show both the outliers and their download counts\n",
        "pptx_outliers_with_counts = pptx_outliers.merge(download_counts_pptx, on='DMSId', how='left')\n",
        "\n",
        "# Display the pptx outliers along with their download counts\n",
        "print(\"\\nOutlier pptx files with download counts:\")\n",
        "print(pptx_outliers_with_counts[['DMSId', 'FileName', 'FileSize', 'Number of Downloads']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uYV2RY-QiPwC",
      "metadata": {
        "id": "uYV2RY-QiPwC"
      },
      "source": [
        "**Data cleaning - Remove outlier**\n",
        "\n",
        "After checking the assets link,  asset \"Plenary 5 Pete Goss.pptx\" contains personal photos unrelated to actuarial content, and it cannot be categorized under any relevant topic. The content is about a person driving a boat across Europe, making it irrelevant and of little value to the purpose of Acuaries CDP. Therefore, I have decided to delete this outlier. I will proceed with data cleaning in the next section."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oZw7uKkPUMc-",
      "metadata": {
        "id": "oZw7uKkPUMc-"
      },
      "source": [
        "**Summary of FileSize EDA**:\n",
        "\n",
        "File sizes are heavily skewed to the right, with a mean of approximately 4MB and significant outliers contributing to high skewness.\n",
        "\n",
        "Significant outliers were identified, particularly in 2018 and 2019, where some files are much larger than others. These outliers are mostly MP3s and PDFs, which are considered reasonable. However, unusually large PPTX files were flagged for further investigation.\n",
        "\n",
        "**Outlier Decision:**\n",
        "\n",
        "The asset \"Plenary 5 Pete Goss.pptx\" was identified as irrelevant and of little value, containing personal photos unrelated to actuarial content. This outlier will be removed from the dataset.\n",
        "\n",
        "**FileSize might not be a useful Feature:**\n",
        " From the plot between download volumne and file size no clear relationship is found.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "l6D8XA3_mlAY",
      "metadata": {
        "id": "l6D8XA3_mlAY"
      },
      "source": [
        "#### **Q1a Step 10 EDA Title**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function.\n",
        "- Check for any missing or invalid values.\n",
        "- Since FileType is a Categorical variable, visualize the distribution using a histogram and boxplot chart.\n",
        "\n",
        "My Expectation\n",
        "- Titles should provide additional insight into what the asset is related to.\n",
        "\n",
        "Interrelationships:\n",
        "- Title might be related to filenames and file types.\n",
        "- Potential Data Leak: If titles are highly dependent on filenames or file types, this could lead to redundancy and potential data leakage in predictive models.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "L2zMThwQTxSE",
      "metadata": {
        "id": "L2zMThwQTxSE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "632165b5-4bec-4734-978a-7a2c95bee5f3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 5669\n",
            "Missing Values: 4\n",
            "Invalid and Error Values: 4\n",
            "Most Common Values: Title\n",
            "Actuaries Magazine                             21\n",
            "Actuaries Digital Magazine                     12\n",
            "Plenary 2 - The far future of healthcare        7\n",
            "GIS2018 - Plenary 4 - Ethics and Leadership     7\n",
            "Plenary 4 -The road ahead                       7\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Title values that appear more than once: 658\n",
            "---------------\n",
            "Data type: <class 'str'>, Count: 6065\n",
            "Sample values: ['FSF18 - C5B - Australian Retail TPD', 'Managing Operational Risk in Financial Reporting', 'AAIC15 - P1 - The computation of one year Capital and the risk margin out of the P and C Triangle - Audio']\n",
            "\n",
            "Data type: <class 'float'>, Count: 4\n",
            "Sample values: [nan, nan, nan]\n",
            "\n",
            "------------------------------\n",
            " Display 5 rows of Title, FileName, and FileType\n",
            "------------------------------\n",
            "                                               Title  \\\n",
            "0                    International CPD opportunities   \n",
            "1  Disability Committee - 1995 Report of the Disa...   \n",
            "2  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3  Australian Actuarial Journal 1997 (September E...   \n",
            "4  Disability Committee - The 1997 Report into mo...   \n",
            "5  Disability Committee - The 1997 Report into mo...   \n",
            "6  Disability Committee - The 2002 Report Of Disa...   \n",
            "7  Disability Committee - The 2002 Report of the ...   \n",
            "8  Australian Actuarial Journal 2005 Volume 11 (I...   \n",
            "9  Australian Actuarial Journal 2006 Volume 12 (I...   \n",
            "\n",
            "                                            FileName FileType  \n",
            "0               international-cpd-opportunities.html     html  \n",
            "1        1995 Report of the Disability Committee.pdf      pdf  \n",
            "2                           1997 - March Edition.pdf      pdf  \n",
            "3                       1997 - September Edition.pdf      pdf  \n",
            "4  1997 Report of the Disability Committee-The Da...      pdf  \n",
            "5        1997 Report of the Disability Committee.pdf      pdf  \n",
            "6  2002 Report of the Disability Committee-The Da...      pdf  \n",
            "7        2002 Report of the Disability Committee.pdf      pdf  \n",
            "8         2005_AAJ_Volume11_Issue_3(final060315).pdf      pdf  \n",
            "9     2006_AAJ_Volume12_Issue_1_final_corrected_.pdf      pdf  \n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df,'Title')\n",
        "\n",
        "print(\"---------------\")\n",
        "eda_column_dtypes(df,'Title')\n",
        "\n",
        "# Dispaly 5 rows of Title, FileName, File Type\n",
        "print(\"------------------------------\")\n",
        "print(\" Display 5 rows of Title, FileName, and FileType\")\n",
        "print(\"------------------------------\")\n",
        "print(docs_train[['Title', 'FileName', 'FileType']].head(10))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-oKyw0G2Wta3",
      "metadata": {
        "id": "-oKyw0G2Wta3"
      },
      "source": [
        "**Findings:**\n",
        "- There are 5,669 unique titles in the dataset, indicating a wide range of different assets.\n",
        "- There are 4 missing values (NaNs) and 4 invalid or erroneous values in the Title column that need to be addressed.\n",
        "- There are 658 instances where titles are not unique, with some titles appearing multiple times, such as \"Actuaries Magazine\" and \"Actuaries Digital Magazine.\" These duplicates may represent different versions or related content.\n",
        "- The most common titles appear multiple times, likely indicating a series or recurring publication."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aiQaem0ZYA3b",
      "metadata": {
        "id": "aiQaem0ZYA3b"
      },
      "source": [
        "**Display the 4 rows of missing titles**\n",
        "We want to see which 4 assets are missing titles. From the initial EDA assets appear to have missing values in other columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8bgVz8uUWYHO",
      "metadata": {
        "id": "8bgVz8uUWYHO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "969c7c93-e414-48a6-e13e-56237ab20452"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "------------------------------\n",
            "Rows with missing Title values:\n",
            "------------------------------\n",
            "        DMSId                                           FileName FileType  \\\n",
            "2529  4325486                Innovation in the 21st Century.docx     docx   \n",
            "5787  4364500  Transcript - The role of actuaries in crunchin...      pdf   \n",
            "5797  4364531                               final transcript.pdf      pdf   \n",
            "5803  4364544  Life Insurance and Wealth Management Practice ...      pdf   \n",
            "\n",
            "      FileSize Title StartPublishing StopPublishing  Cpd Format Level  ...  \\\n",
            "2529     11886   NaN      2018-11-13              0    0    NaN   NaN  ...   \n",
            "5787     47519   NaN      2020-06-11              0    0    NaN   NaN  ...   \n",
            "5797     59324   NaN      2020-06-18              0    0    NaN   NaN  ...   \n",
            "5803     70186   NaN      2020-06-21              0    0    NaN   NaN  ...   \n",
            "\n",
            "     Copyright EventAgeGroup EventType  \\\n",
            "2529       NaN           NaN       NaN   \n",
            "5787       NaN           NaN       NaN   \n",
            "5797       NaN           NaN       NaN   \n",
            "5803       NaN           NaN       NaN   \n",
            "\n",
            "                                                   Link  \\\n",
            "2529  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "5787  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "5797  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "5803  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "\n",
            "                                                   Tags DownloadCount  \\\n",
            "2529  Business Acumen|Business Acumen - Disruption|P...             1   \n",
            "5787                                                NaN           108   \n",
            "5797                                                NaN           629   \n",
            "5803                                                NaN             7   \n",
            "\n",
            "     DurationOnDMS  AnnualizedDownloadCount  YearOfStartPublishing  Year  \n",
            "2529           5.5                     0.18                   2018  2018  \n",
            "5787           3.9                    27.69                   2020  2020  \n",
            "5797           3.9                   161.28                   2020  2020  \n",
            "5803           3.9                     1.79                   2020  2020  \n",
            "\n",
            "[4 rows x 22 columns]\n"
          ]
        }
      ],
      "source": [
        "# Get rows with missing values in the 'Title' column\n",
        "missing_title_rows = docs_train[docs_train['Title'].isnull()]\n",
        "\n",
        "# Display the rows with missing Title values\n",
        "print(\"------------------------------\")\n",
        "print(\"Rows with missing Title values:\")\n",
        "print(\"------------------------------\")\n",
        "print(missing_title_rows)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rMnhXHuDYr1s",
      "metadata": {
        "id": "rMnhXHuDYr1s"
      },
      "source": [
        "We have identified 4 rows with missing values in the Title column. These missing entries align with our initial findings in Q1a Step 1.\n",
        "\n",
        "We will manually repair these missing titles by referring to the links provided for these assets. This will ensure that all titles as well as other missing columns are complete and accurate.\n",
        "\n",
        "This data adjustment will be performed in the data cleaning section, once we have completed the entire EDA process."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Cr6GDYEyVraj",
      "metadata": {
        "id": "Cr6GDYEyVraj"
      },
      "source": [
        "##### **Analysis of High Correlation Between FileName and Title + FileType**\n",
        "FileName appears to be closely related to the combination of Title and FileType.\n",
        "\n",
        "We will verify the Title-Filename relationship to confirm if FileName is derived directly from Title plus FileType. This verification is crucial because if FileName is indeed dependent on other columns, it could lead to data leakage in predictive models.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iShGnQlVZtHa",
      "metadata": {
        "id": "iShGnQlVZtHa"
      },
      "source": [
        "Investigation of Title-Filename Relationship:\n",
        "**Objective**: To check if FileName is derived from Title and FileType.\n",
        "\n",
        "**Method**:  \n",
        "- Word-Based Matching: checking if significant words from the Title appear in the FileName, accounting for differences in formatting, such as underscores, hyphens, and other symbols.\n",
        "- Preprocessing: removing common stopwords and lowercasing text to focus on the essential words in the Title and FileName.\n",
        "\n",
        "**Rationale for Word-Based Matching:** This method ensures that if the FileName and Title differ in formatting or word order, they are still considered a match if they share significant words. For example, \"Australian Actuarial Journal 1997 (March Edition)\" might be matched to \"1997 - March Edition.pdf\" because the key words are present, even if the exact structure is not.\n",
        "\n",
        "**Rationale for chosing this Method**: Exact matching can miss valid relationships between Title and FileName due to formatting differences. This word-based approach allows us to more effectively determine the relationship between the two by focusing on the presence of key terms rather than their exact order or formatting."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0sM8UqGrZlad",
      "metadata": {
        "collapsed": true,
        "id": "0sM8UqGrZlad",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7a5a886f-1d9b-4786-d9e5-a6e4d87425b4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Proportion of rows where significant Title words are found in FileName: 36.25%\n",
            "\n",
            "Rows where significant Title words are not found in FileName:\n",
            "                                     FileName  \\\n",
            "10         46305_AAJ_v16i1_comp.sflb.ashx.pdf   \n",
            "11                50230_Journ_v16i2_combo.pdf   \n",
            "13                  2012AAJVolume18Issue1.PDF   \n",
            "14                  2012AAJVolume18Issue2.pdf   \n",
            "15                     AAJ_Vol17_iss1_web.pdf   \n",
            "...                                       ...   \n",
            "6062                  Presentation Slides.pdf   \n",
            "6068                  Final transcript(5).pdf   \n",
            "6074  AI - GI Pandemic discussion - final.pdf   \n",
            "6075          ICAAP AA Survey Feedback v3.pdf   \n",
            "6094        DITF Insights 14 Dec 20 - 1-1.pdf   \n",
            "\n",
            "                                                  Title  IsMatch  \n",
            "10    Australian Actuarial Journal 2010 Volume 16 (I...    False  \n",
            "11    Australian Actuarial Journal 2010 Volume 16 (I...    False  \n",
            "13    Australian Actuarial Journal 2012 Volume 18 (I...    False  \n",
            "14    Australian Actuarial Journal 2012 Volume 18 (I...    False  \n",
            "15    Australian Actuarial Journal 2011 Volume 17 (I...    False  \n",
            "...                                                 ...      ...  \n",
            "6062   Automated Decisioning Systems - Information Note    False  \n",
            "6068  Unpacking the YDAWG's 2020 US Presidential Ele...    False  \n",
            "6074    COVID-19 - What Have we Learned for the Future?    False  \n",
            "6075    COVID-19 - What Have we Learned for the Future?    False  \n",
            "6094  The Actuaries Institute Disability Insurance T...    False  \n",
            "\n",
            "[3869 rows x 3 columns]\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS as stop_words\n",
        "\n",
        "# Convert to DataFrame\n",
        "docs_train_df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Preprocess the title to remove common stopwords and lowercase\n",
        "def preprocess(text):\n",
        "    return ' '.join([word.lower() for word in text.split() if word.lower() not in stop_words])\n",
        "\n",
        "# Create a derived column to compare title and filename\n",
        "docs_train_df['ProcessedTitle'] = docs_train_df['Title'].apply(lambda x: preprocess(x) if pd.notnull(x) else '')\n",
        "\n",
        "# Function to check if significant words from Title are in FileName\n",
        "def is_match(row):\n",
        "    title_words = set(row['ProcessedTitle'].split())\n",
        "    filename_words = set(row['FileName'].lower().replace('_', ' ').replace('-', ' ').replace('.', ' ').split())\n",
        "    common_words = title_words.intersection(filename_words)\n",
        "    return len(common_words) > 0\n",
        "\n",
        "# Apply the matching function\n",
        "docs_train_df['IsMatch'] = docs_train_df.apply(is_match, axis=1)\n",
        "\n",
        "# Calculate the proportion of matches\n",
        "match_count = docs_train_df['IsMatch'].sum()\n",
        "total_count = docs_train_df.shape[0]\n",
        "match_proportion = (match_count / total_count) * 100\n",
        "\n",
        "# Print the proportion of matches\n",
        "print(f\"Proportion of rows where significant Title words are found in FileName: {match_proportion:.2f}%\")\n",
        "\n",
        "# Show rows where there is no match\n",
        "non_matching_rows = docs_train_df[~docs_train_df['IsMatch']]\n",
        "print(\"\\nRows where significant Title words are not found in FileName:\")\n",
        "print(non_matching_rows[['FileName', 'Title', 'IsMatch']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "C42vyYGPag7L",
      "metadata": {
        "id": "C42vyYGPag7L"
      },
      "source": [
        "**Observation:**\n",
        "Significant title words are found in the file name for 35% of the rows, indicating that nearly half of the dataset has a clear alignment between Title and FileName.\n",
        "\n",
        "There are 4,195 rows where the Title does not directly match the FileName using the current method. For instance:\n",
        "\n",
        "- The file name \"2c_GIS2012PaperLee.pdf\" does not directly match the title \"GIS12 - C2C - Gauging the Tort Temperature and...\" due to differences in naming conventions, such as abbreviations and shorthand.\n",
        "\n",
        "- Similarly, \"PHINEWSLETTER277(1).pdf\" corresponds to \"PHI Newsletter #277,\" where the abbreviation \"PHI\" and shorthand numbering prevent a direct match.\n",
        "\n",
        "\n",
        "**Conclusion:**\n",
        "\n",
        "The current matching approach may not fully capture the relationship between Title and FileName due to abbreviations and shorthand used in filenames. Despite this, it's evident that there is a strong relationship between Title and FileName. Therefore, to avoid data leakage, FileName should not be included in feature selection."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rxeyyj81bmyZ",
      "metadata": {
        "id": "rxeyyj81bmyZ"
      },
      "source": [
        "**Investigation of Title-FileType Relationship:**\n",
        "\n",
        "We check if the FileType (e.g., \"pdf\", \"mp3\") is explicitly mentioned in the FileName. The purpose is to assess the consistency between the file type and how the file is named, which can help identify potential discrepancies or errors in the dataset.\n",
        "\n",
        "How does it work:\n",
        "- The FileType and FileName are converted to lowercase to ensure the comparison is case-insensitive.\n",
        "- The code checks if the FileType appears as a word in the FileName.\n",
        "- It calculates the proportion of rows where the FileType is found in the FileName and identifies the rows where there is no match, potentially highlighting inconsistencies."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yQqUAlljblwo",
      "metadata": {
        "id": "yQqUAlljblwo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "da56c5a2-6486-49d6-c76c-3954a9dd626d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Proportion of rows where FileType is found in FileName: 99.98%\n",
            "\n",
            "Rows where FileType is not found in FileName:\n",
            "                                               FileName FileType  \\\n",
            "5841  Actuaries Summit 2020 - Disability Income Insu...      pdf   \n",
            "\n",
            "      IsFileTypeMatch  \n",
            "5841            False  \n"
          ]
        }
      ],
      "source": [
        "# Convert to DataFrame\n",
        "docs_train_df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Preprocess the FileType to lowercase (in case FileType isn't already lowercase)\n",
        "docs_train_df['ProcessedFileType'] = docs_train_df['FileType'].str.lower()\n",
        "\n",
        "# Function to check if the FileType is found in the FileName\n",
        "def is_filetype_in_filename(row):\n",
        "    filetype = row['ProcessedFileType']\n",
        "    filename_words = set(row['FileName'].lower().replace('_', ' ').replace('-', ' ').replace('.', ' ').split())\n",
        "    return filetype in filename_words\n",
        "\n",
        "# Apply the matching function\n",
        "docs_train_df['IsFileTypeMatch'] = docs_train_df.apply(is_filetype_in_filename, axis=1)\n",
        "\n",
        "# Calculate the proportion of matches\n",
        "filetype_match_count = docs_train_df['IsFileTypeMatch'].sum()\n",
        "total_count = docs_train_df.shape[0]\n",
        "filetype_match_proportion = (filetype_match_count / total_count) * 100\n",
        "\n",
        "# Print the proportion of matches\n",
        "print(f\"Proportion of rows where FileType is found in FileName: {filetype_match_proportion:.2f}%\")\n",
        "\n",
        "# Show rows where there is no match\n",
        "non_matching_filetype_rows = docs_train_df[~docs_train_df['IsFileTypeMatch']]\n",
        "print(\"\\nRows where FileType is not found in FileName:\")\n",
        "print(non_matching_filetype_rows[['FileName', 'FileType', 'IsFileTypeMatch']])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0MITQxMvblbJ",
      "metadata": {
        "id": "0MITQxMvblbJ"
      },
      "source": [
        "**Obesrvation of FileName and FileType matching**:\n",
        "\n",
        "High Match Consistency: 99.88% of the rows show that the FileType consistently matches what is indicated in the FileName, reflecting strong alignment across most of the dataset.\n",
        "\n",
        "\n",
        "**Conclusion:**\n",
        "There is strong evidence that FileName is highly dependent on FileType. To prevent data leakage, FileName should not be included in later feature engineering.bold text"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uM_Tbl-Ijepj",
      "metadata": {
        "id": "uM_Tbl-Ijepj"
      },
      "source": [
        "#### **Q1a Step 10 EDA  StartPublishing**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function.\n",
        "- We have done some data cleaning before spliting into training dataset\n",
        "- Check for any missing or invalid values.\n",
        "- Since FileType is a Date variable, visualize the distribution using a bar chart.\n",
        "\n",
        "Expecation:\n",
        "- All Date Data: The StartPublishing column should only contain valid date entries, with no missing or invalid dates.\n",
        "- Reasonable Date Range: Given that the DMS only started in August 2018, any StartPublishing date earlier than this would be considered unreasonable.\n",
        "- No Future Dates: Data dictionary provides range up to Mar2024, we do not expect any StartPublishing dates later than this."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "IZSt8XdfzL0G",
      "metadata": {
        "id": "IZSt8XdfzL0G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "f97ac44b-27fe-4f21-b5d9-ef9304970143"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-2-00436247ac95>:452: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.barplot(x=month_counts_df['Month-Year'].astype(str), y=month_counts_df['Number of Assets'], palette='Blues_d')\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1600x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 43888 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Month-Year  Number of Assets\n",
            "0     2018-08              1986\n",
            "1     2018-09               259\n",
            "2     2018-10                25\n",
            "3     2018-11               183\n",
            "4     2018-12                68\n",
            "5     2019-01               268\n",
            "6     2019-02               283\n",
            "7     2019-03               848\n",
            "8     2019-04               547\n",
            "9     2019-05               142\n",
            "10    2019-06               161\n",
            "11    2019-07               109\n",
            "12    2019-08               269\n",
            "13    2019-09                25\n",
            "14    2019-10                23\n",
            "15    2019-11                96\n",
            "16    2019-12               102\n",
            "17    2020-01                 4\n",
            "18    2020-02               280\n",
            "19    2020-03                 5\n",
            "20    2020-04                25\n",
            "21    2020-05                32\n",
            "22    2020-06                40\n",
            "23    2020-07                27\n",
            "24    2020-08               158\n",
            "25    2020-09                 4\n",
            "26    2020-10                 8\n",
            "27    2020-11                58\n",
            "28    2020-12                24\n",
            "29    2021-01                10\n",
            "--------------------\n",
            "Check for Date Outliers:\n",
            "--------------------\n",
            "2018-08-01 00:00:00 2021-01-28 00:00:00\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "asset_counts_df = examine_start_publishing(df)\n",
        "print(asset_counts_df)\n",
        "\n",
        "print('--------------------')\n",
        "print(\"Check for Date Outliers:\")\n",
        "print('--------------------')\n",
        "print(df['StartPublishing'].min(), df['StartPublishing'].max())\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZN5Mo_LxppZF",
      "metadata": {
        "id": "ZN5Mo_LxppZF"
      },
      "source": [
        "**Obeservation of StartPublishing EDA & Data Cleaning**\n",
        "-  We have done some data cleaning before spliting into training dataset\n",
        "- The all of StartPublishing dates fall between August 2018 and March 2024, which aligns with our expectations.\n",
        "- There is a significant concentration of assets with a StartPublishing date in August 2018. This is reasonable, as any assets published before August 2018 would have their start dates truncated to the launch of the DMS."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bbr-sxA7wYox",
      "metadata": {
        "id": "bbr-sxA7wYox"
      },
      "source": [
        "#### **Q1a Step 11 EDA  StopPublishing**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function.\n",
        "- Histogram is used to visulise the distrubtion.\n",
        "- We have identiifed a known data type isse from inital EDA. While the data dictionary suggests that StopPublishing a date with range between Nov 2023 and Nov 3023,  our initial EDA suggests it is stored as a object format.\n",
        "- Before converting values to date format, I want to review the data to understand what the current values represent.\n",
        "- Some assets might have \"zero\" in StopPublishing, which could indicate that they never expire."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ZWctvHCdyQLf",
      "metadata": {
        "id": "ZWctvHCdyQLf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 817
        },
        "outputId": "a96536ab-4a41-44b1-ccf6-91ed358552ce"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 53\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: StopPublishing\n",
            "0                      5684\n",
            "2040-12-29 00:00:00     199\n",
            "3018-12-15 00:00:00      30\n",
            "3018-12-16 00:00:00      26\n",
            "3019-01-22 00:00:00      21\n",
            "Name: count, dtype: int64\n",
            "Number of rows with StopPublishing values that appear more than once: 6042\n",
            "------\n",
            "Data type: <class 'int'>, Count: 5684\n",
            "Sample values: [0, 0, 0]\n",
            "\n",
            "Data type: <class 'datetime.datetime'>, Count: 385\n",
            "Sample values: [datetime.datetime(2040, 12, 29, 0, 0), datetime.datetime(2040, 12, 29, 0, 0), datetime.datetime(2040, 12, 29, 0, 0)]\n",
            "\n",
            "------\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 1 Axes>"
            ],
            "image/png": "[encoded data, 27732 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'StopPublishing')\n",
        "print('------')\n",
        "eda_column_dtypes(df, 'StopPublishing')\n",
        "print('------')\n",
        "\n",
        "\n",
        "\n",
        "# Histogram for distribution of date values\n",
        "# Convert StopPublishing to datetime, setting errors='coerce' to handle non-date values\n",
        "df['StopPublishing'] = pd.to_datetime(df['StopPublishing'], errors='coerce')\n",
        "\n",
        "# Plot the histogram of the date values\n",
        "plt.figure(figsize=(10, 5))\n",
        "df['StopPublishing'].hist(bins=50)\n",
        "plt.title('Distribution of StopPublishing Dates')\n",
        "plt.xlabel('Date')\n",
        "plt.ylabel('Frequency')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qBJYcEAEgTSl",
      "metadata": {
        "id": "qBJYcEAEgTSl"
      },
      "source": [
        "**Analysis of StopPublishing Dates Distribution:**\n",
        "\n",
        "The majority of StopPublishing dates are clustered around the year 1970. This likely corresponds to the 0 value, indicating assets with no stop date or assets that are available indefinitely.\n",
        "\n",
        "A small number of StopPublishing dates are set far in the future, around 2040 and beyond, which is unusual and may represent placeholder dates or data entry anomalies."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Y8hcN1Xo6drQ",
      "metadata": {
        "id": "Y8hcN1Xo6drQ"
      },
      "source": [
        "Obesevation:\n",
        "- We noticed that 5684 entries are marked as 1970-01-01, which is known as the Unix epoch (the starting point of Unix time). This usually happens when a date value is stored as 0 in some systems and gets interpreted as 1970-01-01 during data conversion.\n",
        "- The 0 likely indicates \"no expiry date.\"\n",
        "-  missing values, which also likely represent \"no expiry date.\"\n",
        "- we don't see any missing value in training set, but this might not be the same case in Training and testing set. We should also confirm those two set.\n",
        "- Some assets have a stop date after 01 January 3000, likely indicating \"no expiry date.\"\n",
        "\n",
        "\n",
        "\n",
        "**Feature Engineering Consideration:**\n",
        "\n",
        "We can convert the StopPublishing column into categorical variables, such as \"Never Expiry,\" \"Expires Within 1 Year,\" and \"Expires After 1 Year,\" to simplify and enhance our analysis.\n",
        "\n",
        "\n",
        "**Do We Need Data Cleaning?**\n",
        "\n",
        "Even if we plan to perform feature engineering, data cleaning is still necessary. It is because  without cleaning, incorrect or placeholder dates might lead to misclassification in the new categorical variables, which could negatively impact the analysis.\n",
        "\n",
        "**Planned Cleaning Steps:**\n",
        "\n",
        "- Ensure the StopPublishing dates are consistent with the data dictionary, converting the StopPublishing field into a proper date format where necessary.\n",
        "- Handling Missing or Zero Values: Change all missing or zero values to a standardized maximum date, indicating that the file does not have an expiry date.\n",
        "\n",
        "\n",
        "**This cleaning of the StopPublishing dates will be completed after we finish the EDA for all variables.**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "l0FxXSHmjIYW",
      "metadata": {
        "id": "l0FxXSHmjIYW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ebaf3a51-a8a4-4ea6-fea0-728b7b146498"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 44\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: StopPublishing\n",
            "0                      500\n",
            "2040-08-29 00:00:00     94\n",
            "3022-01-16 00:00:00     36\n",
            "2032-08-16 00:00:00     23\n",
            "3022-08-24 00:00:00     23\n",
            "Name: count, dtype: int64\n",
            "Number of rows with StopPublishing values that appear more than once: 734\n",
            "------\n",
            "Data type: <class 'int'>, Count: 500\n",
            "Sample values: [0, 0, 0]\n",
            "\n",
            "Data type: <class 'datetime.datetime'>, Count: 257\n",
            "Sample values: [datetime.datetime(2032, 8, 16, 0, 0), datetime.datetime(2040, 8, 29, 0, 0), datetime.datetime(3022, 8, 24, 0, 0)]\n",
            "\n",
            "------\n",
            "Unique Values: 44\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: StopPublishing\n",
            "3023-07-22 00:00:00    184\n",
            "0                      160\n",
            "3023-07-30 00:00:00    160\n",
            "3023-07-26 00:00:00     58\n",
            "2034-02-27 00:00:00     29\n",
            "Name: count, dtype: int64\n",
            "Number of rows with StopPublishing values that appear more than once: 701\n",
            "------\n",
            "Data type: <class 'datetime.datetime'>, Count: 565\n",
            "Sample values: [datetime.datetime(3023, 7, 26, 0, 0), datetime.datetime(3023, 7, 30, 0, 0), datetime.datetime(2032, 9, 5, 0, 0)]\n",
            "\n",
            "Data type: <class 'int'>, Count: 160\n",
            "Sample values: [0, 0, 0]\n",
            "\n",
            "------\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_val)\n",
        "\n",
        "eda_describe_column(df, 'StopPublishing')\n",
        "print('------')\n",
        "\n",
        "eda_column_dtypes(df, 'StopPublishing')\n",
        "print('------')\n",
        "\n",
        "\n",
        "df = pd.DataFrame(docs_test)\n",
        "\n",
        "eda_describe_column(df, 'StopPublishing')\n",
        "print('------')\n",
        "\n",
        "eda_column_dtypes(df, 'StopPublishing')\n",
        "print('------')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J1ctaMLtUC5J",
      "metadata": {
        "id": "J1ctaMLtUC5J"
      },
      "source": [
        "#### **Q1a Step 12 EDA  Cpd**\n",
        "\n",
        "**Outline**\n",
        "- Start with a basic EDA using the tutorial function, which comes with a Histogram to visulise the distrubtion.\n",
        "- I added a boxplot to visualise the distribution of cpd variable to examinate outliers.\n",
        "\n",
        "Expectation:\n",
        "- The CPD values should be integers ranging between 0 and 10.\n",
        "- We expect all values to fall within this range with no missing or out-of-range data.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "aMgIh83QUwmU",
      "metadata": {
        "id": "aMgIh83QUwmU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "fb7ab364-8f6f-49f6-cc15-76001acd4410"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Data type: <class 'int'>, Count: 6069\n",
            "Sample values: [1, 1, 1]\n",
            "\n",
            "Unique Values: 7\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: Cpd\n",
            "1    4383\n",
            "2    1624\n",
            "3      48\n",
            "0       7\n",
            "4       4\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Cpd values that appear more than once: 6068\n",
            "Mean: 1.2865381446696325\n",
            "Standard Deviation: 0.485231829692131\n",
            "Skewness: 1.5613832903055669\n",
            "Minimum: 0\n",
            "Maximum: 7\n",
            "Maximum Number of Decimal Places: 0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 16992 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 15464 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "eda_column_dtypes(df, 'Cpd')\n",
        "eda_describe_column(df, 'Cpd')\n",
        "eda_numeric_column(df, 'Cpd')\n",
        "\n",
        "# Create the boxplot for 'Cpd'\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.boxplot(docs_train['Cpd'])\n",
        "\n",
        "# Set the title and labels\n",
        "plt.title('Distribution of Cpd')\n",
        "plt.xlabel('Cpd')\n",
        "\n",
        "# Display the plot\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Z-J5RmY0HTST",
      "metadata": {
        "id": "Z-J5RmY0HTST"
      },
      "source": [
        "The EDA suggests that CPD points generally align with the data dictionary.\n",
        "The Cpd column has 11 unique values, with a mean of 1.27 and a high right skewness of 2.81.\n",
        "Most assets have CPD values of 1 or 2, and this aligns  with my expectation\n",
        "\n",
        "However, the previous charts make it difficult to clearly identify outliers.\n",
        "\n",
        "\n",
        "To address this, we will **add a box plot of CPD points by their FileType.**\n",
        "\n",
        "Purpose:\n",
        "- This allow us to understand the relationship between cpd and file type. We want to know is there are particular file type will worth more cpd point.\n",
        "- Separating CPD points by their fileType date makes outliers more visible, avoiding the issue of blending too much data together, which can obscure patterns.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Ev6Q1vQ0H7zN",
      "metadata": {
        "id": "Ev6Q1vQ0H7zN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "2c84e35c-683c-4b9d-d54e-0ea0990922ec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 34216 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average CPD Points by File Type:\n",
            "FileType\n",
            "ics     2.000000\n",
            "wmv     2.000000\n",
            "wma     1.937500\n",
            "mp3     1.736512\n",
            "xlsx    1.500000\n",
            "docx    1.187500\n",
            "pdf     1.146108\n",
            "html    1.000000\n",
            "ppt     1.000000\n",
            "pptx    1.000000\n",
            "txt     1.000000\n",
            "zip     1.000000\n",
            "Name: Cpd, dtype: float64\n"
          ]
        }
      ],
      "source": [
        "# Create a box plot to show the distribution of CPD points for each file type\n",
        "plt.figure(figsize=(12, 8))\n",
        "sns.boxplot(data=docs_train, x='FileType', y='Cpd')\n",
        "\n",
        "plt.xlabel('File Type')\n",
        "plt.ylabel('CPD Points')\n",
        "plt.title('Distribution of CPD Points by File Type')\n",
        "plt.tight_layout()\n",
        "plt.show()\n",
        "\n",
        "# Optionally, calculate and display the average CPD points for each file type\n",
        "avg_cpd_by_filetype = docs_train.groupby('FileType')['Cpd'].mean().sort_values(ascending=False)\n",
        "print(\"Average CPD Points by File Type:\")\n",
        "print(avg_cpd_by_filetype)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nW-fLENzJHbI",
      "metadata": {
        "id": "nW-fLENzJHbI"
      },
      "source": [
        "Comment:\n",
        "\n",
        " - MP3 files show a wider range of CPD points compared to other file types, with some reaching as high as 7 points. This indicates that audio content may cover a broader range of CPD value, possibly due to varying lengths or content depth.\n",
        "\n",
        " - Most other file types, particularly PDFs, have CPD points consistently around 1. This suggests that these file types tend to provide similar CPD value, with fewer outliers.\n",
        "\n",
        "- Some file types, like xlsx and wma, show outliers that suggest a few exceptional files with higher CPD points. This could indicate specialized or particularly comprehensive content within those formats."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-eTLxs10ghQM",
      "metadata": {
        "id": "-eTLxs10ghQM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5e2208b7-bf10-49da-cf79-abdacfde0367"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "        DMSId                             FileName FileType  FileSize  \\\n",
            "5794  4364518  PSWG Final Report 26 April 2020.pdf      pdf    545182   \n",
            "\n",
            "                                               Title StartPublishing  \\\n",
            "5794  Retail Advised Lump Sum Product Sustainability      2020-06-15   \n",
            "\n",
            "           StopPublishing  Cpd       Format Level  ...  Copyright  \\\n",
            "5794  3020-06-15 00:00:00    7  Publication   All  ...  Institute   \n",
            "\n",
            "     EventAgeGroup EventType  \\\n",
            "5794           All     Major   \n",
            "\n",
            "                                                   Link            Tags  \\\n",
            "5794  https://actuaries.logicaldoc.cloud/download-ti...  Life Insurance   \n",
            "\n",
            "     DownloadCount DurationOnDMS  AnnualizedDownloadCount  \\\n",
            "5794           724           3.9                   185.64   \n",
            "\n",
            "      YearOfStartPublishing  Year  \n",
            "5794                   2020  2020  \n",
            "\n",
            "[1 rows x 22 columns]\n"
          ]
        }
      ],
      "source": [
        "# Filter the DataFrame to include only the rows with Cpd value of 6 or higher\n",
        "high_cpd_df = df[df['Cpd'] >= 6]\n",
        "\n",
        "# Display the filtered DataFrame\n",
        "print(high_cpd_df)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "IKYfryg7Jp3R",
      "metadata": {
        "id": "IKYfryg7Jp3R"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "id": "tXYwYIYpMpsa",
      "metadata": {
        "id": "tXYwYIYpMpsa"
      },
      "source": [
        "We've identified an asset with the following unusual characteristics:\n",
        "\n",
        "**Outlier: The Post-Pandemic Actuarial World virtual Seminar**\n",
        "\n",
        "This is highly unusual, as assets with higher CPD points typically attract more downloads, especially over a period of 2.6 years. DMSID:4366648                                  \n",
        "\n",
        "The asset is labeled as a PDF but is actually a YouTube video, indicating a potential data entry error or misclassification.\n",
        "\n",
        "**Recommendation:**\n",
        "\n",
        "The finding suggests that this asset may indeed be an outlier and possibly a data error.\n",
        "\n",
        "We will remove this asset from the dataset. Retaining it could skew the analysis and lead to inaccurate conclusions, especially if it doesn't align with the typical characteristics of other assets.\n",
        "\n",
        "We will remove this assets in the next step after EDA."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "psR4v66QUC5O",
      "metadata": {
        "id": "psR4v66QUC5O"
      },
      "source": [
        "**Summary**  \n",
        "*Provide a **clear** and **concise** summary here of your conclusions based on the EDA and cleaning conducted above.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZpiGPmrSUC5O",
      "metadata": {
        "id": "ZpiGPmrSUC5O"
      },
      "source": [
        "**Overall summary**  \n",
        "*Provide a **clear** and **concise** overall summary here of your conclusions from this section of the notebook.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "W9slHmKqZ1Ux",
      "metadata": {
        "id": "W9slHmKqZ1Ux"
      },
      "source": [
        "#### **Q1a Step 13 EDA  Format**\n",
        "\n",
        "**Outline**\n",
        "- The Format values should be strings representing one of 11 categories (e.g., Article, Video, Workshop).\n",
        "\n",
        "- Start with a basic EDA using the tutorial function, which comes with a bar chart and pie chart to visulise the distrubtion.  \n",
        "\n",
        "Expectation:\n",
        "- We anticipate some relationship between Format and FileType, where certain formats align with specific file types (e.g., Video with MP4, Article with PDF)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yzt3nMj0b3C0",
      "metadata": {
        "id": "yzt3nMj0b3C0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "abf9a49a-8180-4181-dc5c-3d4ffad2e347"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "Least Common Values:\n",
            "\tEvents: 148\n",
            "\tVideo: 101\n",
            "\tTool and Resources: 58\n",
            "\tOther: 3\n",
            "\teLearning: 2\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 40180 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 46216 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "Number of missing values in the 'Format' column: 4\n",
            "---------\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'Format')\n",
        "print(\"---------\")\n",
        "# Check for missing values in the Format column\n",
        "missing_format_values = docs_data['Format'].isnull().sum()\n",
        "print(f\"Number of missing values in the 'Format' column: {missing_format_values}\")\n",
        "print(\"---------\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZfVj3HJXgECF",
      "metadata": {
        "id": "ZfVj3HJXgECF"
      },
      "source": [
        "Summary of Findings:\n",
        "\n",
        "The 4 missing values correspond to the 4 rows identified in the filename EDA. These will be addressed during the data cleaning step.\n",
        "\n",
        "- No other obvious data issues were identified, so no further data cleaning is required.\n",
        "\n",
        "**Consideration in Feature engineering:**\n",
        "- The Format column has high cardinality, suggesting that grouping similar formats might be beneficial if this column is used as an input in modeling.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FKYI6QvQQbfK",
      "metadata": {
        "id": "FKYI6QvQQbfK"
      },
      "source": [
        "#####**Cross Variable analysis**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1s_OxjLqQPz2",
      "metadata": {
        "id": "1s_OxjLqQPz2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 862
        },
        "outputId": "63997e99-47cf-4484-bf13-52522d450d40"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 2 Axes>"
            ],
            "image/png": "[encoded data, 80032 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Create a crosstab to show the relationship between FileType and Format\n",
        "filetype_format_crosstab = pd.crosstab(docs_train['FileType'], docs_train['Format'])\n",
        "\n",
        "# Generate the heatmap\n",
        "plt.figure(figsize=(12, 8))\n",
        "sns.heatmap(filetype_format_crosstab, annot=True, cmap='Blues', fmt='d')\n",
        "plt.title('Heatmap of Correlation Between FileType and Format')\n",
        "plt.xlabel('Format')\n",
        "plt.ylabel('FileType')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TK6kHFshQfWx",
      "metadata": {
        "id": "TK6kHFshQfWx"
      },
      "source": [
        "**Strong association is found**:\n",
        "\n",
        "MP3 files are strongly associated with the Podcast/Vodcast/Audio format, indicating that these two features are closely linked.\n",
        "\n",
        "PDF files are commonly found in Presentation Slides and Publications formats, showing a strong correlation between these file types and formats.\n",
        "\n",
        "HTML files also appear frequently in the Podcast/Vodcast/Audio format, suggesting a connection.\n",
        "\n",
        "Some formats like Video, Workshops, and eLearning have little to no representation in certain file types, indicating that these formats are specialized and not widely spread across different file types.\n",
        "\n",
        "**Consideration in feature selection**\n",
        "\n",
        "- This heatmap suggest that FileType is not a reliable souce to determine what format an assets belong, such as vedio or text based. For example, 219 of asset has a docx file type but they are actually video.\n",
        "- This suggest we should rely more on format to determine what the assets nature rather then using filetype.\n",
        "\n",
        "We consider using one of these features in the model, especially in areas where there's high correlation, to avoid redundancy."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WNaQAk14RLIj",
      "metadata": {
        "id": "WNaQAk14RLIj"
      },
      "source": [
        "##### Summary of Format EDA:\n",
        "\n",
        "Presentation Slides (31.8%) and Podcast/Vodcast/Audio (21.9%) are the most frequent formats.Tools and Resources, Webinars, Other, Workshops, and eLearning are the least common, with fewer than 50 assets each.\n",
        "\n",
        "**4 missing values** were identified in the Format column, and they will be addressed in later data cleaning section.\n",
        "\n",
        "The Format column shows a wide variety of values, indicating **high cardinality**, but with some formats being significantly more frequent than others.\n",
        "\n",
        "**Strong correlations were observed between Format and FileType**, particularly between MP3 files and Podcast/Vodcast/Audio, and between PDF files and Presentation Slides/Publications. T\n",
        "\n",
        "Consider using only one of these features in your model where strong correlations exist to avoid redundancy. However, both features may still be useful for capturing distinct information in other cases."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "N7Sc35fEZ_Jw",
      "metadata": {
        "id": "N7Sc35fEZ_Jw"
      },
      "source": [
        "#### **Q1a Step 14 EDA  Level , Region and CopyRight**\n",
        "\n",
        "**Outline**\n",
        "Focus: This EDA focuses on three categorical columns: \"Level,\" \"Region,\" and \"Copyright.\"\n",
        "\n",
        "Categorical Nature: All three columns are categorical, containing a limited number of predefined options (low cardinality).\n",
        "\n",
        "Level is expected to contain only the categories \"Introductory,\" \"Advanced,\" and \"All\" with no missing values.\n",
        "\n",
        "Region is expected to have three categories: \"International,\" \"Domestic,\" and \"Domestic and International.\"\n",
        "\n",
        "The Copyright column is expected to have four categories: \"Institute,\" \"Other,\" \"Public Domain,\" and \"Speaker/Author.\n",
        "\n",
        "Visualization: We will use bar charts and pie charts to visualize the distribution of categories within each column, ensuring they align with the data dictionary and identifying any potential issues with dominance or skewness."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "BCj6HyWnc3tN",
      "metadata": {
        "id": "BCj6HyWnc3tN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c2be8fe-d4d3-4e96-9217-a9d91f5500e1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 3\n",
            "Missing Values: 4\n",
            "Invalid and Error Values: 4\n",
            "Most Common Values: Level\n",
            "All             6030\n",
            "Advanced          28\n",
            "Introductory       7\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Level values that appear more than once: 6069\n",
            "---------\n",
            "---------\n",
            "Least Common Values:\n",
            "\tAll: 6030\n",
            "\tAdvanced: 28\n",
            "\tIntroductory: 7\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 23172 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 21920 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "------------\n",
            "Number of missing values in the 'Level' column: 4\n",
            "---------\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'Level')\n",
        "print(\"---------\")\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'Level')\n",
        "print(\"---------\")\n",
        "\n",
        "# Check for missing values in the Copyright column\n",
        "missing_Copyright_values = docs_train['Level'].isnull().sum()\n",
        "print('------------')\n",
        "print(f\"Number of missing values in the 'Level' column: {missing_Copyright_values}\")\n",
        "print(\"---------\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aW-jZCW4d3KL",
      "metadata": {
        "id": "aW-jZCW4d3KL"
      },
      "source": [
        "Observation:\n",
        "\n",
        "The EDA output shows no missing data, and the bar chart displays only the three expected categories for the Level column. This aligns with expectations, indicating no apparent data issues.\n",
        "No data cleaning for Level.\n",
        "Consideration for Feature Engineering:\n",
        "\n",
        "We observed that 99.5% of the entries belong to the \"All\" level, which suggests a lack of variation. However, the \"Introductory\" level could be particularly relevant for junior audiences, such as university students or individuals with limited knowledge of certain topics.\n",
        "\n",
        "Adding an \"Introductory\" flag might help in predicting popularity and enhancing engagement summaries. This is a hypothesis and more investigation is still required."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "P8Kuhv62aGjr",
      "metadata": {
        "id": "P8Kuhv62aGjr"
      },
      "source": [
        "We now apply the same method to examinate Region\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "hdi1cGqhkvPn",
      "metadata": {
        "id": "hdi1cGqhkvPn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e66fb000-638d-4d02-9819-9f7b2fc0a8e3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 3\n",
            "Missing Values: 4\n",
            "Invalid and Error Values: 4\n",
            "Most Common Values: Region\n",
            "Domestic and International    6058\n",
            "Domestic                         4\n",
            "International                    3\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Region values that appear more than once: 6069\n",
            "---------\n",
            "---------\n",
            "Least Common Values:\n",
            "\tDomestic and International: 6058\n",
            "\tDomestic: 4\n",
            "\tInternational: 3\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 29296 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 25324 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "------------\n",
            "Number of missing values in the 'Region' column: 4\n",
            "---------\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'Region')\n",
        "print(\"---------\")\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'Region')\n",
        "print(\"---------\")\n",
        "\n",
        "\n",
        "\n",
        "# Check for missing values in the Region column\n",
        "missing_Region_values = docs_train['Region'].isnull().sum()\n",
        "print('------------')\n",
        "print(f\"Number of missing values in the 'Region' column: {missing_Region_values}\")\n",
        "print(\"---------\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EgyRtpUjGmnn",
      "metadata": {
        "id": "EgyRtpUjGmnn"
      },
      "source": [
        "\n",
        "**Observation**\n",
        "- The original dataset had 4 missing values in the Region column. These were resolved during data cleaning, leaving no further issues.\n",
        "- No additional data problems were identified, so no further cleaning is needed.\n",
        "\n",
        "**Feature Consideration**\n",
        "- The overwhelming majority (99.9%) of entries fall under \"Domestic and International,\" indicating minimal variation.\n",
        "- This suggests the Region variable is likely to have little predictive value in the model.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7OfgD48daQCS",
      "metadata": {
        "id": "7OfgD48daQCS"
      },
      "source": [
        "We now apply the same method to examinate\n",
        "Copyright"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b7Ct6HR6XSFX",
      "metadata": {
        "id": "b7Ct6HR6XSFX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f7f7c93-dfa9-402c-db86-7a954832ead3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 4\n",
            "Missing Values: 4\n",
            "Invalid and Error Values: 4\n",
            "Most Common Values: Copyright\n",
            "Institute         5895\n",
            "Speaker/Author     156\n",
            "Other                7\n",
            "Public Domain        7\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Copyright values that appear more than once: 6069\n",
            "---------\n",
            "---------\n",
            "Least Common Values:\n",
            "\tInstitute: 5895\n",
            "\tSpeaker/Author: 156\n",
            "\tOther: 7\n",
            "\tPublic Domain: 7\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 29124 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 27592 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "------------\n",
            "Number of missing values in the 'Copyright' column: 4\n",
            "---------\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'Copyright')\n",
        "print(\"---------\")\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'Copyright')\n",
        "print(\"---------\")\n",
        "\n",
        "# Check for missing values in the Copyright column\n",
        "missing_Copyright_values = docs_train['Copyright'].isnull().sum()\n",
        "print('------------')\n",
        "print(f\"Number of missing values in the 'Copyright' column: {missing_Copyright_values}\")\n",
        "print(\"---------\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FiHUtyNBfi_N",
      "metadata": {
        "id": "FiHUtyNBfi_N"
      },
      "source": [
        "\n",
        "**Observation**\n",
        "- The original dataset had 4 missing values in the Copyright column. These were resolved during data cleaning, leaving no further issues.\n",
        "- No additional data problems were identified, so no further cleaning is needed.\n",
        "\n",
        "**Feature Consideration**\n",
        "- The overwhelming majority (97.7%) of entries fall under \"Domestic and International,\" indicating minimal variation.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tYt2q358dozC",
      "metadata": {
        "id": "tYt2q358dozC"
      },
      "source": [
        "**Summary**:\n",
        "\n",
        "All three columns (Level, Region, and Copyright) are heavily dominated by a single category, which limits their variability and potentially reduces their effectiveness as predictive features.\n",
        "\n",
        "The dominance in each of these columns suggests that they might introduce redundancy in the dataset, and their inclusion in a predictive model should be carefully considered."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "M4MacG9KamOQ",
      "metadata": {
        "id": "M4MacG9KamOQ"
      },
      "source": [
        "#### **Q1a Step 15 EDA  EventAgeGroup**\n",
        "\n",
        "**Outline**\n",
        "- The EventAgeGroup column is expected to have three categories: \"All Retired,\" \"Actuaries,\" and \"YAP.\"\n",
        "- However, initial observations in Q1a indicated a high number of missing values.\n",
        "\n",
        "**Approach:**\n",
        "- Instead of using the default function to generate a bar chart, we will group all missing values into a single bucket labeled \"Missing\" and then examine the distribution using a bar chart.\n",
        "-take a snippet to display 5 sample rows with missing values in the EventAgeGroup column to understand why missing mean in the dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ZPMiJnhOldOp",
      "metadata": {
        "id": "ZPMiJnhOldOp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "462193a6-6401-4a79-ebbb-3f9d47bbb627"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 3\n",
            "Missing Values: 2140\n",
            "Invalid and Error Values: 2140\n",
            "Most Common Values: EventAgeGroup\n",
            "All                  3787\n",
            "YAP                    96\n",
            "Retired Actuaries      46\n",
            "Name: count, dtype: int64\n",
            "Number of rows with EventAgeGroup values that appear more than once: 6069\n",
            "---------\n",
            "------\n",
            "        DMSId                                           FileName FileType  \\\n",
            "5207  4362734         pitching-the-future-of-life-insurance.html     html   \n",
            "2202  4210213  when-life-gives-you-lemons-make-lemonade-a-beh...     html   \n",
            "2048  4210050                            i-am-an-actuary-13.html     html   \n",
            "31    4196196                                     AAJ_3_2002.pdf      pdf   \n",
            "2447  4227184                                        april17.pdf      pdf   \n",
            "\n",
            "      FileSize                                              Title  \\\n",
            "5207      7030              Pitching the future of life insurance   \n",
            "2202      8229  When life gives you lemons, make Lemonade - A ...   \n",
            "2048     11078                      I am an actuary - August 2018   \n",
            "31     3235364  Australian Actuarial Journal 2002 Volume 8 (Is...   \n",
            "2447     13611                       LIWMPC Newsletter April 2017   \n",
            "\n",
            "     StartPublishing StopPublishing  Cpd       Format Level  ...  Copyright  \\\n",
            "5207      2019-10-10              0    1      Article   All  ...  Institute   \n",
            "2202      2019-01-29              0    1      Article   All  ...  Institute   \n",
            "2048      2018-09-04              0    1      Article   All  ...  Institute   \n",
            "31        2018-08-20              0    2  Publication   All  ...  Institute   \n",
            "2447      2018-09-19              0    1  Publication   All  ...  Institute   \n",
            "\n",
            "     EventAgeGroup EventType  \\\n",
            "5207           NaN       NaN   \n",
            "2202           NaN       NaN   \n",
            "2048           NaN       NaN   \n",
            "31             NaN       NaN   \n",
            "2447           NaN       NaN   \n",
            "\n",
            "                                                   Link  \\\n",
            "5207  https://www.actuaries.digital/2019/10/10/pitch...   \n",
            "2202  https://www.actuaries.digital/2017/06/14/when-...   \n",
            "2048  https://www.actuaries.digital/2018/08/15/i-am-...   \n",
            "31    https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "2447  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "\n",
            "                                                   Tags DownloadCount  \\\n",
            "5207                                     Life Insurance             0   \n",
            "2202  Business Acumen|Business Acumen - Disruption|G...             0   \n",
            "2048  Personal Effectiveness &amp; Leadership|Person...             0   \n",
            "31    Professional Updates|Professional Updates - No...             5   \n",
            "2447  Life Insurance|Life Insurance - Practice Updat...             1   \n",
            "\n",
            "     DurationOnDMS  AnnualizedDownloadCount  YearOfStartPublishing  Year  \n",
            "5207           4.6                     0.00                   2019  2019  \n",
            "2202           5.3                     0.00                   2019  2019  \n",
            "2048           5.7                     0.00                   2018  2018  \n",
            "31             5.7                     0.88                   2018  2018  \n",
            "2447           5.6                     0.18                   2018  2018  \n",
            "\n",
            "[5 rows x 22 columns]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 40812 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'EventAgeGroup')\n",
        "print(\"---------\")\n",
        "\n",
        "print(\"------\")\n",
        "# Filter rows where EventAgeGroup is missing\n",
        "missing_eventagegroup = df[df['EventAgeGroup'].isnull()]\n",
        "\n",
        "# Display the rows with missing EventAgeGroup, limited to 5 or the maximum available\n",
        "sample_size = min(5, len(missing_eventagegroup))\n",
        "sample_missing_rows = missing_eventagegroup.sample(sample_size)\n",
        "print(sample_missing_rows)\n",
        "\n",
        "\n",
        "\n",
        "# Replace NaN values with a label 'Missing'\n",
        "df['EventAgeGroup'] = df['EventAgeGroup'].fillna('Missing')\n",
        "\n",
        "# Plot a bar chart\n",
        "plt.figure(figsize=(10, 6))\n",
        "df['EventAgeGroup'].value_counts().plot(kind='bar', color='skyblue')\n",
        "plt.title('Distribution of EventAgeGroup (Including Missing Values)')\n",
        "plt.xlabel('Event Age Group')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QtYKIHdj9Ezx",
      "metadata": {
        "id": "QtYKIHdj9Ezx"
      },
      "source": [
        "**Observation and Data cleaning**\n",
        "\n",
        "A significant portion of assets have missing values in the EventAgeGroup column, as illustrated in the bar chart.\n",
        "\n",
        "The chart also shows that the \"All\" category overwhelmingly dominates the dataset.\n",
        "\n",
        "**What does \"missing\" mean ?**\n",
        "\n",
        "Upon reviewing sample rows, it appears that missing values indicate assets meant for a general audience, with no specific age group target, aligning with the \"All\" category.\n",
        "\n",
        "\n",
        "**Data Cleaning Decision:**\n",
        "Based on the training dataset's distribution, missing values likely represent assets intended for a general audience without a specific age group in mind, which aligns with the \"All\" category.\n",
        "\n",
        "**This data cleaning must be applied after data split to avoid data leakage because the cleaning action is based on distribution of train dataset.**\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "kwl66VLIDruA",
      "metadata": {
        "id": "kwl66VLIDruA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a05d4ed4-f9a5-4f3a-c0dc-ac3b2c001908"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of missing values in 'EventAgeGroup' after cleaning: 0\n",
            "EventAgeGroup\n",
            "All                  5927\n",
            "YAP                    96\n",
            "Retired Actuaries      46\n",
            "Name: count, dtype: int64\n",
            "     DMSId                                           FileName EventAgeGroup\n",
            "0  4194310               international-cpd-opportunities.html           All\n",
            "1  4194590        1995 Report of the Disability Committee.pdf           All\n",
            "2  4194592                           1997 - March Edition.pdf           All\n",
            "3  4194593                       1997 - September Edition.pdf           All\n",
            "4  4194594  1997 Report of the Disability Committee-The Da...           All\n",
            "5  4194595        1997 Report of the Disability Committee.pdf           All\n",
            "6  4194648  2002 Report of the Disability Committee-The Da...           All\n",
            "7  4194649        2002 Report of the Disability Committee.pdf           All\n",
            "8  4194662         2005_AAJ_Volume11_Issue_3(final060315).pdf           All\n",
            "9  4194681     2006_AAJ_Volume12_Issue_1_final_corrected_.pdf           All\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-91-f58596fc1e6a>:2: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['EventAgeGroup'] = docs_train['EventAgeGroup'].fillna('All')\n"
          ]
        }
      ],
      "source": [
        "# Replace missing values in the 'EventAgeGroup' column with 'All'\n",
        "docs_train['EventAgeGroup'] = docs_train['EventAgeGroup'].fillna('All')\n",
        "\n",
        "# Check after data cleaning\n",
        "# Verify that the missing values have been replaced\n",
        "missing_values_count = docs_train['EventAgeGroup'].isnull().sum()  # This should output 0, indicating no missing values\n",
        "print(f\"Number of missing values in 'EventAgeGroup' after cleaning: {missing_values_count}\")\n",
        "\n",
        "# Display the distribution of the 'EventAgeGroup' column to ensure 'All' is included\n",
        "print(docs_train['EventAgeGroup'].value_counts())\n",
        "\n",
        "# Check : display the first few rows to confirm the changes\n",
        "print(docs_train[['DMSId', 'FileName', 'EventAgeGroup']].head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "U-xIlfxPWzli",
      "metadata": {
        "id": "U-xIlfxPWzli"
      },
      "source": [
        "There are no missing values, and all entries have been successfully assigned to \"All.\" The code has executed as intended, ensuring that the data is now complete and ready for next steps."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YA6m-U10ZQvc",
      "metadata": {
        "id": "YA6m-U10ZQvc"
      },
      "source": [
        "Observation:\n",
        "\n",
        "- The EDA output shows no missing data, and the bar chart displays only the three expected categories for the Level column. This aligns with expectations, indicating no apparent data issues.\n",
        "- No data cleaning for Level.\n",
        "\n",
        "Consideration for Feature Engineering:\n",
        "\n",
        "- We observed that 99.5% of the entries belong to the \"All\" level, which suggests a lack of variation. However, the \"Introductory\" level could be particularly relevant for junior audiences, such as university students or individuals with limited knowledge of certain topics.\n",
        "\n",
        "- Adding an \"Introductory\" flag might help in predicting popularity and enhancing engagement summaries. This is a hypothesis and more investigation is still required."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ce-fO11Zawh3",
      "metadata": {
        "id": "Ce-fO11Zawh3"
      },
      "source": [
        "#### **Q1a Step 16 EDA  EventType**\n",
        "\n",
        "Expectation:\n",
        "- The EventType column should contain 6 categories, including \"Insight,\" \"Conference,\" \"Major,\" and \"Other.\"\n",
        "- Initial observations in Q1a revealed a significant number of missing values in this column.\n",
        "\n",
        "Approach:\n",
        "- Instead of using the default function for generating a bar chart, we will first group all missing values into a single category labeled \"Missing.\" Then, we will analyze the distribution using a bar chart.\n",
        "- To better understand the implications of missing data in the EventType column, we will display a snippet of 5 sample rows where EventType is missing.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5nrEF0Bs7tpH",
      "metadata": {
        "id": "5nrEF0Bs7tpH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "ea3cec7d-da01-45c8-9b56-73c4fe52dd4c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 6\n",
            "Missing Values: 1945\n",
            "Invalid and Error Values: 1945\n",
            "Most Common Values: EventType\n",
            "Major                   3176\n",
            "Insights/Other           497\n",
            "One/Half Day             435\n",
            "Insights/Conferences      12\n",
            "Other                      3\n",
            "Name: count, dtype: int64\n",
            "Number of rows with EventType values that appear more than once: 6068\n",
            "---------\n",
            "------\n",
            "        DMSId                                          FileName FileType  \\\n",
            "2685  4358158      CareerView - The Ethics of Moving Roles.docx     docx   \n",
            "4699  4361743                  Transcript SPD Sub-Committee.pdf      pdf   \n",
            "2271  4210290                           how-biased-are-you.html     html   \n",
            "2491  4325400  ActuariesInstituteLaunchesClimateIndex081118.pdf      pdf   \n",
            "1825  4207601                     2013SubAPRADraftSuperPPGs.pdf      pdf   \n",
            "\n",
            "      FileSize                                             Title  \\\n",
            "2685     11891           CareerView - The Ethics of Moving Roles   \n",
            "4699     65743            Transcript - SPD Sub-Committee Podcast   \n",
            "2271      9439                               How biased are you?   \n",
            "2491    118142  Actuaries Climate Index Launches - Media Release   \n",
            "1825    102012   Draft Superannuation Prudential Practice Guides   \n",
            "\n",
            "     StartPublishing StopPublishing  Cpd                 Format Level  ...  \\\n",
            "2685      2018-11-19              0    1  Podcast/Vodcast/Audio   All  ...   \n",
            "4699      2019-06-09              0    1            Publication   All  ...   \n",
            "2271      2018-09-11              0    1                Article   All  ...   \n",
            "2491      2018-11-05              0    2            Publication   All  ...   \n",
            "1825      2018-08-21              0    1            Publication   All  ...   \n",
            "\n",
            "      Copyright EventAgeGroup EventType  \\\n",
            "2685  Institute           All       NaN   \n",
            "4699  Institute           All       NaN   \n",
            "2271  Institute           All       NaN   \n",
            "2491  Institute           All       NaN   \n",
            "1825  Institute           All       NaN   \n",
            "\n",
            "                                                   Link  \\\n",
            "2685  https://www.spreaker.com/user/actinst/episode-...   \n",
            "4699  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "2271  https://www.actuaries.digital/2016/06/14/how-b...   \n",
            "2491  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "1825  https://actuaries.logicaldoc.cloud/download-ti...   \n",
            "\n",
            "                                                   Tags DownloadCount  \\\n",
            "2685  Personal Effectiveness &amp; Leadership|Person...             0   \n",
            "4699                   Superannuation - Emerging Topics            42   \n",
            "2271  Personal Effectiveness &amp; Leadership - Effe...             0   \n",
            "2491  General Insurance|General Insurance - Reinsura...             5   \n",
            "1825  Superannuation|Superannuation - Regulation &am...             1   \n",
            "\n",
            "     DurationOnDMS  AnnualizedDownloadCount  YearOfStartPublishing  Year  \n",
            "2685           5.4                     0.00                   2018  2018  \n",
            "4699           4.9                     8.57                   2019  2019  \n",
            "2271           5.6                     0.00                   2018  2018  \n",
            "2491           5.5                     0.91                   2018  2018  \n",
            "1825           5.7                     0.18                   2018  2018  \n",
            "\n",
            "[5 rows x 22 columns]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 52904 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_describe_column(df, 'EventType')\n",
        "print(\"---------\")\n",
        "\n",
        "print(\"------\")\n",
        "# Filter rows where EventType is missing\n",
        "missing_EventType = df[df['EventType'].isnull()]\n",
        "\n",
        "# Display the rows with missing EventType, limited to 5 or the maximum available\n",
        "sample_size = min(5, len(missing_EventType))\n",
        "sample_missing_rows = missing_EventType.sample(sample_size)\n",
        "print(sample_missing_rows)\n",
        "\n",
        "# Replace NaN values with a label 'Missing'\n",
        "df['EventType'] = df['EventType'].fillna('Missing')\n",
        "\n",
        "# Plot a bar chart\n",
        "plt.figure(figsize=(10, 6))\n",
        "df['EventType'].value_counts().plot(kind='bar', color='skyblue')\n",
        "plt.title('Distribution of EventType (Including Missing Values)')\n",
        "plt.xlabel('EventType')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "30L24vGVgwep",
      "metadata": {
        "id": "30L24vGVgwep"
      },
      "source": [
        "Observations on EventType:\n",
        "\n",
        "- Categories like \"Insights/Conference,\" \"Other,\" and \"Majors/Seminars\" have very low counts.\n",
        "- Significant overlap exists between categories. For example:\n",
        "  - \"Major\" could be grouped with \"Majors/Seminars.\"\n",
        "  - \"Insight/Other\" could be merged with \"Other\" or \"Insights/Conference.\"\n",
        "  - \"One/Half Day\" is unclear and needs further investigation.\n",
        "\n",
        "- The \"Missing\" category is the second largest, indicating a substantial number of entries without an assigned EventType.\n",
        "- We need to understand what \"Missing\" represents: Does it mean the asset doesn't belong to any event, or is it an error?\n",
        "\n",
        "\n",
        "Next Steps:\n",
        "Category Grouping:\n",
        "Group similar categories together to reduce redundancy and improve the clarity of the data. For example:\n",
        "\n",
        "\"Major\" + \"Majors/Seminars\"\n",
        "\"Insight/Other\" + \"Other\" or \"Insights/Conference\"\n",
        "Investigate Missing Values:\n",
        "Determine the meaning of \"Missing.\" If it signifies assets not linked to any event, consider whether this is relevant information or if it should be handled differently."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "26gdGfD9oXSm",
      "metadata": {
        "id": "26gdGfD9oXSm"
      },
      "source": [
        "#####**Investigation of missing EventType**\n",
        "\n",
        "To examine **what \"missing\" means** in the context of EventType, we analyzed the titles of assets with missing values. The word cloud visualization highlights the most common keywords found in the file names of these assets.\n",
        "\n",
        "**Expectation:**\n",
        "Given our hypothesis that the missing EventType values might relate to non-event publications, we expected to see keywords like \"publication,\" \"actuary journal,\" \"monthly,\" \"issue,\" and similar terms.\n",
        "\n",
        "\n",
        "**Process Explanation:**\n",
        "\n",
        "- We broke down the file names into individual words, removed common stopwords, and analyzed the frequency of these keywords.\n",
        "\n",
        "- We visualized the most frequent keywords using a word cloud, helping us identify common themes or terms among the files with missing EventType.\n",
        "\n",
        "This approach allows us to determine if the missing EventType values are indeed associated with non-event-related assets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_2fcfFEbn0EK",
      "metadata": {
        "id": "_2fcfFEbn0EK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a9865007-c3bc-4558-b3fa-0158891cfe49"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 357452 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "from wordcloud import WordCloud\n",
        "from collections import Counter\n",
        "import matplotlib.pyplot as plt\n",
        "import string\n",
        "\n",
        "# Filter rows where EventType is missing\n",
        "missing_eventtype_df = docs_train[docs_train['EventType'].isna()]\n",
        "\n",
        "# Function to clean and extract keywords from file names\n",
        "def extract_keywords(file_name_series):\n",
        "    all_keywords = []\n",
        "    for file_name in file_name_series:\n",
        "        # Remove punctuation and split into words\n",
        "        words = file_name.translate(str.maketrans('', '', string.punctuation)).split()\n",
        "        # Convert to lowercase and filter out common stopwords\n",
        "        keywords = [word.lower() for word in words if len(word) > 2]  # filter out short words (length <= 2)\n",
        "        all_keywords.extend(keywords)\n",
        "    return all_keywords\n",
        "\n",
        "# Extract keywords from the FileName column for rows with missing EventType\n",
        "keywords = extract_keywords(missing_eventtype_df['FileName'])\n",
        "\n",
        "# Count keyword frequencies\n",
        "keyword_counts = Counter(keywords)\n",
        "\n",
        "# Generate a word cloud\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(keyword_counts)\n",
        "\n",
        "# Plot the word cloud\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud of Keywords in File Names with Missing EventType')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sk44Zr_Qpi_s",
      "metadata": {
        "id": "sk44Zr_Qpi_s"
      },
      "source": [
        "\n",
        "**Word Cloud for Missing EventType**\n",
        "\n",
        "The word cloud visualization highlights the most common keywords found in the file names of assets with missing EventType values.\n",
        "\n",
        "Words like \"actuary,\" \"issue,\" \"australia,\" and \"insurance\" are among the most prominent. This suggests that many of these assets are likely related to actuarial publications, journals, or reports.\n",
        "\n",
        "Keywords such as \"issue,\" \"transcript,\" \"practice,\" \"volume,\" and \"newsletter\" further indicate that these assets may be recurring publications or documents that aren't tied to specific events.\n",
        "\n",
        "\n",
        "**Conclusion:**\n",
        "The frequent appearance of words related to actuarial issues, insurance, and publications in the file names suggests that the missing EventType values may correspond to assets that are general publications rather than event-related materials.\n",
        "\n",
        "\n",
        "\n",
        "**Data Cleaning and Feature Engineering Plan**\n",
        "\n",
        "Data Cleaning:\n",
        "\n",
        "- We will replace missing EventType values with a new category, \"Non Event Related,\" based on our findings that these assets likely represent general publications or documents not tied to specific events.\n",
        "\n",
        "- This data cleaning step will be performed after completing the EDA for all variables.\n",
        "\n",
        "\n",
        "Feature Engineering:\n",
        "\n",
        "- Given the high cardinality and low frequency of many minor event categories, we propose simplifying the EventType variable.\n",
        "- We plan to consolidate the EventType into a binary variable with two categories: \"Non Event Related\" and \"Event.\" This approach will reduce complexity and improve the model's performance by focusing on whether an asset is tied to an event or not."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "98vPRDohscOv",
      "metadata": {
        "id": "98vPRDohscOv"
      },
      "source": [
        "##### **Summary of EventType Analysis**\n",
        "\n",
        "The EventType column has 6 unique categories with a significant number of missing values (2,045 out of 6,040).\"Major\" is the dominant category, while other event types like \"Insights/Other\" and \"One/Half Day\" are less common.\n",
        "\n",
        "**Missing Event type Investigation**:\n",
        "\n",
        "A word cloud analysis of file names with missing EventType values suggests these assets are likely non-event-related publications, such as actuarial journals and reports. The missing EventType values likely represent assets not tied to specific events, supporting the creation of a new category.\n",
        "\n",
        "**Data cleaning**:\n",
        "\n",
        "We will replace missing EventType values with \"Non Event Related\" to accurately reflect these assets. This step will be completed after the full EDA.\n",
        "\n",
        "Given the high cardinality and low frequency of many event types, we plan to simplify the EventType variable into a binary classification: \"Non Event Related\" and \"Event.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QAhaCSP2a4jW",
      "metadata": {
        "id": "QAhaCSP2a4jW"
      },
      "source": [
        "#### **Q1a Step 17 EDA  Link**\n",
        "Expectations:\n",
        "-  The Link column should contain valid HTML web links, typically starting with \"http://\" or \"https://\".\n",
        "-  Each link should ideally be unique, as it should direct to a specific asset. Duplicate links could suggest duplicated records or incorrectly assigned links.\n",
        "-  No Missing Values: Every asset should have a corresponding link. Missing links would indicate an issue in the data.\n",
        "-  The links should lead to accessible web pages. Invalid links (e.g., broken or malformed URLs) would need correction.\n",
        "\n",
        "\n",
        "Approach:\n",
        "- Check for missing or invalid links.\n",
        "- Identify unique links and detect any duplicates.\n",
        "- Group and examine the frequency of duplicated links, displaying the results using a bar chart."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "n-j9Bwpe9xlp",
      "metadata": {
        "id": "n-j9Bwpe9xlp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 772
        },
        "outputId": "5573c2b2-c2c6-4d94-f454-27f230495b65"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total Links: 6069\n",
            "Unique Links: 6069\n",
            "Missing Links: 0\n",
            "------------------\n",
            "------------------\n",
            "\n",
            "Summary of Link EDA:\n",
            "Total links: 6069\n",
            "Unique links: 6069\n",
            "Missing links: 0\n",
            "Number of duplicate links: 0\n",
            "Number of invalid links: 0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 105360 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "# 1. Check for missing values in the Link column\n",
        "missing_links = docs_train['Link'].isnull().sum()\n",
        "\n",
        "# 2. Check for unique links\n",
        "unique_links = docs_train['Link'].nunique()\n",
        "total_links = docs_train['Link'].count()\n",
        "\n",
        "# 3. Identify duplicate links\n",
        "duplicate_links = docs_train[docs_train.duplicated(subset=['Link'], keep=False)]\n",
        "\n",
        "# 4. Check for valid links (basic validation for 'http://' or 'https://')\n",
        "valid_links = docs_train['Link'].apply(lambda x: x.startswith(('http://', 'https://')) if pd.notnull(x) else False)\n",
        "invalid_links = docs_train[~valid_links]\n",
        "\n",
        "# Display the results\n",
        "print(f\"Total Links: {total_links}\")\n",
        "print(f\"Unique Links: {unique_links}\")\n",
        "print(f\"Missing Links: {missing_links}\")\n",
        "\n",
        "print(\"------------------\")\n",
        "if not duplicate_links.empty:\n",
        "    print(\"\\nDuplicate Links Found:\")\n",
        "    print(duplicate_links[['DMSId', 'FileName', 'Link']])\n",
        "\n",
        "if not invalid_links.empty:\n",
        "    print(\"\\nInvalid Links Found:\")\n",
        "    print(invalid_links[['DMSId', 'FileName', 'Link']])\n",
        "\n",
        "print(\"------------------\")\n",
        "# Summary\n",
        "print(\"\\nSummary of Link EDA:\")\n",
        "print(f\"Total links: {total_links}\")\n",
        "print(f\"Unique links: {unique_links}\")\n",
        "print(f\"Missing links: {missing_links}\")\n",
        "print(f\"Number of duplicate links: {len(duplicate_links)}\")\n",
        "print(f\"Number of invalid links: {len(invalid_links)}\")\n",
        "\n",
        "\n",
        "# bar chart to show the freuqency of duplication\n",
        "# Group by 'Link' and count the occurrences\n",
        "duplicate_links = docs_data.groupby('Link').filter(lambda x: len(x) > 1)\n",
        "\n",
        "# Count the frequency of each duplicated link\n",
        "link_counts = duplicate_links.groupby('Link').size().reset_index(name='Frequency')\n",
        "\n",
        "# Sort the links by frequency for better visualization\n",
        "link_counts = link_counts.sort_values(by='Frequency', ascending=False)\n",
        "\n",
        "# Plotting the bar chart\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.barh(link_counts['Link'], link_counts['Frequency'], color='skyblue')\n",
        "plt.xlabel('Frequency')\n",
        "plt.ylabel('Link')\n",
        "plt.title('Frequency of Duplicated Links')\n",
        "plt.gca().invert_yaxis()  # Invert y-axis to have the most frequent on top\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HJuUVHFNIr7B",
      "metadata": {
        "id": "HJuUVHFNIr7B"
      },
      "source": [
        "**Summary of Link EDA**:\n",
        "\n",
        "All links in the dataset are unique, valid, and complete, with no missing or duplicate entries. This indicates that the Link column is well-maintained and requires no further cleaning or processing.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lNkZ2bN9bD6j",
      "metadata": {
        "id": "lNkZ2bN9bD6j"
      },
      "source": [
        "#### **Q1a Step 18 EDA tags**\n",
        "\n",
        "The tag system is important because it tells us how the current grouping system works for the Institute.\n",
        "\n",
        "**We want to explore the following:**\n",
        "1. How the tag system functions and how complex it is.\n",
        "2. What overlapping issues exist, and whether we can simplify it into something more manageable and understandable.\n",
        "3. What potential insights we can gain from analyzing the tag system.\n",
        "\n",
        "Expectations:\n",
        "- Data Structure: Based on the data dictionary, the \"Tag\" column likely contains strings with multiple tags separated by commas or semicolons.\n",
        "- Based on Q1a part 1 initial eda, there are number of row with missing tags. We should take a sample to understand why there tags are empty and potentially figure out a way to clean or repair it.\n",
        "\n",
        "Approach:\n",
        "i will use the tutoirla function as a starting point to see what key words are high frequency, because it shouw me the most common words, and least commoent words\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Avoiding Data Leakage:**\n",
        "From the busienss context point of view using the tags as an input in later model could introduce data leakage, especially since they reflect pre-existing classifications that are set to be replaced. To prevent overfitting and ensure a robust model, tags will be excluded from the feature set, focusing instead on features like content metadata (e.g., publication date, format, download history)."
      ],
      "metadata": {
        "id": "uyZeaE694IqQ"
      },
      "id": "uyZeaE694IqQ"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Standard EDA"
      ],
      "metadata": {
        "id": "tf7uUFCN4S88"
      },
      "id": "tf7uUFCN4S88"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3h4K9elLRKCO",
      "metadata": {
        "id": "3h4K9elLRKCO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "42d9205c-be38-4102-b52d-1c4f95a82b6f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of empty strings: 184\n",
            "Most common words:\n",
            "\t-: 10867\n",
            "\t&amp;: 4805\n",
            "\tInsurance: 4539\n",
            "\tManagement: 2568\n",
            "\tLeadership: 2284\n",
            "Least common words:\n",
            "\tExploration|Health|Health: 1\n",
            "\tArchitecture|General: 1\n",
            "\tIndustry|Wealth: 1\n",
            "\tStrategy|Professional: 1\n",
            "\tCare|Life: 1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 1 Axes>"
            ],
            "image/png": "[encoded data, 483680 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 1 Axes>"
            ],
            "image/png": "[encoded data, 31728 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "----------\n",
            "Sample of 10 rows with empty tags:\n",
            "        DMSId                                              Title Tags\n",
            "5221  4362803                    Pensions, housing and postcodes  NaN\n",
            "6021  4365247                              Customer expectations  NaN\n",
            "5776  4364472  SG Level, Benefit Adequacy and Investment Perf...  NaN\n",
            "6054  4365312  Electricity, Ethics & Actuaries: A Case Study ...  NaN\n",
            "5282  4362980            Hong Kong insurtech focuses on customer  NaN\n",
            "5821  4364715         Developing the retirement income framework  NaN\n",
            "5396  4363439                   What is a good customer outcome?  NaN\n",
            "5238  4362860  Best Practice Treatment Whilst Protecting Sche...  NaN\n",
            "4878  4362041   Leading insurtech actuary to speak at Sydney JRS  NaN\n",
            "5242  4362864                 Plenary 1 - Customer and Community  NaN\n",
            "----------\n",
            "\n",
            "Sample of 10 rows with non-empty tags:\n",
            "        DMSId                                              Title  \\\n",
            "3504  4359895  INS15 - Pricing Terminal Illness: Is There No ...   \n",
            "4533  4361467  GIS2 - C5C - Convex Optimization Perspective o...   \n",
            "4618  4361581  Insights - A history of loss reserving leading...   \n",
            "1125  4201665  INS16 - Construction of Detailed Correlation S...   \n",
            "2715  4358171                   DAS18 - Welcome and WOMD - Audio   \n",
            "2396  4227131                 How to Avoid 'Death by PowerPoint'   \n",
            "1864  4207640                       Fees, Costs and Transparency   \n",
            "2465  4259859  Q&A with Deputy Chief Actuary at Medibank, Ada...   \n",
            "4558  4361492  GIS12 - Keynote Address - Regulation Matters -...   \n",
            "607   4200298      FSF16 - P4 - Financial Sustainability - Audio   \n",
            "\n",
            "                                                   Tags  \n",
            "3504  Health - Macro Health Industry|Health - Privat...  \n",
            "4533  General Insurance|General Insurance - Risk Man...  \n",
            "4618                     Data Science|General Insurance  \n",
            "1125  General Insurance|General Insurance - Reinsura...  \n",
            "2715                                       Data Science  \n",
            "2396  Personal Effectiveness &amp; Leadership|Person...  \n",
            "1864  Superannuation|Superannuation - Product Design...  \n",
            "2465  Health|Health - Private Health Insurance Industry  \n",
            "4558  General Insurance|General Insurance - Regulati...  \n",
            "607   Business Acumen|Business Acumen - Business Str...  \n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "\n",
        "analyze_words(df, 'Tags')\n",
        "\n",
        "print(\"----------\")\n",
        "# Sample dataframe (replace with your actual dataframe)\n",
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Step 1: Identify rows with empty or missing tags\n",
        "empty_tag_rows = df[(df['Tags'].isna()) | (df['Tags'].str.strip() == '')]\n",
        "\n",
        "# Step 2: Display 10 sample rows with empty tags\n",
        "sample_empty_tag_rows = empty_tag_rows.sample(10, random_state=1) if len(empty_tag_rows) > 10 else empty_tag_rows\n",
        "print(\"Sample of 10 rows with empty tags:\")\n",
        "print(sample_empty_tag_rows[['DMSId',  'Title','Tags']])\n",
        "print(\"----------\")\n",
        "\n",
        "\n",
        "# Sample dataframe (replace with your actual dataframe)\n",
        "df = pd.DataFrame(docs_data)\n",
        "\n",
        "# Step 1: Identify rows with non-empty tags\n",
        "non_empty_tag_rows = df[(df['Tags'].notna()) & (df['Tags'].str.strip() != '')]\n",
        "\n",
        "# Step 2: Display 10 sample rows with non-empty tags\n",
        "sample_non_empty_tag_rows = non_empty_tag_rows.sample(10, random_state=1) if len(non_empty_tag_rows) > 10 else non_empty_tag_rows\n",
        "print(\"\\nSample of 10 rows with non-empty tags:\")\n",
        "print(sample_non_empty_tag_rows[['DMSId', 'Title', 'Tags']])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jeuXRK3qWcZE",
      "metadata": {
        "id": "jeuXRK3qWcZE"
      },
      "source": [
        "**Observation**\n",
        "- 583 empty strings\n",
        "- High Frequency of Stop Words: The most frequent \"words\" in the Tag column are \"-\" and \"&amp;\".\n",
        "- \"-\" is likely being used as a separator or placeholder, not as a meaningful tag.\n",
        "- \"&amp;\" is the HTML entity for the ampersand character (\"&\"), indicating that the data may have been extracted from HTML and not properly cleaned.\n",
        "\n",
        "Action:\n",
        "- We should clean the data by removing stop words\n",
        "- redo EDA for tags"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Comment and next step**\n",
        "\n",
        "The above EDA does not provide much insight due to excessive noise. To gain a clearer understanding, we will focus only on the main field, specifically the first tag before the initial \"|\"."
      ],
      "metadata": {
        "id": "m0XdZEkdWQdD"
      },
      "id": "m0XdZEkdWQdD"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "LrIdFipllBeY",
      "metadata": {
        "id": "LrIdFipllBeY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 550
        },
        "outputId": "0cc85f7a-e921-437a-d3a2-5e1d21385442"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-96-f2a7c4b4f9cb>:3: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['Tag_only'] = docs_train['Tags'].str.split('|').str[0]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 186716 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Create a new column 'Tag_only' containing only the first word before the first '|'\n",
        "docs_train['Tag_only'] = docs_train['Tags'].str.split('|').str[0]\n",
        "\n",
        "# Filter rows where Tag_only is not null\n",
        "non_null_tag_only_df = docs_train[docs_train['Tag_only'].notna()]\n",
        "\n",
        "# Function to clean and extract keywords from Tag_only column\n",
        "def extract_keywords_from_tag_only(tag_only_series):\n",
        "    all_keywords = []\n",
        "    for tag_only in tag_only_series:\n",
        "        # Remove punctuation and split into words\n",
        "        words = tag_only.translate(str.maketrans('', '', string.punctuation)).split()\n",
        "        # Convert to lowercase and filter out common stopwords\n",
        "        keywords = [word.lower() for word in words if word.lower() not in stop_words and len(word) > 2]  # filter out short words (length <= 2)\n",
        "        all_keywords.extend(keywords)\n",
        "    return all_keywords\n",
        "\n",
        "# Extract keywords from the Tag_only column for rows with non-null Tag_only\n",
        "keywords = extract_keywords_from_tag_only(non_null_tag_only_df['Tag_only'])\n",
        "\n",
        "# Count keyword frequencies\n",
        "keyword_counts = Counter(keywords)\n",
        "\n",
        "# Generate a word cloud\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(keyword_counts)\n",
        "\n",
        "# Plot the word cloud\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud of Keywords in Tag_only')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we focus on the main tag, we can see that it is primarily related to business practice fields such as life insurance, superannuation, and general insurance."
      ],
      "metadata": {
        "id": "6qX7Uz7YWmaD"
      },
      "id": "6qX7Uz7YWmaD"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2bChfloMld17",
      "metadata": {
        "id": "2bChfloMld17",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "35c1e708-84cc-4c12-d241-2f15ad23ef27"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag_only\n",
            "Banking                                                                                     287\n",
            "Banking - Capital                                                                             1\n",
            "Banking - Credit Risk                                                                         2\n",
            "Banking - Economy Updates                                                                     4\n",
            "Banking - Emerging Topics                                                                     6\n",
            "Banking - IFRS                                                                                2\n",
            "Banking - Liquidity Risk                                                                      1\n",
            "Banking - Thought Leadership                                                                  4\n",
            "Business Acumen                                                                             195\n",
            "Business Acumen - Business Strategy                                                           5\n",
            "Business Acumen - Problem Solving &amp; Innovation                                            1\n",
            "Business Acumen - Project Management                                                          1\n",
            "Data Science                                                                                279\n",
            "Data Science - Data Analysis                                                                  3\n",
            "Data Science - Emerging Topics                                                                3\n",
            "Data Science - Regulation &amp; Standards                                                     1\n",
            "General Insurance                                                                          1868\n",
            "General Insurance - Capital Management                                                        1\n",
            "General Insurance - Emerging Topics                                                           2\n",
            "General Insurance - Injury Schemes                                                            9\n",
            "General Insurance - Pricing                                                                   2\n",
            "General Insurance - Reinsurance &amp; Catastrophe Risk                                        2\n",
            "General Insurance - Risk Management                                                           2\n",
            "Health                                                                                      520\n",
            "Health - Capital Management                                                                   1\n",
            "Health - Health Reform                                                                        2\n",
            "Health - Macro Health Industry                                                                3\n",
            "Health - Private Health Insurance Industry                                                    1\n",
            "Life Insurance                                                                              578\n",
            "Life Insurance - Emerging Topics                                                              1\n",
            "Life Insurance - Practice Updates                                                             1\n",
            "Life Insurance - Thought Leadership                                                           1\n",
            "Not Applicable                                                                                2\n",
            "Personal Effectiveness &amp; Leadership                                                     477\n",
            "Personal Effectiveness &amp; Leadership - Career Management                                   9\n",
            "Personal Effectiveness &amp; Leadership - Diversity &amp; Inclusion                           3\n",
            "Personal Effectiveness &amp; Leadership - Effective Communication &amp; Decision Making       3\n",
            "Personal Effectiveness &amp; Leadership - General Soft Skills                                 1\n",
            "Personal Effectiveness &amp; Leadership - People Management &amp; Leadership                  3\n",
            "Personal Effectiveness &amp; Leadership - Thought Leadership                                  1\n",
            "Professional Updates                                                                        434\n",
            "Professional Updates - Industry &amp; Economic                                                1\n",
            "Professionalism Training                                                                      7\n",
            "Risk Management                                                                             368\n",
            "Risk Management - Enterprise Risk Management Overview                                         1\n",
            "Risk Management - Practice Updates                                                            2\n",
            "Risk Management - Risk Models                                                                 2\n",
            "Superannuation                                                                              592\n",
            "Superannuation - Emerging Topics                                                              2\n",
            "Superannuation - Retirement Incomes                                                           3\n",
            "Superannuation - Retirement Systems                                                           1\n",
            "Wealth Management                                                                           184\n",
            "Name: Tag_only, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "# A table to show what the Main tag look like\n",
        "\n",
        "import pandas as pd\n",
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Group by 'Tag_only' and count the number of occurrences\n",
        "tag_counts = df.groupby('Tag_only')['Tag_only'].count()\n",
        "\n",
        "# Print the counts\n",
        "print(tag_counts)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Key Insights from Tag Analysis:**\n",
        "\n",
        "1. Dominance of Major Categories:\n",
        "- General Insurance is the most frequently tagged category, with 1,868 occurrences, making it a significant area of focus for the CPD Hub. This is followed by Health, Life Insurance, and Superannuation, indicating these fields dominate the asset categorization.\n",
        "\n",
        "2. Granularity Within Main Categories:\n",
        "- Some main categories, like General Insurance, Personal Effectiveness & Leadership, and Data Science, have subcategories (e.g., \"General Insurance - Reinsurance & Catastrophe Risk,\" \"Personal Effectiveness & Leadership - Career Management\"). These are far less frequent compared to their main tags, suggesting an inconsistent or underutilized tagging approach.\n",
        "\n",
        "3. Sparse Subcategories:\n",
        "- Many subcategories have only a handful of occurrences, which could indicate either an incomplete tagging process or a lack of specificity in how certain assets are classified. For example, General Insurance - Capital Management only appears once, compared to the main General Insurance tag.\n",
        "\n",
        "4. Potential for Tag Simplification:\n",
        "- Certain tags, such as Banking and Business Acumen, have several subcategories with very low usage. Simplifying these categories and reducing overlaps could improve clarity and searchability for users.\n",
        "\n",
        "5. Redundant or Unclear Tags:\n",
        "- Tags like Not Applicable and Professionalism Training seem out of place or underutilized, suggesting potential tagging inconsistencies or redundancies that may need addressing to streamline the system."
      ],
      "metadata": {
        "id": "ythczl-43KSo"
      },
      "id": "ythczl-43KSo"
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Cleaning the tags is not necessary**\n",
        "\n",
        "It is because the Institute plans to replace the existing tag system with a new clustering approach. Since the tag system will no longer be used in the future, including tags as inputs in the model would introduce data leakage."
      ],
      "metadata": {
        "id": "CoxukdKP42gl"
      },
      "id": "CoxukdKP42gl"
    },
    {
      "cell_type": "markdown",
      "source": [
        "The follwoing reduce noice of  tags with only the 1st main tags"
      ],
      "metadata": {
        "id": "WRTRt1cG33gu"
      },
      "id": "WRTRt1cG33gu"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ItxiYZrjnoeD",
      "metadata": {
        "id": "ItxiYZrjnoeD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ffbc0f29-b9e1-4af6-f154-8650612aa23e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-98-cdf002612b06>:4: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['Tag_only'] = docs_train['Tag_only'].str.strip()  # Remove leading/trailing spaces\n",
            "<ipython-input-98-cdf002612b06>:5: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['Tag_only'] = docs_train['Tag_only'].str.lower()  # Convert to lowercase\n",
            "<ipython-input-98-cdf002612b06>:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['Tag_only_common'] = docs_train['Tag_only'].str.split('-').str[0]\n",
            "<ipython-input-98-cdf002612b06>:11: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_train['Tag_only_common'] = docs_train['Tag_only_common'].str.strip()\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd  # Import pandas if not already imported\n",
        "\n",
        "# Clean the 'Tag_only' column to ensure consistent spacing and casing\n",
        "docs_train['Tag_only'] = docs_train['Tag_only'].str.strip()  # Remove leading/trailing spaces\n",
        "docs_train['Tag_only'] = docs_train['Tag_only'].str.lower()  # Convert to lowercase\n",
        "\n",
        "# Now create the 'Tag_only_common' column\n",
        "docs_train['Tag_only_common'] = docs_train['Tag_only'].str.split('-').str[0]\n",
        "\n",
        "# Remove spacing\n",
        "docs_train['Tag_only_common'] = docs_train['Tag_only_common'].str.strip()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "We will further remove the subgroups as we want to focus on the key groupings in the current tagging system. The following code removes any subgroups by merging similar groups together.\n",
        "\n",
        "We wil first look at training set of data."
      ],
      "metadata": {
        "id": "VZD1xTfhW-Eo"
      },
      "id": "VZD1xTfhW-Eo"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "w6AdX3zbnrhr",
      "metadata": {
        "id": "w6AdX3zbnrhr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 759
        },
        "outputId": "43a0238c-4a14-4a35-8db7-ac64688f8f4f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag_only_common\n",
            "banking                                     307\n",
            "business acumen                             202\n",
            "data science                                286\n",
            "general insurance                          1886\n",
            "health                                      527\n",
            "life insurance                              581\n",
            "not applicable                                2\n",
            "personal effectiveness &amp; leadership     497\n",
            "professional updates                        435\n",
            "professionalism training                      7\n",
            "risk management                             373\n",
            "superannuation                              598\n",
            "wealth management                           184\n",
            "Name: Tag_only_common, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 171720 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Data type of 'Tag_only_common': object\n",
            "Number of missing values in 'Tag_only_common': 184\n",
            "Unique values in 'Tag_only_common': 13\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me a world cloub for Tag_only_common, also i need to know how many assets are in each tag_only_common group, and what are the dtype type , missing value, ...etc\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Group by 'Tag_only_common' and count the number of occurrences\n",
        "tag_counts = df.groupby('Tag_only_common')['Tag_only_common'].count()\n",
        "\n",
        "# Print the counts\n",
        "print(tag_counts)\n",
        "\n",
        "# Function to clean and extract keywords from Tag_only_common column\n",
        "def extract_keywords_from_tag_only_common(tag_only_common_series):\n",
        "    all_keywords = []\n",
        "    for tag_only_common in tag_only_common_series:\n",
        "        # Remove punctuation and split into words\n",
        "        if pd.notna(tag_only_common):  # Check if tag_only_common is not null\n",
        "            words = tag_only_common.translate(str.maketrans('', '', string.punctuation)).split()\n",
        "            # Convert to lowercase and filter out common stopwords\n",
        "            keywords = [word.lower() for word in words if word.lower() not in stop_words and len(word) > 2]  # filter out short words (length <= 2)\n",
        "            all_keywords.extend(keywords)\n",
        "    return all_keywords\n",
        "\n",
        "# Extract keywords from the Tag_only_common column for rows with non-null Tag_only_common\n",
        "keywords = extract_keywords_from_tag_only_common(df['Tag_only_common'])\n",
        "\n",
        "# Count keyword frequencies\n",
        "keyword_counts = Counter(keywords)\n",
        "\n",
        "# Generate a word cloud\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(keyword_counts)\n",
        "\n",
        "# Plot the word cloud\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud of Keywords in Tag_only_common')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Data type, missing values, etc. for Tag_only_common\n",
        "print(\"Data type of 'Tag_only_common':\", df['Tag_only_common'].dtype)\n",
        "print(\"Number of missing values in 'Tag_only_common':\", df['Tag_only_common'].isnull().sum())\n",
        "print(\"Unique values in 'Tag_only_common':\", df['Tag_only_common'].nunique())\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We create the Tag_only and Tag_only_common for later Q2 analysis"
      ],
      "metadata": {
        "id": "OWjCJFHFaCb-"
      },
      "id": "OWjCJFHFaCb-"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8_LqqMuxpr5r",
      "metadata": {
        "id": "8_LqqMuxpr5r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "548fb2b8-73f0-438e-83fd-b45603d8696d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-100-47dba8a5407e>:4: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_val['Tag_only'] = docs_val['Tags'].str.split('|').str[0]\n",
            "<ipython-input-100-47dba8a5407e>:7: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_val['Tag_only_common'] = docs_val['Tag_only'].str.split('-').str[0]\n",
            "<ipython-input-100-47dba8a5407e>:8: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_val['Tag_only_common'] = docs_val['Tag_only_common'].str.strip()\n",
            "<ipython-input-100-47dba8a5407e>:9: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_val['Tag_only_common'] = docs_val['Tag_only_common'].str.lower()\n",
            "<ipython-input-100-47dba8a5407e>:13: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_test['Tag_only'] = docs_test['Tags'].str.split('|').str[0]\n",
            "<ipython-input-100-47dba8a5407e>:16: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_test['Tag_only_common'] = docs_test['Tag_only'].str.split('-').str[0]\n",
            "<ipython-input-100-47dba8a5407e>:17: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_test['Tag_only_common'] = docs_test['Tag_only_common'].str.strip()\n",
            "<ipython-input-100-47dba8a5407e>:18: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  docs_test['Tag_only_common'] = docs_test['Tag_only_common'].str.lower()\n"
          ]
        }
      ],
      "source": [
        "# prompt: can you also create Tag_only, and Tag_only_common for docs_val and docs_test for me\n",
        "\n",
        "# Create Tag_only for docs_val\n",
        "docs_val['Tag_only'] = docs_val['Tags'].str.split('|').str[0]\n",
        "\n",
        "# Create Tag_only_common for docs_val\n",
        "docs_val['Tag_only_common'] = docs_val['Tag_only'].str.split('-').str[0]\n",
        "docs_val['Tag_only_common'] = docs_val['Tag_only_common'].str.strip()\n",
        "docs_val['Tag_only_common'] = docs_val['Tag_only_common'].str.lower()\n",
        "\n",
        "\n",
        "# Create Tag_only for docs_test\n",
        "docs_test['Tag_only'] = docs_test['Tags'].str.split('|').str[0]\n",
        "\n",
        "# Create Tag_only_common for docs_test\n",
        "docs_test['Tag_only_common'] = docs_test['Tag_only'].str.split('-').str[0]\n",
        "docs_test['Tag_only_common'] = docs_test['Tag_only_common'].str.strip()\n",
        "docs_test['Tag_only_common'] = docs_test['Tag_only_common'].str.lower()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will now apply some cleaning steps to the tags to give us a clearer picture of the validation and testing datasets.\n",
        "\n",
        "Since tags will not be used as a feature, examining them in the validation and testing sets will not create any data leakage."
      ],
      "metadata": {
        "id": "hkJT_ddaaOOl"
      },
      "id": "hkJT_ddaaOOl"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Iuo6SCKMp-SK",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Iuo6SCKMp-SK",
        "outputId": "51bd917b-a6ce-4ee8-eb9c-7e885f124a2c"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 178332 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Summary for Tag_only_common for Validation Set:\n",
            "Data type of 'Tag_only_common': object\n",
            "Number of missing values in 'Tag_only_common': 239\n",
            "Unique values in 'Tag_only_common': 16\n",
            "\n",
            "Tag Counts:\n",
            " Tag_only_common\n",
            "banking                                     36\n",
            "business acumen                             13\n",
            "climate change                               4\n",
            "climate change working group                 1\n",
            "data analytics                               1\n",
            "data science                                88\n",
            "general insurance                          105\n",
            "health                                      75\n",
            "life insurance                              38\n",
            "not applicable                               3\n",
            "personal effectiveness &amp; leadership     54\n",
            "professional updates                        34\n",
            "professionalism training                     1\n",
            "risk management                             14\n",
            "superannuation                              43\n",
            "wealth management                            8\n",
            "Name: Tag_only_common, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 172508 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Summary for Tag_only_common for Test Set:\n",
            "Data type of 'Tag_only_common': object\n",
            "Number of missing values in 'Tag_only_common': 158\n",
            "Unique values in 'Tag_only_common': 17\n",
            "\n",
            "Tag Counts:\n",
            " Tag_only_common\n",
            "banking                                     23\n",
            "business acumen                              1\n",
            "climate change                               2\n",
            "climate change working group                 1\n",
            "data analytics                               4\n",
            "data science                               111\n",
            "general insurance                          144\n",
            "health                                      75\n",
            "life insurance                              67\n",
            "not applicable                               1\n",
            "other                                        3\n",
            "personal effectiveness &amp; leadership     10\n",
            "professional updates                         2\n",
            "professionalism training                     2\n",
            "risk management                             54\n",
            "superannuation                              52\n",
            "wealth management                           15\n",
            "Name: Tag_only_common, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "def generate_wordcloud_and_summary(df, title):\n",
        "  \"\"\"Generates a word cloud and summary table for a given DataFrame and column.\"\"\"\n",
        "\n",
        "  # Function to clean and extract keywords from Tag_only_common column\n",
        "  def extract_keywords_from_tag_only_common(tag_only_common_series):\n",
        "      all_keywords = []\n",
        "      for tag_only_common in tag_only_common_series:\n",
        "          # Remove punctuation and split into words\n",
        "          if pd.notna(tag_only_common):  # Check if tag_only_common is not null\n",
        "              words = tag_only_common.translate(str.maketrans('', '', string.punctuation)).split()\n",
        "              # Convert to lowercase and filter out common stopwords\n",
        "              keywords = [word.lower() for word in words if word.lower() not in stop_words and len(word) > 2]  # filter out short words (length <= 2)\n",
        "              all_keywords.extend(keywords)\n",
        "      return all_keywords\n",
        "\n",
        "  # Extract keywords from the Tag_only_common column for rows with non-null Tag_only_common\n",
        "  keywords = extract_keywords_from_tag_only_common(df['Tag_only_common'])\n",
        "\n",
        "  # Count keyword frequencies\n",
        "  keyword_counts = Counter(keywords)\n",
        "\n",
        "  # Generate a word cloud\n",
        "  wordcloud = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(keyword_counts)\n",
        "\n",
        "  # Plot the word cloud\n",
        "  plt.figure(figsize=(10, 6))\n",
        "  plt.imshow(wordcloud, interpolation='bilinear')\n",
        "  plt.axis('off')\n",
        "  plt.title(f'Word Cloud of Keywords in Tag_only_common for {title}')\n",
        "  plt.show()\n",
        "\n",
        "  # Summary table\n",
        "  print(f\"Summary for Tag_only_common for {title}:\")\n",
        "  print(\"Data type of 'Tag_only_common':\", df['Tag_only_common'].dtype)\n",
        "  print(\"Number of missing values in 'Tag_only_common':\", df['Tag_only_common'].isnull().sum())\n",
        "  print(\"Unique values in 'Tag_only_common':\", df['Tag_only_common'].nunique())\n",
        "\n",
        "  # Group by 'Tag_only_common' and count the number of occurrences\n",
        "  tag_counts = df.groupby('Tag_only_common')['Tag_only_common'].count()\n",
        "  print(\"\\nTag Counts:\\n\", tag_counts)\n",
        "\n",
        "# Generate for docs_val\n",
        "generate_wordcloud_and_summary(docs_val, \"Validation Set\")\n",
        "\n",
        "# Generate for docs_test\n",
        "generate_wordcloud_and_summary(docs_test, \"Test Set\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Summary of Findings on Tag EDA:**\n",
        "\n",
        "The tag system currently categorizes assets into key areas such as General Insurance, Life Insurance, Health, and Superannuation, which dominate the dataset. However, many subcategories, like \"Emerging Topics\" or \"Thought Leadership,\" appear infrequently and do not significantly differentiate assets.\n",
        "\n",
        "Key insights from the analysis reveal that the tag system, while useful for historical context, has multiple overlapping tags and subcategories that offer minimal value. Given the plan to replace the tag system with a new clustering approach, using tags as an input would introduce potential data leakage. Therefore, the tags will not be included in the future model, focusing instead on more relevant features such as asset metadata and download history.\n",
        "\n",
        "This transition supports the Institute's goal of moving toward a more streamlined and effective classification system that improves asset discovery and member engagement."
      ],
      "metadata": {
        "id": "oNL0itoV4aSx"
      },
      "id": "oNL0itoV4aSx"
    },
    {
      "cell_type": "markdown",
      "id": "9IlL9mhiWA7h",
      "metadata": {
        "id": "9IlL9mhiWA7h"
      },
      "source": [
        "#### Q1a Step 19. Data Cleaning - After data split\n",
        "**Outline**\n",
        "\n",
        "Belowing are the  Cleaning step we will take based on EDA Findings:\n",
        "1. Dealing with incorrect value \"ics\" in FileType\n",
        "2. Remove outlier - Post-Pandemic Acutarial World\n",
        "3. Remove outlier of a massive file that has little relevent with actuary assets\n",
        "4. Hanlding missing valus for 4 rows of transaciton by mannual lookup repair\n",
        "5. Handling missing value in EventAgeGroup\n",
        "6. Handling missing value in EventType\n",
        "7. Addressing Data Type issues and incorrect values for StopPublishing\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wNeMaR5rjpyd",
      "metadata": {
        "id": "wNeMaR5rjpyd"
      },
      "source": [
        "##### 1.Dealing with incorrect value- FileType incorrect value of ics\n",
        "**Outline:**\n",
        "**Data with incorrect value of ICS**\n",
        "\n",
        "The asset titled \"Progressing Your Investment Career\" was initially classified as an ics file type. However, it is actually a PDF file that contains a link to an ICS schedule.\n",
        "\n",
        "\n",
        "**Data Cleaning**:\n",
        "Manually correct the file type from ICS to PDF in the dataset.\n",
        "\n",
        "**Rationale:**\n",
        "Addressing this inconsistency ensures the dataset remains clean, accurate, and reliable, which is essential for effective analysis and decision-making. By correcting this classification, we maintain the integrity of our data, enabling more accurate insights in subsequent analysis.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Wr7HUI57HA5X",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 827
        },
        "id": "Wr7HUI57HA5X",
        "outputId": "5c4bb276-343b-4c5c-a2f3-1224f1248c84"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Unique Values: 11\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: FileType\n",
            "pdf     3560\n",
            "mp3     1594\n",
            "html     735\n",
            "docx     144\n",
            "wma       16\n",
            "Name: count, dtype: int64\n",
            "Number of rows with FileType values that appear more than once: 6067\n",
            "------------------------------\n",
            "Distinct FileTypes: ['html' 'pdf' 'mp3' 'zip' 'ppt' 'wmv' 'pptx' 'wma' 'docx' 'xlsx' 'txt']\n",
            "------------------------------\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 35608 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "docs_train_cleaned = pd.DataFrame(docs_train)\n",
        "\n",
        "# Correct the file type for the specific row\n",
        "docs_train_cleaned.loc[docs_train_cleaned['FileName'] == 'ProgressingYourInvestmentCareer.ics', 'FileType'] = 'pdf'\n",
        "\n",
        "\n",
        "# Checking for Cleaned file\n",
        "df = pd.DataFrame(docs_train_cleaned)\n",
        "eda_describe_column(df, 'FileType')\n",
        "\n",
        "# Get all distinct values of FileType\n",
        "distinct_filetypes = df['FileType'].unique()\n",
        "print(\"------------------------------\")\n",
        "print(f\"Distinct FileTypes: {distinct_filetypes}\")\n",
        "print(\"------------------------------\")\n",
        "\n",
        "\n",
        "# Bar chart of the most common filenames\n",
        "top_filenames = df['FileType'].value_counts().head(12)\n",
        "plt.figure(figsize=(12, 6))\n",
        "top_filenames.plot(kind='bar')\n",
        "plt.title('Top 12 Most Common FileType')\n",
        "plt.xlabel('FileType')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dRgKLU3WN1Pj",
      "metadata": {
        "id": "dRgKLU3WN1Pj"
      },
      "source": [
        "##### **2.Remove Outlier: The Post-Pandemic Actuarial World virtual Seminar**\n",
        "\n",
        "The Post-Pandemic Actuarial World virtual Seminar\n",
        "\n",
        "This is highly unusual, as assets with higher CPD points typically attract more downloads, especially over a period of 2.6 years. DMSID:4366648\n",
        "\n",
        "\n",
        "The finding suggests that this asset may indeed be an outlier and possibly a data error.\n",
        "\n",
        "We will remove this asset from the dataset. Retaining it could skew the analysis and lead to inaccurate conclusions, especially if it doesn't align with the typical characteristics of other assets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "OtEfrXXkN8oH",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OtEfrXXkN8oH",
        "outputId": "8b820707-82a7-41a9-ae86-29a73bc0b785"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Asset with DMSId 4366648 removed from docs_train_cleaned:\n",
            "Empty DataFrame\n",
            "Columns: [DMSId, FileName, FileType, FileSize, Title, StartPublishing, StopPublishing, Cpd, Format, Level, Description, Region, Copyright, EventAgeGroup, EventType, Link, Tags, DownloadCount, DurationOnDMS, AnnualizedDownloadCount, YearOfStartPublishing, Year, Tag_only, Tag_only_common]\n",
            "Index: []\n",
            "\n",
            "[0 rows x 24 columns]\n"
          ]
        }
      ],
      "source": [
        "# Remove the asset from the cleaned training dataset\n",
        "docs_train_cleaned = docs_train_cleaned[docs_train_cleaned['DMSId'] != 4366648]\n",
        "\n",
        "# Verify the removal\n",
        "print(\"Asset with DMSId 4366648 removed from docs_train_cleaned:\")\n",
        "print(docs_train_cleaned[docs_train_cleaned['DMSId'] == 4366648])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uRFbO4-mFolJ",
      "metadata": {
        "id": "uRFbO4-mFolJ"
      },
      "source": [
        "##### **3.Remove outlier of massive size** (must be done after data split)\n",
        "\n",
        "The asset \"Plenary 5 Pete Goss.pptx\" , DMSID:4361700  contains personal photos unrelated to actuarial content, and it cannot be categorized under any relevant topic. The content is about a person driving a boat across Europe, making it irrelevant and of little value to the purpose of Acuaries CDP.\n",
        "\n",
        "Therefore, I have decided to delete this outlier. I will also do a check to make sure this removeal is sucessful.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "MX-IKB0dd4Dl",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MX-IKB0dd4Dl",
        "outputId": "2fc69701-0041-48e9-ba4a-e2e4eb67b35c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Asset with DMSId 4361700 removed from docs_train_cleaned:\n",
            "Empty DataFrame\n",
            "Columns: [DMSId, FileName, FileType, FileSize, Title, StartPublishing, StopPublishing, Cpd, Format, Level, Description, Region, Copyright, EventAgeGroup, EventType, Link, Tags, DownloadCount, DurationOnDMS, AnnualizedDownloadCount, YearOfStartPublishing, Year, Tag_only, Tag_only_common]\n",
            "Index: []\n",
            "\n",
            "[0 rows x 24 columns]\n",
            "Asset with DMSId 4361700 removed from downloads_train_cleaned:\n",
            "Empty DataFrame\n",
            "Columns: [DMSId, DownloadDate, Month, YearMonth]\n",
            "Index: []\n"
          ]
        }
      ],
      "source": [
        "# DMSId of the outlier asset \"Plenary 5 Pete Goss.pptx\"\n",
        "outlier_dmsid = 4361700  # The correct DMSId\n",
        "\n",
        "# Remove the asset from the cleaned training dataset\n",
        "docs_train_cleaned = docs_train_cleaned[docs_train_cleaned['DMSId'] != outlier_dmsid]\n",
        "\n",
        "# Remove the corresponding downloads from the download datasets\n",
        "downloads_train_cleaned = downloads_train[downloads_train['DMSId'] != outlier_dmsid]\n",
        "\n",
        "# Verify the removal from docs_train_cleaned\n",
        "print(\"Asset with DMSId 4361700 removed from docs_train_cleaned:\")\n",
        "print(docs_train_cleaned[docs_train_cleaned['DMSId'] == outlier_dmsid])\n",
        "\n",
        "# Verify the removal from downloads_train_cleaned\n",
        "print(\"Asset with DMSId 4361700 removed from downloads_train_cleaned:\")\n",
        "print(downloads_train_cleaned[downloads_train_cleaned['DMSId'] == outlier_dmsid])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SktLzj_JAnHV",
      "metadata": {
        "id": "SktLzj_JAnHV"
      },
      "source": [
        "##### **4.Handling missing value - 4 rows with missing values by mannual repair**\n",
        "Recap of Data Cleaning done in Q1a Step 2 - EDA FileName:\n",
        "\n",
        "- Issue: We encountered 4 rows with missing file details.\n",
        "- Action Taken: By clicking on the links associated with these records, we were able to manually correcting missing valuess.\n",
        "\n",
        "- We can perform this cleaning before spliting data because manualy correcting does not use statistical information from the dataset, so it would not introduce data leakage.\n",
        "\n",
        "\n",
        "**Data Cleaning Assumptions:**\n",
        "  - StartPublishing Date: We will assume that the StartPublishing date reflects the date when the asset was released or when the interview occurred.\n",
        "\n",
        "- However, since the DMS only started in 20 Aug 2018, any StartPublishing dates that fall before 2018 will be truncated as 20 Aug 2018.\n",
        "\n",
        "- We apply the most common obsevation for Region, Level and Copyright if we cannot find clear indication from the assets.\n",
        "\n",
        "\n",
        "**Check**:\n",
        "We apply check after complete the cleaning step.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dG_tB5v1CXPX",
      "metadata": {
        "id": "dG_tB5v1CXPX"
      },
      "source": [
        "1. \"Innovation in the 21st Century\" is a YouTube video published by the Actuaries Institute on 19th December 2017. YouTube video is accessed via a web browser, and the link to the video is an HTML page\n",
        "\n",
        "Here is some suggestion on what repair to put into each empty columns\n",
        "\n",
        "Title: \"Innovation in the 21st Century\"\n",
        "FileType: HTML\n",
        "StartPublishing: \"2018-08-20\"\n",
        "Format: \"Video\"\n",
        "Level:\"All\"\n",
        "Region: \"Domestic and International\"\n",
        "Copyright: \"Institute\"\n",
        "2. \"The role of actuaries in crunching COVID-19 numbers\" is a podcast interview recorded as a PDF on 19th May 2020.\n",
        "\n",
        "Suggested Repairs for the Empty Columns:\n",
        "\n",
        "Title: \"The Role of Actuaries in Crunching COVID-19 Numbers\"\n",
        "StartPublishing:\"2020-05-19\"\n",
        "Format: \"Podcast/Vodcast/Audio\"\n",
        "Level:\"All\"\n",
        "Region: \"Domestic and International\"\n",
        "Copyright: \"Institute\"\n",
        "3. \"Modelling COVID-19 infections - lessons and reflections\" is a podcast interview recorded as a PDF on 29th May 2020.\n",
        "\n",
        "Suggested Repairs for the Empty Columns:\n",
        "\n",
        "Title: \"Modelling COVID-19 Infections - Lessons and Reflections\"\n",
        "StartPublishing: \"2020-05-29\"\n",
        "Format:\"Podcast/Vodcast/Audio\"\n",
        "Level: \"All\"\n",
        "Region:\"Domestic and International\"\n",
        "Copyright: \"Institute\"\n",
        "\"Life Insurance and Wealth Management Practice Committee practice update podcast.pdf\" is a pod cast interview recored as a PDF on 29th May 2020.\n",
        "\n",
        "Suggested Reapirs for the Empty Columns:\n",
        "\n",
        "Title: \"M\"Life Insurance and Wealth Management Practice Committee Practice Update Podcast\"\n",
        "StartPublishing: \"2020-05-29\"\n",
        "Format:\"Podcast/Vodcast/Audio\"\n",
        "Level: \"All\"\n",
        "Region:\"Domestic and International\"\n",
        "Copyright: \"Institute\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "udGl5255CfBf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "udGl5255CfBf",
        "outputId": "d6d012dd-33e2-400d-d956-e3ea47076a57"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Updated Rows:\n",
            "------------------------------\n",
            "        DMSId                                              Title FileType  \\\n",
            "2529  4325486                     Innovation in the 21st Century     html   \n",
            "5787  4364500  The Role of Actuaries in Crunching COVID-19 Nu...      pdf   \n",
            "5797  4364531  Modelling COVID-19 Infections - Lessons and Re...      pdf   \n",
            "5803  4364544  Life Insurance and Wealth Management Practice ...      pdf   \n",
            "\n",
            "     StartPublishing                 Format Level                      Region  \\\n",
            "2529      2018-08-20                  Video   All  Domestic and International   \n",
            "5787      2020-05-19  Podcast/Vodcast/Audio   All  Domestic and International   \n",
            "5797      2020-05-29  Podcast/Vodcast/Audio   All  Domestic and International   \n",
            "5803      2020-05-29  Podcast/Vodcast/Audio   All  Domestic and International   \n",
            "\n",
            "      Copyright  \n",
            "2529  Institute  \n",
            "5787  Institute  \n",
            "5797  Institute  \n",
            "5803  Institute  \n",
            "------------------------------\n",
            "Verification of Data Cleaning:\n",
            "------------------------------\n",
            "Correct: DMSId 4325486 - Title is Innovation in the 21st Century\n",
            "Correct: DMSId 4325486 - FileType is html\n",
            "Correct: DMSId 4325486 - StartPublishing is 2018-08-20\n",
            "Correct: DMSId 4325486 - Format is Video\n",
            "Correct: DMSId 4325486 - Level is All\n",
            "Correct: DMSId 4325486 - Region is Domestic and International\n",
            "Correct: DMSId 4325486 - Copyright is Institute\n",
            "Correct: DMSId 4364500 - Title is The Role of Actuaries in Crunching COVID-19 Numbers\n",
            "Correct: DMSId 4364500 - StartPublishing is 2020-05-19\n",
            "Correct: DMSId 4364500 - Format is Podcast/Vodcast/Audio\n",
            "Correct: DMSId 4364500 - Level is All\n",
            "Correct: DMSId 4364500 - Region is Domestic and International\n",
            "Correct: DMSId 4364500 - Copyright is Institute\n",
            "Correct: DMSId 4364531 - Title is Modelling COVID-19 Infections - Lessons and Reflections\n",
            "Correct: DMSId 4364531 - StartPublishing is 2020-05-29\n",
            "Correct: DMSId 4364531 - Format is Podcast/Vodcast/Audio\n",
            "Correct: DMSId 4364531 - Level is All\n",
            "Correct: DMSId 4364531 - Region is Domestic and International\n",
            "Correct: DMSId 4364531 - Copyright is Institute\n",
            "Correct: DMSId 4364544 - Title is Life Insurance and Wealth Management Practice Committee Practice Update Podcast\n",
            "Correct: DMSId 4364544 - StartPublishing is 2020-05-29\n",
            "Correct: DMSId 4364544 - Format is Podcast/Vodcast/Audio\n",
            "Correct: DMSId 4364544 - Level is All\n",
            "Correct: DMSId 4364544 - Region is Domestic and International\n",
            "Correct: DMSId 4364544 - Copyright is Institute\n",
            "------------------------------\n",
            "EDA Summary after Data Cleaning:\n",
            "------------------------------\n",
            "Unique Values: 5672\n",
            "Missing Values: 0\n",
            "Invalid and Error Values: 0\n",
            "Most Common Values: Title\n",
            "Actuaries Magazine                             21\n",
            "Actuaries Digital Magazine                     12\n",
            "GIS2018 - Plenary 4 - Ethics and Leadership     7\n",
            "Plenary 4 -The road ahead                       7\n",
            "Plenary 2 - The far future of healthcare        7\n",
            "Name: count, dtype: int64\n",
            "Number of rows with Title values that appear more than once: 654\n"
          ]
        }
      ],
      "source": [
        "# Data cleaning for 4 missing rows\n",
        "# Define the repairs for each DMSId (or other unique identifier)\n",
        "repairs = {\n",
        "    4325486: {\n",
        "        'Title': 'Innovation in the 21st Century',\n",
        "        'FileType': 'html',\n",
        "        'StartPublishing': '2018-08-20',\n",
        "        'Format': 'Video',\n",
        "        'Level': 'All',\n",
        "        'Region': 'Domestic and International',\n",
        "        'Copyright': 'Institute'\n",
        "    },\n",
        "    4364500: {\n",
        "        'Title': 'The Role of Actuaries in Crunching COVID-19 Numbers',\n",
        "        'StartPublishing': '2020-05-19',\n",
        "        'Format': 'Podcast/Vodcast/Audio',\n",
        "        'Level': 'All',\n",
        "        'Region': 'Domestic and International',\n",
        "        'Copyright': 'Institute'\n",
        "    },\n",
        "    4364531: {\n",
        "        'Title': 'Modelling COVID-19 Infections - Lessons and Reflections',\n",
        "        'StartPublishing': '2020-05-29',\n",
        "        'Format': 'Podcast/Vodcast/Audio',\n",
        "        'Level': 'All',\n",
        "        'Region': 'Domestic and International',\n",
        "        'Copyright': 'Institute'\n",
        "    },\n",
        "    4364544: {\n",
        "        'Title': 'Life Insurance and Wealth Management Practice Committee Practice Update Podcast',\n",
        "        'StartPublishing': '2020-05-29',\n",
        "        'Format': 'Podcast/Vodcast/Audio',\n",
        "        'Level': 'All',\n",
        "        'Region': 'Domestic and International',\n",
        "        'Copyright': 'Institute'\n",
        "    }\n",
        "}\n",
        "\n",
        "# Apply the repairs to the DataFrame with date conversion\n",
        "for dmsid, repair in repairs.items():\n",
        "    for column, value in repair.items():\n",
        "        if column == 'StartPublishing':\n",
        "            docs_train_cleaned.loc[docs_train_cleaned['DMSId'] == dmsid, column] = pd.to_datetime(value).strftime('%Y-%m-%d')\n",
        "        else:\n",
        "            docs_train_cleaned.loc[docs_train_cleaned['DMSId'] == dmsid, column] = value\n",
        "\n",
        "# Display the updated rows for verification\n",
        "updated_rows = docs_train_cleaned[docs_train_cleaned['DMSId'].isin(repairs.keys())]\n",
        "print(\"Updated Rows:\")\n",
        "print(\"------------------------------\")\n",
        "print(updated_rows[['DMSId', 'Title', 'FileType', 'StartPublishing', 'Format', 'Level', 'Region', 'Copyright']])\n",
        "\n",
        "\n",
        "# Check after data cleaning\n",
        "# Verify that the repairs were applied correctly\n",
        "def verify_repairs(df, repairs):\n",
        "    for dmsid, expected_values in repairs.items():\n",
        "        row = df[df['DMSId'] == dmsid]\n",
        "        for column, expected_value in expected_values.items():\n",
        "            actual_value = row[column].values[0]\n",
        "            if column == 'StartPublishing':\n",
        "                actual_value = pd.to_datetime(actual_value).strftime('%Y-%m-%d')\n",
        "            if actual_value != expected_value:\n",
        "                print(f\"Error in DMSId {dmsid}: {column} is {actual_value}, expected {expected_value}.\")\n",
        "            else:\n",
        "                print(f\"Correct: DMSId {dmsid} - {column} is {actual_value}\")\n",
        "\n",
        "# Run the verification\n",
        "print(\"------------------------------\")\n",
        "print(\"Verification of Data Cleaning:\")\n",
        "print(\"------------------------------\")\n",
        "verify_repairs(docs_train_cleaned, repairs)\n",
        "\n",
        "# Print the EDA summary after Data cleaning\n",
        "print(\"------------------------------\")\n",
        "print(\"EDA Summary after Data Cleaning:\")\n",
        "print(\"------------------------------\")\n",
        "eda_describe_column(docs_train_cleaned, 'Title')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rMSLsm_yjWI5",
      "metadata": {
        "id": "rMSLsm_yjWI5"
      },
      "source": [
        "#####**5.Handling Missing Value - EventAgeGroup Replace missing to \"All\"**\n",
        "\n",
        "**What does \"missing\" mean ?**\n",
        "Upon reviewing sample rows, it appears that missing values indicate assets meant for a general audience, with no specific age group target, aligning with the \"All\" category.\n",
        "\n",
        "\n",
        "**Data Cleaning Decision:**\n",
        "Based on the training dataset's distribution, missing values likely represent assets intended for a general audience without a specific age group in mind, which aligns with the \"All\" category.\n",
        "\n",
        "We also perform a check to ensure this missing issue is resloved."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5Fxfj9YZjWtH",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Fxfj9YZjWtH",
        "outputId": "b815a565-1d9b-4b7d-9c19-df6e84d071ea"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of missing values in 'EventAgeGroup' before cleaning: 0\n",
            "Number of missing values in 'EventAgeGroup' after cleaning: 0\n",
            "\n",
            "Distribution of 'EventAgeGroup' after cleaning:\n",
            "EventAgeGroup\n",
            "All                  5926\n",
            "YAP                    96\n",
            "Retired Actuaries      46\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-106-d10925b0ce6e>:6: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
            "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
            "\n",
            "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
            "\n",
            "\n",
            "  docs_train_cleaned['EventAgeGroup'].fillna('All', inplace=True)\n"
          ]
        }
      ],
      "source": [
        "# Display the number of missing values before cleaning\n",
        "missing_before = docs_train_cleaned['EventAgeGroup'].isna().sum()\n",
        "print(f\"Number of missing values in 'EventAgeGroup' before cleaning: {missing_before}\")\n",
        "\n",
        "# Replace missing values with \"All\"\n",
        "docs_train_cleaned['EventAgeGroup'].fillna('All', inplace=True)\n",
        "\n",
        "# Display the number of missing values after cleaning to confirm\n",
        "missing_after = docs_train_cleaned['EventAgeGroup'].isna().sum()\n",
        "print(f\"Number of missing values in 'EventAgeGroup' after cleaning: {missing_after}\")\n",
        "\n",
        "# Verify the replacement was successful by checking the distribution\n",
        "print(\"\\nDistribution of 'EventAgeGroup' after cleaning:\")\n",
        "print(docs_train_cleaned['EventAgeGroup'].value_counts())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "RqHjnCjXuH_j",
      "metadata": {
        "id": "RqHjnCjXuH_j"
      },
      "source": [
        "#####**6.Handling Missing Value - Replacing Missing Event type as a new category:**\n",
        "\n",
        "A word cloud analysis of file names with missing EventType values suggests these assets are likely non-event-related publications, such as actuarial journals and reports. The missing EventType values likely represent assets not tied to specific events, supporting the creation of a new category.\n",
        "\n",
        "**Data cleaning:**\n",
        "\n",
        "We replace all NaN values in the EventType column with the string \"Non Event Related\".\n",
        "\n",
        "The value_counts() method is used to check the distribution of the EventType column after the replacement, ensuring that the operation was successful.\n",
        "\n",
        "The bar chart will display the counts of each EventType category after the missing values have been replaced with \"Non Event Related.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "mQ8r6UUUysyn",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 958
        },
        "id": "mQ8r6UUUysyn",
        "outputId": "ea8b1882-456a-4686-e061-c2bebd432db2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Missing EventType values replaced with 'Non Event Related':\n",
            "EventType\n",
            "Major                   3175\n",
            "Non Event Related       1945\n",
            "Insights/Other           497\n",
            "One/Half Day             435\n",
            "Insights/Conferences      12\n",
            "Other                      3\n",
            "Majors/Seminars            1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-107-f203963941d2>:2: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
            "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
            "\n",
            "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
            "\n",
            "\n",
            "  docs_train_cleaned['EventType'].fillna('Non Event Related', inplace=True)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 57336 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Replace missing EventType values with \"Non Event Related\"\n",
        "docs_train_cleaned['EventType'].fillna('Non Event Related', inplace=True)\n",
        "\n",
        "# Verify the replacement\n",
        "print(\"Missing EventType values replaced with 'Non Event Related':\")\n",
        "print(docs_train_cleaned['EventType'].value_counts())\n",
        "\n",
        "\n",
        "# Create a bar chart to visualize the distribution of EventType after replacement\n",
        "plt.figure(figsize=(10, 6))\n",
        "docs_train_cleaned['EventType'].value_counts().plot(kind='bar', color='skyblue')\n",
        "plt.title('Distribution of EventType After Replacing Missing Values')\n",
        "plt.xlabel('EventType')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "IuH-uceLH2R0",
      "metadata": {
        "id": "IuH-uceLH2R0"
      },
      "source": [
        "#####7.Dealing with StopPublishing (addressing Date Type issues & Incorrect values)\n",
        "\n",
        "Outlines:\n",
        "\n",
        "**Data Type Issue:**\n",
        "\n",
        "- The StopPublishing field, intended to represent dates, is currently stored as an object (string) rather than a datetime format. According to the data dictionary, StopPublishing should have values ranging between November 2023 and November 3023.\n",
        "\n",
        "**Incorrect Values:**\n",
        "\n",
        "-  5,071 entries are marked as 1970-01-01, the Unix epoch start date, likely indicating \"no expiry date.\"\n",
        "-  Some assets have stop dates set after January 1, 3000, which also likely represent \"no expiry date.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SPJ6fzxdaOhh",
      "metadata": {
        "id": "SPJ6fzxdaOhh"
      },
      "source": [
        "**Dealing with incorrect values**\n",
        "To align with the data dictionary, we will use 3023-11-30 as the default \"no expiry date\" for assets.\n",
        "\n",
        "Any asset with a StopPublishing date set 100 years or more in the future will be adjusted to 3023-11-30 to standardize the data for later feature engineering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "dRmvKo2XEqjz",
      "metadata": {
        "id": "dRmvKo2XEqjz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "85026e19-a52e-4519-edaa-5449c18a1b5a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "StopPublishing after cleaning:\n",
            "count                    6068\n",
            "unique                      5\n",
            "top       3023-11-30 00:00:00\n",
            "freq                     5866\n",
            "Name: StopPublishing, dtype: object\n"
          ]
        }
      ],
      "source": [
        "import datetime\n",
        "import pandas as pd\n",
        "\n",
        "# Define the largest possible date according to the data dictionary\n",
        "largest_possible_date = pd.Timestamp('3023-11-30')\n",
        "\n",
        "# Convert StopPublishing to datetime format\n",
        "docs_train_cleaned['StopPublishing'] = pd.to_datetime(docs_train_cleaned['StopPublishing'], errors='coerce')\n",
        "\n",
        "# Fix incorrect values\n",
        "docs_train_cleaned['StopPublishing'] = docs_train_cleaned['StopPublishing'].apply(\n",
        "    lambda x: largest_possible_date if pd.isnull(x) or x == pd.Timestamp('1970-01-01') or x > pd.Timestamp('3000-01-01') else x\n",
        ")\n",
        "\n",
        "# Verify the cleaning\n",
        "print(\"StopPublishing after cleaning:\")\n",
        "print(docs_train_cleaned['StopPublishing'].describe())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a2s3UtUkVobH",
      "metadata": {
        "id": "a2s3UtUkVobH"
      },
      "source": [
        "**Dealing Date Type issue**\n",
        "We are now trying to reslove the data type issue by change the StopPublishing column to datatime format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yZ91MDSOVuow",
      "metadata": {
        "id": "yZ91MDSOVuow",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7d56408f-e9c6-43bf-c27b-15878172982a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "datetime64[ns]\n",
            "0   NaT\n",
            "1   NaT\n",
            "2   NaT\n",
            "3   NaT\n",
            "4   NaT\n",
            "Name: StopPublishing, dtype: datetime64[ns]\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Convert the StopPublishing column to datetime format\n",
        "docs_train_cleaned['StopPublishing'] = pd.to_datetime(docs_train_cleaned['StopPublishing'], errors='coerce')\n",
        "\n",
        "# Verify the conversion\n",
        "print(docs_train_cleaned['StopPublishing'].dtype)\n",
        "print(docs_train_cleaned['StopPublishing'].head())\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0ZftSTyzXJBS",
      "metadata": {
        "id": "0ZftSTyzXJBS"
      },
      "source": [
        "**The result is not as expected, and NaT values were created.**\n",
        "\n",
        "- After checking with Gemini, the error \"OutOfBoundsDatetime: Cannot cast 3023-11-30 00:00:00 to unit='ns' without overflow\" occurred because the date 3023-11-30 is outside the valid range for Pandas Timestamps with nanosecond resolution.\n",
        "\n",
        "- By default, this range is limited to approximately the years 1677 to 2262.\n",
        "\n",
        "To work around this issue, we should set the largest possible date within 31 Dec 2261, instead of using 3023.\n",
        "\n",
        "This would not have a material impact to later part of feature enginering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QmPf_ZxZWokL",
      "metadata": {
        "id": "QmPf_ZxZWokL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a723d086-285e-43f2-dc98-07d96224b752"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "datetime64[ns]\n",
            "0   2261-12-31\n",
            "1   2261-12-31\n",
            "2   2261-12-31\n",
            "3   2261-12-31\n",
            "4   2261-12-31\n",
            "Name: StopPublishing, dtype: datetime64[ns]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-110-c6181b51c3e0>:8: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
            "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
            "\n",
            "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
            "\n",
            "\n",
            "  docs_train_cleaned['StopPublishing'].fillna(largest_possible_date, inplace=True)\n"
          ]
        }
      ],
      "source": [
        "# Convert the StopPublishing column to datetime format\n",
        "docs_train_cleaned['StopPublishing'] = pd.to_datetime(docs_train_cleaned['StopPublishing'], errors='coerce')\n",
        "\n",
        "# Define the largest possible date within the valid range for Pandas Timestamps\n",
        "largest_possible_date = pd.Timestamp('2261-12-31')\n",
        "\n",
        "# Replace NaT values with the largest possible date\n",
        "docs_train_cleaned['StopPublishing'].fillna(largest_possible_date, inplace=True)\n",
        "\n",
        "# Verify the changes\n",
        "print(docs_train_cleaned['StopPublishing'].dtype)\n",
        "print(docs_train_cleaned['StopPublishing'].head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DYQs-PM7Y-ZE",
      "metadata": {
        "id": "DYQs-PM7Y-ZE"
      },
      "source": [
        "To check our cleaning , we create a bar chart displaying the frequency of StopPublishing dates by year after the cleaning process"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "j1KNfDrhY1i0",
      "metadata": {
        "id": "j1KNfDrhY1i0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 585
        },
        "outputId": "27d5b08a-811f-4a62-bbb0-e0924b906b30"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 31384 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Extract the year from the StopPublishing column\n",
        "docs_train_cleaned['StopPublishing_Year'] = docs_train_cleaned['StopPublishing'].dt.year\n",
        "\n",
        "# Plot the frequency of StopPublishing by year\n",
        "plt.figure(figsize=(10, 6))\n",
        "docs_train_cleaned['StopPublishing_Year'].value_counts().sort_index().plot(kind='bar', color='skyblue')\n",
        "plt.title('Frequency of StopPublishing Dates by Year After Cleaning')\n",
        "plt.xlabel('Year')\n",
        "plt.ylabel('Frequency')\n",
        "plt.xticks(rotation=45)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MMs_RgI_9RYb",
      "metadata": {
        "id": "MMs_RgI_9RYb"
      },
      "source": [
        "##### Summary of Data Cleaning Findings and Actions\n",
        "1. Dealing with Incorrect FileType:\n",
        "\n",
        "Issue: The asset \"Progressing Your Investment Career\" was incorrectly labeled as an ICS file type.\n",
        "Action: Corrected the file type to PDF to maintain data accuracy.\n",
        "2. Removing Outliers:\n",
        "\n",
        "\"The Post-Pandemic Actuarial World\" Seminar: Identified as an outlier with unusual CPD points and download patterns, removed to prevent skewed analysis.\n",
        "\"Plenary 5 Pete Goss.pptx\": This massive, irrelevant file was removed as it contained non-actuarial content.\n",
        "3. Handling Missing Values in 4 Key Rows:\n",
        "\n",
        "Issue: Four assets had missing information.\n",
        "Action: Manually repaired these rows by referencing links and using logical assumptions for missing fields like Title, FileType, StartPublishing, Format, Level, Region, and Copyright.\n",
        "4. Handling Missing Values in EventAgeGroup:\n",
        "\n",
        "Issue: Missing values indicated a general audience.\n",
        "Action: Replaced missing values with \"All\" to reflect their broad applicability.\n",
        "5. Replacing Missing EventType:\n",
        "\n",
        "Issue: Many assets had missing EventType values.\n",
        "Action: Replaced missing values with \"Non Event Related,\" based on a word cloud analysis suggesting these assets weren't tied to specific events.\n",
        "6. Addressing StopPublishing Date Issues:\n",
        "\n",
        "Data Type Issue: Converted the column from string to datetime format.\n",
        "Incorrect Values: Replaced Unix epoch start dates and future dates beyond 3000 with '2261-12-31,' the largest valid date, to represent \"no expiry date.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Bsd1DWJUa6XT",
      "metadata": {
        "id": "Bsd1DWJUa6XT"
      },
      "source": [
        "#### Q1a Step 20. Checking our cleaned training data and ensure it is good to use:\n",
        "\n",
        "Outlines:\n",
        "- Provides a quick overview of the central tendencies and spread of numerical data, which helps in identifying any anomalies or unusual values.\n",
        "- Checks for any missing data that could affect the analysis or model training.\n",
        "- Ensures that all columns have the correct data types for the intended analysis.\n",
        "-  Visualizes the distribution of important numerical features like FileSize and CPD to spot any outliers or irregularities.\n",
        "- Examines the unique values and distribution of categorical columns to ensure they align with expectations.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-VV1jOzO4oe_",
      "metadata": {
        "id": "-VV1jOzO4oe_"
      },
      "source": [
        "Rationale for Minimal Data Cleaning on Validation and Test Sets\n",
        "\n",
        "We aim to apply minimal data cleaning to the validation and test sets to avoid overly refining these datasets. This approach prevents introducing biases that could lead to overinterpretation or overfitting by the model. By maintaining the natural variability and imperfections present in real-world data, we ensure that our model is evaluated on more realistic conditions, resulting in a more accurate assessment of its generalization ability and performance. This strategy helps avoid a false sense of confidence in the model's effectiveness."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0qk_sJqQb5xb",
      "metadata": {
        "id": "0qk_sJqQb5xb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7be38b8c-6176-43a7-ee10-79e57f097a04"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Data Types of Each Column:\n",
            "DMSId                               int64\n",
            "FileName                           object\n",
            "FileType                           object\n",
            "FileSize                            int64\n",
            "Title                              object\n",
            "StartPublishing            datetime64[ns]\n",
            "StopPublishing             datetime64[ns]\n",
            "Cpd                                 int64\n",
            "Format                             object\n",
            "Level                              object\n",
            "Description                        object\n",
            "Region                             object\n",
            "Copyright                          object\n",
            "EventAgeGroup                      object\n",
            "EventType                          object\n",
            "Link                               object\n",
            "Tags                               object\n",
            "DownloadCount                       int64\n",
            "DurationOnDMS                     float64\n",
            "AnnualizedDownloadCount           float64\n",
            "YearOfStartPublishing               int32\n",
            "Year                                int32\n",
            "Tag_only                           object\n",
            "Tag_only_common                    object\n",
            "StopPublishing_Year                 int32\n",
            "dtype: object\n"
          ]
        }
      ],
      "source": [
        "# Verify the data types of each column\n",
        "print(\"\\nData Types of Each Column:\")\n",
        "print(docs_train_cleaned.dtypes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "RVvCk07ObtuR",
      "metadata": {
        "id": "RVvCk07ObtuR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "00a01162-d9db-4ff6-c07d-013cab08f85e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Missing Values by Column:\n",
            "DMSId                        0\n",
            "FileName                     0\n",
            "FileType                     0\n",
            "FileSize                     0\n",
            "Title                        0\n",
            "StartPublishing              0\n",
            "StopPublishing               0\n",
            "Cpd                          0\n",
            "Format                       0\n",
            "Level                        0\n",
            "Description                433\n",
            "Region                       0\n",
            "Copyright                    0\n",
            "EventAgeGroup                0\n",
            "EventType                    0\n",
            "Link                         0\n",
            "Tags                       184\n",
            "DownloadCount                0\n",
            "DurationOnDMS                0\n",
            "AnnualizedDownloadCount      0\n",
            "YearOfStartPublishing        0\n",
            "Year                         0\n",
            "Tag_only                   184\n",
            "Tag_only_common            184\n",
            "StopPublishing_Year          0\n",
            "dtype: int64\n"
          ]
        }
      ],
      "source": [
        "# Check for missing values in each column\n",
        "print(\"\\nMissing Values by Column:\")\n",
        "print(docs_train_cleaned.isnull().sum())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Vo-bVZC0b0iQ",
      "metadata": {
        "id": "Vo-bVZC0b0iQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 447
        },
        "outputId": "0c2397e9-05b5-4d56-b77e-5a16fb661571"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1400x800 with 2 Axes>"
            ],
            "image/png": "[encoded data, 41520 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Plot distribution of a few key numerical features\n",
        "plt.figure(figsize=(14, 8))\n",
        "\n",
        "# Example: Plotting FileSize\n",
        "plt.subplot(2, 2, 1)\n",
        "sns.histplot(docs_train_cleaned['FileSize'], kde=True, color='blue')\n",
        "plt.title('Distribution of FileSize')\n",
        "\n",
        "# Example: Plotting CPD\n",
        "plt.subplot(2, 2, 2)\n",
        "sns.histplot(docs_train_cleaned['Cpd'], kde=True, color='green')\n",
        "plt.title('Distribution of cpd')\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "T2BsDu8TcMHa",
      "metadata": {
        "id": "T2BsDu8TcMHa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "01c1ed07-30f6-4227-d58d-2c9afa868910"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Unique Values in Categorical Columns:\n",
            "FileType: 11 unique values\n",
            "FileType\n",
            "pdf     3560\n",
            "mp3     1594\n",
            "html     736\n",
            "docx     143\n",
            "wma       16\n",
            "pptx       7\n",
            "zip        4\n",
            "ppt        4\n",
            "xlsx       2\n",
            "wmv        1\n",
            "txt        1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "EventType: 7 unique values\n",
            "EventType\n",
            "Major                   3175\n",
            "Non Event Related       1945\n",
            "Insights/Other           497\n",
            "One/Half Day             435\n",
            "Insights/Conferences      12\n",
            "Other                      3\n",
            "Majors/Seminars            1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Level: 3 unique values\n",
            "Level\n",
            "All             6033\n",
            "Advanced          28\n",
            "Introductory       7\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Region: 3 unique values\n",
            "Region\n",
            "Domestic and International    6061\n",
            "Domestic                         4\n",
            "International                    3\n",
            "Name: count, dtype: int64\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Check unique values for categorical columns\n",
        "print(\"\\nUnique Values in Categorical Columns:\")\n",
        "for col in ['FileType', 'EventType', 'Level', 'Region']:\n",
        "    print(f\"{col}: {docs_train_cleaned[col].nunique()} unique values\")\n",
        "    print(docs_train_cleaned[col].value_counts())\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "h8g7r94AV0Wg",
      "metadata": {
        "id": "h8g7r94AV0Wg"
      },
      "source": [
        "**Summary**  \n",
        "Rationale for Minimal Cleaning on Validation and Test Sets: We aim to keep these sets as close to real-world data as possible, ensuring the model's evaluation is unbiased and accurately reflects its performance on unrefined data.\n",
        "\n",
        "The training data is now clean, well-structured, and ready for further analysis, ensuring a solid foundation for model development..*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xIEDBkj5V0Wg",
      "metadata": {
        "id": "xIEDBkj5V0Wg"
      },
      "source": [
        "###**Overall summary**  \n",
        "Overall Summary of Q1a: Exploratory Data Analysis (EDA)\n",
        "In Q1a, we conducted a thorough analysis of the CPD Hub's dataset, including the documents and download data. Our EDA explored various aspects, such as download patterns, publication trends, and relationships between key variables. Notably, we observed clear data drift in download volumes over time, as well as shifts in asset formats, both in terms of assets published and downloaded.\n",
        "\n",
        "We also reviewed the existing tagging system, which serves as the current clustering mechanism. We confirmed that the system contains overlapping tags and sub-tags, which often result in confusion. Upon removing the sub-tags, we found that the tags mainly focus on key business practices, such as General Insurance, Life Insurance, and Superannuation.\n",
        "\n",
        "After analyzing data drift and concept drift, we decided to split the data into training, validation, and testing sets to prevent potential data leakage, which will be critical for the construction of predictive models in later tasks. We also assessed data quality, identifying issues such as inconsistencies, missing data, incorrect data types, duplication, and outliers. Appropriate cleaning steps were taken to address these issues and prepare the data for further analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f15YWRqswTo8",
      "metadata": {
        "id": "f15YWRqswTo8"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "id": "yNelfxy5UQLm",
      "metadata": {
        "id": "yNelfxy5UQLm"
      },
      "source": [
        "### Q1b - Clean then Vectorise the 'Description' feature\n",
        "#### Purpose:\n",
        "\n",
        "**Purpose:**\n",
        "Convert descriptions into numerical vectors (vectorization) for clustering (Q2) and neural network models (Q3), as they require numerical inputs. This process will apply to all training, validation, and testing datasets.\n",
        "\n",
        "Section Objectives:\n",
        "1. Conduct EDA to understand description characteristics (length, complexity, punctuation).\n",
        "2. Clean text data for vectorization.\n",
        "3. Select and apply an appropriate vectorization method (TF-IDF).\n",
        "4. Validate the vectorization process.\n",
        "5. Final Check Vectorisation\n",
        "\n",
        "\n",
        "#### References:\n",
        "*Outline any references you have used in this section of your notebook here.*\n",
        "\n",
        "We use some of the code from ChatGPT4 and the Gemini on Colab AI"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DYZlTpdBUQLz",
      "metadata": {
        "id": "DYZlTpdBUQLz"
      },
      "source": [
        "#### Q1b Step1 **EDA of Descritpion**\n",
        "\n",
        "**Purpose:**\n",
        "This step aims to assess the 'Description' column's data type, identify missing values, and gain an initial understanding of the data's quality. By examining samples of both non-empty and empty descriptions, we identify potential issues and determine the extent of missing data, informing the necessary cleaning and preprocessing steps for accurate analysis.\n",
        "\n",
        "**Below are the steps:**\n",
        "\n",
        "1. Data Type, Missing Values, and Samples:\n",
        " -  What We Did: Checked the data type, counted missing values, reviewed a sample of 10 non-empty rows for potential issues, and analyzed 10 empty rows with their titles.\n",
        " -  Why: To understand data quality and identify necessary cleaning or adjustments.\n",
        "\n",
        "2. Length Analysis of Descriptions:\n",
        "  - What We Did: Measured min, max, and average lengths, visualized the distribution, and examined outliers.\n",
        "\n",
        "  - Why: To gauge complexity and identify trends for better vectorization choices.\n",
        "\n",
        "3. Common Words Analysis:\n",
        "  - What We Did: Created word clouds to identify frequently used words.\n",
        "\n",
        "  - Why: To spot generic terms that may need exclusion in clustering or modeling.\n",
        "\n",
        "4. Punctuation Analysis:\n",
        "  - What We Did: Analyzed punctuation usage statistically.\n",
        "\n",
        "  - Why: To decide on the necessity of punctuation removal for effective text processing.\n",
        "\n",
        "\n",
        "**5.Correlation Checking:**\n",
        "\n",
        "**What We Did:**\n",
        "We checked for correlations between the level of downloads and the length of descriptions.\n",
        "\n",
        "**Why:**\n",
        "Although this analysis doesn't directly relate to cleaning, it provides early insights into potential relationships between variables, which will be valuable for feature engineering and constructing predictive variables in later stages (Q2 and Q3).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ARYbmk-IfhDo",
      "metadata": {
        "id": "ARYbmk-IfhDo"
      },
      "source": [
        "**Data Type, Missing Values, and samples:**\n",
        "\n",
        "- What We Did: Checked the data type, counted missing values, reviewed a sample of 10 non-empty rows for potential issues, and analyzed 10 empty rows with their titles.\n",
        "- Why: To understand data quality and identify necessary cleaning or adjustments."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_M9fQxPRqFz_",
      "metadata": {
        "id": "_M9fQxPRqFz_"
      },
      "source": [
        "##### Data type and missing values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1sQdMQ-M6yQb",
      "metadata": {
        "id": "1sQdMQ-M6yQb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fca6f4ee-3a19-4dd7-bdc5-1af5298a6603"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Data type: <class 'str'>, Count: 5636\n",
            "Sample values: ['Accident Compensation Seminar 2011. This covers various aspects of the TAC Claims Management Transformation model and the segmentation algorithm.', 'Insights 2015: The appointed actuary role.', \"What does unconscious bias look like in your colleagues, your workplace and up the corporate ladder? Kate O'Reilly from Optimiss Consulting unpacks the conditions that support biased decisions and offers strategies to promote better thinking and practice.\"]\n",
            "\n",
            "Data type: <class 'float'>, Count: 433\n",
            "Sample values: [nan, nan, nan]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "eda_column_dtypes(df, 'Description')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Negxu5plsdWc",
      "metadata": {
        "id": "Negxu5plsdWc"
      },
      "source": [
        "**Findings:**\n",
        "The 'Description' column primarily contains strings (str) with 5,636 entries.\n",
        "There are 433 missing entries (NaN values), indicating that approximately 7.1% of the descriptions are missing."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZUIwXkHxqjEE",
      "metadata": {
        "id": "ZUIwXkHxqjEE"
      },
      "source": [
        "10 Sample of empty descirption, and 10 sample of non empty descpriton."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WzEY3d0m1Tn4",
      "metadata": {
        "id": "WzEY3d0m1Tn4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1d235463-f505-4c0a-8d2c-112cc3976e18"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample of 10 Descriptions:\n",
            "2780                                                  NaN\n",
            "2738                       General Insurance Seminar 2018\n",
            "5410    Demographic changes demand financial literacy ...\n",
            "4460    Financial Services Forum 2012. Concurrent 6F. ...\n",
            "3149    ASTIN, AFIR/ERM, IACA Colloquia 2015. Paper Sy...\n",
            "4226    Actuaries Summit 2013. Concurrent 5C. Emerging...\n",
            "5598    Actuary Australia Magazine December 2010 (Issu...\n",
            "2361    Pension funds, investment managers and now glo...\n",
            "2774                                                  NaN\n",
            "263                              Catastrophe Risk Seminar\n",
            "Name: Description, dtype: object\n"
          ]
        }
      ],
      "source": [
        "sample_descriptions = docs_train_cleaned['Description'].sample(10, random_state=42)\n",
        "print(\"Sample of 10 Descriptions:\")\n",
        "print(sample_descriptions)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ZPbQjIEesQy-",
      "metadata": {
        "id": "ZPbQjIEesQy-"
      },
      "source": [
        "We take 10 sample of assets that have missing description with their title"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "xYey-Q6YkLYf",
      "metadata": {
        "id": "xYey-Q6YkLYf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa50b653-217a-438c-eb35-b557a0478da1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample of 10 rows with empty descriptions:\n",
            "        DMSId                                              Title Description\n",
            "5894  4364857               Actuaries Summit 2020 - Happy Ageing         NaN\n",
            "5889  4364847  Actuaries Summit 2020 - PLENARY: COVID-19 - Un...         NaN\n",
            "3523  4359914                             INS15 - - Presentation         NaN\n",
            "5794  4364518     Retail Advised Lump Sum Product Sustainability         NaN\n",
            "5400  4363448                                  IDSS 2019 Edition         NaN\n",
            "5823  4364719  Spending in retirement and the taper rate tran...         NaN\n",
            "5865  4364823  Actuaries Summit 2020 - The Hidden Risks in Ov...         NaN\n",
            "5323  4363080  Psychological distress in Australian workers' ...         NaN\n",
            "5768  4364453    Pandemic Briefing - Stress and scenario testing         NaN\n",
            "55    4196538                                2014 Year in Review         NaN\n",
            "----------\n"
          ]
        }
      ],
      "source": [
        "# Sample dataframe (replace with your actual dataframe)\n",
        "df = pd.DataFrame(docs_train)\n",
        "\n",
        "# Step 1: Identify rows with empty or missing descriptions\n",
        "empty_description_rows = df[(df['Description'].isna()) | (df['Description'].str.strip() == '')]\n",
        "\n",
        "# Step 2: Display 10 sample rows with empty descriptions\n",
        "sample_empty_description_rows = empty_description_rows.sample(10, random_state=1) if len(empty_description_rows) > 10 else empty_description_rows\n",
        "print(\"Sample of 10 rows with empty descriptions:\")\n",
        "print(sample_empty_description_rows[['DMSId', 'Title', 'Description']])\n",
        "print(\"----------\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QosbS5flsj4W",
      "metadata": {
        "id": "QosbS5flsj4W"
      },
      "source": [
        "**Finding on Sample Review of Descriptions:**\n",
        "\n",
        "- Non-empty descriptions vary in detail, ranging from brief titles like \"General Insurance Seminar 2018\" to more informative content, such as \"Demographic changes demand financial literacy.\"\n",
        "\n",
        "- Empty descriptions often correspond to titles indicating presentations, seminars, or general topics, suggesting that these assets may not have detailed descriptions provided."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CQ3_0jIqsvfW",
      "metadata": {
        "id": "CQ3_0jIqsvfW"
      },
      "source": [
        "#####**Lenght of descroption**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YXw7BLqepSRy",
      "metadata": {
        "id": "YXw7BLqepSRy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 877
        },
        "outputId": "e216184c-3444-4787-9728-4c62c15b7511"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Description Length Statistics:\n",
            "count    5635.000000\n",
            "mean      205.869920\n",
            "std       251.508703\n",
            "min        12.000000\n",
            "25%        72.000000\n",
            "50%       113.000000\n",
            "75%       243.000000\n",
            "max      2498.000000\n",
            "Name: Description_Length, dtype: float64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 40676 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Potential Outliers (very long/short descriptions):\n",
            "        DMSId                                              Title  \\\n",
            "4638  4361637  A Data Analytics Paradigm for the Construction...   \n",
            "2393  4227128                   I am an Actuary - September 2014   \n",
            "\n",
            "                                            Description  Description_Length  \n",
            "4638  Actuaries Summit 2019 Humanity has made, and c...              2498.0  \n",
            "2393                                       I am Actuary                12.0  \n"
          ]
        }
      ],
      "source": [
        "# prompt: do me code to see the length of descpriton\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Calculate the length of each description\n",
        "docs_train_cleaned['Description_Length'] = docs_train_cleaned['Description'].str.len()\n",
        "\n",
        "# Display basic statistics (min, max, mean, etc.)\n",
        "print(\"Description Length Statistics:\")\n",
        "print(docs_train_cleaned['Description_Length'].describe())\n",
        "\n",
        "# Visualize the distribution of description lengths\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.histplot(docs_train_cleaned['Description_Length'], kde=True)\n",
        "plt.title('Distribution of Description Lengths')\n",
        "plt.xlabel('Description Length')\n",
        "plt.ylabel('Frequency')\n",
        "plt.show()\n",
        "\n",
        "# Identify potential outliers (e.g., very long or very short descriptions)\n",
        "print(\"Potential Outliers (very long/short descriptions):\")\n",
        "print(docs_train_cleaned.loc[[\n",
        "    docs_train_cleaned['Description_Length'].idxmax(),\n",
        "    docs_train_cleaned['Description_Length'].idxmin()\n",
        "], ['DMSId', 'Title', 'Description', 'Description_Length']])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HfN00aE4s_u1",
      "metadata": {
        "id": "HfN00aE4s_u1"
      },
      "source": [
        "**Findings on Description Length Analysis:**\n",
        "\n",
        "- The average description length is approximately 206 characters, with a wide variation (standard deviation of about 251 characters).\n",
        "- The shortest description is 12 characters, while the longest reaches 2,498 characters.\n",
        "- Most descriptions fall within a range of 72 to 243 characters (25th to 75th percentile).\n",
        "\n",
        "Potential Outliers:\n",
        "- One description is notably long (2,498 characters), while another is very short (12 characters).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "w8UxNKSOuCdQ",
      "metadata": {
        "id": "w8UxNKSOuCdQ"
      },
      "source": [
        "**lenght of descrption with year of publication**\n",
        "\n",
        "This analysis helps identify trends in how descriptive content has evolved, suggesting that more recent assets provide more detailed descriptions.\n",
        "\n",
        "Rationale: While this analysis does not directly help with next cleaning step, understanding this trend is important for vectorization and later modelling, as longer descriptions might carry more context or relevant information, impacting how they are processed and represented in the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "KG8QDkO2tLXR",
      "metadata": {
        "id": "KG8QDkO2tLXR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6546d4bb-44a4-49ce-e63b-3df61d5bc18c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-120-cb1587eaadde>:15: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.boxplot(x='Year', y='Description_Length', data=docs_train_cleaned, palette='viridis')\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/_base.py:949: FutureWarning: When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas. Pass `(name,)` instead of `name` to silence this warning.\n",
            "  data_subset = grouped_data.get_group(pd_key)\n",
            "/usr/local/lib/python3.10/dist-packages/seaborn/categorical.py:640: FutureWarning: SeriesGroupBy.grouper is deprecated and will be removed in a future version of pandas.\n",
            "  positions = grouped.grouper.result_index.to_numpy(dtype=float)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1400x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 78628 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Ensure 'Description_Length' is calculated\n",
        "docs_train_cleaned['Description_Length'] = docs_train_cleaned['Description'].apply(lambda x: len(x.split()) if isinstance(x, str) else 0)\n",
        "\n",
        "# Extract the year from the StartPublishing date if not already done\n",
        "docs_train_cleaned['Year'] = pd.to_datetime(docs_train_cleaned['StartPublishing']).dt.year\n",
        "\n",
        "# Count the number of assets per year\n",
        "year_counts = docs_train_cleaned['Year'].value_counts().sort_index()\n",
        "\n",
        "# Create custom labels for the x-axis, including the number of assets\n",
        "x_labels = [f\"{year}\\n(N={count})\" for year, count in year_counts.items()]\n",
        "\n",
        "# Plot a box plot for the word counts split by year of publication\n",
        "plt.figure(figsize=(14, 8))\n",
        "sns.boxplot(x='Year', y='Description_Length', data=docs_train_cleaned, palette='viridis')\n",
        "\n",
        "# Set custom labels on the x-axis\n",
        "plt.xticks(ticks=range(len(x_labels)), labels=x_labels, rotation=45)\n",
        "plt.xlabel('Year of Publication')\n",
        "plt.ylabel('Number of Words in Description')\n",
        "plt.title('Box Plot of Word Counts in Descriptions by Year of Publication')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "x925Hk8-wfzq",
      "metadata": {
        "id": "x925Hk8-wfzq"
      },
      "source": [
        "**findings:**\n",
        "The scatter plot reveals that description lengths have increased over time, with assets published in more recent years showing significantly longer descriptions. This trend is evident as descriptions from around 2018 onwards display greater variability and longer lengths compared to earlier years.\n",
        "\n",
        "Implication\n",
        "There seems to be an increasing trend in the median length of descriptions in more recent years . This could suggest that more recent assets tend to have longer descriptions.\n",
        "\n",
        "Summary and Explanation: **Suggesting \"TF_IDF\"**\n",
        "\n",
        "The increasing length of descriptions suggests that the text data could contain a lot of common terms, potentially diluting the importance of unique or informative words. TF-IDF (Term Frequency-Inverse Document Frequency) is well-suited for this scenario as it down-weights common words and highlights rarer, more distinctive terms. This approach ensures that the most relevant features are emphasized, making it an ideal method for vectorizing the descriptions, especially given their growing complexity and verbosity."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_XhWdfaYuSLE",
      "metadata": {
        "id": "_XhWdfaYuSLE"
      },
      "source": [
        "##### **Common words**\n",
        "We will then examine the the EDA on common word on descroption\n",
        "the following codes is generated using Germini on Colab AI with the insturuciton of \"do me an EDA on the common word from Descirption\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YOARV_O0s5lV",
      "metadata": {
        "id": "YOARV_O0s5lV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 499
        },
        "outputId": "9488cbd4-52de-4d28-cb13-4c5e87073220"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Most common words:\n",
            "[('insurance', 1920), ('actuaries', 1756), ('seminar', 1392), ('concurrent', 1392), ('risk', 1295), ('financial', 925), ('general', 873), ('paper', 709), ('management', 659), ('2015', 655), ('actuarial', 640), ('services', 635), ('report', 572), ('life', 567), ('institute', 563), ('convention', 559), ('summit', 558), ('actuary', 554), ('health', 539), ('plenary', 537)]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x500 with 1 Axes>"
            ],
            "image/png": "[encoded data, 389460 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me an EDA on the common word from descroption\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Create a list of all descriptions (excluding NaN values)\n",
        "all_descriptions = docs_train_cleaned['Description'].dropna().tolist()\n",
        "\n",
        "# Combine all descriptions into a single string\n",
        "text = ' '.join(all_descriptions)\n",
        "\n",
        "# Remove punctuation\n",
        "text = text.translate(str.maketrans('', '', string.punctuation))\n",
        "\n",
        "# Convert to lowercase\n",
        "text = text.lower()\n",
        "\n",
        "# Tokenize the text\n",
        "words = text.split()\n",
        "\n",
        "# Remove stop words\n",
        "words = [word for word in words if word not in stop_words]\n",
        "\n",
        "# Count word frequencies\n",
        "word_counts = Counter(words)\n",
        "\n",
        "# Print the most common words\n",
        "print(\"Most common words:\")\n",
        "print(word_counts.most_common(20))\n",
        "\n",
        "\n",
        "# Generate a word cloud\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate_from_frequencies(word_counts)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 5))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud of Descriptions')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BD7OLqVTxPMh",
      "metadata": {
        "id": "BD7OLqVTxPMh"
      },
      "source": [
        "**Findings**\n",
        "\n",
        "- The most frequently occurring words in the descriptions are 'insurance,' 'actuaries,' 'seminar,' 'concurrent,' and 'risk.'\n",
        "\n",
        "- Many of these terms, such as 'actuaries,' 'seminar,' 'paper,' 'management,' and 'convention,' are quite generic or industry-specific but do not necessarily differentiate between different topics or content types.\n",
        "\n",
        "**Implications:**\n",
        "- The word cloud confirms that several high-frequency words, such as \"actuaries,\" \"insurance,\" \"seminar,\" and \"concurrent,\" dominate the descriptions. These terms are common across many documents and likely offer little distinguishing power for clustering or modeling.\n",
        "\n",
        "- To improve vectorization, we'll manually filter out high-frequency words using actuarial knowledge, keeping only unique and informative terms. This ensures the model can better differentiate between descriptions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VpQO9B2ys2Vp",
      "metadata": {
        "id": "VpQO9B2ys2Vp"
      },
      "source": [
        "#####**Punctuation Usage Analysis:**\n",
        "Punctuation analysis helps identify how frequently different punctuation marks appear in descriptions, guiding text cleaning decisions. By understanding punctuation usage, we can determine which marks may add noise or inconsistencies to the data.\n",
        "\n",
        "We generate the following codes using Gemini"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zrZS4s28yPKn",
      "metadata": {
        "id": "zrZS4s28yPKn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9c21ce80-8621-48e3-bdd7-f287b21093b8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Punctuation Usage Analysis:\n",
            "!: 68.0\n",
            "\": 377.0\n",
            "#: 2.0\n",
            "$: 18.0\n",
            "%: 50.0\n",
            "&: 182.0\n",
            "': 1432.0\n",
            "(: 1085.0\n",
            "): 1090.0\n",
            "*: 4.0\n",
            "+: 4.0\n",
            ",: 5707.0\n",
            "-: 3974.0\n",
            ".: 14086.0\n",
            "/: 715.0\n",
            ":: 1155.0\n",
            ";: 125.0\n",
            "<: 0.0\n",
            "=: 2.0\n",
            ">: 1.0\n",
            "?: 549.0\n",
            "@: 3.0\n",
            "[: 108.0\n",
            "\\: 7.0\n",
            "]: 111.0\n",
            "^: 1.0\n",
            "_: 0.0\n",
            "`: 0.0\n",
            "{: 0.0\n",
            "|: 0.0\n",
            "}: 0.0\n",
            "~: 6.0\n"
          ]
        }
      ],
      "source": [
        "import string\n",
        "import re\n",
        "\n",
        "# Function to analyze punctuation usage in a series of text\n",
        "def punctuation_analysis(text_series):\n",
        "  \"\"\"Analyzes punctuation usage in a series of text.\n",
        "\n",
        "  Args:\n",
        "      text_series: A pandas Series containing text data.\n",
        "\n",
        "  Returns:\n",
        "      A dictionary containing punctuation counts.\n",
        "  \"\"\"\n",
        "  punctuation_counts = {}\n",
        "  for punctuation_mark in string.punctuation:\n",
        "    # Escape special characters in punctuation marks for regex\n",
        "    punctuation_mark_escaped = re.escape(punctuation_mark)\n",
        "    punctuation_counts[punctuation_mark] = text_series.str.count(punctuation_mark_escaped).sum()\n",
        "  return punctuation_counts\n",
        "\n",
        "# Apply the function to the 'Description' column\n",
        "punctuation_counts = punctuation_analysis(docs_train_cleaned['Description'])\n",
        "\n",
        "# Print the results\n",
        "print(\"Punctuation Usage Analysis:\")\n",
        "for punctuation, count in punctuation_counts.items():\n",
        "  print(f\"{punctuation}: {count}\")\n",
        "\n",
        "# You can further visualize the results using a bar chart or other suitable plot\n",
        "# to compare the frequency of different punctuation marks."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ZRPba05yiTh",
      "metadata": {
        "id": "1ZRPba05yiTh"
      },
      "source": [
        "**Findings**\n",
        "Punctuation Usage Analysis:\n",
        "\n",
        "The most frequently used punctuation marks are periods (.) with 14,086 occurrences, followed by commas (,) with 5,707 occurrences, and hyphens (-) with 3,974 occurrences.\n",
        "Less common punctuation includes symbols like *, +, <, =, >, ^, {, |, }, ~, which appear very infrequently or not at all.\n",
        "Implications:\n",
        "\n",
        "The high frequency of periods, commas, and hyphens suggests these marks are heavily embedded in the descriptions and might need handling during cleaning to avoid skewing the vectorization process.\n",
        "Rare punctuation marks likely have minimal impact but can be removed to ensure cleaner, more consistent text data for modeling. This step helps maintain focus on the more informative content in descriptions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8znnU9un1BZr",
      "metadata": {
        "id": "8znnU9un1BZr"
      },
      "source": [
        "##### Relation between download volume and description length\n",
        "\n",
        "The purpose of this analysis is to understand whether the length of an asset's description influences its popularity, as measured by the number of downloads. By examining this relationship, we can identify patterns or trends that may inform how descriptive content impacts user engagement. This understanding helps determine whether description length should be considered as a feature in predictive modeling, ensuring that our approach aligns with factors that drive asset popularity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "FIHLUoIf1EST",
      "metadata": {
        "id": "FIHLUoIf1EST",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "outputId": "cb940ed6-03c2-4cb3-e3be-262dc1f3ef15"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 60184 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Ensure that the 'Number of Downloads' is available in the dataset\n",
        "if 'Number of Downloads' not in docs_train_cleaned.columns:\n",
        "    downloads_count = downloads_train.groupby('DMSId').size().reset_index(name='Number of Downloads')\n",
        "    docs_train_cleaned = docs_train_cleaned.merge(downloads_count, on='DMSId', how='left')\n",
        "    docs_train_cleaned['Number of Downloads'] = docs_train_cleaned['Number of Downloads'].fillna(0)  # Fill NaN with 0 if no downloads\n",
        "\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.scatterplot(x='Description_Length', y='Number of Downloads', data=docs_train_cleaned, alpha=0.6, color='blue')\n",
        "plt.xlabel('Number of Words in Description')\n",
        "plt.ylabel('Number of Downloads')\n",
        "plt.title('Relationship Between Description Length and Number of Downloads')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OIm4MQC71NqE",
      "metadata": {
        "id": "OIm4MQC71NqE"
      },
      "source": [
        "**Findings:**\n",
        "\n",
        "- The scatter plot shows that assets with shorter descriptions (fewer than 50 words) tend to have a higher number of downloads, with some reaching over 1,200 downloads.\n",
        "- As the description length increases, the number of downloads generally decreases, indicating that longer descriptions may not necessarily contribute to an asset's popularity.\n",
        "\n",
        "**Implications:**\n",
        "\n",
        "- Shorter, concise descriptions seem more engaging or appealing to users, suggesting that the amount of detail provided does not always correlate with higher popularity.\n",
        "\n",
        "- This insight may guide future modeling by considering description length as a potential feature influencing download patterns. Additionally, it reinforces the importance of balancing detail with conciseness in creating descriptions for assets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3U2zBfJtRZKg",
      "metadata": {
        "id": "3U2zBfJtRZKg"
      },
      "source": [
        "#### Q1b Step 2 Data Cleaning\n",
        "\n",
        "Purpose:\n",
        "We will now conduct cleaning steps on the descriptions to prepare them for TF-IDF processing.\n",
        "\n",
        "Instead of using a standard tutorial formula, we will carry out the cleaning step by step for the following reasons:\n",
        "- It allows greater flexibility.\n",
        "- It's easier to explain the rationale behind each step.\n",
        "\n",
        "\n",
        "Reference:\n",
        "The code below was generated using ChatGPT."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "CJ2IrW7Xm-CN",
      "metadata": {
        "id": "CJ2IrW7Xm-CN"
      },
      "source": [
        "Here is the Outline of what we will do and the rationale of each cleaning\n",
        "\n",
        "**Cleaning Steps:**\n",
        "1. Handle Missing Descriptions:\n",
        "\n",
        "**What We Did:** Replaced missing or empty descriptions with a placeholder (\"No Description Provided\").\n",
        "\n",
        "**Why:** This step ensures consistency in the dataset, preventing issues during further text processing and vectorization. Missing values can disrupt the analysis and lead to errors, so filling them is essential.\n",
        "\n",
        "2.Fixing Quote Spillage:\n",
        "\n",
        "**What We Did:** Removed or closed unmatched quotes in the descriptions.\n",
        "\n",
        "**Why:** Inconsistent quotation marks can interfere with tokenization and phrase extraction, leading to inaccurate vectorization and modeling. Cleaning these ensures proper segmentation and tokenization of the text.\n",
        "\n",
        "3.Lowercasing:\n",
        "\n",
        "**What We Did:** Converted all text in the descriptions to lowercase.\n",
        "\n",
        "**Why:** This step standardizes the text, ensuring that words are treated consistently, regardless of capitalization. It reduces redundancy in the token set and improves the accuracy of vectorization.\n",
        "\n",
        "4.Substituting and Removing Corrupted Characters:\n",
        "\n",
        "**What We Did:** Replaced or removed non-standard or corrupted characters from the descriptions.\n",
        "\n",
        "**Why:** These characters can disrupt tokenization and lead to incorrect vectorization. Cleaning them ensures that the text is ready for accurate processing.\n",
        "\n",
        "5.Expanding Contractions:\n",
        "**What We Did:** Expanded contractions (e.g., \"can't\" to \"cannot\") to their full form.\n",
        "\n",
        "**Why:** This ensures consistency in word representation, improving token matching and the accuracy of the vectorization process.\n",
        "\n",
        "6.Removing Punctuation:\n",
        "**What We Did:** Removed punctuation marks from the descriptions.\n",
        "\n",
        "**Why:** Punctuation can create unnecessary tokens that do not contribute to the meaning of the text. Removing them simplifies tokenization and reduces the dimensionality of the feature space.\n",
        "\n",
        "7.Remove Short or Irrelevant Terms:\n",
        "**What We Did:** Removed words that are shorter than three characters or irrelevant to the analysis.\n",
        "\n",
        "**Why:** Short or irrelevant terms often do not add meaningful information and can clutter the vector space. Removing them enhances the quality of the text data for vectorization.\n",
        "\n",
        "8.Tokenization:\n",
        "**What We Did:** Split the text into individual words or tokens.\n",
        "\n",
        "**Why:** Tokenization is a fundamental step in preparing text for vectorization techniques like TF-IDF. It breaks down the text into manageable units, allowing for precise and effective processing.\n",
        "\n",
        "9.Removing Stop Words:\n",
        "**What We Did:** Removed common stop words (e.g., \"the,\" \"and\") from the token lists.\n",
        "\n",
        "**Why:** Stop words are often not meaningful in the context of analysis and can clutter the feature space. Removing them focuses the vectorization on more informative words.\n",
        "\n",
        "10.Lemmatization:\n",
        "\n",
        "**What We Did:** Reduced words to their base or root form (e.g., \"running\" to \"run\").\n",
        "\n",
        "**Why:** Lemmatization helps group similar words together, reducing redundancy in the text data and improving the accuracy of vectorization.\n",
        "\n",
        "11.Reconstruct the Cleaned Description:\n",
        "\n",
        "**What We Did:** Rejoined the cleaned and tokenized text into a single string format.\n",
        "\n",
        "**Why:** TF-IDF and other vectorization methods typically require entire text strings rather than lists of tokens. Reconstructing the cleaned text ensures compatibility with these techniques."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xpHjRazQ4t3t",
      "metadata": {
        "id": "xpHjRazQ4t3t"
      },
      "source": [
        "#####Data Set up:\n",
        "\n",
        "**Rationale for Applying Text Cleaning to Validation and Test Sets**\n",
        "\n",
        "While the goal is to maintain minimal data cleaning on the validation and test sets to avoid introducing biases or overfitting, some level of cleaning is still essential, especially for text data. Given that description vectorization requires standardized text input, we need to apply necessary text cleaning steps to ensure consistency across the training, validation, and test sets.\n",
        "\n",
        "Applying consistent text cleaning (e.g., lowercasing, removing punctuation, handling missing values) ensures that the vectorization process treats all datasets similarly, preventing discrepancies that could impact model performance. This balanced approach maintains the integrity of the validation and test sets while still enabling accurate and effective vectorization, ensuring that the model is evaluated under realistic yet standardized conditions.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Esr0uqW_23n7",
      "metadata": {
        "id": "Esr0uqW_23n7"
      },
      "outputs": [],
      "source": [
        "# Data set preparation\n",
        "docs_train_cleaned_desc = docs_train_cleaned.copy()\n",
        "\n",
        "docs_val_cleaned_desc = docs_val.copy()\n",
        "docs_test_cleaned_desc = docs_test.copy()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gMbJLods2ENW",
      "metadata": {
        "id": "gMbJLods2ENW"
      },
      "source": [
        "#####Handle Missing Descriptions\n",
        "What We Did: Replaced missing or empty descriptions with the corresponding asset title instead of a generic placeholder.\n",
        "\n",
        "Why (Based on EDA Findings): EDA identified 433 missing descriptions. Using the title as a replacement provides relevant context for these assets, ensuring that the description retains valuable information. This approach preserves the integrity of the data and prevents gaps during vectorization while still acknowledging missing descriptions for future analysis.\n",
        "\n",
        "Expected Result: All rows will have meaningful text in the 'Description' column, enabling smooth processing and modeling, while allowing us to retain and utilize the insight that these entries originally lacked descriptions.\n",
        "\n",
        "Reference: the following code is generated by Gemini"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "EZ9Y7bnk2ke4",
      "metadata": {
        "id": "EZ9Y7bnk2ke4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9dc38f85-e071-431d-dc97-e56aae3540ec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Missing descriptions in training set: 0\n",
            "Missing descriptions in validation set: 0\n",
            "Missing descriptions in test set: 0\n"
          ]
        }
      ],
      "source": [
        "# prompt: we want to do data cleaning on description for train, validation and test , if descrption is missing, get it from title, if still missing, give \"no description provided\"  also do me check to ensure they are working.\n",
        "\n",
        "def clean_description(df):\n",
        "  df['Description'] = df['Description'].fillna(df['Title'])\n",
        "  df['Description'] = df['Description'].fillna('No Description Provided')\n",
        "  return df\n",
        "\n",
        "docs_train_cleaned_desc = clean_description(docs_train_cleaned_desc)\n",
        "docs_val_cleaned_desc = clean_description(docs_val_cleaned_desc)\n",
        "docs_test_cleaned_desc = clean_description(docs_test_cleaned_desc)\n",
        "\n",
        "\n",
        "# Check if there are any missing descriptions after cleaning\n",
        "print(\"Missing descriptions in training set:\", docs_train_cleaned_desc['Description'].isnull().sum())\n",
        "print(\"Missing descriptions in validation set:\", docs_val_cleaned_desc['Description'].isnull().sum())\n",
        "print(\"Missing descriptions in test set:\", docs_test_cleaned_desc['Description'].isnull().sum())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ePa4BTQpTcCn",
      "metadata": {
        "id": "ePa4BTQpTcCn"
      },
      "source": [
        "#####Fixing Quote Spillage\n",
        "\n",
        "Reason: Unclosed or mismatched quotes disrupt text segmentation and lead to incorrect tokenization.\n",
        "\n",
        "EDA Observation: Descriptions contain unbalanced quotes, causing fragmented sentences.\n",
        "\n",
        "Relation to Vectorization: Fixing quotes ensures accurate tokenization, allowing vectorization to capture the true meaning of the text.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4Fvug9kZn8dO",
      "metadata": {
        "id": "4Fvug9kZn8dO"
      },
      "outputs": [],
      "source": [
        "# Fix quote spillage by removing or closing unmatched quotes\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].apply(\n",
        "    lambda x: x.replace('\"', '').replace(\"'\", \"\")\n",
        ")\n",
        "\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].apply(\n",
        "    lambda x: x.replace('\"', '').replace(\"'\", \"\")\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].apply(\n",
        "    lambda x: x.replace('\"', '').replace(\"'\", \"\")\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rwxarbeI3L8g",
      "metadata": {
        "id": "rwxarbeI3L8g"
      },
      "source": [
        "Check: Verify No Unmatched Quotes\n",
        "\n",
        "Explanation: This check looks for any remaining unmatched or single quotes that could disrupt tokenization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Z9r8eifV3HL6",
      "metadata": {
        "id": "Z9r8eifV3HL6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a2457aba-97f0-4dca-b72b-a03e169b6651"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "before -- Unmatched Quotes Found: 1026\n",
            "after --  Unmatched Quotes Found: 0\n"
          ]
        }
      ],
      "source": [
        "unmatched_quotes = docs_train_cleaned['Description'].str.contains(r'[\"\\']').sum()\n",
        "print(f\"before -- Unmatched Quotes Found: {unmatched_quotes}\")\n",
        "\n",
        "unmatched_quotes = docs_train_cleaned_desc['Description'].str.contains(r'[\"\\']').sum()\n",
        "print(f\"after --  Unmatched Quotes Found: {unmatched_quotes}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FvG68_ZpTi4S",
      "metadata": {
        "id": "FvG68_ZpTi4S"
      },
      "source": [
        "#####Lowercasing\n",
        "\n",
        "Reason for Cleaning: Converting text to lowercase ensures that words are treated consistently, regardless of their capitalization.\n",
        "\n",
        "EDA Observation: Variation in capitalization can result in the same word being treated as different tokens (e.g., \"Actuarial\" vs. \"actuarial\").\n",
        "\n",
        "Relation to Vectorization: Improves the accuracy of vectorization by reducing redundancy in the token set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wr6Qob3dTl6d",
      "metadata": {
        "id": "wr6Qob3dTl6d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "054da970-0c79-4a55-ba73-4e91ff6f8187"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before cleaning --All Descriptions Lowercase: False\n",
            "Before cleaning --All Descriptions Lowercase: False\n",
            "Before cleaning --All Descriptions Lowercase: False\n"
          ]
        }
      ],
      "source": [
        "train_lowercase_check = docs_train_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"Before cleaning --All Descriptions Lowercase: {train_lowercase_check == len(docs_train_cleaned_desc)}\")\n",
        "\n",
        "# Convert descriptions to lowercase\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].str.lower()\n",
        "\n",
        "\n",
        "val_lowercase_check = docs_val_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"Before cleaning --All Descriptions Lowercase: {val_lowercase_check == len(docs_val_cleaned_desc)}\")\n",
        "\n",
        "# Convert descriptions to lowercase\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].str.lower()\n",
        "\n",
        "\n",
        "test_lowercase_check = docs_test_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"Before cleaning --All Descriptions Lowercase: {test_lowercase_check == len(docs_test_cleaned_desc)}\")\n",
        "\n",
        "# Convert descritpions to lowercase\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].str.lower()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6gZr5KJL3eBc",
      "metadata": {
        "id": "6gZr5KJL3eBc"
      },
      "source": [
        "Check: Verify All Text is Lowercase\n",
        "\n",
        "Explanation: This check ensures that all text in the Description column is lowercase."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Sk0elE3q3htu",
      "metadata": {
        "id": "Sk0elE3q3htu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f81e28fe-e829-4491-f822-b9db7bec6b6d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "All Descriptions Lowercase: True\n",
            "Before cleaning --All Descriptions Lowercase: True\n",
            "Before cleaning --All Descriptions Lowercase: True\n"
          ]
        }
      ],
      "source": [
        "train_lowercase_check = docs_train_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"All Descriptions Lowercase: {train_lowercase_check == len(docs_train_cleaned_desc)}\")\n",
        "\n",
        "val_lowercase_check = docs_val_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"Before cleaning --All Descriptions Lowercase: {val_lowercase_check == len(docs_val_cleaned_desc)}\")\n",
        "\n",
        "test_lowercase_check = docs_test_cleaned_desc['Description'].apply(lambda x: x.islower()).sum()\n",
        "print(f\"Before cleaning --All Descriptions Lowercase: {test_lowercase_check == len(docs_test_cleaned_desc)}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "r3jAS7r-7_WH",
      "metadata": {
        "id": "r3jAS7r-7_WH"
      },
      "source": [
        "The checks suggest that all 3 dataset have been converted to lower case."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KKxP5gXwTroG",
      "metadata": {
        "id": "KKxP5gXwTroG"
      },
      "source": [
        "#####Substituting and Removing Corrupted Characters\n",
        "\n",
        "**Reason:** Remove or replace corrupted and non-standard characters to ensure smooth tokenization.\n",
        "\n",
        "**EDA Observation:** No obvious corrupted characters were observed during the EDA, but we will conduct a thorough check across training, validation, and test sets to confirm their presence. Any identified issues will be cleaned.\n",
        "\n",
        "**Expectation:** All corrupted characters will be removed, resulting in cleaner, standardized text.\n",
        "\n",
        "Relation to Vectorization: Ensures that tokenization is error-free, allowing vectorization to accurately capture meaningful information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WT0YxqK7TtlH",
      "metadata": {
        "id": "WT0YxqK7TtlH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "561f25fd-b88d-41fc-b9a3-7264abe615ef"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before cleaning -- Corrupted Characters Found: 5391\n",
            "Before cleaning -- Corrupted Characters Found: 688\n",
            "Before cleaning -- Corrupted Characters Found: 671\n"
          ]
        }
      ],
      "source": [
        "train_corrupted_characters_check = docs_train_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"Before cleaning -- Corrupted Characters Found: {train_corrupted_characters_check}\")\n",
        "\n",
        "val_corrupted_characters_check = docs_val_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"Before cleaning -- Corrupted Characters Found: {val_corrupted_characters_check}\")\n",
        "\n",
        "test_corrupted_characters_check = docs_test_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"Before cleaning -- Corrupted Characters Found: {test_corrupted_characters_check}\")\n",
        "\n",
        "\n",
        "\n",
        "# Replace corrupted characters\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].str.replace(\n",
        "    r'[^A-Za-z0-9\\s]', '', regex=True\n",
        ")\n",
        "\n",
        "\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].str.replace(\n",
        "    r'[^A-Za-z0-9\\s]', '', regex=True\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].str.replace(\n",
        "    r'[^A-Za-z0-9\\s]', '', regex=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eVKWXfjL3owl",
      "metadata": {
        "id": "eVKWXfjL3owl"
      },
      "source": [
        "\n",
        "Check: Verify No Corrupted Characters\n",
        "\n",
        "Explanation: This check scans for any remaining non-standard characters that could disrupt processing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "gswenS8F3qus",
      "metadata": {
        "id": "gswenS8F3qus",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30eb785c-dd8b-471d-89a1-bd29b27b284c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "after cleaning -- Corrupted Characters Found: 0\n",
            "after cleaning -- Corrupted Characters Found: 0\n",
            "after cleaning -- Corrupted Characters Found: 0\n"
          ]
        }
      ],
      "source": [
        "train_corrupted_characters_check = docs_train_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"after cleaning -- Corrupted Characters Found: {train_corrupted_characters_check}\")\n",
        "\n",
        "val_corrupted_characters_check = docs_val_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"after cleaning -- Corrupted Characters Found: {val_corrupted_characters_check}\")\n",
        "\n",
        "test_corrupted_characters_check = docs_test_cleaned_desc['Description'].str.contains(r'[^A-Za-z0-9\\s]').sum()\n",
        "print(f\"after cleaning -- Corrupted Characters Found: {test_corrupted_characters_check}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xfwp8IQp8bfO",
      "metadata": {
        "id": "xfwp8IQp8bfO"
      },
      "source": [
        "the zero corrupted charaters sugget we have sucessfully cleaned the corrupted character"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iXSNcZZOTzdf",
      "metadata": {
        "id": "iXSNcZZOTzdf"
      },
      "source": [
        "#####Expanding Contractions and Shortened Words**\n",
        "\n",
        "**Reason:** Expanding contractions (e.g., \"can't\" to \"cannot\") ensures consistent word representation, improving token matching.\n",
        "\n",
        "**EDA Observation:** Although no obvious contractions were found during EDA, we will still perform this step to maintain consistency.\n",
        "\n",
        "**Relation to Vectorization:** Enhances tokenization accuracy and frequency counts, leading to more reliable vectorization, especially in models like TF-IDF."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2eqkExW3T2-1",
      "metadata": {
        "id": "2eqkExW3T2-1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9bf55d7f-2323-43f4-c50a-1142e6a05262"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting contractions\n",
            "  Downloading contractions-0.1.73-py2.py3-none-any.whl.metadata (1.2 kB)\n",
            "Collecting textsearch>=0.0.21 (from contractions)\n",
            "  Downloading textsearch-0.0.24-py2.py3-none-any.whl.metadata (1.2 kB)\n",
            "Collecting anyascii (from textsearch>=0.0.21->contractions)\n",
            "  Downloading anyascii-0.3.2-py3-none-any.whl.metadata (1.5 kB)\n",
            "Collecting pyahocorasick (from textsearch>=0.0.21->contractions)\n",
            "  Downloading pyahocorasick-2.1.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (13 kB)\n",
            "Downloading contractions-0.1.73-py2.py3-none-any.whl (8.7 kB)\n",
            "Downloading textsearch-0.0.24-py2.py3-none-any.whl (7.6 kB)\n",
            "Downloading anyascii-0.3.2-py3-none-any.whl (289 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m289.9/289.9 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyahocorasick-2.1.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (110 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.7/110.7 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: pyahocorasick, anyascii, textsearch, contractions\n",
            "Successfully installed anyascii-0.3.2 contractions-0.1.73 pyahocorasick-2.1.0 textsearch-0.0.24\n"
          ]
        }
      ],
      "source": [
        "!pip install contractions\n",
        "import contractions\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].apply(\n",
        "    lambda x: contractions.fix(x)\n",
        ")\n",
        "\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].apply(\n",
        "    lambda x: contractions.fix(x)\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].apply(\n",
        "    lambda x: contractions.fix(x)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "t2Nj8AJz39Pc",
      "metadata": {
        "id": "t2Nj8AJz39Pc"
      },
      "source": [
        "Check: Verify Contractions Are Expanded\n",
        "\n",
        "Explanation: This check looks for any common contractions that should have been expanded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Jjv29jHB399H",
      "metadata": {
        "id": "Jjv29jHB399H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8047c2ab-4b51-4ab1-eb44-3c0e429e7d69"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Contractions Remaining: 0\n",
            "Contractions Remaining: 0\n",
            "Contractions Remaining: 0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-133-4145091c7c80>:1: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.\n",
            "  train_contractions_check = docs_train_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n",
            "<ipython-input-133-4145091c7c80>:4: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.\n",
            "  val_contractions_check = docs_val_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n",
            "<ipython-input-133-4145091c7c80>:7: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.\n",
            "  test_contractions_check = docs_test_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n"
          ]
        }
      ],
      "source": [
        "train_contractions_check = docs_train_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n",
        "print(f\"Contractions Remaining: {train_contractions_check}\")\n",
        "\n",
        "val_contractions_check = docs_val_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n",
        "print(f\"Contractions Remaining: {val_contractions_check}\")\n",
        "\n",
        "test_contractions_check = docs_test_cleaned_desc['Description'].str.contains(r\"\\b(can't|won't|n't|it's)\\b\").sum()\n",
        "print(f\"Contractions Remaining: {test_contractions_check}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zdRGECsf9_Gh",
      "metadata": {
        "id": "zdRGECsf9_Gh"
      },
      "source": [
        "We are happy with the cehck showing zero contractions remaining suggest the cleaning step is doing as we expected."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lYE2wzerUCn0",
      "metadata": {
        "id": "lYE2wzerUCn0"
      },
      "source": [
        "#####Removing Punctuation\n",
        "\n",
        "Reason for Cleaning: Punctuation can create unnecessary tokens that may not contribute to the understanding of the text.\n",
        "\n",
        "EDA Observation: Descriptions contain various punctuation marks that might not add value to the analysis.\n",
        "\n",
        "Relation to Vectorization: Simplifies the tokenization process and reduces the dimensionality of the feature space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6BXI-cuIUE7x",
      "metadata": {
        "id": "6BXI-cuIUE7x",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8a2f88ca-0cfe-4a6d-cecd-3cdaff21ef83"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Before cleaning --Punctuation Remaining: 0\n",
            "Before cleaning --Punctuation Remaining: 0\n",
            "Before cleaning --Punctuation Remaining: 0\n"
          ]
        }
      ],
      "source": [
        "# Check of punctuation\n",
        "train_punctuation_check = docs_train_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Before cleaning --Punctuation Remaining: {train_punctuation_check}\")\n",
        "\n",
        "val_punctuation_check = docs_val_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Before cleaning --Punctuation Remaining: {val_punctuation_check}\")\n",
        "\n",
        "test_contractions_check = docs_test_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Before cleaning --Punctuation Remaining: {test_contractions_check}\")\n",
        "\n",
        "\n",
        "# Remove punctuation\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].str.replace(\n",
        "    r'[^\\w\\s]', '', regex=True\n",
        ")\n",
        "\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].str.replace(\n",
        "    r'[^\\w\\s]', '', regex=True\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].str.replace(\n",
        "    r'[^\\w\\s]', '', regex=True\n",
        ")\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "m9yVcvbI4MPJ",
      "metadata": {
        "id": "m9yVcvbI4MPJ"
      },
      "source": [
        "Check: Verify Punctuation Removal\n",
        "\n",
        "Explanation: This check ensures that all punctuation has been removed from the descriptions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2paQgTrY4NJc",
      "metadata": {
        "id": "2paQgTrY4NJc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "edc1b3fd-cc48-407b-ce7d-4531967283b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Punctuation Remaining: 0\n",
            "Punctuation Remaining: 0\n",
            "Punctuation Remaining: 0\n"
          ]
        }
      ],
      "source": [
        "train_punctuation_check = docs_train_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Punctuation Remaining: {train_punctuation_check}\")\n",
        "\n",
        "val_punctuation_check = docs_val_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Punctuation Remaining: {val_punctuation_check}\")\n",
        "\n",
        "test_contractions_check = docs_test_cleaned_desc['Description'].str.contains(r'[^\\w\\s]').sum()\n",
        "print(f\"Punctuation Remaining: {test_contractions_check}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eEis9S5BVRB3",
      "metadata": {
        "id": "eEis9S5BVRB3"
      },
      "source": [
        "#####Removing Number - non appliable in this case\n",
        "\n",
        "Reason: Instead of removing all numbers, we should selectively retain numbers that are part of key actuarial terms or phrases.Such as AASB17\n",
        "\n",
        "Relation to Vectorization: This approach ensures that important terms with numerical significance are preserved, maintaining their value for accurate vectorization and capturing essential actuarial concepts. This balance helps avoid losing critical information while still reducing irrelevant numerical clutter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "khXDTt5uUoD3",
      "metadata": {
        "id": "khXDTt5uUoD3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "170d0db3-9a5b-42aa-a8c9-ac3e0fdf99ac"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Numbers remaining in training set: 0\n",
            "Numbers remaining in validation set: 0\n",
            "Numbers remaining in test set: 0\n"
          ]
        }
      ],
      "source": [
        "# prompt: generate code to remove number in the descroption for trainin, val and test data set\n",
        "\n",
        "# Remove numbers from the 'Description' column in all datasets\n",
        "docs_train_cleaned_desc['Description'] = docs_train_cleaned_desc['Description'].str.replace('\\d+', '', regex=True)\n",
        "docs_val_cleaned_desc['Description'] = docs_val_cleaned_desc['Description'].str.replace('\\d+', '', regex=True)\n",
        "docs_test_cleaned_desc['Description'] = docs_test_cleaned_desc['Description'].str.replace('\\d+', '', regex=True)\n",
        "\n",
        "# Check if numbers are removed\n",
        "print(\"Numbers remaining in training set:\", docs_train_cleaned_desc['Description'].str.contains('\\d+').sum())\n",
        "print(\"Numbers remaining in validation set:\", docs_val_cleaned_desc['Description'].str.contains('\\d+').sum())\n",
        "print(\"Numbers remaining in test set:\", docs_test_cleaned_desc['Description'].str.contains('\\d+').sum())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xJmCHfYIwbXS",
      "metadata": {
        "id": "xJmCHfYIwbXS"
      },
      "source": [
        "#####Tokenization\n",
        "Split the text into individual words or tokens.\n",
        "\n",
        "Reason for Cleaning: Breaking down text into individual words or tokens is crucial for most text vectorization techniques.\n",
        "\n",
        "Relation to Vectorization: Fundamental step to prepare the text data for techniques like TF-IDF.\n",
        "\n",
        "\n",
        "We take Tokenization before stop word removal  because it allows for precise and effective removal of stop words, which is crucial for accurate vectorization.\n",
        "\n",
        "If stop word removal is done before tokenization, there might be a risk of partial or incorrect removal of stop words, which could affect the quality of the text preprocessing and subsequent analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uds94jtjwbXS",
      "metadata": {
        "id": "uds94jtjwbXS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6927c1ab-c660-4d75-c9f9-4de7e9e0dbf9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.8.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.9.11)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.66.5)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ],
      "source": [
        "!pip install nltk\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "docs_train_cleaned_desc['Tokens'] = docs_train_cleaned_desc['Description'].apply(word_tokenize)\n",
        "docs_val_cleaned_desc['Tokens'] = docs_val_cleaned_desc['Description'].apply(word_tokenize)\n",
        "docs_test_cleaned_desc['Tokens'] = docs_test_cleaned_desc['Description'].apply(word_tokenize)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6Xahf8HHwbXS",
      "metadata": {
        "id": "6Xahf8HHwbXS"
      },
      "source": [
        "Check: Verify Tokenization\n",
        "\n",
        "Explanation: This check ensures that the descriptions have been successfully tokenized into words.\n",
        "\n",
        "Purpose: Ensures that the text has been properly tokenized, which is necessary for effective vectorization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YdUSrS6-wbXS",
      "metadata": {
        "id": "YdUSrS6-wbXS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3481008-4333-49d2-99aa-aa2777f8cc1b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tokenization Successful: True\n",
            "Tokenization Successful: True\n",
            "Tokenization Successful: True\n"
          ]
        }
      ],
      "source": [
        "train_tokenization_check = docs_train_cleaned_desc['Tokens'].apply(lambda x: isinstance(x, list)).sum()\n",
        "print(f\"Tokenization Successful: {train_tokenization_check == len(docs_train_cleaned_desc)}\")\n",
        "\n",
        "val_tokenization_check = docs_val_cleaned_desc['Tokens'].apply(lambda x: isinstance(x, list)).sum()\n",
        "print(f\"Tokenization Successful: {val_tokenization_check == len(docs_val_cleaned_desc)}\")\n",
        "\n",
        "test_tokenization_check = docs_test_cleaned_desc['Tokens'].apply(lambda x: isinstance(x, list)).sum()\n",
        "print(f\"Tokenization Successful: {test_tokenization_check == len(docs_test_cleaned_desc)}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3zC5azhnObCh",
      "metadata": {
        "id": "3zC5azhnObCh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "281b6994-2df6-4804-868e-002ce631d1dd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Document 1 Tokens: ['there', 'are', 'over', 'committees', 'subcommittees', 'task', 'forces', 'and', 'working', 'groups', 'of', 'the', 'international', 'actuarial', 'association', 'these', 'groups', 'all', 'commission', 'create', 'and', 'share', 'content', 'that', 'may', 'be', 'relevant', 'to', 'actuaries', 'institute', 'members', 'links', 'to', 'some', 'of', 'the', 'content', 'discussed', 'at', 'the', 'recent', 'meetings', 'in', 'berlin', 'germany', 'are', 'provided', 'below']\n",
            "\n",
            "Document 2 Tokens: ['the', 'report', 'is', 'based', 'on', 'data', 'in', 'respect', 'of', 'individual', 'disability', 'income', 'business', 'for', 'the', 'period', 'and', 'largely', 'follows', 'the', 'format', 'of', 'the', 'previous', 'report']\n",
            "\n",
            "Document 3 Tokens: ['inadequacy', 'of', 'liasb', 'mortality', 'reserves', 'allen', 'truslove', 'investment', 'returns', 'inflation', 'and', 'assetallocation', 'models', 'm', 'sherris', 'profiting', 'from', 'asset', 'price', 'trends', 'p', 'whelan']\n",
            "\n",
            "Document 4 Tokens: ['analysis', 'of', 'surplus', 'mixing', 'realism', 'with', 'conservatism', 'jim', 'farmer', 'actuarial', 'standards', 'workshop', 'on', 'mos', 'and', 'disability', 'income', 'richard', 'land']\n",
            "\n",
            "Document 5 Tokens: ['the', 'report', 'into', 'morbidity', 'for', 'insured', 'and', 'other', 'lives', 'data', 'tables', 'a', 'to', 'p', 'december', 'revised', 'march']\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# Print the first 5 tokenized descriptions to inspect the output\n",
        "for i, tokens in enumerate(docs_train_cleaned_desc['Tokens'].head(5)):\n",
        "    print(f\"Document {i + 1} Tokens: {tokens}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ipHnlwEIPEGs",
      "metadata": {
        "id": "ipHnlwEIPEGs"
      },
      "source": [
        "he tokenization process appears to have split many key phrases into individual tokens, which can dilute their meaning and relevance. For instance, important phrases like \"General Insurance,\" \"Actuary Institute,\" and \"Injury Disability Scheme\" should remain intact to preserve their context. Additionally, a high presence of stop words (e.g., \"there,\" \"are,\" \"of,\" \"the,\" \"groups\") adds noise to the analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3lE23cntwbXS",
      "metadata": {
        "id": "3lE23cntwbXS"
      },
      "source": [
        "#####Removing Stop Words\n",
        "\n",
        "**Rationale:**\n",
        "Stop words (e.g., \"the,\" \"and\") are frequently occurring words that usually carry little to no meaningful information. Including them can clutter the text data, adding noise and making it harder to identify valuable patterns.\n",
        "\n",
        "**Custom Stop Words:**\n",
        "In addition to standard stop words, we included a custom list of domain-specific stop words (e.g., \"paper,\" \"seminar,\" \"actuarial\") that frequently appear in our data but do not help in distinguishing descriptions. These words were identified based on our actuarial knowledge to ensure that only the most relevant and distinguishing terms remain.\n",
        "\n",
        "**Relation to Vectorization:**\n",
        "Removing these stop words, both standard and custom, reduces the dimensionality of the feature space, allowing the vectorization process to focus on more meaningful words that contribute to differentiating assets. This leads to cleaner and more representative vectors, enhancing the efficiency and effectiveness of the vectorization.\n",
        "\n",
        "**Impact on Later Modeling:**\n",
        "By filtering out non-informative words, the resulting vectors capture more relevant information, improving the model's ability to learn patterns, identify clusters, and make accurate predictions. This refinement ultimately leads to better model performance and more insightful analysis."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Yx3r6juPwbXT",
      "metadata": {
        "id": "Yx3r6juPwbXT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31ec1eb6-efcd-4956-a240-317380b2c8fa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (3.8.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk) (8.1.7)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk) (1.4.2)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk) (2024.9.11)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from nltk) (4.66.5)\n",
            " before Descriptions with Stop Words Remaining: 5076\n",
            " before Descriptions with Stop Words Remaining: 705\n",
            " before Descriptions with Stop Words Remaining: 714\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ],
      "source": [
        "!pip install nltk\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords') # Download stopwords data\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.corpus import stopwords # Import the stopwords module\n",
        "\n",
        "#Tokenization Process:\n",
        "\n",
        "#The nltk library's word_tokenize function was used to split the text in the 'Description' column into individual words (tokens) for the training, validation, and test sets.\n",
        "# The process checks for successful tokenization by ensuring that each entry in the 'Tokens' column is a list, confirming that the descriptions were properly converted into tokens.\n",
        "\n",
        "\n",
        "train_stop_words_remaining = docs_train_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\" before Descriptions with Stop Words Remaining: {train_stop_words_remaining}\")\n",
        "\n",
        "val_stop_words_remaining = docs_val_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\" before Descriptions with Stop Words Remaining: {val_stop_words_remaining}\")\n",
        "\n",
        "test_stop_words_remaining = docs_test_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\" before Descriptions with Stop Words Remaining: {test_stop_words_remaining}\")\n",
        "\n",
        "\n",
        "\n",
        "# Download and create the stop words list from NLTK\n",
        "stop_words = set(stopwords.words('english'))\n",
        "\n",
        "# Define your custom stop words list\n",
        "custom_stop_words = set([\n",
        "    'member', 'paper', 'presentation', 'challenge','report', 'new', 'change', 'plenary', 'need', 'impact',\n",
        "    'issue','actuary','actuarial', 'approach', 'description_provided','discus', 'discussion','program','review',\n",
        "    'australia','australian','institute','actuary_australia','may','insight','industry','actuary_institute',\n",
        "    'concurrent','risk','seminar','system','update','work','year', 'time','data','future','forum','model',\n",
        "    'plenary','seminar_plenary', 'first', 'value','also','using','comment','young_actuary','analysis','based',\n",
        "    'biennial_convention','customer','current','many','one','world', 'provide','well','used','company','area',\n",
        "    'different','same','including','result','research','seminar_concurrent','group','fund','business','outcome',\n",
        "    'practice','cost','use','product','market','policy','practice_committee'\n",
        "])\n",
        "\n",
        "# Combine the NLTK stop words with your custom stop words\n",
        "all_stop_words = stop_words.union(custom_stop_words)\n",
        "\n",
        "# Apply the combined stop words removal to training, validation, and test sets\n",
        "docs_train_cleaned_desc['Tokens'] = docs_train_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [word for word in x if word.lower() not in all_stop_words]\n",
        ")\n",
        "\n",
        "docs_val_cleaned_desc['Tokens'] = docs_val_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [word for word in x if word.lower() not in all_stop_words]\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Tokens'] = docs_test_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [word for word in x if word.lower() not in all_stop_words]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Zvtpqk_gwbXT",
      "metadata": {
        "id": "Zvtpqk_gwbXT"
      },
      "source": [
        "Check: Verify Stop Words Removal\n",
        "\n",
        "Explanation: This check ensures that common stop words have been removed from the token lists."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uewK0-HuwbXT",
      "metadata": {
        "id": "uewK0-HuwbXT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9ac8e4a-1790-4ac2-a61d-2f74dd6a62b6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Descriptions with Stop Words Remaining: 0\n",
            "Descriptions with Stop Words Remaining: 0\n",
            "Descriptions with Stop Words Remaining: 0\n"
          ]
        }
      ],
      "source": [
        "train_stop_words_remaining = docs_train_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\"Descriptions with Stop Words Remaining: {train_stop_words_remaining}\")\n",
        "\n",
        "val_stop_words_remaining = docs_val_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\"Descriptions with Stop Words Remaining: {val_stop_words_remaining}\")\n",
        "\n",
        "test_stop_words_remaining = docs_test_cleaned_desc['Tokens'].apply(lambda x: any(word in stop_words for word in x)).sum()\n",
        "print(f\"Descriptions with Stop Words Remaining: {test_stop_words_remaining}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ROMCRvFKDCGX",
      "metadata": {
        "id": "ROMCRvFKDCGX"
      },
      "source": [
        "Check show zeros, indicate our stop word removal is sucessful."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SudI6Wy6wbXT",
      "metadata": {
        "id": "SudI6Wy6wbXT"
      },
      "source": [
        "#####Lemmatization\n",
        "\n",
        "Reason for Cleaning: Reduces words to their base or root form (e.g., \"running\" to \"run\"), which helps in grouping similar words together.\n",
        "\n",
        "**Why taking Lemmatization After Stop Word Removal**:\n",
        "By lemmatizing after stop word removal, we ensure that only the meaningful words are reduced to their base forms. This process helps in reducing redundancy.\n",
        "\n",
        "\n",
        "Relation to Vectorization: Improves the representation of text data by reducing the number of unique tokens, leading to better generalization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "UbEpkRKVwbXT",
      "metadata": {
        "id": "UbEpkRKVwbXT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "922fab91-1667-4a20-d68e-324e7f469651"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n"
          ]
        }
      ],
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('wordnet') # Download the WordNet corpus\n",
        "\n",
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "# Lemmatize the tokens\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "docs_train_cleaned_desc['Tokens'] = docs_train_cleaned_desc['Tokens'].apply(lambda x: [lemmatizer.lemmatize(word) for word in x])\n",
        "docs_val_cleaned_desc['Tokens'] = docs_val_cleaned_desc['Tokens'].apply(lambda x: [lemmatizer.lemmatize(word) for word in x])\n",
        "docs_test_cleaned_desc['Tokens'] = docs_test_cleaned_desc['Tokens'].apply(lambda x: [lemmatizer.lemmatize(word) for word in x])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oUh8yJN3wbXT",
      "metadata": {
        "id": "oUh8yJN3wbXT"
      },
      "source": [
        "Check: Verify Lemmatization\n",
        "\n",
        "Explanation: This check ensures that words have been reduced to their base forms."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "GA68Hqn1Dafs",
      "metadata": {
        "id": "GA68Hqn1Dafs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "823cf5c0-6ed9-41dc-b120-e218295d3d8f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 'running' in document 5392. Lemmatization check failed.\n",
            "Found 'running' in document 6245. Lemmatization check failed.\n",
            "Found 'running' in document 6147. Lemmatization check failed.\n",
            "Unique word count in train: 9308\n",
            "Unique word count in validation: 4664\n",
            "Unique word count in test: 5904\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me some sensible check for both train, val and test, to ensure the lemmatization was done correctly\n",
        "\n",
        "# Check for Lemmatization - Example: Check if \"running\" is converted to \"run\" in a sample\n",
        "def check_lemmatization(df):\n",
        "  for index, row in df.iterrows():\n",
        "    tokens = row['Tokens']\n",
        "    for token in tokens:\n",
        "      if token == \"running\":\n",
        "        print(f\"Found 'running' in document {index}. Lemmatization check failed.\")\n",
        "        return\n",
        "  print(\"Lemmatization check passed (for 'running' example).\")\n",
        "\n",
        "\n",
        "check_lemmatization(docs_train_cleaned_desc)\n",
        "check_lemmatization(docs_val_cleaned_desc)\n",
        "check_lemmatization(docs_test_cleaned_desc)\n",
        "\n",
        "\n",
        "# Another check: you can check if the number of unique words reduced after lemmatization\n",
        "def unique_word_count(df):\n",
        "  all_words = []\n",
        "  for tokens in df['Tokens']:\n",
        "      all_words.extend(tokens)\n",
        "  return len(set(all_words))\n",
        "\n",
        "print(f\"Unique word count in train: {unique_word_count(docs_train_cleaned_desc)}\")\n",
        "print(f\"Unique word count in validation: {unique_word_count(docs_val_cleaned_desc)}\")\n",
        "print(f\"Unique word count in test: {unique_word_count(docs_test_cleaned_desc)}\")\n",
        "\n",
        "# It's hard to check if Lemmatization was perfect without having a manually\n",
        "# curated ground truth of what the lemmas should be in each case.\n",
        "# The above checks provide a reasonable level of confidence in the process.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nH5FdPQ3EMRv",
      "metadata": {
        "id": "nH5FdPQ3EMRv"
      },
      "source": [
        "**Action Based on Lemmatization Check Findings**\n",
        "The initial check indicates that the lemmatization process didn't perform as expected, failing to convert words like \"running\" to \"run\" consistently. Therefore, we will incorporate Part-of-Speech (POS) tagging to improve lemmatization accuracy.\n",
        "\n",
        "**Rationale for Using POS Tagging**\n",
        "Why POS Tagging Helps: The WordNetLemmatizer default behavior treats all words as nouns, which results in incorrect transformations for verbs and adjectives. By incorporating POS tagging, we provide the lemmatizer with the correct grammatical role of each word (e.g., verb, noun, adjective, adverb), ensuring more accurate conversion to the base form.\n",
        "\n",
        "\n",
        "Reference from ChatGPT4\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8_2s6ZiDEalz",
      "metadata": {
        "id": "8_2s6ZiDEalz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca10dda8-93e8-405a-e8e7-0ea9fbe77a0d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/bin/bash: line 1: import: command not found\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
          ]
        }
      ],
      "source": [
        "!import nltk\n",
        "# Download the 'averaged_perceptron_tagger' resource:\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "from nltk.corpus import wordnet\n",
        "from nltk import pos_tag\n",
        "\n",
        "# Function to convert NLTK POS tags to WordNet POS tags\n",
        "def get_wordnet_pos(treebank_tag):\n",
        "    if treebank_tag.startswith('J'):\n",
        "        return wordnet.ADJ\n",
        "    elif treebank_tag.startswith('V'):\n",
        "        return wordnet.VERB\n",
        "    elif treebank_tag.startswith('N'):\n",
        "        return wordnet.NOUN\n",
        "    elif treebank_tag.startswith('R'):\n",
        "        return wordnet.ADV\n",
        "    else:\n",
        "        return wordnet.NOUN\n",
        "\n",
        "# Apply lemmatization with POS tagging\n",
        "docs_train_cleaned_desc['Tokens'] = docs_train_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [lemmatizer.lemmatize(word, get_wordnet_pos(pos)) for word, pos in pos_tag(x)]\n",
        ")\n",
        "\n",
        "docs_val_cleaned_desc['Tokens'] = docs_val_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [lemmatizer.lemmatize(word, get_wordnet_pos(pos)) for word, pos in pos_tag(x)]\n",
        ")\n",
        "\n",
        "docs_test_cleaned_desc['Tokens'] = docs_test_cleaned_desc['Tokens'].apply(\n",
        "    lambda x: [lemmatizer.lemmatize(word, get_wordnet_pos(pos)) for word, pos in pos_tag(x)]\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nSiEC3uiFMvH",
      "metadata": {
        "id": "nSiEC3uiFMvH"
      },
      "source": [
        "We re-check the result of lemmatization after perfroming POS to see the result."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0w6IhWAVE-a-",
      "metadata": {
        "id": "0w6IhWAVE-a-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "049299b4-e450-44f0-8a96-072cf148a57b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Lemmatization check passed (for 'running' example).\n",
            "Lemmatization check passed (for 'running' example).\n",
            "Lemmatization check passed (for 'running' example).\n",
            "Unique word count in train: 8524\n",
            "Unique word count in validation: 4197\n",
            "Unique word count in test: 5296\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Re- Check for Lemmatization - Example: Check if \"running\" is converted to \"run\" in a sample\n",
        "def check_lemmatization(df):\n",
        "  for index, row in df.iterrows():\n",
        "    tokens = row['Tokens']\n",
        "    for token in tokens:\n",
        "      if token == \"running\":\n",
        "        print(f\"Found 'running' in document {index}. Lemmatization check failed.\")\n",
        "        return\n",
        "  print(\"Lemmatization check passed (for 'running' example).\")\n",
        "\n",
        "\n",
        "check_lemmatization(docs_train_cleaned_desc)\n",
        "check_lemmatization(docs_val_cleaned_desc)\n",
        "check_lemmatization(docs_test_cleaned_desc)\n",
        "\n",
        "\n",
        "# Another check: you can check if the number of unique words reduced after lemmatization\n",
        "def unique_word_count(df):\n",
        "  all_words = []\n",
        "  for tokens in df['Tokens']:\n",
        "      all_words.extend(tokens)\n",
        "  return len(set(all_words))\n",
        "\n",
        "print(f\"Unique word count in train: {unique_word_count(docs_train_cleaned_desc)}\")\n",
        "print(f\"Unique word count in validation: {unique_word_count(docs_val_cleaned_desc)}\")\n",
        "print(f\"Unique word count in test: {unique_word_count(docs_test_cleaned_desc)}\")\n",
        "\n",
        "# It's hard to check if Lemmatization was perfect without having a manually\n",
        "# curated ground truth of what the lemmas should be in each case.\n",
        "# The above checks provide a reasonable level of confidence in the process."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c-0Jij0PwbXT",
      "metadata": {
        "id": "c-0Jij0PwbXT"
      },
      "source": [
        "#####Reconstruct the Cleaned Description\n",
        "Reconstruct the cleaned text:\n",
        "\n",
        " After tokenization and cleaning, reconstruct the description from tokens for final use.\n",
        "\n",
        " TF-IDF methods typically take entire text strings as input rather than lists of tokens. Rejoining the tokens into a string format ensures compatibility with these vectorization techniques."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8kGuAjzjwbXT",
      "metadata": {
        "id": "8kGuAjzjwbXT"
      },
      "outputs": [],
      "source": [
        "docs_train_cleaned_desc['Cleaned_Description'] = docs_train_cleaned_desc['Tokens'].apply(lambda x: ' '.join(x))\n",
        "\n",
        "docs_val_cleaned_desc['Cleaned_Description'] = docs_val_cleaned_desc['Tokens'].apply(lambda x: ' '.join(x))\n",
        "docs_test_cleaned_desc['Cleaned_Description'] = docs_test_cleaned_desc['Tokens'].apply(lambda x: ' '.join(x))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3tX_uXhpwbXT",
      "metadata": {
        "id": "3tX_uXhpwbXT"
      },
      "source": [
        "Vectorization Method Recommendation: Given the cleaned text, TF-IDF is recommended for vectorization because it accounts for both the frequency of words within each document and across the entire corpus, which is useful in distinguishing common words from more informative ones, especially in lengthy descriptions."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Summary\n",
        "We applied cleaning to the training, validation, and testing sets using a consistent approach. This ensures that all sets can be vectorized uniformly later on.\n",
        "\n",
        "We conducted checks at each cleaning step and confirmed there were no data errors.\n"
      ],
      "metadata": {
        "id": "r0AYCzFpc6VE"
      },
      "id": "r0AYCzFpc6VE"
    },
    {
      "cell_type": "markdown",
      "id": "19Ox0dd7wbXT",
      "metadata": {
        "id": "19Ox0dd7wbXT"
      },
      "source": [
        "####Q1 b Step 3 Vectorisation\n",
        "\n",
        "**Purpose:** Transform the cleaned text data into a structured numerical format using vectorization, making it ready for analysis and machine learning models.\n",
        "\n",
        "**Approach:** We will begin with the TF-IDF (Term Frequency-Inverse Document Frequency) method and later explore advanced vectorization techniques such as Word2Vec and BERT.\n",
        "\n",
        "**Internal Validation:**\n",
        "For each vectorization method, we will perform internal validation to assess:\n",
        "\n",
        "Relevance and Distinguishing Power: How effectively each method captures meaningful features.\n",
        "Suitability for Clustering: Evaluate how well the vectorized representations support our later clustering tasks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Justification for Using TF-IDF as Our Primary Vectorization Technique**\n",
        "\n",
        "\n",
        "**Why TF-IDF Is Suitable for Description Vectorization:**\n",
        "\n",
        "The descriptions on the CPD Hub are well-structured, as they are professionally written and free from unstructured text like emojis, misspellings, slang, or casual writing. This makes TF-IDF an effective choice since it works best with structured, clear text. While descriptions might not always be short, TF-IDF is still a logical starting point for vectorization.\n",
        "\n",
        "**Other Benefits of Using TF-IDF:**\n",
        "- TF-IDF is less resource-intensive compared to advanced methods like BERT, which is important given the limited computational resources (no paid subscription on Colab).\n",
        "- TF-IDF is easy to implement and understand. For the CPD Hub, where descriptions are concise but meaningful, it captures the significance of key terms without being overly complex.\n",
        "- In the CPD Hub, certain terms (e.g., mortality,” “superannuation”) are crucial in distinguishing assets. TF-IDF emphasizes these by giving more weight to important terms, making it ideal for differentiating between topics and fields.\n"
      ],
      "metadata": {
        "id": "NrH_A5E6FdPN"
      },
      "id": "NrH_A5E6FdPN"
    },
    {
      "cell_type": "markdown",
      "id": "-rkFNpiRMhWv",
      "metadata": {
        "id": "-rkFNpiRMhWv"
      },
      "source": [
        "##### Vecotrisation TF-IDF\n",
        "\n",
        "**Objective:** To convert the cleaned text data into a structured format that can be effectively utilized in machine learning models, ensuring that the data is ready for analysis and subsequent processing.\n",
        "\n",
        "Reason for Vectorization: TF-IDF helps in identifying the importance of words relative to the document and the entire corpus, which is useful for distinguishing common words from more informative ones.\n",
        "\n",
        "**Why Start with TF-IDF?**\n",
        "Simplicity and Interpretability: TF-IDF is a straightforward, interpretable method that efficiently highlights important words, making it an ideal starting point for capturing key patterns in our dataset.\n",
        "\n",
        "Our EDA revealed a mix of common and unique terms in the descriptions, making TF-IDF suitable for emphasizing distinguishing words while downweighting frequent, less informative terms.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3N1CPMvNildz",
      "metadata": {
        "id": "3N1CPMvNildz"
      },
      "source": [
        "Define a function to show the sucess of vectorisaiton\n",
        "why we selectthis two metrix\n",
        "- Cosine Similarity: The function computes the pairwise cosine similarity for all documents and returns the average similarity score, excluding self-similarity (diagonal values).\n",
        "\n",
        "- Matrix Sparsity: The function also computes the mean value of all elements in the TF-IDF matrix to assess how sparse the matrix is.\n",
        "\n",
        "**why using these two metrics**\n",
        "Balanced Evaluation: Cosine similarity measures how well the vectorization process captures document distinctions, while matrix sparsity evaluates how effectively the process identifies and retains the most informative terms.\n",
        "Complementary Insights: Together, they provide a comprehensive evaluation of vectorization:\n",
        "\n",
        "- If cosine similarity is low, it means documents are distinct, indicating that the vectorization process effectively differentiates between them.\n",
        "- If matrix sparsity is also low, it confirms that TF-IDF is capturing only the most meaningful words, leading to a more efficient representation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ErvrVYNoih6-",
      "metadata": {
        "id": "ErvrVYNoih6-"
      },
      "outputs": [],
      "source": [
        "def evaluate_vectorization(tfidf_matrix):\n",
        "    from sklearn.metrics.pairwise import cosine_similarity\n",
        "    import numpy as np\n",
        "\n",
        "    # Compute the cosine similarity matrix for all documents\n",
        "    cosine_sim_matrix = cosine_similarity(tfidf_matrix)\n",
        "\n",
        "    # Mask the diagonal (set it to 0) to exclude self-similarity\n",
        "    np.fill_diagonal(cosine_sim_matrix, 0)\n",
        "\n",
        "    # Calculate the mean of the off-diagonal elements (average cosine similarity)\n",
        "    average_cosine_similarity = np.mean(cosine_sim_matrix[cosine_sim_matrix != 0])\n",
        "\n",
        "    # Calculate the mean value of all elements in the TF-IDF matrix (matrix sparsity)\n",
        "    mean_value = np.mean(tfidf_matrix.toarray())\n",
        "\n",
        "    return average_cosine_similarity, mean_value"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MgKemUeQiqAN",
      "metadata": {
        "id": "MgKemUeQiqAN"
      },
      "source": [
        "First trial of vecotrisation, suing 1000 feautes as a starting point - trail and erorr approahc.\n",
        "\n",
        "**Why Start with 1,000 Features?**\n",
        " Starting with 1,000 features strikes a balance between capturing enough meaningful words/phrases and maintaining manageable dimensionality. This helps ensure that the model can effectively process and learn from the data without being overwhelmed by too many features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eY-L7lAXPk7X",
      "metadata": {
        "id": "eY-L7lAXPk7X",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "85a10777-58a7-4788-ed8e-f28b2176ede8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF-IDF Matrix Shape (Training): (6068, 1000)\n",
            "    aa  aasb  acc  access  accident  account  accounting  accumulation  \\\n",
            "0  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "1  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "2  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "3  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "4  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "\n",
            "   achieve  across  ...      work  worker  workplace  workshop  would  write  \\\n",
            "0      0.0     0.0  ...  0.170284     0.0        0.0  0.000000    0.0    0.0   \n",
            "1      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "2      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "3      0.0     0.0  ...  0.000000     0.0        0.0  0.549663    0.0    0.0   \n",
            "4      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "\n",
            "   year  yet  young  zealand  \n",
            "0   0.0  0.0    0.0      0.0  \n",
            "1   0.0  0.0    0.0      0.0  \n",
            "2   0.0  0.0    0.0      0.0  \n",
            "3   0.0  0.0    0.0      0.0  \n",
            "4   0.0  0.0    0.0      0.0  \n",
            "\n",
            "[5 rows x 1000 columns]\n",
            "TF-IDF Matrix Shape (Validation): (757, 1000)\n",
            "    aa  aasb  acc  access  accident  account  accounting  accumulation  \\\n",
            "0  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "1  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "2  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "3  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "4  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "\n",
            "   achieve  across  ...      work  worker  workplace  workshop  would  write  \\\n",
            "0      0.0     0.0  ...  0.384516     0.0        0.0  0.000000    0.0    0.0   \n",
            "1      0.0     0.0  ...  0.000000     0.0        0.0  0.370732    0.0    0.0   \n",
            "2      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "3      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "4      0.0     0.0  ...  0.000000     0.0        0.0  0.000000    0.0    0.0   \n",
            "\n",
            "   year  yet  young  zealand  \n",
            "0   0.0  0.0    0.0      0.0  \n",
            "1   0.0  0.0    0.0      0.0  \n",
            "2   0.0  0.0    0.0      0.0  \n",
            "3   0.0  0.0    0.0      0.0  \n",
            "4   0.0  0.0    0.0      0.0  \n",
            "\n",
            "[5 rows x 1000 columns]\n",
            "TF-IDF Matrix Shape (Test): (725, 1000)\n",
            "    aa  aasb  acc  access  accident  account  accounting  accumulation  \\\n",
            "0  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "1  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "2  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "3  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "4  0.0   0.0  0.0     0.0       0.0      0.0         0.0           0.0   \n",
            "\n",
            "   achieve  across  ...      work  worker  workplace  workshop  would  write  \\\n",
            "0      0.0     0.0  ...  0.000000     0.0        0.0       0.0    0.0    0.0   \n",
            "1      0.0     0.0  ...  0.000000     0.0        0.0       0.0    0.0    0.0   \n",
            "2      0.0     0.0  ...  0.233831     0.0        0.0       0.0    0.0    0.0   \n",
            "3      0.0     0.0  ...  0.273355     0.0        0.0       0.0    0.0    0.0   \n",
            "4      0.0     0.0  ...  0.000000     0.0        0.0       0.0    0.0    0.0   \n",
            "\n",
            "   year  yet  young  zealand  \n",
            "0   0.0  0.0    0.0      0.0  \n",
            "1   0.0  0.0    0.0      0.0  \n",
            "2   0.0  0.0    0.0      0.0  \n",
            "3   0.0  0.0    0.0      0.0  \n",
            "4   0.0  0.0    0.0      0.0  \n",
            "\n",
            "[5 rows x 1000 columns]\n"
          ]
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "import pandas as pd\n",
        "\n",
        "# Ensure your dataframes 'docs_train_cleaned_desc', 'docs_val_cleaned_desc', and 'docs_test_cleaned_desc' are loaded\n",
        "\n",
        "# Initialize TF-IDF vectorizer with a maximum of 1000 features\n",
        "tfidf_vectorizer = TfidfVectorizer(max_features=1000)\n",
        "\n",
        "# Vectorize the training set\n",
        "tfidf_matrix_train = tfidf_vectorizer.fit_transform(docs_train_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_train = pd.DataFrame(tfidf_matrix_train.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Training): {tfidf_df_train.shape}\")\n",
        "print(tfidf_df_train.head())\n",
        "\n",
        "# Apply the fitted vectorizer to the validation set\n",
        "tfidf_matrix_val = tfidf_vectorizer.transform(docs_val_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_val = pd.DataFrame(tfidf_matrix_val.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Validation): {tfidf_df_val.shape}\")\n",
        "print(tfidf_df_val.head())\n",
        "\n",
        "# Apply the fitted vectorizer to the test set\n",
        "tfidf_matrix_test = tfidf_vectorizer.transform(docs_test_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_test = pd.DataFrame(tfidf_matrix_test.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Test): {tfidf_df_test.shape}\")\n",
        "print(tfidf_df_test.head())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uzfNCT34l_7n",
      "metadata": {
        "id": "uzfNCT34l_7n",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "outputId": "8ace088a-1f59-42fa-87e0-a6d985c7f349"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 59244 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.decomposition import PCA\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(tfidf_df_train)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XOh11DhWmLlW",
      "metadata": {
        "id": "XOh11DhWmLlW"
      },
      "source": [
        "The cummulative variance by PCA suggest 400 PC could have explainted 80% of variance,\n",
        "hence it indicates that not all 1000 feature used in the TF-IDF are signicnalty contribution to overall variance,. This suggest we should reduce the Max feature around 500 without lossing too much info"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "YPDKBGMO7fN4",
      "metadata": {
        "id": "YPDKBGMO7fN4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "173a7841-f0c4-4f71-b6b6-ca70dac71839"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity: 0.10114701761802085\n",
            "Matrix Sparsity: 0.9899345748187212\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 28804 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 88980 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me code to evaluate the performance of tfidf_matrix_0, with suitable visual display\n",
        "\n",
        "import numpy as np\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "# we define tfidf from train set as tfidif_matrix_0\n",
        "tfidf_matrix_0 = tfidf_matrix_train\n",
        "\n",
        "# Calculate the cosine similarity matrix\n",
        "cosine_sim_matrix = cosine_similarity(tfidf_matrix_0)\n",
        "\n",
        "# Calculate the average cosine similarity (excluding self-similarity)\n",
        "np.fill_diagonal(cosine_sim_matrix, 0)\n",
        "average_cosine_similarity = np.mean(cosine_sim_matrix[cosine_sim_matrix != 0])\n",
        "\n",
        "# Calculate the sparsity of the matrix\n",
        "sparsity = 1.0 - np.count_nonzero(tfidf_matrix_0.toarray()) / tfidf_matrix_0.toarray().size\n",
        "\n",
        "print(f\"Average Cosine Similarity: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity: {sparsity}\")\n",
        "\n",
        "\n",
        "# Visualize the distribution of cosine similarity\n",
        "plt.figure(figsize=(8, 6))\n",
        "sns.histplot(cosine_sim_matrix[cosine_sim_matrix != 0], bins=30)\n",
        "plt.title(\"Distribution of Cosine Similarity (excluding self-similarity)\")\n",
        "plt.xlabel(\"Cosine Similarity\")\n",
        "plt.ylabel(\"Frequency\")\n",
        "plt.show()\n",
        "\n",
        "# Visualize the sparsity with a heatmap (optional)\n",
        "plt.figure(figsize=(8, 6))\n",
        "sns.heatmap(tfidf_matrix_0.toarray(), cmap=\"viridis\", cbar_kws={'label': 'TF-IDF Value'})\n",
        "plt.title(\"TF-IDF Matrix Heatmap\")\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Visualizations comment**\n",
        "\n",
        "1. **Cosine Similarity Distribution (Histogram)**\n",
        "\n",
        "What It Shows:\n",
        "- The histogram shows the distribution of cosine similarities between pairs of documents (excluding self-similarity). The majority of document pairs have a very low similarity (close to 0), which means that the descriptions are largely distinct.\n",
        "\n",
        "Significance:\n",
        "- This is a good outcome because it indicates that the vectorized representations of the documents are capturing unique features, allowing the model to effectively differentiate between them.\n",
        "\n",
        "\n",
        "2. **TF-IDF Matrix Heatmap**\n",
        "What It Shows:\n",
        "- The heatmap displays the TF-IDF values for the terms across all documents.\n",
        "- Each row represents a document, and each column represents a term. The matrix is mostly dark, showing that most values are zero (high sparsity), but there are occasional non-zero values where relevant terms are present.\n",
        "\n",
        "Significance:\n",
        "- The heatmap confirms that the TF-IDF vectorization is assigning values to only the most important terms in each document. The sparsity of the matrix indicates that irrelevant terms are being effectively filtered out."
      ],
      "metadata": {
        "id": "8DZ_JxAiN9_D"
      },
      "id": "8DZ_JxAiN9_D"
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Check on Cosine and sparsity value**"
      ],
      "metadata": {
        "id": "4a3bTSozO9hk"
      },
      "id": "4a3bTSozO9hk"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "vNXAo4c0iy8c",
      "metadata": {
        "id": "vNXAo4c0iy8c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "25d4b85b-5dcd-4d9f-a3d2-b92466908a0d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.10114701761802085\n",
            "Matrix Sparsity (Mean Value): 0.002726580295339703\n"
          ]
        }
      ],
      "source": [
        "# Example usage with your TF-IDF matrix\n",
        "average_cosine_similarity, mean_value = evaluate_vectorization(tfidf_matrix_0)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Check on Cosine Similarty**\n",
        "Average Cosine Similarity: 0.101\n",
        "\n",
        "**What It Means:**\n",
        "- Low Similarity: A cosine similarity of 0.101 indicates that most documents are distinct from each other. This suggests that the descriptions of the assets have enough variation, and the TF-IDF vectorization is successfully capturing the differences.\n",
        "\n",
        "- Ideal for Grouping: In grouping tasks, it's important for descriptions to be distinct enough to form clear, separate groups. A low average cosine similarity means the descriptions are not overly similar, which is beneficial for forming accurate groups.\n",
        "\n",
        "- Meaningful Distinctions: The low similarity score shows that the TF-IDF vectorization is highlighting the key terms that differentiate one description from another, which is essential for identifying unique categories."
      ],
      "metadata": {
        "id": "dxfr_QDUN-2r"
      },
      "id": "dxfr_QDUN-2r"
    },
    {
      "cell_type": "markdown",
      "source": [
        "2. Check on Matrix Sparsity: 0.9899 (or 98.99%)\n",
        "\n",
        "**What It Means:**\n",
        "A 98.99% sparsity means that almost 99% of the matrix contains zeros. This is typical for TF-IDF matrices since it assigns non-zero values only to the terms that are present in a given document, ignoring irrelevant terms. The high sparsity indicates that only a small number of terms are relevant in each document, which aligns with how TF-IDF is supposed to function.\n",
        "\n",
        "**Significance:**\n",
        "High sparsity ensures that the model is focusing on a select set of important terms for each document, which leads to an efficient and meaningful representation of the text data."
      ],
      "metadata": {
        "id": "AuaPJTP2OPZh"
      },
      "id": "AuaPJTP2OPZh"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "CslX7_14bnS2",
      "metadata": {
        "id": "CslX7_14bnS2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3f8f2ac4-52f2-4199-c596-6717fa0c0753"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Combined TF-IDF Matrix Shape: (7550, 1000)\n"
          ]
        }
      ],
      "source": [
        "# prompt: i want to create a matric that combined all three train, val and test tifidif matrix, that can later use in cluster model\n",
        "\n",
        "from scipy.sparse import vstack\n",
        "\n",
        "# Combine the TF-IDF matrices for train, validation, and test sets\n",
        "combined_tfidf_matrix = vstack([tfidf_matrix_train, tfidf_matrix_val, tfidf_matrix_test])\n",
        "\n",
        "print(f\"Combined TF-IDF Matrix Shape: {combined_tfidf_matrix.shape}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nKB2cIeRoO9S",
      "metadata": {
        "id": "nKB2cIeRoO9S"
      },
      "source": [
        "**Conclusion:**\n",
        "The results show that the vectorization process is well-suited for grouping asset descriptions, providing meaningful distinctions between documents without unnecessary overlap. This approach should be effective for your goal of improving content grouping on the CPD Hub.\n",
        "\n",
        "However, as we will use this as input in Q2 clustering, having too many vectors might result in too many features for K-Means, even after applying dimensionality reduction techniques like PCA. To address this, we will also test TF-IDF with a lower max number of features, optimizing it for the purpose of Q2b clustering. This approach will help ensure that the clustering process remains efficient and effective."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LV6FUzdTmoYM",
      "metadata": {
        "id": "LV6FUzdTmoYM"
      },
      "source": [
        "#####Vectorization TF-IDF Iteration with Lower Max Features\n",
        "We reduced the maximum number of features to 500, then applied PCA to examine how many principal components are needed to capture 80% of the variance.\n",
        "\n",
        "The rationale behind this is that, since we will be using K-Means clustering, reducing dimensionality efficiently is crucial for effective results. If the data’s dimensionality cannot be reduced efficiently, this lower max feature count will help compensate, leading to more manageable and meaningful K-Means clustering outcomes."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "FCLLh7FmWsdz"
      },
      "id": "FCLLh7FmWsdz"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WakhnSEpmteG",
      "metadata": {
        "id": "WakhnSEpmteG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5756b8fb-954b-4795-b536-ddfa924425cb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF-IDF Matrix Shape (Training): (6068, 500)\n",
            "   aasb  accident  account  achieve  across  act  action  activity   actuary  \\\n",
            "0   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.118494   \n",
            "1   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "2   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "3   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "4   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "\n",
            "   address  ...  whether  within  wood      work  worker  would  write  year  \\\n",
            "0      0.0  ...      0.0     0.0   0.0  0.238216     0.0    0.0    0.0   0.0   \n",
            "1      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "2      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "3      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "4      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "\n",
            "   young  zealand  \n",
            "0    0.0      0.0  \n",
            "1    0.0      0.0  \n",
            "2    0.0      0.0  \n",
            "3    0.0      0.0  \n",
            "4    0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n",
            "TF-IDF Matrix Shape (Validation): (757, 500)\n",
            "   aasb  accident  account  achieve  across  act  action  activity  actuary  \\\n",
            "0   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.11293   \n",
            "1   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.00000   \n",
            "2   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.00000   \n",
            "3   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.00000   \n",
            "4   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.00000   \n",
            "\n",
            "   address  ...  whether  within  wood      work  worker  would  write  year  \\\n",
            "0      0.0  ...      0.0     0.0   0.0  0.454059     0.0    0.0    0.0   0.0   \n",
            "1      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "2      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "3      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "4      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "\n",
            "   young  zealand  \n",
            "0    0.0      0.0  \n",
            "1    0.0      0.0  \n",
            "2    0.0      0.0  \n",
            "3    0.0      0.0  \n",
            "4    0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n",
            "TF-IDF Matrix Shape (Test): (725, 500)\n",
            "   aasb  accident  account  achieve  across  act  action  activity   actuary  \\\n",
            "0   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "1   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.000000   \n",
            "2   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.232627   \n",
            "3   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.320485   \n",
            "4   0.0       0.0      0.0      0.0     0.0  0.0     0.0       0.0  0.171957   \n",
            "\n",
            "   address  ...  whether  within  wood      work  worker  would  write  year  \\\n",
            "0      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "1      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "2      0.0  ...      0.0     0.0   0.0  0.233831     0.0    0.0    0.0   0.0   \n",
            "3      0.0  ...      0.0     0.0   0.0  0.322144     0.0    0.0    0.0   0.0   \n",
            "4      0.0  ...      0.0     0.0   0.0  0.000000     0.0    0.0    0.0   0.0   \n",
            "\n",
            "   young  zealand  \n",
            "0    0.0      0.0  \n",
            "1    0.0      0.0  \n",
            "2    0.0      0.0  \n",
            "3    0.0      0.0  \n",
            "4    0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n"
          ]
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "import pandas as pd\n",
        "\n",
        "# Ensure your dataframes 'docs_train_cleaned_desc', 'docs_val_cleaned_desc', and 'docs_test_cleaned_desc' are loaded\n",
        "\n",
        "# Initialize TF-IDF vectorizer with a maximum of 1000 features\n",
        "tfidf_vectorizer = TfidfVectorizer(max_features=500)\n",
        "\n",
        "# Vectorize the training set\n",
        "tfidf_matrix_train1 = tfidf_vectorizer.fit_transform(docs_train_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_train1 = pd.DataFrame(tfidf_matrix_train1.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Training): {tfidf_df_train1.shape}\")\n",
        "print(tfidf_df_train1.head())\n",
        "\n",
        "# Apply the fitted vectorizer to the validation set\n",
        "tfidf_matrix_val1 = tfidf_vectorizer.transform(docs_val_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_val1 = pd.DataFrame(tfidf_matrix_val1.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Validation): {tfidf_df_val1.shape}\")\n",
        "print(tfidf_df_val1.head())\n",
        "\n",
        "# Apply the fitted vectorizer to the test set\n",
        "tfidf_matrix_test1 = tfidf_vectorizer.transform(docs_test_cleaned_desc['Cleaned_Description'])\n",
        "tfidf_df_test1 = pd.DataFrame(tfidf_matrix_test1.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Test): {tfidf_df_test1.shape}\")\n",
        "print(tfidf_df_test1.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Comments:\n",
        "\n",
        "Many of the vectors, such as \"activity,\" \"actuary,\" \"account,\" and \"achieve,\" do not offer strong predictive power for clustering. These terms are too generic and do not provide meaningful distinctions between assets. They do not help identify the core subject matter of each asset, which is essential for effective grouping and clustering. As a result, it would be beneficial to further refine the vectorization process by either adjusting the stop words list or reducing the feature set to focus on more relevant, specific terms."
      ],
      "metadata": {
        "id": "eWylevRCSsV9"
      },
      "id": "eWylevRCSsV9"
    },
    {
      "cell_type": "markdown",
      "source": [
        "We applied PCA to assess how much dimensionality reduction can be achieved, as it's important to balance this with the results for the upcoming K-Means clustering.\n",
        "\n",
        "The goal is to retain enough variance to maintain meaningful information while reducing the number of features to improve the efficiency and effectiveness of the clustering process. By applying PCA, we can determine the optimal number of components needed to strike this balance, ensuring that the K-Means model performs well without being overwhelmed by too many irrelevant dimensions."
      ],
      "metadata": {
        "id": "B24JJnwJS-90"
      },
      "id": "B24JJnwJS-90"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "GnYvFW8UnBE4",
      "metadata": {
        "id": "GnYvFW8UnBE4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "outputId": "9f6e3d3c-4510-4c22-a605-13367536a8f6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 55812 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(tfidf_df_train1)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Left Graph: Explained Variance by Principal Components\n",
        "- This bar chart shows the variance captured by each principal component. The first few components capture a lot of variance, while the remaining components capture significantly less.\n",
        "- Most of the variance is explained by the first few components, indicating that a large number of the later components contribute very little to the overall variance. This suggests that dimensionality reduction can be achieved without losing much important information.\n",
        "\n",
        "\n",
        "Right Graph: Cumulative Explained Variance\n",
        "- This line chart shows how much cumulative variance is captured as the number of principal components increases. It indicates how many components are needed to reach a given level of variance.\n",
        "- To capture 80% of the variance, the curve likely suggests that fewer than 300 components are required, which is much fewer than the original 500 features.\n",
        "- However, this might be still too many for K-mean\n",
        "\n",
        "\n",
        "**Conclusion:**\n",
        "- we need to reduce the max feature in TFIDF to ensure k-mean in Q2 can have a reasonable input.\n"
      ],
      "metadata": {
        "id": "UP1_FWDySPst"
      },
      "id": "UP1_FWDySPst"
    },
    {
      "cell_type": "markdown",
      "id": "-lxc8t3vU2Xa",
      "metadata": {
        "id": "-lxc8t3vU2Xa"
      },
      "source": [
        "#####TFIDF on Phrase Detection (Bigram and Trigram Detection)\n",
        "\n",
        "-----------------------------------------------------------------------\n",
        "**This part is not longer relevence because we end up not using TFIDF on Phrase detection as it does not improve Clustering performance**\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "-------------------------------------------------------------------------\n",
        "Since the Cleaned_Description is created by rejoining the tokens from Tokens, we'll work with the Tokens column directly for bigram detection and then rejoin the tokens afterward."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1F3H5OuvU0qa",
      "metadata": {
        "id": "1F3H5OuvU0qa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a73e765e-f286-4ca7-dfc9-dae15f40e88b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Bigrams applied successfully to validation and test datasets.\n"
          ]
        }
      ],
      "source": [
        "from gensim.models import Phrases\n",
        "from gensim.models.phrases import Phraser\n",
        "\n",
        "# Assume 'Tokens' column contains tokenized text data\n",
        "\n",
        "# Detect bigrams\n",
        "phrases = Phrases(docs_train_cleaned_desc['Tokens'], min_count=2, threshold=5)\n",
        "bigram = Phraser(phrases)\n",
        "\n",
        "# Apply the bigram model to the tokenized text\n",
        "docs_train_cleaned_desc['Bigram_Description'] = docs_train_cleaned_desc['Tokens'].apply(lambda x: bigram[x])\n",
        "docs_val_cleaned_desc['Bigram_Description'] = docs_val_cleaned_desc['Tokens'].apply(lambda x: bigram[x])\n",
        "docs_test_cleaned_desc['Bigram_Description'] = docs_test_cleaned_desc['Tokens'].apply(lambda x: bigram[x])\n",
        "\n",
        "\n",
        "\n",
        "# Join the bigrams back into a single string for TF-IDF vectorization\n",
        "docs_train_cleaned_desc['Bigram_Description'] = docs_train_cleaned_desc['Bigram_Description'].apply(lambda x: ' '.join(x))\n",
        "docs_val_cleaned_desc['Bigram_Description'] = docs_val_cleaned_desc['Bigram_Description'].apply(lambda x: ' '.join(x))\n",
        "docs_test_cleaned_desc['Bigram_Description'] = docs_test_cleaned_desc['Bigram_Description'].apply(lambda x: ' '.join(x))\n",
        "\n",
        "print(\"Bigrams applied successfully to validation and test datasets.\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MOWfInXiSh0a",
      "metadata": {
        "id": "MOWfInXiSh0a"
      },
      "source": [
        "Clustering Purpose:\n",
        "The aim of clustering is to group similar items together. High cosine similarity between assets within the same cluster indicates that these assets share significant features, meaning they are similar in content or characteristics.\n",
        "Reduced Dimensionality: The matrix now contains 200 features instead of 1,000, meaning each document is represented by a vector of 200 dimensions. This reduction helps in focusing on the most informative terms while making the data more manageable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1wx7d-K_YWkw",
      "metadata": {
        "collapsed": true,
        "id": "1wx7d-K_YWkw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6caf26d5-3528-4c96-dcac-4cac0ae83f36"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF-IDF Matrix Shape: (6068, 500)\n",
            "   aasb  accident_compensation  achieve  act  activity   actuary  \\\n",
            "0   0.0                    0.0      0.0  0.0       0.0  0.152197   \n",
            "1   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "2   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "3   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "4   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "\n",
            "   actuary take_lead  actuary_digital  actuary_magazine  actuary_summit  ...  \\\n",
            "0                0.0              0.0               0.0             0.0  ...   \n",
            "1                0.0              0.0               0.0             0.0  ...   \n",
            "2                0.0              0.0               0.0             0.0  ...   \n",
            "3                0.0              0.0               0.0             0.0  ...   \n",
            "4                0.0              0.0               0.0             0.0  ...   \n",
            "\n",
            "   week  welcome  wood  work  worker_compensation  write  write_apra  year  \\\n",
            "0   0.0      0.0   0.0   0.0                  0.0    0.0         0.0   0.0   \n",
            "1   0.0      0.0   0.0   0.0                  0.0    0.0         0.0   0.0   \n",
            "2   0.0      0.0   0.0   0.0                  0.0    0.0         0.0   0.0   \n",
            "3   0.0      0.0   0.0   0.0                  0.0    0.0         0.0   0.0   \n",
            "4   0.0      0.0   0.0   0.0                  0.0    0.0         0.0   0.0   \n",
            "\n",
            "   young_actuary  zealand  \n",
            "0            0.0      0.0  \n",
            "1            0.0      0.0  \n",
            "2            0.0      0.0  \n",
            "3            0.0      0.0  \n",
            "4            0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n",
            "TF-IDF Matrix Shape (Validation): (757, 500)\n",
            "   aasb  accident_compensation  achieve  act  activity   actuary  \\\n",
            "0   0.0                    0.0      0.0  0.0       0.0  0.181962   \n",
            "1   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "2   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "3   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "4   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "\n",
            "   actuary take_lead  actuary_digital  actuary_magazine  actuary_summit  ...  \\\n",
            "0                0.0              0.0               0.0             0.0  ...   \n",
            "1                0.0              0.0               0.0             0.0  ...   \n",
            "2                0.0              0.0               0.0             0.0  ...   \n",
            "3                0.0              0.0               0.0             0.0  ...   \n",
            "4                0.0              0.0               0.0             0.0  ...   \n",
            "\n",
            "   week  welcome  wood      work  worker_compensation  write  write_apra  \\\n",
            "0   0.0      0.0   0.0  0.292472                  0.0    0.0         0.0   \n",
            "1   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "2   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "3   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "4   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "\n",
            "   year  young_actuary  zealand  \n",
            "0   0.0            0.0      0.0  \n",
            "1   0.0            0.0      0.0  \n",
            "2   0.0            0.0      0.0  \n",
            "3   0.0            0.0      0.0  \n",
            "4   0.0            0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n",
            "TF-IDF Matrix Shape (Test): (725, 500)\n",
            "   aasb  accident_compensation  achieve  act  activity   actuary  \\\n",
            "0   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "1   0.0                    0.0      0.0  0.0       0.0  0.000000   \n",
            "2   0.0                    0.0      0.0  0.0       0.0  0.348391   \n",
            "3   0.0                    0.0      0.0  0.0       0.0  0.520279   \n",
            "4   0.0                    0.0      0.0  0.0       0.0  0.349735   \n",
            "\n",
            "   actuary take_lead  actuary_digital  actuary_magazine  actuary_summit  ...  \\\n",
            "0                0.0              0.0               0.0             0.0  ...   \n",
            "1                0.0              0.0               0.0             0.0  ...   \n",
            "2                0.0              0.0               0.0             0.0  ...   \n",
            "3                0.0              0.0               0.0             0.0  ...   \n",
            "4                0.0              0.0               0.0             0.0  ...   \n",
            "\n",
            "   week  welcome  wood      work  worker_compensation  write  write_apra  \\\n",
            "0   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "1   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "2   0.0      0.0   0.0  0.279989                  0.0    0.0         0.0   \n",
            "3   0.0      0.0   0.0  0.418128                  0.0    0.0         0.0   \n",
            "4   0.0      0.0   0.0  0.000000                  0.0    0.0         0.0   \n",
            "\n",
            "   year  young_actuary  zealand  \n",
            "0   0.0            0.0      0.0  \n",
            "1   0.0            0.0      0.0  \n",
            "2   0.0            0.0      0.0  \n",
            "3   0.0            0.0      0.0  \n",
            "4   0.0            0.0      0.0  \n",
            "\n",
            "[5 rows x 500 columns]\n",
            "TF-IDF vectorization successfully applied to validation and test datasets.\n",
            "Selected Features (Terms/Phrases) in TF-IDF Vectorization:\n",
            "1: aasb\n",
            "2: accident_compensation\n",
            "3: achieve\n",
            "4: act\n",
            "5: activity\n",
            "6: actuary\n",
            "7: actuary take_lead\n",
            "8: actuary_digital\n",
            "9: actuary_magazine\n",
            "10: actuary_summit\n",
            "11: address\n",
            "12: advice\n",
            "13: affect\n",
            "14: age\n",
            "15: aim\n",
            "16: allow\n",
            "17: alternative\n",
            "18: analytics\n",
            "19: andrew\n",
            "20: andrew_brown\n",
            "21: annual\n",
            "22: annuity\n",
            "23: appetite\n",
            "24: application\n",
            "25: apply\n",
            "26: appoint\n",
            "27: approach\n",
            "28: appropriate\n",
            "29: apra\n",
            "30: apras\n",
            "31: area\n",
            "32: arrangement\n",
            "33: article\n",
            "34: asia\n",
            "35: ask\n",
            "36: ask_gae\n",
            "37: aspect\n",
            "38: assessment\n",
            "39: asset\n",
            "40: assist\n",
            "41: associate\n",
            "42: assumption\n",
            "43: astin_afirerm\n",
            "44: astin_afirerm iaca_colloquium\n",
            "45: audio\n",
            "46: australia\n",
            "47: australian\n",
            "48: author\n",
            "49: available\n",
            "50: award\n",
            "51: balance\n",
            "52: bank\n",
            "53: banking\n",
            "54: barry_rafe\n",
            "55: basis\n",
            "56: behaviour\n",
            "57: benefit\n",
            "58: best\n",
            "59: biennial_convention\n",
            "60: big\n",
            "61: board\n",
            "62: book\n",
            "63: build\n",
            "64: calendar_upcoming\n",
            "65: calendar_upcoming event\n",
            "66: capital\n",
            "67: care\n",
            "68: career\n",
            "69: case\n",
            "70: case_study\n",
            "71: catastrophe\n",
            "72: ceo\n",
            "73: ceo_column\n",
            "74: challenge\n",
            "75: change\n",
            "76: choice\n",
            "77: claim\n",
            "78: claim_management\n",
            "79: client\n",
            "80: climate\n",
            "81: code\n",
            "82: come\n",
            "83: comment\n",
            "84: commission\n",
            "85: committee\n",
            "86: communication\n",
            "87: community\n",
            "88: company\n",
            "89: compare\n",
            "90: compensation\n",
            "91: concept\n",
            "92: concern\n",
            "93: condition\n",
            "94: conference\n",
            "95: consider\n",
            "96: consideration\n",
            "97: considers\n",
            "98: consultation\n",
            "99: consumer\n",
            "100: content\n",
            "101: context\n",
            "102: continue\n",
            "103: contribution\n",
            "104: convention\n",
            "105: cost\n",
            "106: council\n",
            "107: course\n",
            "108: cover\n",
            "109: covid\n",
            "110: create\n",
            "111: crisis\n",
            "112: culture\n",
            "113: currently\n",
            "114: customer\n",
            "115: day\n",
            "116: deal\n",
            "117: december\n",
            "118: decision\n",
            "119: define_benefit\n",
            "120: deliver\n",
            "121: demonstrate\n",
            "122: design\n",
            "123: determine\n",
            "124: develop\n",
            "125: development\n",
            "126: difference\n",
            "127: director\n",
            "128: disability\n",
            "129: disability_income\n",
            "130: discus\n",
            "131: discuss\n",
            "132: distribution\n",
            "133: dr\n",
            "134: draft\n",
            "135: driver\n",
            "136: economic\n",
            "137: economy\n",
            "138: education\n",
            "139: education_philip\n",
            "140: effect\n",
            "141: effective\n",
            "142: emerge\n",
            "143: employer\n",
            "144: enterprise_management\n",
            "145: environment\n",
            "146: equity\n",
            "147: erm\n",
            "148: establish\n",
            "149: estimate\n",
            "150: event\n",
            "151: event_notice\n",
            "152: examine\n",
            "153: example\n",
            "154: exist\n",
            "155: experience\n",
            "156: explain\n",
            "157: explanatory_memorandum\n",
            "158: explore\n",
            "159: face\n",
            "160: factor\n",
            "161: fair\n",
            "162: feature\n",
            "163: ffa\n",
            "164: fia\n",
            "165: field\n",
            "166: finance\n",
            "167: financial\n",
            "168: financial_condition\n",
            "169: financial_inquiry\n",
            "170: financial_service\n",
            "171: flood\n",
            "172: flood_resilience\n",
            "173: focus\n",
            "174: follow\n",
            "175: forecast\n",
            "176: form\n",
            "177: framework\n",
            "178: function\n",
            "179: fund\n",
            "180: funding\n",
            "181: general\n",
            "182: general_insurance\n",
            "183: general_insurance committee\n",
            "184: general_insurance pricing\n",
            "185: genevieve_hayes\n",
            "186: gi\n",
            "187: gi_session\n",
            "188: global\n",
            "189: good\n",
            "190: governance\n",
            "191: government\n",
            "192: great\n",
            "193: growth\n",
            "194: guidance\n",
            "195: guide\n",
            "196: guideline\n",
            "197: head\n",
            "198: health\n",
            "199: health_committee\n",
            "200: health_committee newsletter\n",
            "201: health_insurance\n",
            "202: healthcare\n",
            "203: help\n",
            "204: high\n",
            "205: highlight\n",
            "206: historical\n",
            "207: hold\n",
            "208: home\n",
            "209: iaaust\n",
            "210: iaca_colloquium\n",
            "211: iaca_colloquium synopsis\n",
            "212: icaap\n",
            "213: idea\n",
            "214: identify\n",
            "215: ifrs\n",
            "216: illustrate\n",
            "217: impact\n",
            "218: implement\n",
            "219: implementation\n",
            "220: implication\n",
            "221: importance\n",
            "222: important\n",
            "223: improve\n",
            "224: include\n",
            "225: income\n",
            "226: increase\n",
            "227: index\n",
            "228: individual\n",
            "229: inflation\n",
            "230: influence\n",
            "231: information\n",
            "232: information_note\n",
            "233: injury\n",
            "234: injury_disability\n",
            "235: injury_scheme\n",
            "236: innovation\n",
            "237: insight\n",
            "238: insight_session\n",
            "239: institute\n",
            "240: insurance\n",
            "241: insurance scheme\n",
            "242: insurance_contract\n",
            "243: insurer\n",
            "244: international\n",
            "245: interview\n",
            "246: introduce\n",
            "247: introduction\n",
            "248: investment\n",
            "249: investor\n",
            "250: involve\n",
            "251: issue\n",
            "252: journey\n",
            "253: july\n",
            "254: june\n",
            "255: key\n",
            "256: know\n",
            "257: large\n",
            "258: late\n",
            "259: lead\n",
            "260: leader\n",
            "261: leadership\n",
            "262: learn\n",
            "263: legislation\n",
            "264: lesson\n",
            "265: letter\n",
            "266: level\n",
            "267: liability\n",
            "268: life\n",
            "269: life_insurance\n",
            "270: life_insurance wealth_management\n",
            "271: life_insurer\n",
            "272: like\n",
            "273: line\n",
            "274: long\n",
            "275: long_term\n",
            "276: longevity\n",
            "277: look\n",
            "278: loss\n",
            "279: major\n",
            "280: make\n",
            "281: make_submission\n",
            "282: manage\n",
            "283: management\n",
            "284: management_committee\n",
            "285: manager\n",
            "286: march\n",
            "287: margin\n",
            "288: margin_puzzle\n",
            "289: margin_puzzle genevieve_hayes\n",
            "290: market\n",
            "291: martin_mulcare\n",
            "292: mean\n",
            "293: measure\n",
            "294: medium\n",
            "295: meet\n",
            "296: meeting\n",
            "297: melbourne\n",
            "298: melinda_howe\n",
            "299: member\n",
            "300: mental_health\n",
            "301: method\n",
            "302: methodology\n",
            "303: model\n",
            "304: month\n",
            "305: mortality\n",
            "306: national\n",
            "307: natural_disaster\n",
            "308: need\n",
            "309: newsletter\n",
            "310: note\n",
            "311: note_president\n",
            "312: notice\n",
            "313: november\n",
            "314: nsw\n",
            "315: number\n",
            "316: obituary\n",
            "317: objective\n",
            "318: occasional_newsletter\n",
            "319: occasional_newsletter general_insurance\n",
            "320: operational\n",
            "321: opportunity\n",
            "322: option\n",
            "323: organisation\n",
            "324: outcome\n",
            "325: outline\n",
            "326: overview\n",
            "327: pandemic\n",
            "328: panel\n",
            "329: part_iii\n",
            "330: participant\n",
            "331: particular\n",
            "332: particularly\n",
            "333: pension\n",
            "334: people\n",
            "335: performance\n",
            "336: period\n",
            "337: perspective\n",
            "338: peter\n",
            "339: phi\n",
            "340: place\n",
            "341: plan\n",
            "342: play\n",
            "343: point\n",
            "344: policy\n",
            "345: population\n",
            "346: portfolio\n",
            "347: position\n",
            "348: possible\n",
            "349: postcard\n",
            "350: potential\n",
            "351: practical\n",
            "352: practice\n",
            "353: premium\n",
            "354: prepare\n",
            "355: present\n",
            "356: president\n",
            "357: president_column\n",
            "358: presidential_address\n",
            "359: price\n",
            "360: pricing\n",
            "361: principle\n",
            "362: private_health\n",
            "363: private_health insurance\n",
            "364: prize\n",
            "365: problem\n",
            "366: process\n",
            "367: product\n",
            "368: profession\n",
            "369: professional\n",
            "370: professional_standard\n",
            "371: profile\n",
            "372: program\n",
            "373: project\n",
            "374: projection\n",
            "375: proposal\n",
            "376: propose\n",
            "377: provide\n",
            "378: provider\n",
            "379: public\n",
            "380: pulse\n",
            "381: pulse_survey\n",
            "382: purpose\n",
            "383: puzzle\n",
            "384: question\n",
            "385: range\n",
            "386: rate\n",
            "387: recent\n",
            "388: recently\n",
            "389: reduce\n",
            "390: reflect\n",
            "391: reform\n",
            "392: regard\n",
            "393: regulation\n",
            "394: regulatory\n",
            "395: reinsurance\n",
            "396: relate\n",
            "397: relation\n",
            "398: relationship\n",
            "399: release\n",
            "400: relevant\n",
            "401: report\n",
            "402: require\n",
            "403: requirement\n",
            "404: reserve\n",
            "405: response\n",
            "406: result\n",
            "407: retiree\n",
            "408: retirement\n",
            "409: retirement_income\n",
            "410: return\n",
            "411: review\n",
            "412: rise\n",
            "413: risk\n",
            "414: robinson\n",
            "415: role\n",
            "416: royal_commission\n",
            "417: save\n",
            "418: scheme\n",
            "419: science\n",
            "420: sector\n",
            "421: seek\n",
            "422: september\n",
            "423: series\n",
            "424: service\n",
            "425: session\n",
            "426: set\n",
            "427: share\n",
            "428: significant\n",
            "429: skill\n",
            "430: social\n",
            "431: society\n",
            "432: solution\n",
            "433: spotlight\n",
            "434: stakeholder\n",
            "435: standard\n",
            "436: start\n",
            "437: state\n",
            "438: stay_ahead\n",
            "439: stochastic\n",
            "440: strategy\n",
            "441: structure\n",
            "442: student\n",
            "443: student_column\n",
            "444: study\n",
            "445: subject\n",
            "446: submission\n",
            "447: success\n",
            "448: suggest\n",
            "449: summary\n",
            "450: super\n",
            "451: superannuation\n",
            "452: superannuation_committee\n",
            "453: superannuation_fund\n",
            "454: support\n",
            "455: survey\n",
            "456: sustainability\n",
            "457: swim_stephen\n",
            "458: sydney\n",
            "459: synopsis\n",
            "460: take_lead\n",
            "461: taskforce\n",
            "462: tax\n",
            "463: team\n",
            "464: technique\n",
            "465: technology\n",
            "466: test\n",
            "467: theory\n",
            "468: think\n",
            "469: today\n",
            "470: topic\n",
            "471: traditional\n",
            "472: travel\n",
            "473: treasury\n",
            "474: trend\n",
            "475: two_duck\n",
            "476: two_duck swim_stephen\n",
            "477: uk\n",
            "478: uncertainty\n",
            "479: understand\n",
            "480: understanding\n",
            "481: unearth\n",
            "482: unearthed\n",
            "483: update\n",
            "484: valuation\n",
            "485: value\n",
            "486: various\n",
            "487: view\n",
            "488: want\n",
            "489: way\n",
            "490: wealth_management\n",
            "491: week\n",
            "492: welcome\n",
            "493: wood\n",
            "494: work\n",
            "495: worker_compensation\n",
            "496: write\n",
            "497: write_apra\n",
            "498: year\n",
            "499: young_actuary\n",
            "500: zealand\n"
          ]
        }
      ],
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "import pandas as pd\n",
        "\n",
        "# Set up TF-IDF vectorization parameters\n",
        "tfidf_vectorizer = TfidfVectorizer(\n",
        "    max_features=500,      # Limit to 100 features (words/terms)\n",
        "    min_df=3,              # Exclude terms that appear in fewer than 3 documents\n",
        "    max_df=0.85,           # Exclude terms that appear in more than 85% of documents\n",
        "    ngram_range=(1, 2),    # Use both unigrams and bigrams\n",
        "    stop_words='english'   # Remove common English stop words\n",
        ")\n",
        "\n",
        "# Perform TF-IDF vectorization on the bigram-processed descriptions\n",
        "tfidf_matrix_train_2 = tfidf_vectorizer.fit_transform(docs_train_cleaned_desc['Bigram_Description'])\n",
        "\n",
        "# Convert the TF-IDF matrix to a DataFrame for easier inspection\n",
        "tfidf_df = pd.DataFrame(tfidf_matrix_train_2.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "\n",
        "# Display the shape of the TF-IDF matrix and the first few rows\n",
        "print(f\"TF-IDF Matrix Shape: {tfidf_df.shape}\")\n",
        "print(tfidf_df.head())\n",
        "\n",
        "\n",
        "# Apply the TF-IDF vectorizer to the validation set\n",
        "tfidf_matrix_val_2 = tfidf_vectorizer.transform(docs_val_cleaned_desc['Bigram_Description'])\n",
        "tfidf_df_val = pd.DataFrame(tfidf_matrix_val_2.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Validation): {tfidf_df_val.shape}\")\n",
        "print(tfidf_df_val.head())\n",
        "\n",
        "# Apply the TF-IDF vectorizer to the test set\n",
        "tfidf_matrix_test_2 = tfidf_vectorizer.transform(docs_test_cleaned_desc['Bigram_Description'])\n",
        "tfidf_df_test = pd.DataFrame(tfidf_matrix_test_2.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "print(f\"TF-IDF Matrix Shape (Test): {tfidf_df_test.shape}\")\n",
        "print(tfidf_df_test.head())\n",
        "\n",
        "# Confirmation message\n",
        "print(\"TF-IDF vectorization successfully applied to validation and test datasets.\")\n",
        "\n",
        "\n",
        "\n",
        "# Display the features (terms/phrases) selected in the TF-IDF vectorizer, from the training set\n",
        "selected_features = tfidf_vectorizer.get_feature_names_out()\n",
        "\n",
        "# Print out the 60 selected features\n",
        "print(\"Selected Features (Terms/Phrases) in TF-IDF Vectorization:\")\n",
        "for idx, feature in enumerate(selected_features):\n",
        "    print(f\"{idx + 1}: {feature}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "jNUOQmT3QxWJ",
      "metadata": {
        "id": "jNUOQmT3QxWJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc64c73c-0cd7-4332-8c3d-0416eb722167"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Combined TF-IDF Matrix Shape: (7550, 500)\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me code to cobined the tfidf_matrix_train2, tfiid_matrix_val2,tfiid_matrix_test2 into a single matric call tfidif_matrix_all2, that can be use in later clustering model\n",
        "\n",
        "from scipy.sparse import vstack\n",
        "\n",
        "tfidf_matrix_all_2 = vstack([tfidf_matrix_train_2, tfidf_matrix_val_2, tfidf_matrix_test_2])\n",
        "print(f\"Combined TF-IDF Matrix Shape: {tfidf_matrix_all_2.shape}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "RwyrPy377Gck",
      "metadata": {
        "id": "RwyrPy377Gck",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "94f3a557-3a79-45c7-a54b-3cec660457cf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity: 0.20119033126912314\n",
            "Matrix Sparsity: 0.9904765985497693\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 34272 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 87268 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.20119033126912314\n",
            "Matrix Sparsity (Mean Value): 0.0037280409833987363\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me codes to evaluate the tfidf_matrix_train_2 , and with suitable visual plot\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "# Calculate the cosine similarity matrix\n",
        "cosine_sim_matrix = cosine_similarity(tfidf_matrix_train_2)\n",
        "\n",
        "# Calculate the average cosine similarity (excluding self-similarity)\n",
        "np.fill_diagonal(cosine_sim_matrix, 0)\n",
        "average_cosine_similarity = np.mean(cosine_sim_matrix[cosine_sim_matrix != 0])\n",
        "\n",
        "# Calculate the sparsity of the matrix\n",
        "sparsity = 1.0 - np.count_nonzero(tfidf_matrix_train_2.toarray()) / tfidf_matrix_train_2.toarray().size\n",
        "\n",
        "print(f\"Average Cosine Similarity: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity: {sparsity}\")\n",
        "\n",
        "\n",
        "# Visualize the distribution of cosine similarity\n",
        "plt.figure(figsize=(8, 6))\n",
        "sns.histplot(cosine_sim_matrix[cosine_sim_matrix != 0], bins=30)\n",
        "plt.title(\"Distribution of Cosine Similarity (excluding self-similarity)\")\n",
        "plt.xlabel(\"Cosine Similarity\")\n",
        "plt.ylabel(\"Frequency\")\n",
        "plt.show()\n",
        "\n",
        "# Visualize the sparsity with a heatmap (optional)\n",
        "plt.figure(figsize=(8, 6))\n",
        "sns.heatmap(tfidf_matrix_train_2.toarray(), cmap=\"viridis\", cbar_kws={'label': 'TF-IDF Value'})\n",
        "plt.title(\"TF-IDF Matrix Heatmap\")\n",
        "plt.show()\n",
        "\n",
        "# Example usage with your TF-IDF matrix\n",
        "average_cosine_similarity, mean_value = evaluate_vectorization(tfidf_matrix_train_2)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "l7fIA1f_ox8h",
      "metadata": {
        "id": "l7fIA1f_ox8h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f7b808fd-e365-4138-b58f-76c4decf1c7c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.20119033126912314\n",
            "Matrix Sparsity (Mean Value): 0.0037280409833987363\n"
          ]
        }
      ],
      "source": [
        "# Example usage with your TF-IDF matrix\n",
        "average_cosine_similarity, mean_value = evaluate_vectorization(tfidf_matrix_train_2)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zfUqPm1BaGOb",
      "metadata": {
        "id": "zfUqPm1BaGOb"
      },
      "source": [
        "Trade-Offs to Consider:\n",
        "Higher Similarity but Less Specificity: While reducing the max_features increases similarity, it may also reduce the specificity of the document representations. This can be problematic if you need fine-grained distinctions between documents.\n",
        "Loss of Important Details: If the feature count is reduced too much, you might lose important terms that are critical for capturing the nuances of certain documents, which could negatively impact downstream tasks like clustering or classification."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "IrI6hpiMbYbK",
      "metadata": {
        "id": "IrI6hpiMbYbK"
      },
      "source": [
        "Increase N-gram Range:\n",
        "Problem: Some useful multi-word phrases might not be captured by just using bigrams.\n",
        "Solution: Consider increasing the ngram_range to (1, 3) to capture trigrams as well, which can provide more context and specificity.\n",
        "\n",
        "\n",
        "Use Domain-Specific Stop Words:\n",
        "Problem: Some terms like \"member,\" \"paper,\" \"presentation,\" and \"report\" may be less informative because they are common in many contexts.\n",
        "Solution: Create a custom stop word list that includes domain-specific terms that you deem too generic or uninformative."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "PKD21XmybST3",
      "metadata": {
        "id": "PKD21XmybST3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7dbacae5-7f6e-4cec-d7eb-80e781d93b85"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py:406: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['plenary'] not in stop_words.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TF-IDF Matrix Shape: (6068, 50)\n",
            "   accident_compensation  actuary_summit  address  analytics  \\\n",
            "0                    0.0             0.0      0.0        0.0   \n",
            "1                    0.0             0.0      0.0        0.0   \n",
            "2                    0.0             0.0      0.0        0.0   \n",
            "3                    0.0             0.0      0.0        0.0   \n",
            "4                    0.0             0.0      0.0        0.0   \n",
            "\n",
            "   biennial_convention  capital  claim  climate  committee   content  ...  \\\n",
            "0                  0.0      0.0    0.0      0.0   0.356354  0.765612  ...   \n",
            "1                  0.0      0.0    0.0      0.0   0.000000  0.000000  ...   \n",
            "2                  0.0      0.0    0.0      0.0   0.000000  0.000000  ...   \n",
            "3                  0.0      0.0    0.0      0.0   0.000000  0.000000  ...   \n",
            "4                  0.0      0.0    0.0      0.0   0.000000  0.000000  ...   \n",
            "\n",
            "   profession   provide    recent  retirement_income  role  scheme  session  \\\n",
            "0         0.0  0.381272  0.376141                0.0   0.0     0.0      0.0   \n",
            "1         0.0  0.000000  0.000000                0.0   0.0     0.0      0.0   \n",
            "2         0.0  0.000000  0.000000                0.0   0.0     0.0      0.0   \n",
            "3         0.0  0.000000  0.000000                0.0   0.0     0.0      0.0   \n",
            "4         0.0  0.000000  0.000000                0.0   0.0     0.0      0.0   \n",
            "\n",
            "   strategy  superannuation  young_actuary  \n",
            "0       0.0             0.0            0.0  \n",
            "1       0.0             0.0            0.0  \n",
            "2       0.0             0.0            0.0  \n",
            "3       0.0             0.0            0.0  \n",
            "4       0.0             0.0            0.0  \n",
            "\n",
            "[5 rows x 50 columns]\n",
            "Selected Features (Terms/Phrases) in TF-IDF Vectorization:\n",
            "1: accident_compensation\n",
            "2: actuary_summit\n",
            "3: address\n",
            "4: analytics\n",
            "5: biennial_convention\n",
            "6: capital\n",
            "7: claim\n",
            "8: climate\n",
            "9: committee\n",
            "10: content\n",
            "11: develop\n",
            "12: development\n",
            "13: education\n",
            "14: enterprise_management\n",
            "15: event\n",
            "16: experience\n",
            "17: fia\n",
            "18: financial\n",
            "19: financial_service\n",
            "20: focus\n",
            "21: general_insurance\n",
            "22: government\n",
            "23: health\n",
            "24: iaca_colloquium\n",
            "25: injury_scheme\n",
            "26: insurance\n",
            "27: insurer\n",
            "28: investment\n",
            "29: life\n",
            "30: life_insurance\n",
            "31: look\n",
            "32: make\n",
            "33: management\n",
            "34: model\n",
            "35: opportunity\n",
            "36: perspective\n",
            "37: present\n",
            "38: pricing\n",
            "39: private_health\n",
            "40: product\n",
            "41: profession\n",
            "42: provide\n",
            "43: recent\n",
            "44: retirement_income\n",
            "45: role\n",
            "46: scheme\n",
            "47: session\n",
            "48: strategy\n",
            "49: superannuation\n",
            "50: young_actuary\n"
          ]
        }
      ],
      "source": [
        "custom_stop_words = set(['member', 'paper', 'presentation', 'report', 'new',\n",
        "                         'change', ' plenary','need','impact', 'issue','actuary','actuarial',\n",
        "                          'approach', 'astin_afirerm','description_provided','discus', 'discussion','program','review',\n",
        "                         'australia','australian','institute','actuary_australia','may','insight','industry',\n",
        "                         'actuary_institute','concurrent','risk','seminar','system','update' ,'work','year','time'])\n",
        "\n",
        "tfidf_vectorizer = TfidfVectorizer(\n",
        "    max_features=50,\n",
        "    min_df=3,\n",
        "    max_df=0.5,\n",
        "    ngram_range=(1, 4),  # Use unigrams and bigrams\n",
        "    stop_words=list(custom_stop_words.union(set('english'))) # Convert the set of stop words to a list\n",
        ")\n",
        "\n",
        "\n",
        "# Perform TF-IDF vectorization on the bigram-processed descriptions\n",
        "tfidf_matrix_2 = tfidf_vectorizer.fit_transform(docs_train_cleaned_desc['Bigram_Description'])\n",
        "\n",
        "# Convert the TF-IDF matrix to a DataFrame for easier inspection\n",
        "tfidf_df_1 = pd.DataFrame(tfidf_matrix_2.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "\n",
        "# Display the shape of the TF-IDF matrix and the first few rows\n",
        "print(f\"TF-IDF Matrix Shape: {tfidf_df_1.shape}\")\n",
        "print(tfidf_df_1.head())\n",
        "\n",
        "# Display the features (terms/phrases) selected in the TF-IDF vectorizer\n",
        "selected_features = tfidf_vectorizer.get_feature_names_out()\n",
        "\n",
        "# Print out the 60 selected features\n",
        "print(\"Selected Features (Terms/Phrases) in TF-IDF Vectorization:\")\n",
        "for idx, feature in enumerate(selected_features):\n",
        "    print(f\"{idx + 1}: {feature}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "aW24-ggaqGNn",
      "metadata": {
        "id": "aW24-ggaqGNn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "431015ba-d78b-41b2-ad47-b0f2a4ac02d3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.5122173599741022\n",
            "Matrix Sparsity (Mean Value): 0.019968421625219113\n"
          ]
        }
      ],
      "source": [
        "# Example usage with your TF-IDF matrix\n",
        "average_cosine_similarity, mean_value = evaluate_vectorization(tfidf_matrix_2)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eIBqNKT3qeWf",
      "metadata": {
        "id": "eIBqNKT3qeWf"
      },
      "source": [
        "Matrix Sparsity (Mean Value): 0.0191\n",
        "What It Indicates:\n",
        "\n",
        "The matrix sparsity, with a mean value of 0.0191, suggests that the TF-IDF matrix is quite sparse. This means that most of the elements in the matrix are zero, indicating that most terms do not appear in most documents.\n",
        "A mean value of 0.0191 implies that, on average, about 1.91% of the terms in any given document are non-zero.\n",
        "\n",
        "Average Cosine Similarity: 0.4285\n",
        "What It Indicates:\n",
        "The average cosine similarity score of 0.4285 suggests that, on average, the documents are somewhat similar to each other.\n",
        "A value around 0.4 indicates moderate similarity, meaning that while there are shared features among the documents, they are not overwhelmingly similar. This level of similarity could be expected in a collection of documents that share some common topics or themes but still have distinct content.\n",
        "Interpretation:\n",
        "If your goal is to group similar documents together (e.g., in clustering), this level of similarity might be appropriate, especially if the documents are supposed to be related but not identical.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "N2EI9zDsynuu",
      "metadata": {
        "id": "N2EI9zDsynuu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d579034f-b331-46e4-d313-8adc7cbc80e4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cosine Similarity between description 1 and 2: 0.0000\n"
          ]
        }
      ],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import numpy as np\n",
        "\n",
        "# Example: Select specific descriptions to compare\n",
        "index1 = 1  # Replace with your specific row index\n",
        "index2 = 2  # Replace with your specific row index\n",
        "\n",
        "# Assuming 'tfidf_matrix' from the previous code block is your TF-IDF matrix\n",
        "tfidf_train_final = tfidf_matrix_2\n",
        "\n",
        "# Extract the vector for the chosen descriptions\n",
        "vec1 = tfidf_train_final[index1].toarray()\n",
        "vec2 = tfidf_train_final[index2].toarray()\n",
        "\n",
        "# Calculate the cosine similarity between the two selected descriptions\n",
        "cosine_sim = cosine_similarity(vec1, vec2)\n",
        "print(f\"Cosine Similarity between description {index1} and {index2}: {cosine_sim[0][0]:.4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xJZIcTewz7a0",
      "metadata": {
        "id": "xJZIcTewz7a0"
      },
      "source": [
        "The average Cosine Similary is close to zero,\n",
        "\n",
        "Descriptions are largely dissimilar, which could mean that the vectorization isn’t capturing enough meaningful features or that the dataset is highly diverse.\n",
        "\n",
        "Action:\n",
        "**Adjust TF-IDF Parameters:**\n",
        "\n",
        "Increase max_df: If you lowered max_df to filter out very common words, consider raising it slightly to allow more shared terms between descriptions.\n",
        "\n",
        "Decrease min_df: Lowering min_df might allow more rare but meaningful terms to be included in the vectorization, which could increase similarity.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7iQgBOQMiHPT",
      "metadata": {
        "id": "7iQgBOQMiHPT"
      },
      "source": [
        "##### Vectorisation BERT\n",
        "\n",
        "---------------------------------------------------------------------------\n",
        "This part is no longer Relevence because we end up not using BERT in cluster feaure\n",
        "\n",
        "------------------------------------------------------------------------------\n",
        "\n",
        "\n",
        "bert does not require data split because the it work is based on large mode\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VYN4udtbiHPT",
      "metadata": {
        "id": "VYN4udtbiHPT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d477d108-5ba3-46a1-91ea-e9c3521d32a9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 30 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    7550 non-null   int64         \n",
            " 1   FileName                 7550 non-null   object        \n",
            " 2   FileType                 7550 non-null   object        \n",
            " 3   FileSize                 7550 non-null   int64         \n",
            " 4   Title                    7550 non-null   object        \n",
            " 5   StartPublishing          7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing           7550 non-null   object        \n",
            " 7   Cpd                      7550 non-null   int64         \n",
            " 8   Format                   7550 non-null   object        \n",
            " 9   Level                    7550 non-null   object        \n",
            " 10  Description              7550 non-null   object        \n",
            " 11  Region                   7550 non-null   object        \n",
            " 12  Copyright                7550 non-null   object        \n",
            " 13  EventAgeGroup            6840 non-null   object        \n",
            " 14  EventType                6947 non-null   object        \n",
            " 15  Link                     7550 non-null   object        \n",
            " 16  Tags                     6969 non-null   object        \n",
            " 17  DownloadCount            7550 non-null   int64         \n",
            " 18  DurationOnDMS            7550 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  7550 non-null   float64       \n",
            " 20  YearOfStartPublishing    7550 non-null   int32         \n",
            " 21  Year                     7550 non-null   int32         \n",
            " 22  Tag_only                 6969 non-null   object        \n",
            " 23  Tag_only_common          6969 non-null   object        \n",
            " 24  StopPublishing_Year      6068 non-null   float64       \n",
            " 25  Description_Length       6068 non-null   float64       \n",
            " 26  Number of Downloads      6068 non-null   float64       \n",
            " 27  Tokens                   7550 non-null   object        \n",
            " 28  Cleaned_Description      7550 non-null   object        \n",
            " 29  Bigram_Description       7550 non-null   object        \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(4), object(18)\n",
            "memory usage: 1.7+ MB\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me a combined_set, with is joined from docs_train_cleaned, docs_val_cleaned and docs_test_cleaned\n",
        "\n",
        "import pandas as pd\n",
        "combined_set = pd.concat([docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc], ignore_index=True)\n",
        "\n",
        "combined_set.info()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SWzxwtJQiHPb",
      "metadata": {
        "id": "SWzxwtJQiHPb"
      },
      "source": [
        "BERT is quite robust and doesn't require extensive cleaning like traditional NLP models. However, some minimal preprocessing might be beneficial depending on your data and task.\n",
        "\n",
        "Here are some considerations for BERT preprocessing:\n",
        "\n",
        "Handle special characters:\n",
        "You generally don't need to remove punctuation or special characters as BERT can learn from them.\n",
        "However, if your text contains excessive noise or unusual characters, cleaning might help.\n",
        "Lowercasing:\n",
        "Lowercasing can be helpful, especially if your task is not case-sensitive (e.g., sentiment analysis).\n",
        "For tasks where case is important (e.g., named entity recognition), avoid lowercasing.\n",
        "Tokenization:\n",
        "BERT has its own tokenizer (like WordPiece), so use the appropriate one for your BERT model. Don't rely on standard tokenizers.\n",
        "Important Note: Avoid common practices like stemming, lemmatization, and stop word removal with BERT. These can actually harm performance as BERT relies on the original word forms and context.\n",
        "\n",
        "In summary: Minimal cleaning is generally sufficient for BERT. Focus on handling noisy characters and consider lowercasing if appropriate. Use the correct BERT tokenizer and avoid traditional text cleaning techniques."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8XruaYvAiHPb",
      "metadata": {
        "id": "8XruaYvAiHPb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "65e77515-74d0-413c-d86b-60cd026a6080"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                         Description  \\\n",
            "0  there are over  committees subcommittees task ...   \n",
            "1  the report is based on data in respect of indi...   \n",
            "2  inadequacy of liasb mortality reserves allen t...   \n",
            "3  analysis of surplus mixing realism with conser...   \n",
            "4  the  report into morbidity for insured and oth...   \n",
            "\n",
            "                             basic_clean_description  \n",
            "0  there are over  committees subcommittees task ...  \n",
            "1  the report is based on data in respect of indi...  \n",
            "2  inadequacy of liasb mortality reserves allen t...  \n",
            "3  analysis of surplus mixing realism with conser...  \n",
            "4  the  report into morbidity for insured and oth...  \n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        }
      ],
      "source": [
        "# prompt: for combined_set, do a new cloumn that is basic_clean_description. this column only perform cleaning that enoguh for BERT. and this should come from descroption column\n",
        "\n",
        "# Ensure necessary NLTK data is downloaded\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')\n",
        "nltk.download('wordnet')\n",
        "\n",
        "def basic_clean_descriptions_for_bert(df, column_name):\n",
        "    # Step 1: Handle Missing Descriptions by inserting \"No Description Provided\"\n",
        "    df[column_name] = df[column_name].fillna(\"No Description Provided\")\n",
        "\n",
        "    # Step 2: Fix Quote Spillage\n",
        "    df[column_name] = df[column_name].apply(lambda x: x.replace('\"', '').replace(\"'\", \"\"))\n",
        "\n",
        "    return df\n",
        "\n",
        "# Apply the basic cleaning function to the combined set\n",
        "combined_set = basic_clean_descriptions_for_bert(combined_set.copy(), 'Description')\n",
        "combined_set['basic_clean_description'] = combined_set['Description']\n",
        "\n",
        "# Now both docs_train_cleaned and docs_val_cleaned have the 'Cleaned_Description' column\n",
        "print(combined_set[['Description', 'basic_clean_description']].head())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WXdWE2PgiHPb",
      "metadata": {
        "id": "WXdWE2PgiHPb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b519747b-a885-409b-e6f6-2e0223c3914e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 31 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    7550 non-null   int64         \n",
            " 1   FileName                 7550 non-null   object        \n",
            " 2   FileType                 7550 non-null   object        \n",
            " 3   FileSize                 7550 non-null   int64         \n",
            " 4   Title                    7550 non-null   object        \n",
            " 5   StartPublishing          7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing           7550 non-null   object        \n",
            " 7   Cpd                      7550 non-null   int64         \n",
            " 8   Format                   7550 non-null   object        \n",
            " 9   Level                    7550 non-null   object        \n",
            " 10  Description              7550 non-null   object        \n",
            " 11  Region                   7550 non-null   object        \n",
            " 12  Copyright                7550 non-null   object        \n",
            " 13  EventAgeGroup            6840 non-null   object        \n",
            " 14  EventType                6947 non-null   object        \n",
            " 15  Link                     7550 non-null   object        \n",
            " 16  Tags                     6969 non-null   object        \n",
            " 17  DownloadCount            7550 non-null   int64         \n",
            " 18  DurationOnDMS            7550 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  7550 non-null   float64       \n",
            " 20  YearOfStartPublishing    7550 non-null   int32         \n",
            " 21  Year                     7550 non-null   int32         \n",
            " 22  Tag_only                 6969 non-null   object        \n",
            " 23  Tag_only_common          6969 non-null   object        \n",
            " 24  StopPublishing_Year      6068 non-null   float64       \n",
            " 25  Description_Length       6068 non-null   float64       \n",
            " 26  Number of Downloads      6068 non-null   float64       \n",
            " 27  Tokens                   7550 non-null   object        \n",
            " 28  Cleaned_Description      7550 non-null   object        \n",
            " 29  Bigram_Description       7550 non-null   object        \n",
            " 30  basic_clean_description  7550 non-null   object        \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(4), object(19)\n",
            "memory usage: 1.7+ MB\n"
          ]
        }
      ],
      "source": [
        "combined_set.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "hzZm764ViHPb",
      "metadata": {
        "id": "hzZm764ViHPb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "51e1c245-f7e6-42a3-fe28-b352d5765618"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.44.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.24.7)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.5)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (2024.6.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30)\n"
          ]
        }
      ],
      "source": [
        "# prompt: i have the basic_clearn_descroption, use it as an input of BERT. I know it will take time to run, save the BERT output into my colbab gogle drive, and include me codes to recall it later\n",
        "\n",
        "!pip install transformers\n",
        "\n",
        "from transformers import BertTokenizer, BertModel\n",
        "import torch\n",
        "from google.colab import drive\n",
        "import pickle\n",
        "\n",
        "# Mount your Google Drive\n",
        "#drive.mount('/content/drive')\n",
        "\n",
        "# Path to save BERT embeddings\n",
        "#bert_embeddings_path = '/content/drive/MyDrive/bert_embeddings.pkl'\n",
        "\n",
        "# Load pre-trained BERT model and tokenizer\n",
        "#model_name = 'bert-base-uncased'\n",
        "#tokenizer = BertTokenizer.from_pretrained(model_name)\n",
        "#model = BertModel.from_pretrained(model_name)\n",
        "\n",
        "#def get_bert_embeddings(text):\n",
        "#  \"\"\"Generates BERT embeddings for a given text.\"\"\"\n",
        "#  inputs = tokenizer(text, return_tensors=\"pt\", padding=True, truncation=True)\n",
        "#  outputs = model(**inputs)\n",
        "#  embeddings = outputs.last_hidden_state[:, 0, :].detach().numpy()  # Use [CLS] token embeddings\n",
        "#  return embeddings\n",
        "\n",
        "#try:\n",
        "  # Try to load existing embeddings from Google Drive\n",
        "#  with open(bert_embeddings_path, 'rb') as f:\n",
        "#    bert_embeddings = pickle.load(f)\n",
        "#  print(\"BERT embeddings loaded from Google Drive.\")\n",
        "\n",
        "#except FileNotFoundError:\n",
        "  # If embeddings not found, generate and save them\n",
        "#  bert_embeddings = {}\n",
        "#  for i, row in combined_set.iterrows():\n",
        "#    text = row['basic_clean_description']\n",
        "#    embeddings = get_bert_embeddings(text)\n",
        "#    bert_embeddings[i] = embeddings\n",
        "#    print(f\"Processed row {i}\")\n",
        "\n",
        "  # Save the embeddings to Google Drive\n",
        "#  with open(bert_embeddings_path, 'wb') as f:\n",
        "#    pickle.dump(bert_embeddings, f)\n",
        "#  print(\"BERT embeddings generated and saved to Google Drive.\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-LFaSnvNiHPb",
      "metadata": {
        "id": "-LFaSnvNiHPb"
      },
      "source": [
        "above codes it to generate Bert embddings , how as the codes take long time to run, we have # the code, and saved the output in colab dirve. and only record the rsult for the reamining anauslsus in Q2"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LJifLp3eiHPb",
      "metadata": {
        "id": "LJifLp3eiHPb"
      },
      "source": [
        "The following code extract Bert output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "nU7N55WsiHPc",
      "metadata": {
        "id": "nU7N55WsiHPc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1b19bd9a-fed9-45e5-8587-18b56085db0c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n",
            "BERT embeddings loaded successfully!\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me code to extract the bert output from mydrive ,bert_embeddings.pkl, call this bert_embeddings.pkl\n",
        "\n",
        "import pickle\n",
        "\n",
        "# Mount your Google Drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Path to your BERT embeddings file\n",
        "bert_embeddings_path = '/content/drive/MyDrive/bert_embeddings.pkl'\n",
        "\n",
        "try:\n",
        "  # Load BERT embeddings from Google Drive\n",
        "  with open(bert_embeddings_path, 'rb') as f:\n",
        "    bert_embeddings = pickle.load(f)\n",
        "  print(\"BERT embeddings loaded successfully!\")\n",
        "\n",
        "except FileNotFoundError:\n",
        "  print(f\"Error: BERT embeddings file not found at {bert_embeddings_path}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ZInlJDqCiHPc",
      "metadata": {
        "id": "ZInlJDqCiHPc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a379d77f-63bc-492c-a536-03e363ef8d5a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.7871415019035339\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x800 with 2 Axes>"
            ],
            "image/png": "[encoded data, 1403984 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 28500 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me codes to examine the performance of bert_embeddings, with suitable visulisation\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import numpy as np\n",
        "\n",
        "def examine_bert_embeddings(bert_embeddings):\n",
        "  \"\"\"Examines the performance of BERT embeddings using cosine similarity and visualization.\"\"\"\n",
        "\n",
        "  # Calculate cosine similarity between all pairs of embeddings\n",
        "  embedding_list = [bert_embeddings[i][0] for i in bert_embeddings]\n",
        "  cosine_sim_matrix = cosine_similarity(embedding_list)\n",
        "\n",
        "  # Calculate the average cosine similarity (excluding self-similarity)\n",
        "  average_cosine_similarity = np.mean(cosine_sim_matrix[cosine_sim_matrix != 1])\n",
        "  print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "\n",
        "  # Visualize the cosine similarity matrix\n",
        "  plt.figure(figsize=(10, 8))\n",
        "  sns.heatmap(cosine_sim_matrix, cmap='viridis', annot=False)\n",
        "  plt.title('Cosine Similarity Matrix of BERT Embeddings')\n",
        "  plt.xlabel('Document Index')\n",
        "  plt.ylabel('Document Index')\n",
        "  plt.show()\n",
        "\n",
        "  # Histogram of cosine similarities\n",
        "  plt.figure(figsize=(8, 6))\n",
        "  plt.hist(cosine_sim_matrix[cosine_sim_matrix != 1], bins=20)\n",
        "  plt.title('Distribution of Cosine Similarities')\n",
        "  plt.xlabel('Cosine Similarity')\n",
        "  plt.ylabel('Frequency')\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "# Example usage:\n",
        "examine_bert_embeddings(bert_embeddings)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Ca5Be0z9TsJ-",
      "metadata": {
        "id": "Ca5Be0z9TsJ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0a4bbad7-d867-4dfd-c003-c331fc2a361c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of BERT embedding array: (9032, 768)\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me an output for bert embdeing that can go into the cluster model later\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "# Assuming 'bert_embeddings' is your dictionary of BERT embeddings\n",
        "embedding_list = [bert_embeddings[i][0] for i in bert_embeddings]\n",
        "\n",
        "# Convert the list of embeddings to a NumPy array\n",
        "bert_embedding_array = np.array(embedding_list)\n",
        "\n",
        "print(f\"Shape of BERT embedding array: {bert_embedding_array.shape}\")\n",
        "\n",
        "# Now, 'bert_embedding_array' is ready to be used in your clustering model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Lf6WV7DmHba3",
      "metadata": {
        "id": "Lf6WV7DmHba3"
      },
      "source": [
        "Test"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dUhq1gL8enur",
      "metadata": {
        "id": "dUhq1gL8enur"
      },
      "source": [
        "##### TFIDF for Q2b on whole dataset\n",
        "---------------------------------------------------------\n",
        "This part of codes is importance because we use this to build our inital k-mean clustering model in Q2b\n",
        "\n",
        "--------------------------------------------------------\n",
        "\n",
        "\"In this section, we chose a low max_features value of 100 for TF-IDF because a higher value resulted in negative silhouette scores in K-means clustering, likely due to generating too many vectors that diminished clustering quality. As shown in part 2c, the Random Forest model identified only key TF-IDF features, like 'general insurance' and 'life insurance,' as important for clustering. Therefore, it's efficient to reduce the TF-IDF vectors and instead capture relevant information using binary features for each file.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LDxQWaraiM81",
      "metadata": {
        "id": "LDxQWaraiM81"
      },
      "source": [
        "This process enriches the text data by capturing meaningful bigram phrases, ensuring that important two-word combinations (e.g., \"general_insurance,\" \"life_insurance\") are treated as single entities in further analysis. This improves the effectiveness of downstream tasks, such as clustering or classification, by preserving important context in the text.\n",
        "\n",
        "**Rationale of selection:**\n",
        "- min_count=2: Ensures only bigrams appearing at least twice are included, capturing frequently relevant phrases.\n",
        "- threshold=5: Filters out less important bigrams; chosen to avoid noisy or rare combinations that could dilute clustering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "lcMIQfksenu4",
      "metadata": {
        "id": "lcMIQfksenu4"
      },
      "outputs": [],
      "source": [
        "#  'Tokens' column contains tokenized text data\n",
        "docs_cleaned_desc = pd.concat([docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc], ignore_index=True)\n",
        "\n",
        "# Detect bigrams\n",
        "phrases = Phrases(docs_cleaned_desc['Tokens'], min_count=2, threshold=5)\n",
        "bigram = Phraser(phrases)\n",
        "\n",
        "# Validation set\n",
        "# Apply the bigram model to the tokenized text\n",
        "docs_cleaned_desc['Bigram_Description'] = docs_cleaned_desc['Tokens'].apply(lambda x: bigram[x])\n",
        "\n",
        "# Join the bigrams back into a single string for TF-IDF vectorization\n",
        "docs_cleaned_desc['Bigram_Description'] = docs_cleaned_desc['Bigram_Description'].apply(lambda x: ' '.join(x))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "NrefTEEyiYc9",
      "metadata": {
        "id": "NrefTEEyiYc9"
      },
      "source": [
        "This code initializes the final TF-IDF vectorizer with specific parameters to transform text data into numerical features, capturing important terms and phrases while excluding irrelevant or overly common ones.\n",
        "\n",
        "Combines custom stop words with the default English stop words to filter out common, non-informative terms. The custom stop words list was tailored to remove terms irrelevant to clustering (e.g., 'member', 'report', 'industry'). This selection is done on our domaint knowledge.\n",
        "\n",
        "**Rationale of selection:**\n",
        "- max_features=100: Limits the TF-IDF matrix to the top 100 most important terms. This helps avoid overwhelming the model with too many features and focuses on the most relevant terms for better clustering or classification.\n",
        "- min_df = 10 : Includes only terms that appear in at least 10 documents. This ensures that less frequent (potentially irrelevant) terms are filtered out, improving the overall quality and consistency of the features.\n",
        "- ngram_range (1,4): Captures unigrams to 4-grams (single words to four-word phrases). This range is set to identify important phrases, ensuring that both individual words and meaningful multi-word combinations are included.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zywMO1Efenu4",
      "metadata": {
        "id": "zywMO1Efenu4"
      },
      "outputs": [],
      "source": [
        "# Initialize the final TF-IDF Vectorizer\n",
        "custom_stop_words = set(['member', 'paper', 'presentation', 'challenge','report', 'new',\n",
        "                         'change', ' plenary','need','impact', 'issue','actuary','actuarial',\n",
        "                          'approach', 'description_provided','discus', 'discussion','program','review',\n",
        "                         'australia','australian','institute','actuary_australia','may','insight','industry',\n",
        "                         'actuary_institute','concurrent','risk','seminar','system','update' ,'work','year',\n",
        "                         'time','data','future','forum','model','plenary','seminar_plenary',\n",
        "                         'first',\n",
        "                         'value','also','using','comment','young_actuary','analysis','based','biennial_convention',\n",
        "                         'customer','currnet','many','one','world',\n",
        "                         'provide','well','used','company','area','different','same','including','result','research',\n",
        "                         'seminar_concurrent','group','fund','business','outcome','practice','cost','use','product','market','policy','practice_committee'])\n",
        "\n",
        "\n",
        "# Initialize the final TF-IDF Vectorizer\n",
        "tfidf_vectorizer = TfidfVectorizer(\n",
        "    max_features=100,  # Adjust or remove this to explore more features\n",
        "    min_df=10,  # Lowered to include less frequent terms\n",
        "    max_df=0.3,  # Slightly increased to allow more common terms\n",
        "    ngram_range=(1, 4),  # Ensure it captures the desired phrases\n",
        "    stop_words=list(custom_stop_words.union(set('english'))),  # Convert the set of stop words to a list\n",
        "\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xfdv36-2jU4r",
      "metadata": {
        "id": "xfdv36-2jU4r"
      },
      "source": [
        "The following code perfrom TFIDF and the selected metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4r3PbFY-enu4",
      "metadata": {
        "id": "4r3PbFY-enu4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "284c47dc-cc66-4f7c-b673-77f8bf6c4fd3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.3511517618076747\n",
            "Matrix Sparsity (Mean Value): 0.012660192113833664\n"
          ]
        }
      ],
      "source": [
        "# Perform TF-IDF vectorization on the bigram-processed descriptions\n",
        "tfidf_matrix_All = tfidf_vectorizer.fit_transform(docs_cleaned_desc['Bigram_Description'])\n",
        "\n",
        "# Evaluate the vectorization (assuming you have a function evaluate_vectorization)\n",
        "average_cosine_similarity, mean_value = evaluate_vectorization(tfidf_matrix_All)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")\n",
        "\n",
        "# Convert the TF-IDF matrix to a DataFrame for easier inspection\n",
        "tfidf_df_all = pd.DataFrame(tfidf_matrix_All.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "\n",
        "# Display the features (terms/phrases) selected in the TF-IDF vectorizer\n",
        "selected_features = tfidf_vectorizer.get_feature_names_out()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check 1  : Average Cosine Similarity\n",
        "\n",
        "A cosine similarity of 0.351 indicates that the documents have moderate similarity. but there is still enough distinction between them. This is a positive sign because it means TF-IDF is differentiating the documents based on their content.\n",
        "\n",
        "\n",
        "Next Steps: This indicates improved differentiation between documents, so there is no need for further adjustments."
      ],
      "metadata": {
        "id": "WoixfZkeMvLO"
      },
      "id": "WoixfZkeMvLO"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check 2 : Matrix sapsity\n",
        "Matrix Sparsity A sparsity of 1.2% means that only a small fraction of the TF-IDF matrix contains non-zero values, which is typical for text data. This shows that TF-IDF is working as expected, focusing on the most relevant terms and ignoring irrelevant ones.\n",
        "\n",
        "No issues here; this indicates an efficient vectorization process with a good balance between detail and sparsity."
      ],
      "metadata": {
        "id": "R-15Pm2AMuzu"
      },
      "id": "R-15Pm2AMuzu"
    },
    {
      "cell_type": "markdown",
      "id": "qkUftkcAjbwD",
      "metadata": {
        "id": "qkUftkcAjbwD"
      },
      "source": [
        "PCA dimensional reduction\n",
        "\n",
        "This code applies Principal Component Analysis (PCA) to the TF-IDF matrix (tfidf_matrix_All) to reduce dimensionality and identify the most important components that explain the variance in the dataset. The plots illustrate how much variance each principal component explains, as well as the cumulative variance explained as more components are included."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VN8AXStjk3vZ",
      "metadata": {
        "id": "VN8AXStjk3vZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "outputId": "c7da5192-5670-44bd-b419-c58b8d276897"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 51476 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(tfidf_matrix_All)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eZNAnqTdenu4",
      "metadata": {
        "id": "eZNAnqTdenu4"
      },
      "source": [
        "Background Search for the most suitable hyper parameter.\n",
        "Working on seaching the most sutiable hyper paratmer"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "86nhYO8ZOpah",
      "metadata": {
        "id": "86nhYO8ZOpah"
      },
      "source": [
        "##### TFIDF for Q2b on whole dataset -with only max 60\n",
        "\n",
        "---------------------------------------------------------\n",
        "This part is also important becuase we use this part to build the imporved version of k-mean clustering model in Q2b\n",
        "\n",
        "\n",
        "---------------------------------------------------------\n",
        "\n",
        "\"In this section, we chose a low max_features value of 60 for TF-IDF because a higher value resulted in negative silhouette scores in K-means clustering, likely due to generating too many vectors that diminished clustering quality. As shown in part 2c, the Random Forest model identified only key TF-IDF features, like 'general insurance' and 'life insurance,' as important for clustering. Therefore, it's efficient to reduce the TF-IDF vectors and instead capture relevant information using binary features for each file.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "wtMS10IqOpan",
      "metadata": {
        "id": "wtMS10IqOpan"
      },
      "source": [
        "This process enriches the text data by capturing meaningful bigram phrases, ensuring that important two-word combinations (e.g., \"general_insurance,\" \"life_insurance\") are treated as single entities in further analysis. This improves the effectiveness of downstream tasks, such as clustering or classification, by preserving important context in the text.\n",
        "\n",
        "**Rationale of selection:**\n",
        "- min_count=2: Ensures only bigrams appearing at least twice are included, capturing frequently relevant phrases.\n",
        "- threshold=5: Filters out less important bigrams; chosen to avoid noisy or rare combinations that could dilute clustering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "tUENB5RSOpan",
      "metadata": {
        "id": "tUENB5RSOpan"
      },
      "outputs": [],
      "source": [
        "#  'Tokens' column contains tokenized text data\n",
        "docs_cleaned_desc = pd.concat([docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc], ignore_index=True)\n",
        "\n",
        "# Detect bigrams\n",
        "phrases = Phrases(docs_cleaned_desc['Tokens'], min_count=2, threshold=5)\n",
        "bigram = Phraser(phrases)\n",
        "\n",
        "# Validation set\n",
        "# Apply the bigram model to the tokenized text\n",
        "docs_cleaned_desc['Bigram_Description'] = docs_cleaned_desc['Tokens'].apply(lambda x: bigram[x])\n",
        "\n",
        "# Join the bigrams back into a single string for TF-IDF vectorization\n",
        "docs_cleaned_desc['Bigram_Description'] = docs_cleaned_desc['Bigram_Description'].apply(lambda x: ' '.join(x))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lx-OqLk7Opan",
      "metadata": {
        "id": "lx-OqLk7Opan"
      },
      "source": [
        "This code initializes the final TF-IDF vectorizer with specific parameters to transform text data into numerical features, capturing important terms and phrases while excluding irrelevant or overly common ones.\n",
        "\n",
        "Combines custom stop words with the default English stop words to filter out common, non-informative terms. The custom stop words list was tailored to remove terms irrelevant to clustering (e.g., 'member', 'report', 'industry'). This selection is done on our domaint knowledge.\n",
        "\n",
        "**Rationale of selection:**\n",
        "- max_features=100: Limits the TF-IDF matrix to the top 100 most important terms. This helps avoid overwhelming the model with too many features and focuses on the most relevant terms for better clustering or classification.\n",
        "- min_df = 10 : Includes only terms that appear in at least 10 documents. This ensures that less frequent (potentially irrelevant) terms are filtered out, improving the overall quality and consistency of the features.\n",
        "- ngram_range (1,4): Captures unigrams to 4-grams (single words to four-word phrases). This range is set to identify important phrases, ensuring that both individual words and meaningful multi-word combinations are included.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "W02s0n_rOpan",
      "metadata": {
        "id": "W02s0n_rOpan"
      },
      "outputs": [],
      "source": [
        "# Initialize the final TF-IDF Vectorizer\n",
        "custom_stop_words = set(['member', 'paper', 'presentation', 'challenge','report', 'new',\n",
        "                         'change', ' plenary','need','impact', 'issue','actuary','actuarial',\n",
        "                          'approach', 'description_provided','discus', 'discussion','program','review',\n",
        "                         'australia','australian','institute','actuary_australia','may','insight','industry',\n",
        "                         'actuary_institute','concurrent','risk','seminar','system','update' ,'work','year',\n",
        "                         'time','data','future','forum','model','plenary','seminar_plenary',\n",
        "                         'first', 'general_insurance','life_insurance', 'injury_scheme', 'actuary_summit','enterprise_management','biennial convention','accident compensation',\n",
        "                         'value','also','using','comment','young_actuary','analysis','based','biennial_convention',\n",
        "                         'customer','currnet','many','one','world',\n",
        "                         'provide','well','used','company','area','different','same','including','result','research',\n",
        "                         'seminar_concurrent','group','fund','business','outcome','practice','cost','use','product','market','policy','practice_committee'])\n",
        "\n",
        "\n",
        "# Initialize the final TF-IDF Vectorizer\n",
        "tfidf_vectorizer2 = TfidfVectorizer(\n",
        "    max_features=50,  # Adjust or remove this to explore more features\n",
        "    min_df=10,  # Lowered to include less frequent terms\n",
        "    max_df=0.3,  # Slightly increased to allow more common terms\n",
        "    ngram_range=(1, 4),  # Ensure it captures the desired phrases\n",
        "    stop_words=list(custom_stop_words.union(set('english'))),  # Convert the set of stop words to a list\n",
        "\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "62U4PzWzOpan",
      "metadata": {
        "id": "62U4PzWzOpan"
      },
      "source": [
        "The following code perfrom TFIDF and the selected metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "le-EWXGPOpan",
      "metadata": {
        "id": "le-EWXGPOpan",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "060518c3-d6d4-4f21-c86d-1932fd8ec94c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/sklearn/feature_extraction/text.py:406: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['accident', 'biennial', 'compensation', 'convention'] not in stop_words.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.4141388510437662\n",
            "Matrix Sparsity (Mean Value): 0.017264302277605434\n"
          ]
        }
      ],
      "source": [
        "# Perform TF-IDF vectorization on the bigram-processed descriptions\n",
        "tfidf_matrix_all_2 = tfidf_vectorizer2.fit_transform(docs_cleaned_desc['Bigram_Description'])\n",
        "\n",
        "# Evaluate the vectorization (assuming you have a function evaluate_vectorization)\n",
        "average_cosine_similarity2, mean_value = evaluate_vectorization(tfidf_matrix_all_2)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity2}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")\n",
        "\n",
        "# Convert the TF-IDF matrix to a DataFrame for easier inspection\n",
        "tfidf_df_all2 = pd.DataFrame(tfidf_matrix_all_2.toarray(), columns=tfidf_vectorizer2.get_feature_names_out())\n",
        "\n",
        "# Display the features (terms/phrases) selected in the TF-IDF vectorizer\n",
        "selected_features2 = tfidf_vectorizer2.get_feature_names_out()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check on  **Warning: Stop Words Inconsistency**\n",
        "\n",
        "What It Means: The warning suggests that there are some inconsistencies between the stop words you provided and the tokens generated during preprocessing. Specifically, words like \"accident,\" \"biennial,\" \"compensation,\" and \"convention\" were tokenized but not found in your stop words list.\n",
        "\n",
        "What Should Be Done:\n",
        "We should double-check the stop words list and preprocessing steps (e.g., lowercasing, stemming) to ensure they are properly aligned.\n",
        "\n",
        "Next Steps:\n",
        "We attempted to fix the issue, but the warning persists. Since only four stop words are affected, we believe this will not significantly impact the results, so we will proceed with the current setup."
      ],
      "metadata": {
        "id": "EQTS8iqxLcN-"
      },
      "id": "EQTS8iqxLcN-"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check 1. Cosine Similarity Check\n",
        "\n",
        "\n",
        "What Should Be Done:\n",
        "Interpretation: A cosine similarity of 0.414 across all documents suggests that the descriptions are moderately similar to each other, but there is still enough distinction between them. This is a positive sign because it means TF-IDF is differentiating the documents based on their content.\n",
        "\n",
        "This value is within a reasonable range, so no immediate action is needed\n"
      ],
      "metadata": {
        "id": "bliAYUxdMGSW"
      },
      "id": "bliAYUxdMGSW"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check 2:  Matrix Sparsity\n",
        "A sparsity of 1.7% means that only a small fraction of the TF-IDF matrix contains non-zero values, which is typical for text data. This shows that TF-IDF is working as expected, focusing on the most relevant terms and ignoring irrelevant ones.\n",
        "\n",
        "No issues here; this indicates an efficient vectorization process with a good balance between detail and sparsity."
      ],
      "metadata": {
        "id": "YaUN7y2HMfJs"
      },
      "id": "YaUN7y2HMfJs"
    },
    {
      "cell_type": "markdown",
      "id": "b4RxxJd7Opao",
      "metadata": {
        "id": "b4RxxJd7Opao"
      },
      "source": [
        "This code applies Principal Component Analysis (PCA) to the TF-IDF matrix (tfidf_matrix_All) to reduce dimensionality and identify the most important components that explain the variance in the dataset. The plots illustrate how much variance each principal component explains, as well as the cumulative variance explained as more components are included."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Two key check on TF_IDF"
      ],
      "metadata": {
        "id": "zvhPdY3LLGfG"
      },
      "id": "zvhPdY3LLGfG"
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ak1gcUCTLMKj"
      },
      "id": "ak1gcUCTLMKj"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "khKWbtFdOpao",
      "metadata": {
        "id": "khKWbtFdOpao",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "outputId": "c1e4ccb6-958f-48d7-a043-9b3d96a14061"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 51544 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(tfidf_matrix_all_2)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sn4tIlviOpao",
      "metadata": {
        "id": "sn4tIlviOpao"
      },
      "source": [
        "Background Search for the most suitable hyper parameter.\n",
        "Working on seaching the most sutiable hyper paratmer"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aOXDF_DLkFAF",
      "metadata": {
        "id": "aOXDF_DLkFAF"
      },
      "source": [
        "##### Word 2 Vec for Q2b on whole dataset\n",
        "\n",
        "**This part is no longer relevence to the whole project**\n",
        "\n",
        "**We End up not using Word 2 Vec in Q2b becuase TFIDF appear to be good enough**\n",
        "\n",
        "\n",
        "\n",
        "------------------------------------------------------------------\n",
        "\n",
        "\n",
        "**Word2Vec Implementation for Q2b on the Entire Dataset**\n",
        "\n",
        "This section prepares for Q2 clustering by applying Word2Vec to the entire dataset, as clustering requires comprehensive data coverage. The goal is to use Word2Vec embeddings, combined with domain knowledge of actuarial terms, to enhance the clustering process.\n",
        "\n",
        "By using Word2Vec, we create document embeddings that capture the semantic meaning of each description. This allows us to utilize K-Means clustering to group documents with similar themes more effectively.\n",
        "\n",
        "Why Word2Vec Is Relevant  Descriptions in Clustering\n",
        "\n",
        "**Capturing Semantic Relationships:**\n",
        "Word2Vec: Understands the context and relationships between words, capturing nuanced meanings. For example, terms like \"insurance,\" \"premium,\" and \"risk\" will have similar vector representations, reflecting their related meanings in actuarial contexts. This helps create clusters that are more meaningful and representative of the actual content."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1MHIRfqjkFAL",
      "metadata": {
        "id": "1MHIRfqjkFAL"
      },
      "source": [
        "Domain-Specific Knowledge Integration:\n",
        "\n",
        "Domain-specific terms related to actuarial fields (e.g., general insurance, life insurance, data science) were defined and added to the corpus. This step ensures that the Word2Vec model captures domain-relevant semantics, enhancing the quality of the embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "HqtiJ3FJkFAL",
      "metadata": {
        "id": "HqtiJ3FJkFAL"
      },
      "outputs": [],
      "source": [
        "# Grouped domain-specific terms\n",
        "domain_terms = {\n",
        "    'general_insurance': ['general insurance', 'reinsurance', 'catastrophe', 'IBNR', 'flood', 'storm', 'bushfire', 'motor', 'property', 'professional indemnity'],\n",
        "    'life_insurance': ['life insurance', 'mortality', 'annuity', 'morbidity', 'disability income', 'death', 'endowment', 'longevity'],\n",
        "    'health_disability': ['health insurance', 'disability insurance', 'NDIS', 'participant', 'care', 'modification', 'medical costs'],\n",
        "    'erm': ['ERM', 'enterprise risk management', 'governance', 'compliance', 'operational risk', 'regulatory risk', 'holistic', 'risk appetite'],\n",
        "    'data_science': ['data science', 'machine learning', 'predictive modeling', 'big data', 'analytics', 'statistical'],\n",
        "    'superannuation_wealth': ['superannuation', 'wealth management', 'retirement', 'pensions', 'investment', 'capital guarantee', 'investment objective', 'diversification', 'risk return', 'passive investment', 'active investment', 'retired']\n",
        "}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "hACuEKNNkFAL",
      "metadata": {
        "id": "hACuEKNNkFAL"
      },
      "source": [
        "We integrate domain-specific knowledge with the primary dataset to improve the quality of the Word2Vec model training.\n",
        "\n",
        "- Step 1: Get the Tokenized Sentences from the Main Dataset\n",
        "- Step 2: Prepare Domain-Specific Sentences\n",
        "- Step 3: Combine the Primary Dataset with Domain-Specific Sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "DRMa2v1TkFAL",
      "metadata": {
        "id": "DRMa2v1TkFAL"
      },
      "outputs": [],
      "source": [
        "# Step 1: Get the tokenized sentences from your main dataset\n",
        "primary_sentences = docs_cleaned_desc['Tokens'].tolist()\n",
        "\n",
        "# Step 2: Prepare domain-specific sentences\n",
        "domain_sentences = []\n",
        "for key, terms in domain_terms.items():\n",
        "    for term in terms:\n",
        "        term_tokens = term.split()\n",
        "        for _ in range(10):  # Adjust the number of repetitions as needed, less than before\n",
        "            domain_sentences.append(term_tokens)\n",
        "\n",
        "# Step 3: Combine the primary dataset with the domain-specific sentences\n",
        "combined_sentences = primary_sentences + domain_sentences\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Ra0zlUBikFAL",
      "metadata": {
        "id": "Ra0zlUBikFAL"
      },
      "source": [
        "Creating Document Vectors:\n",
        "\n",
        "For each document, an embedding was generated by averaging the Word2Vec vectors of the words within the document. This step converts the textual data into a numerical form that can be used for clustering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zv5ECkq8kFAL",
      "metadata": {
        "id": "zv5ECkq8kFAL"
      },
      "outputs": [],
      "source": [
        "from gensim.models import Word2Vec\n",
        "\n",
        "# Train Word2Vec model using the combined corpus\n",
        "model = Word2Vec(combined_sentences, vector_size=20, window=4, min_count=1, workers=4, sg=0)\n",
        "\n",
        "# Create document vectors by averaging word vectors\n",
        "def document_vector(tokens):\n",
        "    word_vecs = [model.wv[word] for word in tokens if word in model.wv]\n",
        "    return np.mean(word_vecs, axis=0) if word_vecs else np.zeros(model.vector_size)\n",
        "\n",
        "docs_cleaned_desc['doc_vector'] = docs_cleaned_desc['Tokens'].apply(document_vector)\n",
        "doc_matrix = np.vstack(docs_cleaned_desc['doc_vector'].values)\n",
        "doc_matrix = np.nan_to_num(doc_matrix)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XmselMFhkFAL",
      "metadata": {
        "id": "XmselMFhkFAL"
      },
      "source": [
        "performing Principal Component Analysis (PCA) to understand the variance captured by different principal components from the document vector matrix\n",
        "\n",
        "The key steps includes :\n",
        "1. Apply PCA to the Document Vector Matrix:\n",
        "2. Calculate Explained Variance for Each Principal Component:\n",
        "3. Calculate Cumulative Explained Variance:\n",
        "4. Plotting Explained Variance and Cumulative Explained Variance:\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "vFtVacztkFAL",
      "metadata": {
        "id": "vFtVacztkFAL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 607
        },
        "outputId": "d2733ffc-6152-4aae-be69-405906c616ca"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 49624 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(doc_matrix)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XrbzkeMskFAM",
      "metadata": {
        "id": "XrbzkeMskFAM"
      },
      "source": [
        "Explained Variance Plot (Left):\n",
        "\n",
        "The first principal component explains the highest amount of variance, around 30%, with the subsequent components contributing significantly less.\n",
        "This suggests that the majority of the information in your dataset can be captured by just a few principal components.\n",
        "Cumulative Explained Variance Plot (Right):\n",
        "\n",
        "The cumulative explained variance quickly approaches 100%, meaning that around 10 principal components capture most of the variance in the data.\n",
        "This indicates that you can effectively reduce the dimensionality of your data to 4-5 components while retaining most of the original information.\n",
        "\n",
        "**So, what does it mean**:\n",
        "we should use around 4 to 5 principal components. These components capture most of the variance in the data, ensuring you retain the essential information while reducing dimensionality. This range balances the trade-off between capturing the majority of the variance and avoiding unnecessary complexity.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mEHeEiu1kFAN",
      "metadata": {
        "id": "mEHeEiu1kFAN"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "id": "2Ffx8FYIkFAN",
      "metadata": {
        "id": "2Ffx8FYIkFAN"
      },
      "source": [
        "**Summary**  \n",
        "*Provide a **clear** and **concise** summary here of your conclusions based on the output of the clustering step undertaken above. Is the output what was expected? If not, why not?*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BkAnUyHlkFAN",
      "metadata": {
        "id": "BkAnUyHlkFAN"
      },
      "source": [
        "**Overall summary**  \n",
        "*Provide a **clear** and **concise** overall summary here of your conclusions from this section of the notebook.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Vs8EisR3kFAN",
      "metadata": {
        "id": "Vs8EisR3kFAN"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "####Q1b Step4 Final Check for Vectorisation\n",
        "\n",
        "As the TF-IDF vectorization with a max feature limit of 60 will serve as input for the later K-Means clustering, an additional check is necessary to ensure the vectorization aligns with the purpose. This final validation helps confirm that the TF-IDF matrix effectively represents the descriptions and will contribute to meaningful clustering results.\n",
        "\n",
        "Key Checks:\n",
        "1. Cosine Similarity Check:\n",
        "- Evaluate the average cosine similarity to ensure the descriptions remain distinct enough after the reduction to 60 features. Low similarity indicates that the vectorization is capturing meaningful differences between descriptions.\n",
        "\n",
        "2. Matrix Sparsity Check:\n",
        "- Ensure that the matrix sparsity remains high enough to filter out irrelevant terms but not so high that meaningful information is lost. This ensures the input to K-Means remains both efficient and informative.\n",
        "\n",
        "By performing these checks, the TF-IDF vectorization will be confirmed as suitable for input into the K-Means model, ensuring that it contributes to effective grouping in the later steps.\n"
      ],
      "metadata": {
        "id": "o33Dcel8JY1E"
      },
      "id": "o33Dcel8JY1E"
    },
    {
      "cell_type": "code",
      "source": [
        "# Perform TF-IDF vectorization on the bigram-processed descriptions\n",
        "tfidf_matrix_all_2 = tfidf_vectorizer2.fit_transform(docs_cleaned_desc['Bigram_Description'])\n",
        "\n",
        "# Evaluate the vectorization (assuming you have a function evaluate_vectorization)\n",
        "average_cosine_similarity2, mean_value = evaluate_vectorization(tfidf_matrix_all_2)\n",
        "\n",
        "# Display the results\n",
        "print(f\"Average Cosine Similarity across all documents: {average_cosine_similarity2}\")\n",
        "print(f\"Matrix Sparsity (Mean Value): {mean_value}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gl24MxZRJxoo",
        "outputId": "f7d8f42b-7530-475a-844e-615b58ed6c69"
      },
      "id": "Gl24MxZRJxoo",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Average Cosine Similarity across all documents: 0.4141388510437662\n",
            "Matrix Sparsity (Mean Value): 0.017264302277605434\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Interpretation of the Final Check Results for Vectorization:**\n",
        "\n",
        "1. Average Cosine Similarity: 0.414\n",
        "- What It Means: This indicates that the document vectors have a moderate level of similarity. While it's not extremely high, it suggests that there is some overlap between the descriptions, but they are still distinct enough to allow for meaningful differentiation.\n",
        "- For K-Means clustering, this level of similarity is acceptable. It means that the descriptions are not too similar to be indistinguishable, which is a positive outcome for the purpose of grouping the assets.\n",
        "\n",
        "\n",
        "2. Matrix Sparsity: 0.017 (1.72%)\n",
        "- What It Means: This means that only 1.72% of the matrix contains non-zero values, which is typical for a TF-IDF matrix. A sparse matrix like this ensures that only the most relevant terms are contributing to the vectorization, filtering out less important words.\n",
        "-  This level of sparsity is expected and indicates that the TF-IDF matrix is capturing relevant terms while maintaining efficiency. It’s important for the input to K-Means to be compact and focused, and this result confirms that the matrix is suitable for that."
      ],
      "metadata": {
        "id": "CRGDerguYwCX"
      },
      "id": "CRGDerguYwCX"
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Conclusion:**\n",
        "The final check confirms that the TF-IDF vectorization is appropriate for the later use in K-Means clustering. The moderate cosine similarity ensures that descriptions are distinct enough for clustering, and the high sparsity confirms that the matrix is efficiently capturing the most important features. This vectorization meets the needs of the next steps in the analysis."
      ],
      "metadata": {
        "id": "5Kc9hbSvZDBr"
      },
      "id": "5Kc9hbSvZDBr"
    },
    {
      "cell_type": "markdown",
      "id": "UA8v6oZgmciq",
      "metadata": {
        "id": "UA8v6oZgmciq"
      },
      "source": [
        "#### Summary for Q1b\n",
        "In this section, we conducted an initial EDA of the descriptions, followed by thorough data cleaning to prepare them for consistent TF-IDF vectorization across the training, validation, and testing datasets.\n",
        "\n",
        "We applied TF-IDF to the cleaned descriptions and examined key metrics such as sparsity and cosine similarity. Using PCA, we assessed how many vectors would be sufficient to capture the majority of the variance, ensuring an efficient representation of the data.\n",
        "\n",
        "To optimize the vectorization process, we experimented with different max_features values to balance usefulness and complexity. We also tested TF-IDF on bigrams to capture more meaningful phrases.\n",
        "\n",
        "Finally, we applied TF-IDF, BERT, and Word2Vec to the entire dataset in preparation for Q2 clustering, ensuring that our vectorization techniques are ready for the upcoming clustering analysis on the full dataset. This comprehensive approach allows us to identify the most effective vectorization method for capturing the nuances in actuarial descriptions."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Saving output to my Drive\n",
        "\n",
        "\"The following code saves our work to Colab Drive, allowing me to avoid re-running Q1 when working on later parts of the questions.\"\n",
        "\n",
        "\n",
        "**Note to Marker:**\n",
        "You might comment out this section because the purpose of doing this is to ensure I don't need to rerun everything, as Colab frequently disconnects."
      ],
      "metadata": {
        "id": "UJkcmeJoD6ew"
      },
      "id": "UJkcmeJoD6ew"
    },
    {
      "cell_type": "code",
      "source": [
        "# Code to save downloads_data to Google Drive\n",
        "from google.colab import drive\n",
        "import pickle\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Assuming 'downloads_data' is the variable you want to save\n",
        "file_path = '/content/drive/MyDrive/downloads_data.pkl'  # Specify the desired path\n",
        "\n",
        "with open(file_path, 'wb') as f:\n",
        "  pickle.dump(downloads_data, f)\n",
        "\n",
        "print(f\"downloads_data saved to {file_path}\")\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uLsG3vCMCked",
        "outputId": "53f702d8-c490-4f38-b9e0-0fff1f6c05cd"
      },
      "id": "uLsG3vCMCked",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "downloads_data saved to /content/drive/MyDrive/downloads_data.pkl\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: do me  code to save downloads_clean in my drive, and code to recall tgehis so all needed packa, two seperate section\n",
        "\n",
        "import pickle\n",
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Assuming 'downloads_clean' is your DataFrame\n",
        "def save_dataframe_to_drive(df, filename):\n",
        "  with open(f'/content/drive/MyDrive/{filename}.pkl', 'wb') as f:\n",
        "    pickle.dump(df, f)\n",
        "\n",
        "# Example usage:\n",
        "# Replace 'downloads_clean' with your actual DataFrame variable\n",
        "# Replace 'downloads_clean' with the desired filename in Google Drive\n",
        "save_dataframe_to_drive(downloads_clean, 'downloads_clean')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gq8djF7jJSUp",
        "outputId": "5cf881d2-a806-40ce-ea49-923893f5a06c"
      },
      "id": "Gq8djF7jJSUp",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: do me codes, 2 sperae, docs_data_cleaned, save this  to my colab my drive, and code to recall\n",
        "\n",
        "import pickle\n",
        "from google.colab import drive\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Assuming 'docs_data_cleaned' is the DataFrame you want to save\n",
        "\n",
        "# Save the DataFrame to your Google Drive\n",
        "def save_dataframe_to_drive(df, filename):\n",
        "  with open(f'/content/drive/MyDrive/{filename}.pkl', 'wb') as f:\n",
        "    pickle.dump(df, f)\n",
        "\n",
        "# Replace 'docs_data_cleaned' with the desired filename in Google Drive\n",
        "save_dataframe_to_drive(docs_data_cleaned, 'docs_data_cleaned')"
      ],
      "metadata": {
        "id": "8CXa7Kl5J3BN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d4b8dc0-10dc-4ee0-c8cc-cb4679fb9936"
      },
      "id": "8CXa7Kl5J3BN",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: do me code to save downloads_train_cleaned to colab my drive, also do me seperate code , inclduing package to recall them\n",
        "\n",
        "from google.colab import drive\n",
        "import pickle\n",
        "\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Save the DataFrames to your Google Drive\n",
        "def save_df_to_drive(df, filename):\n",
        "  with open(f'/content/drive/My Drive/{filename}.pkl', 'wb') as f:\n",
        "    pickle.dump(df, f)\n",
        "\n",
        "\n",
        "save_df_to_drive(downloads_train_cleaned, 'downloads_train_cleaned')\n",
        "\n",
        "\n",
        "print(\"DataFrames saved to Google Drive.\")"
      ],
      "metadata": {
        "id": "Q4r5UshTMKUB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "888c10ea-5b25-4917-fe63-99201851f25a"
      },
      "id": "Q4r5UshTMKUB",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "DataFrames saved to Google Drive.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# prompt: do me code to save docs_val_cleaned_desc, docs_train_cleaned_desc, and docs_train_cleaned_desc  into google drive, and then do me code how to recall them , in colab my drive, dont do csv\n",
        "\n",
        "from google.colab import drive\n",
        "import pickle\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "# Save DataFrames to Google Drive\n",
        "def save_dataframe_to_drive(df, filename):\n",
        "  with open(f'/content/drive/MyDrive/{filename}.pkl', 'wb') as f:\n",
        "    pickle.dump(df, f)\n",
        "\n",
        "save_dataframe_to_drive(docs_val_cleaned_desc, 'docs_val_cleaned_desc')\n",
        "save_dataframe_to_drive(docs_train_cleaned_desc, 'docs_train_cleaned_desc')\n",
        "save_dataframe_to_drive(docs_test_cleaned_desc, 'docs_test_cleaned_desc')\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "mRqiotdO82yg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e01f17c1-e11c-48a2-def2-9d2c597790a5"
      },
      "id": "mRqiotdO82yg",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "r2D2yMinUS_R",
      "metadata": {
        "id": "r2D2yMinUS_R"
      },
      "source": [
        "### Q1c - Summarise the dataset\n",
        "#### Purpose:\n",
        "We will discuss the key characteristics of the dataset\n",
        "\n",
        "#### References:\n",
        "*Outline any references you have used in this section of your notebook here.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_O-pIPbrUS_m",
      "metadata": {
        "id": "_O-pIPbrUS_m"
      },
      "source": [
        "#### Q1c Step 1\n",
        "**Outline**\n",
        "\n",
        "**Outline Business Goal:**\n",
        "The main purpose is to enhance member engagement on the CPD Hub by predicting which assets will become popular and grouping assets in a simpler, more effective way.\n",
        "\n",
        "**Download Patterns:**\n",
        "The CPD Hub hosts over 7,000 assets, but only a small fraction of these drive the majority of downloads. By accurately predicting and highlighting popular assets, engagement can significantly improve.\n",
        "\n",
        "**Key Observations:**\n",
        "- Many assets were uploaded in 2018 and 2019, likely due to the introduction of the CPD Hub.\n",
        "- Downloads surged in mid-2020 but have gradually declined since 2022, which may indicate reduced engagement.\n",
        "- Newer assets attract more attention, with downloads decreasing as assets age. Assets from 2020 remain particularly popular.\n",
        "- There is a monthly seasonal pattern in asset downloads, with lower downloads during school holidays.\n",
        "- Shifting Preferences: Asset formats, file types, and CPD point mix have changed over time, reflecting shifts in member preferences and content creation.\n",
        "\n",
        "**Tagging System:**\n",
        "The current tagging system is overly complex, with multiple overlapping tags, often related to actuarial fields like general insurance or life insurance. It doesn’t account for content recency, making it harder for members to assess relevance. This suggests that a simpler tagging approach would make navigation easier and boost engagement.\n",
        "\n",
        "**Description Challenges:**\n",
        "Descriptions are crucial for engagement, yet 7% of assets are missing them, and many descriptions are just copies of titles. Despite descriptions getting longer over time, engagement hasn’t improved. Improving the quality and informativeness of descriptions is essential.\n",
        "Additionally, the descriptions are well-structured text, meaning they do not contain emojis, misspellings, slang, or casual writing. This makes descriptions a suitable input for later predictive and classifier models.\n",
        "\n",
        "**Other Asset Characteristics: **\n",
        "- The majority of assets are Institute-owned and relevant across all member levels, suggesting that some data columns may not add much insight.\n",
        "- A number of assets have missing EventType, which we believe is due to some assets being regular publications and not related to a specific event.\n",
        "- The asset filenames are a combination of the file title and file type, indicating that the filename might not provide much additional insight.\n",
        "- Regarding CPD points, the majority of assets have points around 0, 1, or 2, though there are some high outliers. However, the number of these outliers appears to decrease across the more recent publication years.\n",
        "\n",
        "**Data Quality:**\n",
        "Several data issues were found, like missing fields, incorrect publishing dates, and duplicate records. Some assets had publishing dates that were earlier than their first download. These issues have been fixed.\n",
        "\n",
        "Additionally, many assets had missing or incorrect StopPublishing dates. Since the StopPublishing field was often left blank or had errors, it was assumed that these assets have no explicit expiration date.\n",
        "\n",
        "**Outlier:**\n",
        "We discovered one outlier—an oversized PowerPoint file containing many photos. Since it does not have actuarial content, we decided to exclude it from the analysis.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tXIkXc2hVZg1",
      "metadata": {
        "id": "tXIkXc2hVZg1"
      },
      "source": [
        "## Q2 - Choose a simpler, more effective way to group the CPD Hub's Assets"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "EKWXjyBPWBew",
      "metadata": {
        "id": "EKWXjyBPWBew"
      },
      "source": [
        "### Q2a - Suggest features for clustering\n",
        "#### Purpose:\n",
        "*Provide a **clear** and **concise** statement of the purpose of this section. (500 words or less)*\n",
        "\n",
        "#### References:\n",
        "*Outline any references you have used in this section of your notebook here.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e74eb619",
      "metadata": {
        "id": "e74eb619"
      },
      "source": [
        "**Defining the Business Problem and Objective:**\n",
        "\n",
        "The current tagging system in the CPD Hub is complicated, making it difficult for members to find relevant content. The goal is to create a simpler system that organizes assets into 6-8 clear clusters based on key business practice areas. This new clustering model will help the Institute categorize assets more intuitively, replacing the existing tagging system and improving the user experience.\n",
        "\n",
        "**Logic of Feature Selection:**\n",
        "We selected features that reflect members' professional needs and interests. The aim is for the clusters to align with major actuarial fields, learning preferences, and CPD requirements. To ensure the features are relevant and impactful, we built a random forest model using key features against the existing tagging system. This allowed us to assess feature importance and identify which features contribute the most to the current grouping structure. We then combined these results with our professional knowledge to make necessary adjustments, ensuring the selected features align with both data insights and actuarial business goals.\n",
        "\n",
        "**Rationale Behind Each Feature Selection:**\n",
        "\n",
        "Business Field Flags:\n",
        "- Members' interests vary by field of practice. Including key fields like General Insurance, Life Insurance, and Actuary Summits ensures that the clusters reflect the primary areas of actuarial work.\n",
        "\n",
        "Description:\n",
        "- Members often rely on descriptions to decide whether to download an asset. Descriptions provide critical insights into content, helping clusters guide members to relevant resources more efficiently.\n",
        "\n",
        "Recency:\n",
        "- Actuaries need to stay up to date for professional development. Including the recency of an asset helps capture trends, ensuring members find the most current and relevant content.\n",
        "\n",
        "Format:\n",
        "- Members have different content preferences, whether for documents, videos, or presentations. Including format ensures the clusters align with members’ preferred learning styles, making the system more user-friendly.\n",
        "\n",
        "Missing Description:\n",
        "- A binary indicator for missing descriptions ensures completeness. Since members depend on descriptions for understanding asset content, identifying missing descriptions helps improve asset selection.\n",
        "\n",
        "Event-Related Features:\n",
        "-  Many actuarial assets are tied to events. By categorizing assets as \"Non-Event Related,\" \"Major,\" or \"Other,\" clustering reflects members' interest in event-based content, which is crucial for those seeking insights from key industry events.\n",
        "\n",
        "CPD Points:\n",
        "- CPD points are essential for professional development, helping actuaries meet their 200-point requirement. Including this feature allows members to easily find high-CPD-point assets.\n",
        "\n",
        "Features Excluded:\n",
        "\n",
        "- Expiry Status: This feature is more about asset availability rather than relevance to content, making it less useful for clustering by business areas.\n",
        "\n",
        "- Level, Ownership, Region, Event Age Group: Since most assets fall under a single category (e.g., most are \"Institute\" owned), these features don't provide much differentiation between clusters and would add unnecessary complexity.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "15ababee",
      "metadata": {
        "id": "15ababee"
      },
      "source": [
        "### Q2b - Clustering\n",
        "#### Purpose:\n",
        "In this section, we will construct a clustering model as an alternative to the Institute's existing tagging system.\n",
        "\n",
        "outltine:\n",
        "1. Feature Importance with Random Forest:\n",
        "- Use Random Forest to identify key features for clustering.\n",
        "2. Feature Engineering:\n",
        " - Develop and refine features to improve cluster quality.\n",
        "3. Data Preparation:\n",
        "- Scale data and apply PCA for dimensionality reduction.\n",
        "4. Initial K-Means Trial:\n",
        "- Conduct initial clustering with K-Means and perform internal validation.\n",
        "5. Feature Selection Tuning:\n",
        "- Fine-tune feature selection for better clustering performance.\n",
        "6. K-Means Improvements:\n",
        "- Experiment with alternative features, varying hyperparameters (number of clusters, distance metrics).\n",
        "7. Alternative Vectorization Techniques:\n",
        "- Explore different vectorization techniques for improved clustering results.\n",
        "\n",
        "\n",
        "#### References:\n",
        "Codes in the sections were generated using ChatGPT and Gemnini\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "txOrBQZurA9w",
      "metadata": {
        "id": "txOrBQZurA9w"
      },
      "source": [
        "#### Q2b step 1 Random forest for selecting features\n",
        "\n",
        "**Purpose:**\n",
        "We will build a single random forest model using key features or columns against the existing tagging system to understand how the features relate to the current groups. This will provide insights for creating features for Q2a (yes, we did this part before Q2a).\n",
        "\n",
        "**Why Random Forest?**\n",
        "Random forest is a flexible model that doesn’t require extensive data cleaning or type conversion. Additionally, it provides feature importance, which is exactly what we need for this analysis."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "import pickle\n",
        "\n",
        "drive.mount('/content/drive')\n",
        "# Recall DataFrames from Google Drive\n",
        "def load_dataframe_from_drive(filename):\n",
        "  with open(f'/content/drive/MyDrive/{filename}.pkl', 'rb') as f:\n",
        "    df = pickle.load(f)\n",
        "  return df\n",
        "\n",
        "# Example of how to recall the DataFrames\n",
        "docs_val_cleaned_desc = load_dataframe_from_drive('docs_val_cleaned_desc')\n",
        "docs_train_cleaned_desc = load_dataframe_from_drive('docs_train_cleaned_desc')\n",
        "docs_test_cleaned_desc = load_dataframe_from_drive('docs_test_cleaned_desc')"
      ],
      "metadata": {
        "id": "ynQpFa3j9Xgq"
      },
      "id": "ynQpFa3j9Xgq",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Take a look to understand the strucutre of the dataset"
      ],
      "metadata": {
        "id": "rYdd-2hDgUvN"
      },
      "id": "rYdd-2hDgUvN"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "k6tZfr5KrEO-",
      "metadata": {
        "id": "k6tZfr5KrEO-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31cdf9f5-14c8-4f9b-b6e3-8e8ce098c046"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 6068 entries, 0 to 6067\n",
            "Data columns (total 30 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    6068 non-null   int64         \n",
            " 1   FileName                 6068 non-null   object        \n",
            " 2   FileType                 6068 non-null   object        \n",
            " 3   FileSize                 6068 non-null   int64         \n",
            " 4   Title                    6068 non-null   object        \n",
            " 5   StartPublishing          6068 non-null   datetime64[ns]\n",
            " 6   StopPublishing           6068 non-null   datetime64[ns]\n",
            " 7   Cpd                      6068 non-null   int64         \n",
            " 8   Format                   6068 non-null   object        \n",
            " 9   Level                    6068 non-null   object        \n",
            " 10  Description              6068 non-null   object        \n",
            " 11  Region                   6068 non-null   object        \n",
            " 12  Copyright                6068 non-null   object        \n",
            " 13  EventAgeGroup            6068 non-null   object        \n",
            " 14  EventType                6068 non-null   object        \n",
            " 15  Link                     6068 non-null   object        \n",
            " 16  Tags                     5884 non-null   object        \n",
            " 17  DownloadCount            6068 non-null   int64         \n",
            " 18  DurationOnDMS            6068 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  6068 non-null   float64       \n",
            " 20  YearOfStartPublishing    6068 non-null   int32         \n",
            " 21  Year                     6068 non-null   int32         \n",
            " 22  Tag_only                 5884 non-null   object        \n",
            " 23  Tag_only_common          5884 non-null   object        \n",
            " 24  StopPublishing_Year      6068 non-null   int32         \n",
            " 25  Description_Length       6068 non-null   int64         \n",
            " 26  Number of Downloads      6068 non-null   float64       \n",
            " 27  Tokens                   6068 non-null   object        \n",
            " 28  Cleaned_Description      6068 non-null   object        \n",
            " 29  Bigram_Description       6068 non-null   object        \n",
            "dtypes: datetime64[ns](2), float64(3), int32(3), int64(5), object(17)\n",
            "memory usage: 1.3+ MB\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "#  you have three DataFrames: docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc\n",
        "docs_train_cleaned_desc_trail=docs_train_cleaned_desc.copy()\n",
        "\n",
        "# Concatenate the DataFrames into one combined DataFrame\n",
        "\n",
        "\n",
        "docs_train_cleaned_desc_trail.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We take a look of the Tag_only_common as we will use it as our response variable in the randon forest."
      ],
      "metadata": {
        "id": "UBu8-3uvgaro"
      },
      "id": "UBu8-3uvgaro"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ToK5ihN1sEQS",
      "metadata": {
        "id": "ToK5ihN1sEQS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45a4d6dd-3801-4770-8a24-9ae19a9a08bb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag_only_common Value Counts:\n",
            " Tag_only_common\n",
            "general insurance                          1886\n",
            "superannuation                              598\n",
            "life insurance                              581\n",
            "health                                      527\n",
            "personal effectiveness &amp; leadership     497\n",
            "professional updates                        435\n",
            "risk management                             373\n",
            "banking                                     306\n",
            "data science                                286\n",
            "business acumen                             202\n",
            "wealth management                           184\n",
            "professionalism training                      7\n",
            "not applicable                                2\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 65012 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Top 10 most frequent tags:\n",
            "Tag_only_common\n",
            "general insurance                          1886\n",
            "superannuation                              598\n",
            "life insurance                              581\n",
            "health                                      527\n",
            "personal effectiveness &amp; leadership     497\n",
            "professional updates                        435\n",
            "risk management                             373\n",
            "banking                                     306\n",
            "data science                                286\n",
            "business acumen                             202\n",
            "Name: count, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "# EDA on Tag_only_common\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Check the distribution of Tag_only_common\n",
        "tag_counts = docs_train_cleaned_desc_trail['Tag_only_common'].value_counts()\n",
        "print(\"Tag_only_common Value Counts:\\n\", tag_counts)\n",
        "\n",
        "# Visualize the distribution\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=docs_train_cleaned_desc_trail)\n",
        "plt.title('Distribution of Tag_only_common')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=90)  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "# Analyze the most frequent tags\n",
        "print(\"\\nTop 10 most frequent tags:\")\n",
        "print(tag_counts.head(10))\n",
        "\n",
        "# Explore the relationship between Tag_only_common and other features (e.g., Description length, Format)\n",
        "# You can use groupby and aggregation functions to do this.\n",
        "# For example, to see the average description length for each tag:\n",
        "# average_description_length_by_tag = docs_cleaned_desc.groupby('Tag_only_common')['Description_Length'].mean()\n",
        "# print(\"\\nAverage Description Length by Tag:\\n\", average_description_length_by_tag)\n",
        "\n",
        "# You can further investigate the relationship between 'Tag_only_common' and other variables\n",
        "# by creating scatter plots, box plots, or heatmaps.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that business fields are the key distinguishing factor under the current tagging system."
      ],
      "metadata": {
        "id": "wcdb3xGdgj4N"
      },
      "id": "wcdb3xGdgj4N"
    },
    {
      "cell_type": "markdown",
      "source": [
        "We want to group the fields into the most common 8 group."
      ],
      "metadata": {
        "id": "Bj9-NWokgve6"
      },
      "id": "Bj9-NWokgve6"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "z-pwG66ItJSM",
      "metadata": {
        "id": "z-pwG66ItJSM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4220f5c1-c5c6-4a04-ada9-97c01cc4e754"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tag_only_common_8 Value Counts:\n",
            " Tag_only_common_8\n",
            "general insurance                          1886\n",
            "Other                                       865\n",
            "superannuation                              598\n",
            "life insurance                              581\n",
            "health                                      527\n",
            "personal effectiveness &amp; leadership     497\n",
            "professional updates                        435\n",
            "risk management                             373\n",
            "banking                                     306\n",
            "Name: count, dtype: int64\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me avariable is Tag_only_common_8 on docs_train_cleaned_desc_trail, keep the same name for the top 8 frequency , rest (including missing) all called \"Other\" , do me a summary to see the freenct tags for this new variable\n",
        "\n",
        "# Get the top 8 most frequent tags\n",
        "top_8_tags = docs_train_cleaned_desc_trail['Tag_only_common'].value_counts().head(8).index.tolist()\n",
        "\n",
        "# Create a new variable 'Tag_only_common_8'\n",
        "docs_train_cleaned_desc_trail['Tag_only_common_8'] = docs_train_cleaned_desc_trail['Tag_only_common'].apply(lambda x: x if x in top_8_tags else 'Other')\n",
        "\n",
        "# Summary of the new variable\n",
        "tag_counts_8 = docs_train_cleaned_desc_trail['Tag_only_common_8'].value_counts()\n",
        "print(\"Tag_only_common_8 Value Counts:\\n\", tag_counts_8)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, we can run the random forest model with Tag_only_common_8 as the response variable. Our aim is not to build a perfect model, but to focus on examining feature importance."
      ],
      "metadata": {
        "id": "8L71vuOQg28u"
      },
      "id": "8L71vuOQg28u"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54nD99DUtrnd",
      "metadata": {
        "id": "54nD99DUtrnd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5469cce0-0485-489c-ed8b-de236634cc8b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-165-a52ccf9f06b2>:22: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  X[['FileType', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType']] = imputer.fit_transform(X[['FileType', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType']])\n",
            "<ipython-input-165-a52ccf9f06b2>:40: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  X['StartPublishing_year'] = pd.to_numeric(X['StartPublishing_year'], errors='coerce')\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 0.6087314662273476\n",
            "                                         precision    recall  f1-score   support\n",
            "\n",
            "                                  Other       0.47      0.56      0.51       172\n",
            "                                banking       0.41      0.39      0.40        61\n",
            "                      general insurance       0.69      0.76      0.72       380\n",
            "                                 health       0.82      0.59      0.69       120\n",
            "                         life insurance       0.54      0.53      0.54       105\n",
            "personal effectiveness &amp; leadership       0.52      0.45      0.48        99\n",
            "                   professional updates       0.62      0.80      0.70        87\n",
            "                        risk management       0.68      0.50      0.58        68\n",
            "                         superannuation       0.56      0.45      0.50       122\n",
            "\n",
            "                               accuracy                           0.61      1214\n",
            "                              macro avg       0.59      0.56      0.57      1214\n",
            "                           weighted avg       0.61      0.61      0.61      1214\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me a tree model, using tag_only_common_8 as y,  for x variable, use FileType, FileSize, StartPublishing_year( this is a new feature creasted only using year) , Cpd, Format, Level ,Region, Copyright, EventAgeGroup, EventType. also do a tfidf on Bigram_description ) they all as X, apply mode to all missing, do one hot code , turn all non cateragoial to numeric\n",
        "\n",
        "import pandas as pd\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.metrics import accuracy_score, classification_report\n",
        "\n",
        "# Prepare the data\n",
        "docs_train_cleaned_desc_trail['StartPublishing_year'] = pd.to_datetime(docs_train_cleaned_desc_trail['StartPublishing']).dt.year\n",
        "\n",
        "X = docs_train_cleaned_desc_trail[['FileType', 'StartPublishing_year', 'Cpd', 'Format', 'Level',\n",
        "                                   'Region', 'Copyright', 'EventAgeGroup', 'EventType']]\n",
        "\n",
        "y = docs_train_cleaned_desc_trail['Tag_only_common_8']\n",
        "\n",
        "\n",
        "# Impute missing values with mode\n",
        "imputer = SimpleImputer(strategy='most_frequent')\n",
        "X[['FileType', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType']] = imputer.fit_transform(X[['FileType', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType']])\n",
        "\n",
        "\n",
        "# One-hot encode categorical features\n",
        "categorical_features = ['FileType', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType']\n",
        "ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False)\n",
        "X_encoded = ohe.fit_transform(X[categorical_features])\n",
        "feature_names = ohe.get_feature_names_out(categorical_features)\n",
        "X_encoded_df = pd.DataFrame(X_encoded, columns=feature_names)\n",
        "\n",
        "\n",
        "# TF-IDF for clean_description\n",
        "tfidf_vectorizer = TfidfVectorizer(ngram_range=(2, 2))\n",
        "tfidf_matrix = tfidf_vectorizer.fit_transform(docs_train_cleaned_desc_trail['Cleaned_Description'])\n",
        "tfidf_df = pd.DataFrame(tfidf_matrix.toarray(), columns=tfidf_vectorizer.get_feature_names_out())\n",
        "\n",
        "\n",
        "# Convert FileSize and StartPublishing_year to numeric\n",
        "X['StartPublishing_year'] = pd.to_numeric(X['StartPublishing_year'], errors='coerce')\n",
        "\n",
        "# Combine encoded features and numerical features\n",
        "X_combined = pd.concat([X_encoded_df, X[[ 'StartPublishing_year', 'Cpd']], tfidf_df], axis=1)\n",
        "X_combined = X_combined.fillna(0)  # Fill remaining NaNs with 0\n",
        "\n",
        "\n",
        "# Split the data\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_combined, y, test_size=0.2, random_state=42)\n",
        "\n",
        "# Train the model\n",
        "model = DecisionTreeClassifier(random_state=42)\n",
        "model.fit(X_train, y_train)\n",
        "\n",
        "# Make predictions\n",
        "y_pred = model.predict(X_test)\n",
        "\n",
        "# Evaluate the model\n",
        "accuracy = accuracy_score(y_test, y_pred)\n",
        "print(\"Accuracy:\", accuracy)\n",
        "print(classification_report(y_test, y_pred))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will construct the feature importance plot from the above model."
      ],
      "metadata": {
        "id": "mnps_ANchG40"
      },
      "id": "mnps_ANchG40"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Vrp6ASdtvZ7e",
      "metadata": {
        "id": "Vrp6ASdtvZ7e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48846e73-7a5f-46e0-8ea7-098f7cc09ee5"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 1 Axes>"
            ],
            "image/png": "[encoded data, 157836 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a feature importance for the above model , so me vislaustion, i want to see what are the most impotantt 50 feature\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "importances = model.feature_importances_\n",
        "feature_names = X_combined.columns\n",
        "feature_importances = pd.DataFrame({'Feature': feature_names, 'Importance': importances})\n",
        "feature_importances = feature_importances.sort_values(by='Importance', ascending=False)\n",
        "\n",
        "# Select the top 50 features\n",
        "top_50_features = feature_importances.head(50)\n",
        "\n",
        "# Visualize the top 50 feature importances\n",
        "plt.figure(figsize=(12, 8))\n",
        "sns.barplot(x='Importance', y='Feature', data=top_50_features)\n",
        "plt.title('Top 50 Feature Importances')\n",
        "plt.xlabel('Importance')\n",
        "plt.ylabel('Feature')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The top 5-6 key features identified by the random forest model are:\n",
        "\n",
        "- General Insurance: This feature has the highest importance, indicating that it plays a crucial role in distinguishing between asset groups.\n",
        "- StartPublishing Year: The year an asset was published strongly impacts its classification and relevance.\n",
        "- Injury Scheme: A highly significant feature, reflecting the importance of this category within the tagging system.\n",
        "- Actuary Summit: Assets related to Actuary Summits are key in the current system, showing strong influence on groupings.\n",
        "- Life Insurance: Another critical feature, highlighting its relevance in categorizing assets.\n",
        "- Enterprise Management: Also highly important, this feature likely helps differentiate assets focused on business management topics.\n",
        "\n",
        "These features play a central role in determining how assets are grouped under the current tagging system."
      ],
      "metadata": {
        "id": "fix92SF4hgF7"
      },
      "id": "fix92SF4hgF7"
    },
    {
      "cell_type": "markdown",
      "source": [
        "The table below shows the source of each feature, helping us understand how to create new features based on this information."
      ],
      "metadata": {
        "id": "1_4WDNAFhQpC"
      },
      "id": "1_4WDNAFhQpC"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "JVegW3JtwyOA",
      "metadata": {
        "id": "JVegW3JtwyOA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24163b97-f656-4634-f3cf-733ed5c75ead"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                     Feature  Importance                    Source\n",
            "18394      general insurance    0.063080           TF-IDF Features\n",
            "40      StartPublishing_year    0.040898        Numerical Features\n",
            "22209          injury scheme    0.035845           TF-IDF Features\n",
            "1016          actuary summit    0.028707           TF-IDF Features\n",
            "26285         life insurance    0.025359           TF-IDF Features\n",
            "14762  enterprise management    0.023498           TF-IDF Features\n",
            "41                       Cpd    0.023411        Numerical Features\n",
            "5179     biennial convention    0.019833           TF-IDF Features\n",
            "242    accident compensation    0.017187           TF-IDF Features\n",
            "16        Format_Publication    0.017024  One-Hot Encoded Features\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-167-1eccadff43a4>:17: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame.\n",
            "Try using .loc[row_indexer,col_indexer] = value instead\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  top_10_features['Source'] = top_10_features['Feature'].apply(get_feature_source)\n"
          ]
        }
      ],
      "source": [
        "# prompt: check which 10 most imporatn feature ,including a cloumn indicate where they come from\n",
        "\n",
        "# Assuming 'feature_importances' is the DataFrame containing feature importances as calculated in the previous code.\n",
        "top_10_features = feature_importances.head(10)\n",
        "\n",
        "# Extract the source of each feature\n",
        "def get_feature_source(feature_name):\n",
        "  if feature_name in X_encoded_df.columns:\n",
        "    return \"One-Hot Encoded Features\"\n",
        "  elif feature_name in ['StartPublishing_year', 'Cpd']:\n",
        "    return \"Numerical Features\"\n",
        "  elif feature_name in tfidf_df.columns:\n",
        "    return \"TF-IDF Features\"\n",
        "  else:\n",
        "    return \"Unknown\"\n",
        "\n",
        "top_10_features['Source'] = top_10_features['Feature'].apply(get_feature_source)\n",
        "\n",
        "# Display the top 10 features and their sources\n",
        "print(top_10_features[['Feature', 'Importance', 'Source']])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "KY8_5Wu9wLk2",
      "metadata": {
        "id": "KY8_5Wu9wLk2"
      },
      "source": [
        "Summary"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jSTLTiopzrh6",
      "metadata": {
        "id": "jSTLTiopzrh6"
      },
      "source": [
        "#### Q2b Step 2 Creating Features\n",
        "**Outline**\n",
        "\n",
        "Outline\n",
        "\n",
        "The following section focuses on creating features that will be used in the clustering model. Based on the random forest feature selection and the discussions in Q2a, the following features have been included:\n",
        "\n",
        "- Business Field: A binary flag indicating whether certain key business-related terms appear in the description. This helps capture important business-related information for clustering.\n",
        "\n",
        "- Recency: The 'date published' has been transformed into buckets based on recency up to April 30, 2024. One-hot encoding is then applied to these recency buckets, making this feature ready for the clustering model.\n",
        "\n",
        "- Missing Description: A binary flag to indicate whether the description is missing. This is crucial for distinguishing between assets with complete versus incomplete information.\n",
        "\n",
        "- Format: Asset formats are grouped into broader categories and then one-hot encoded to ensure that different formats are effectively represented without overwhelming the clustering model.\n",
        "\n",
        "- Event-Related Feature: Event types are also grouped into broader categories and one-hot encoded to capture the association between assets and specific events.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Qktnbur5xk1H",
      "metadata": {
        "id": "Qktnbur5xk1H"
      },
      "source": [
        "We created docs_cleaned_desc, a dataset that consolidates all relevant features, as clustering will be applied to the entire dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "yrk9WDxLz5_n",
      "metadata": {
        "id": "yrk9WDxLz5_n"
      },
      "outputs": [],
      "source": [
        "# prompt: create docs_cleaned_desc, from docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc\n",
        "\n",
        "import pandas as pd\n",
        "# Assuming you have docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc DataFrames\n",
        "\n",
        "# Concatenate the DataFrames into one combined DataFrame\n",
        "docs_cleaned_desc = pd.concat([docs_train_cleaned_desc, docs_val_cleaned_desc, docs_test_cleaned_desc], ignore_index=True)\n",
        "\n",
        "# Now 'docs_cleaned_desc' contains all the data from the three DataFrames.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-d19kBdA0jZ9",
      "metadata": {
        "id": "-d19kBdA0jZ9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1b2656e-232c-4b41-8d57-87893f649c32"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 30 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    7550 non-null   int64         \n",
            " 1   FileName                 7550 non-null   object        \n",
            " 2   FileType                 7550 non-null   object        \n",
            " 3   FileSize                 7550 non-null   int64         \n",
            " 4   Title                    7550 non-null   object        \n",
            " 5   StartPublishing          7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing           7550 non-null   object        \n",
            " 7   Cpd                      7550 non-null   int64         \n",
            " 8   Format                   7550 non-null   object        \n",
            " 9   Level                    7550 non-null   object        \n",
            " 10  Description              7550 non-null   object        \n",
            " 11  Region                   7550 non-null   object        \n",
            " 12  Copyright                7550 non-null   object        \n",
            " 13  EventAgeGroup            6840 non-null   object        \n",
            " 14  EventType                6947 non-null   object        \n",
            " 15  Link                     7550 non-null   object        \n",
            " 16  Tags                     6969 non-null   object        \n",
            " 17  DownloadCount            7550 non-null   int64         \n",
            " 18  DurationOnDMS            7550 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  7550 non-null   float64       \n",
            " 20  YearOfStartPublishing    7550 non-null   int32         \n",
            " 21  Year                     7550 non-null   int32         \n",
            " 22  Tag_only                 6969 non-null   object        \n",
            " 23  Tag_only_common          6969 non-null   object        \n",
            " 24  StopPublishing_Year      6068 non-null   float64       \n",
            " 25  Description_Length       6068 non-null   float64       \n",
            " 26  Number of Downloads      6068 non-null   float64       \n",
            " 27  Tokens                   7550 non-null   object        \n",
            " 28  Cleaned_Description      7550 non-null   object        \n",
            " 29  Bigram_Description       7550 non-null   object        \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(4), object(18)\n",
            "memory usage: 1.7+ MB\n"
          ]
        }
      ],
      "source": [
        "docs_cleaned_desc.info()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "siIqW5YVhGre",
      "metadata": {
        "id": "siIqW5YVhGre"
      },
      "source": [
        "Check looks sensible with all the columns we needed to create the following features"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Wd8R8_3C-r_p",
      "metadata": {
        "id": "Wd8R8_3C-r_p"
      },
      "source": [
        "##### Feature: Business Field\n",
        "\n",
        "**Rationale:**\n",
        "Our random forest model identified that most of the top 10 important features were related to specific business fields. While the TF-IDF vectorization process includes these terms, there's a concern that some critical business-related keywords might not be captured due to the low max_features limit set for TF-IDF. This means not all essential terms are guaranteed to be retained in the final vector representation. To ensure these important business field terms are effectively captured and to maintain their influence in our clustering model, we created explicit binary features for each field.\n",
        "\n",
        "\n",
        "**Implementation:**\n",
        "We applied a simple logic to create flags for each field of practice. If the cleaned description contains the exact wording of a particular business field, we assigned a value of 1; otherwise, it was set to 0. This ensures that essential business fields are explicitly represented in the clustering model.\n",
        "\n",
        "\n",
        "**Potential Issues and Considerations:**\n",
        "Redundancy with TF-IDF: There’s a possibility that these terms might still be captured by the TF-IDF process, which could lead to redundancy. To avoid this, we plan to treat these business field terms as stop words in the TF-IDF vectorization, ensuring they don’t dominate the feature space or reappear unnecessarily."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "SlKwUFtYyxmS",
      "metadata": {
        "id": "SlKwUFtYyxmS"
      },
      "outputs": [],
      "source": [
        "# prompt: create me a feture called general insurance in docs_cleaned_desc, if the Cleaned_Description contain the word \"general insurance\" give 1, other wise 0 , similarly , do injury scheme, actuary summit, life insurance, enterprise management, biennial convention, accident compensation\n",
        "\n",
        "docs_cleaned_desc['general_insurance'] = docs_cleaned_desc['Cleaned_Description'].str.contains('general insurance').astype(int)\n",
        "docs_cleaned_desc['injury_scheme'] = docs_cleaned_desc['Cleaned_Description'].str.contains('injury scheme').astype(int)\n",
        "docs_cleaned_desc['actuary_summit'] = docs_cleaned_desc['Cleaned_Description'].str.contains('actuary summit').astype(int)\n",
        "docs_cleaned_desc['life_insurance'] = docs_cleaned_desc['Cleaned_Description'].str.contains('life insurance').astype(int)\n",
        "docs_cleaned_desc['enterprise_management'] = docs_cleaned_desc['Cleaned_Description'].str.contains('enterprise management').astype(int)\n",
        "docs_cleaned_desc['biennial_convention'] = docs_cleaned_desc['Cleaned_Description'].str.contains('biennial convention').astype(int)\n",
        "docs_cleaned_desc['accident_compensation'] = docs_cleaned_desc['Cleaned_Description'].str.contains('accident compensation').astype(int)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gnEv2SaoDSat",
      "metadata": {
        "id": "gnEv2SaoDSat"
      },
      "source": [
        "Check to make sure new field are added"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "qPdXZqb0zTQl",
      "metadata": {
        "id": "qPdXZqb0zTQl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "33279906-6b1c-4c5f-a17b-838805c2977e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 37 columns):\n",
            " #   Column                   Non-Null Count  Dtype         \n",
            "---  ------                   --------------  -----         \n",
            " 0   DMSId                    7550 non-null   int64         \n",
            " 1   FileName                 7550 non-null   object        \n",
            " 2   FileType                 7550 non-null   object        \n",
            " 3   FileSize                 7550 non-null   int64         \n",
            " 4   Title                    7550 non-null   object        \n",
            " 5   StartPublishing          7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing           7550 non-null   object        \n",
            " 7   Cpd                      7550 non-null   int64         \n",
            " 8   Format                   7550 non-null   object        \n",
            " 9   Level                    7550 non-null   object        \n",
            " 10  Description              7550 non-null   object        \n",
            " 11  Region                   7550 non-null   object        \n",
            " 12  Copyright                7550 non-null   object        \n",
            " 13  EventAgeGroup            6840 non-null   object        \n",
            " 14  EventType                6947 non-null   object        \n",
            " 15  Link                     7550 non-null   object        \n",
            " 16  Tags                     6969 non-null   object        \n",
            " 17  DownloadCount            7550 non-null   int64         \n",
            " 18  DurationOnDMS            7550 non-null   float64       \n",
            " 19  AnnualizedDownloadCount  7550 non-null   float64       \n",
            " 20  YearOfStartPublishing    7550 non-null   int32         \n",
            " 21  Year                     7550 non-null   int32         \n",
            " 22  Tag_only                 6969 non-null   object        \n",
            " 23  Tag_only_common          6969 non-null   object        \n",
            " 24  StopPublishing_Year      6068 non-null   float64       \n",
            " 25  Description_Length       6068 non-null   float64       \n",
            " 26  Number of Downloads      6068 non-null   float64       \n",
            " 27  Tokens                   7550 non-null   object        \n",
            " 28  Cleaned_Description      7550 non-null   object        \n",
            " 29  Bigram_Description       7550 non-null   object        \n",
            " 30  general_insurance        7550 non-null   int64         \n",
            " 31  injury_scheme            7550 non-null   int64         \n",
            " 32  actuary_summit           7550 non-null   int64         \n",
            " 33  life_insurance           7550 non-null   int64         \n",
            " 34  enterprise_management    7550 non-null   int64         \n",
            " 35  biennial_convention      7550 non-null   int64         \n",
            " 36  accident_compensation    7550 non-null   int64         \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(11), object(18)\n",
            "memory usage: 2.1+ MB\n"
          ]
        }
      ],
      "source": [
        "docs_cleaned_desc.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X_lJ8mBADVfr",
      "metadata": {
        "id": "X_lJ8mBADVfr"
      },
      "source": [
        "The check looks sensible an meet our expecation of extra 7 field is added to capture the busienss field."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Kwi04fh6MFbx",
      "metadata": {
        "id": "Kwi04fh6MFbx"
      },
      "source": [
        "##### Feature for Recency\n",
        "\n",
        "**Rationale of using Recency**\n",
        "Our initial analysis indicated that the asset's publish date was an important feature in understanding asset behavior and classification. However, date formats aren't directly suitable for clustering models like K-Means. To address this, we transformed the publish date into a \"Recency\" feature, which represents the number of days since each asset was published up to a fixed point (30 April 2024). This conversion allows us to capture how long an asset has been available, making it easier to include this information in our clustering model.\n",
        "\n",
        "**Heavily skewed**\n",
        "During the analysis, we noticed that the recency feature was heavily skewed, with a high concentration of assets having been published more recently and fewer older assets. This skewness posed a problem because K-Means clustering is sensitive to feature scales and could be disproportionately influenced by the wide range of recency values, leading to less meaningful clusters.\n",
        "\n",
        "\n",
        "**Grouped and Applied One-Hot Encoding:**\n",
        "- To address the skewness and ensure the recency feature didn't dominate the clustering process, we grouped the recency values into broader, more manageable categories, such as \"Recent,\" \"Moderately Recent,\" \"Old,\" and \"Very Old.\" This grouping helped to reduce the skewness and make the feature more interpretable.\n",
        "\n",
        "- We then applied one-hot encoding to convert these categories into binary features. One-hot encoding ensures that each recency category is treated independently and equally by the clustering algorithm, avoiding any assumptions of an ordinal relationship between them. This approach allows the model to better capture the differences in asset age without being overwhelmed by the original skewed distribution, resulting in more balanced and meaningful clusters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ZLnAC0VsQuSb",
      "metadata": {
        "id": "ZLnAC0VsQuSb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e4f0ca9-a904-4fcf-ff20-5f5e86cef8bd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId  Recency_Numerical\n",
            "0  4194310               2065\n",
            "1  4194590               2080\n",
            "2  4194592               2080\n",
            "3  4194593               2080\n",
            "4  4194594               2080\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "\n",
        "# Define \"today\" as 30 April 2024\n",
        "today = datetime(2024, 4, 30)\n",
        "\n",
        "# Ensure that 'StartPublishing' is in datetime format\n",
        "docs_cleaned_desc['StartPublishing'] = pd.to_datetime(docs_cleaned_desc['StartPublishing'])\n",
        "\n",
        "# Calculate the number of days since StartPublishing (Numerical Recency)\n",
        "docs_cleaned_desc['Recency_Numerical'] = (today - docs_cleaned_desc['StartPublishing']).dt.days\n",
        "\n",
        "\n",
        "# Check the result\n",
        "print(docs_cleaned_desc[['DMSId', 'Recency_Numerical']].head())\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Fh8hMabEQy41",
      "metadata": {
        "id": "Fh8hMabEQy41",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "490be348-fe96-4dba-99a7-84e33813118e"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 38556 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# plot to show the Recency_Numerical\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.histplot(docs_cleaned_desc['Recency_Numerical'], bins=30, kde=True)\n",
        "plt.xlabel('Recency (Days)')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of Document Recency')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rLotJEnqATZL",
      "metadata": {
        "id": "rLotJEnqATZL"
      },
      "source": [
        "The above chart clearly shows a skewed distribution, primarily due to a high volume of asset publications in 2018 and 2019, likely caused by a backlog upload into the new system.\n",
        "\n",
        "The code below demonstrates how we transformed the numerical recency feature into categorical buckets, followed by applying one-hot encoding:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "E6eWy_er_sLr",
      "metadata": {
        "id": "E6eWy_er_sLr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f08ddc71-33cd-4c0b-92ea-d7f41b9a81f5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                           FileName FileType  \\\n",
            "0  4194310               international-cpd-opportunities.html     html   \n",
            "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
            "2  4194592                           1997 - March Edition.pdf      pdf   \n",
            "3  4194593                       1997 - September Edition.pdf      pdf   \n",
            "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
            "\n",
            "   FileSize                                              Title  \\\n",
            "0      8892                    International CPD opportunities   \n",
            "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
            "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
            "4    209993  Disability Committee - The 1997 Report into mo...   \n",
            "\n",
            "  StartPublishing       StopPublishing  Cpd       Format         Level  ...  \\\n",
            "0      2018-09-04  2261-12-31 00:00:00    1      Article  Introductory  ...   \n",
            "1      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "2      2018-08-20  2261-12-31 00:00:00    2  Publication           All  ...   \n",
            "3      2018-08-20  2261-12-31 00:00:00    1  Publication           All  ...   \n",
            "4      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "\n",
            "  life_insurance enterprise_management biennial_convention  \\\n",
            "0              0                     0                   0   \n",
            "1              0                     0                   0   \n",
            "2              0                     0                   0   \n",
            "3              0                     0                   0   \n",
            "4              0                     0                   0   \n",
            "\n",
            "  accident_compensation Recency_Numerical Recency_Category  \\\n",
            "0                     0              2065         Very Old   \n",
            "1                     0              2080         Very Old   \n",
            "2                     0              2080         Very Old   \n",
            "3                     0              2080         Very Old   \n",
            "4                     0              2080         Very Old   \n",
            "\n",
            "  Recency_Moderately Recent  Recency_Old  Recency_Recent  Recency_Very Old  \n",
            "0                     False        False           False              True  \n",
            "1                     False        False           False              True  \n",
            "2                     False        False           False              True  \n",
            "3                     False        False           False              True  \n",
            "4                     False        False           False              True  \n",
            "\n",
            "[5 rows x 43 columns]\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me code, i will group recency_numerical based on into buckets such as:\n",
        "# 0-500 days: Recent\n",
        "# 501-1000 days: Moderately Recent\n",
        "# 1001-1500 days: Old\n",
        "# 1501+ days: Very Old, we then create on hot encoding\n",
        "\n",
        "import pandas as pd\n",
        "# Create a function to categorize recency into buckets\n",
        "def categorize_recency(days):\n",
        "  if 0 <= days <= 500:\n",
        "    return 'Recent'\n",
        "  elif 501 <= days <= 1000:\n",
        "    return 'Moderately Recent'\n",
        "  elif 1001 <= days <= 1500:\n",
        "    return 'Old'\n",
        "  else:\n",
        "    return 'Very Old'\n",
        "\n",
        "# Apply the function to create a new column 'Recency_Category'\n",
        "docs_cleaned_desc['Recency_Category'] = docs_cleaned_desc['Recency_Numerical'].apply(categorize_recency)\n",
        "\n",
        "# Create one-hot encoded features for Recency_Category\n",
        "recency_encoded = pd.get_dummies(docs_cleaned_desc['Recency_Category'], prefix='Recency')\n",
        "\n",
        "# Concatenate the one-hot encoded features with the original DataFrame\n",
        "docs_cleaned_desc = pd.concat([docs_cleaned_desc, recency_encoded], axis=1)\n",
        "\n",
        "\n",
        "# Display the updated DataFrame with the one-hot encoded features\n",
        "print(docs_cleaned_desc.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uEvfSBSNAan1",
      "metadata": {
        "id": "uEvfSBSNAan1"
      },
      "source": [
        "The check inidicates the new created recency one hot encoded are True/False.\n",
        "\n",
        "The following codes is to convert the one-hot encoded recency feature from True/False to 0/1,"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "XD3kYQqpAAfB",
      "metadata": {
        "id": "XD3kYQqpAAfB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d29142e6-9f6d-4e27-e6b4-575ac2c1eb66"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                           FileName FileType  \\\n",
            "0  4194310               international-cpd-opportunities.html     html   \n",
            "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
            "2  4194592                           1997 - March Edition.pdf      pdf   \n",
            "3  4194593                       1997 - September Edition.pdf      pdf   \n",
            "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
            "\n",
            "   FileSize                                              Title  \\\n",
            "0      8892                    International CPD opportunities   \n",
            "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
            "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
            "4    209993  Disability Committee - The 1997 Report into mo...   \n",
            "\n",
            "  StartPublishing       StopPublishing  Cpd       Format         Level  ...  \\\n",
            "0      2018-09-04  2261-12-31 00:00:00    1      Article  Introductory  ...   \n",
            "1      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "2      2018-08-20  2261-12-31 00:00:00    2  Publication           All  ...   \n",
            "3      2018-08-20  2261-12-31 00:00:00    1  Publication           All  ...   \n",
            "4      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "\n",
            "  life_insurance enterprise_management biennial_convention  \\\n",
            "0              0                     0                   0   \n",
            "1              0                     0                   0   \n",
            "2              0                     0                   0   \n",
            "3              0                     0                   0   \n",
            "4              0                     0                   0   \n",
            "\n",
            "  accident_compensation Recency_Numerical Recency_Category  \\\n",
            "0                     0              2065         Very Old   \n",
            "1                     0              2080         Very Old   \n",
            "2                     0              2080         Very Old   \n",
            "3                     0              2080         Very Old   \n",
            "4                     0              2080         Very Old   \n",
            "\n",
            "  Recency_Moderately Recent  Recency_Old  Recency_Recent  Recency_Very Old  \n",
            "0                         0            0               0                 1  \n",
            "1                         0            0               0                 1  \n",
            "2                         0            0               0                 1  \n",
            "3                         0            0               0                 1  \n",
            "4                         0            0               0                 1  \n",
            "\n",
            "[5 rows x 43 columns]\n"
          ]
        }
      ],
      "source": [
        "# prompt: turn Recnecy_moderatly, recency old, recency recnt , recent_Very old into 0, or 1, i dont want ture or false\n",
        "\n",
        "docs_cleaned_desc['Recency_Moderately Recent'] = docs_cleaned_desc['Recency_Moderately Recent'].astype(int)\n",
        "docs_cleaned_desc['Recency_Old'] = docs_cleaned_desc['Recency_Old'].astype(int)\n",
        "docs_cleaned_desc['Recency_Recent'] = docs_cleaned_desc['Recency_Recent'].astype(int)\n",
        "docs_cleaned_desc['Recency_Very Old'] = docs_cleaned_desc['Recency_Very Old'].astype(int)\n",
        "\n",
        "print(docs_cleaned_desc.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5PN6r9NeOnbI",
      "metadata": {
        "id": "5PN6r9NeOnbI"
      },
      "source": [
        "##### Feature for missing Description\n",
        "The rationale for including a binary feature for missing descriptions is rooted in data completeness and accuracy. Assets without descriptions represent incomplete information, which can impact the clustering process, potentially leading to less precise groupings.\n",
        "\n",
        "By introducing this feature, we explicitly inform the model about the presence of missing data, allowing it to account for this gap during clustering. This ensures that clusters are formed with an awareness of data completeness, improving the overall quality and interpretability of the clustering results, especially in practical business contexts where missing descriptions might indicate different asset characteristics or priorities.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "xEaK1FqReVfr",
      "metadata": {
        "id": "xEaK1FqReVfr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "11216683-b3fc-4c5d-c63e-94cac584c863"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                        Description  \\\n",
            "0  4194310  there are over  committees subcommittees task ...   \n",
            "1  4194590  the report is based on data in respect of indi...   \n",
            "2  4194592  inadequacy of liasb mortality reserves allen t...   \n",
            "3  4194593  analysis of surplus mixing realism with conser...   \n",
            "4  4194594  the  report into morbidity for insured and oth...   \n",
            "\n",
            "   Missing_Description  \n",
            "0                    0  \n",
            "1                    0  \n",
            "2                    0  \n",
            "3                    0  \n",
            "4                    0  \n",
            "      DMSId                      Title            Description  \\\n",
            "47  4196529  2009 Financial Statements   financial statements   \n",
            "48  4196530  2011 Financial Statements   financial statements   \n",
            "50  4196533         2010 Annual Review          annual review   \n",
            "53  4196536  2013 Financial Statements   financial statements   \n",
            "55  4196538        2014 Year in Review         year in review   \n",
            "\n",
            "    Missing_Description  \n",
            "47                    1  \n",
            "48                    1  \n",
            "50                    1  \n",
            "53                    1  \n",
            "55                    1  \n"
          ]
        }
      ],
      "source": [
        "# Step 1: Create a binary indicator for missing descriptions in the original data\n",
        "docs_data['Missing_Description'] = docs_data['Description'].apply(lambda x: 1 if pd.isna(x) or x.strip() == \"\" else 0)\n",
        "\n",
        "# Step 2: Merge this flag back into the cleaned dataset\n",
        "# Ensure that you merge on a unique identifier that both datasets share, e.g., 'DMSId'\n",
        "docs_cleaned_desc = pd.merge(docs_cleaned_desc, docs_data[['DMSId', 'Missing_Description']], on='DMSId', how='left')\n",
        "\n",
        "# Step 3: Verify the result\n",
        "print(docs_cleaned_desc[['DMSId', 'Description', 'Missing_Description']].head())\n",
        "\n",
        "\n",
        "# Filter the cleaned dataset to show rows where Missing_Description is 1\n",
        "missing_desc_rows = docs_cleaned_desc[docs_cleaned_desc['Missing_Description'] == 1]\n",
        "\n",
        "# Display the first 5 rows with missing descriptions\n",
        "print(missing_desc_rows[['DMSId', 'Title', 'Description', 'Missing_Description']].head())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jQ2e0Le6TFhN",
      "metadata": {
        "id": "jQ2e0Le6TFhN"
      },
      "source": [
        "##### feature for format\n",
        "\n",
        "**Rationale for Using Format:**\n",
        "\n",
        "Our random forest model highlighted that certain formats play a crucial role in the existing tagging system for classifying assets. Therefore, capturing this information as a feature is essential.\n",
        "\n",
        "**High Cardinality**:\n",
        "To reduce complexity while retaining key distinctions, I grouped the original format types into broader categories: Text-Based, Video/Audio-Based, Presentation, and Other.\n",
        "\n",
        "\n",
        "**High Cardinality Consideration**:\n",
        "In the Q1a EDA, we observed high cardinality in the format feature. To reduce complexity while retaining key distinctions, I grouped the original format types into broader categories: Text-Based, Video/Audio-Based, Presentation, and Other. This approach simplifies the data while preserving essential differences between asset formats."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "upz20jMiD2Iq",
      "metadata": {
        "id": "upz20jMiD2Iq"
      },
      "source": [
        "Below codes are from Q1a EDA, to help use decide how to do the grouping."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Vi-4M2orTTau",
      "metadata": {
        "id": "Vi-4M2orTTau",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4ca69652-e5cc-4d35-b3e4-20d18022d148"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "Least Common Values:\n",
            "\tTool and Resources: 60\n",
            "\tWebinar: 56\n",
            "\tOther: 4\n",
            "\tWorkshops: 4\n",
            "\teLearning: 2\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 40948 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 48748 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_cleaned_desc)\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'Format')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yvPwXMfCEDbt",
      "metadata": {
        "id": "yvPwXMfCEDbt"
      },
      "source": [
        "Below codes are group them into different bucket and then applies One Hot encoding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "aEHmSgOrxc3h",
      "metadata": {
        "id": "aEHmSgOrxc3h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "25261727-d300-4bb9-8726-7eeb92878f3a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-178-56963c3eaf0f>:20: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(data=docs_cleaned_desc, x='Simplified_Format', palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 29660 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                           FileName FileType  \\\n",
            "0  4194310               international-cpd-opportunities.html     html   \n",
            "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
            "2  4194592                           1997 - March Edition.pdf      pdf   \n",
            "3  4194593                       1997 - September Edition.pdf      pdf   \n",
            "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
            "\n",
            "   FileSize                                              Title  \\\n",
            "0      8892                    International CPD opportunities   \n",
            "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
            "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
            "4    209993  Disability Committee - The 1997 Report into mo...   \n",
            "\n",
            "  StartPublishing       StopPublishing  Cpd       Format         Level  ...  \\\n",
            "0      2018-09-04  2261-12-31 00:00:00    1      Article  Introductory  ...   \n",
            "1      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "2      2018-08-20  2261-12-31 00:00:00    2  Publication           All  ...   \n",
            "3      2018-08-20  2261-12-31 00:00:00    1  Publication           All  ...   \n",
            "4      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "\n",
            "  Recency_Moderately Recent Recency_Old Recency_Recent Recency_Very Old  \\\n",
            "0                         0           0              0                1   \n",
            "1                         0           0              0                1   \n",
            "2                         0           0              0                1   \n",
            "3                         0           0              0                1   \n",
            "4                         0           0              0                1   \n",
            "\n",
            "  Missing_Description Simplified_Format Format_Other  Format_Presentation  \\\n",
            "0                   0        Text-Based            0                    0   \n",
            "1                   0        Text-Based            0                    0   \n",
            "2                   0        Text-Based            0                    0   \n",
            "3                   0        Text-Based            0                    0   \n",
            "4                   0        Text-Based            0                    0   \n",
            "\n",
            "   Format_Text-Based  Format_Video/Audio-Based  \n",
            "0                  1                         0  \n",
            "1                  1                         0  \n",
            "2                  1                         0  \n",
            "3                  1                         0  \n",
            "4                  1                         0  \n",
            "\n",
            "[5 rows x 49 columns]\n"
          ]
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Simplified format grouping function\n",
        "def simplify_format(format):\n",
        "    if format in ['Article', 'Publication', 'Tool and Resources']:\n",
        "        return 'Text-Based'\n",
        "    elif format in ['Video', 'Podcast/Vodcast/Audio']:\n",
        "        return 'Video/Audio-Based'\n",
        "    elif format in ['Presentation Slides']:\n",
        "        return 'Presentation'\n",
        "    else:\n",
        "        return 'Other'  # Capture everything else as 'Other'\n",
        "\n",
        "# Apply the function to create the Simplified_Format column\n",
        "docs_cleaned_desc['Simplified_Format'] = docs_cleaned_desc['Format'].apply(simplify_format)\n",
        "\n",
        "# Visualize the distribution of the simplified format\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(data=docs_cleaned_desc, x='Simplified_Format', palette='viridis')\n",
        "plt.title('Distribution of Simplified Format')\n",
        "plt.xlabel('Simplified Format')\n",
        "plt.ylabel('Count')\n",
        "plt.show()\n",
        "\n",
        "# One-Hot Encode the Simplified_Format feature\n",
        "simplified_format_ohe = pd.get_dummies(docs_cleaned_desc['Simplified_Format'], prefix='Format', dtype=int)\n",
        "\n",
        "# Add the encoded features back to the combined DataFrame\n",
        "docs_cleaned_desc = pd.concat([docs_cleaned_desc, simplified_format_ohe], axis=1)\n",
        "\n",
        "# Check the result\n",
        "print(docs_cleaned_desc.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "A4oMhBcrELEV",
      "metadata": {
        "id": "A4oMhBcrELEV"
      },
      "source": [
        "Check suggest that the new fields for format are corrected, with expected value of 0/1."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Kxhx1Wz1UP2l",
      "metadata": {
        "id": "Kxhx1Wz1UP2l"
      },
      "source": [
        "##### Feature: Event-Related Grouping\n",
        "\n",
        "\n",
        "**Rationale of having this feature**\n",
        "Our initial random forest model indicated that certain event types were significant, ranking within the top 30 most important features. Based on this insight, we decided to create an event-related feature.\n",
        "\n",
        "**Treatment of Missing Values:**\n",
        "In Q1a, we observed that a sizable number of assets lacked event information. Using domain knowledge, we recognized that this likely indicates these assets are not event-related, a step we addressed during the Q1a cleaning process.\n",
        "\n",
        "**High Cardinality:**\n",
        "To manage the high cardinality of event types, I grouped them into broader categories: \"Major Event,\" \"Non-Event Related,\" and \"Other Event.\" This simplification reduces complexity while retaining key distinctions.\n",
        "\n",
        "**Why One-Hot Encoding?**\n",
        " One-hot encoding then converts these categories into binary features, allowing the algorithm to effectively differentiate between event-related aspects, enhancing the model's ability to form meaningful clusters."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "g7czR5Jwz5XR",
      "metadata": {
        "id": "g7czR5Jwz5XR"
      },
      "source": [
        "a quick check of what values is contained in the EventType"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "H3Av2d7CUcaZ",
      "metadata": {
        "id": "H3Av2d7CUcaZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ebb9e3cc-a6ac-4c71-f76f-059bc2a85754"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "---------\n",
            "Least Common Values:\n",
            "\tInsights/Other: 570\n",
            "\tOne/Half Day: 439\n",
            "\tInsights/Conferences: 12\n",
            "\tOther: 3\n",
            "\tMajors/Seminars: 1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 41080 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x400 with 1 Axes>"
            ],
            "image/png": "[encoded data, 39484 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "df = pd.DataFrame(docs_cleaned_desc)\n",
        "print(\"---------\")\n",
        "eda_categorical_column(df, 'EventType')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "T0bB3MWrEd9u",
      "metadata": {
        "id": "T0bB3MWrEd9u"
      },
      "source": [
        "below codes ae\n",
        "\n",
        "- To manage the high cardinality of event types, I grouped them into broader categories: \"Major Event,\" \"Non-Event Related,\" and \"Other Event.\" This simplification reduces complexity while retaining key distinctions.\n",
        "\n",
        "- One-hot encoding then converts these categories into binary features, allowing the algorithm to effectively differentiate between event-related aspects, enhancing the model's ability to form meaningful clusters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "hVUWkR5oU6CH",
      "metadata": {
        "id": "hVUWkR5oU6CH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "35c1b477-ddb5-4b06-af34-187732161919"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                           FileName FileType  \\\n",
            "0  4194310               international-cpd-opportunities.html     html   \n",
            "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
            "2  4194592                           1997 - March Edition.pdf      pdf   \n",
            "3  4194593                       1997 - September Edition.pdf      pdf   \n",
            "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
            "\n",
            "   FileSize                                              Title  \\\n",
            "0      8892                    International CPD opportunities   \n",
            "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
            "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
            "4    209993  Disability Committee - The 1997 Report into mo...   \n",
            "\n",
            "  StartPublishing       StopPublishing  Cpd       Format         Level  ...  \\\n",
            "0      2018-09-04  2261-12-31 00:00:00    1      Article  Introductory  ...   \n",
            "1      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "2      2018-08-20  2261-12-31 00:00:00    2  Publication           All  ...   \n",
            "3      2018-08-20  2261-12-31 00:00:00    1  Publication           All  ...   \n",
            "4      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "\n",
            "  Recency_Very Old Missing_Description Simplified_Format Format_Other  \\\n",
            "0                1                   0        Text-Based            0   \n",
            "1                1                   0        Text-Based            0   \n",
            "2                1                   0        Text-Based            0   \n",
            "3                1                   0        Text-Based            0   \n",
            "4                1                   0        Text-Based            0   \n",
            "\n",
            "  Format_Presentation Format_Text-Based  Format_Video/Audio-Based  \\\n",
            "0                   0                 1                         0   \n",
            "1                   0                 1                         0   \n",
            "2                   0                 1                         0   \n",
            "3                   0                 1                         0   \n",
            "4                   0                 1                         0   \n",
            "\n",
            "   Event_Major Event  Event_Non-Event Related  Event_Other Event  \n",
            "0                  0                        0                  1  \n",
            "1                  0                        1                  0  \n",
            "2                  0                        1                  0  \n",
            "3                  0                        1                  0  \n",
            "4                  0                        1                  0  \n",
            "\n",
            "[5 rows x 51 columns]\n"
          ]
        }
      ],
      "source": [
        "# Example categorization function for Event-Related Features based on your provided event types\n",
        "def categorize_event(event_type):\n",
        "    if event_type in ['Major', 'Majors/Seminars']:\n",
        "        return 'Major Event'\n",
        "    elif event_type == 'Non Event Related':\n",
        "        return 'Non-Event Related'\n",
        "    else:\n",
        "        return 'Other Event'\n",
        "\n",
        "# Apply the function to create the Event_Category column\n",
        "docs_cleaned_desc['Event_Category'] = docs_cleaned_desc['EventType'].apply(categorize_event)\n",
        "\n",
        "# One-Hot Encode the Event_Category feature\n",
        "event_ohe = pd.get_dummies(docs_cleaned_desc['Event_Category'], prefix='Event', dtype=int)\n",
        "\n",
        "# Add the encoded features back to the combined DataFrame\n",
        "docs_cleaned_desc = pd.concat([docs_cleaned_desc, event_ohe], axis=1)\n",
        "\n",
        "# Drop the original 'EventType' and 'Event_Category' columns if no longer needed\n",
        "docs_cleaned_desc.drop(['EventType', 'Event_Category'], axis=1, inplace=True)\n",
        "\n",
        "# Check the result\n",
        "print(docs_cleaned_desc.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gNpsuVM-Em8k",
      "metadata": {
        "id": "gNpsuVM-Em8k"
      },
      "source": [
        "The above check is meeting our expectation , new event fields are all 0/1."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QbmUjcYcyMmn",
      "metadata": {
        "id": "QbmUjcYcyMmn"
      },
      "source": [
        "##### Feature CPD\n",
        "\n",
        "**Rationale:**\n",
        "Our initial random forest model identified CPD points as an important feature for classifying assets.\n",
        "\n",
        "**Challenges for drfiting**\n",
        "Additionally, our Question 1 EDA showed that CPD points have experienced changes over time, with a noticeable reduction in high outliers, indicating data drift.\n",
        "\n",
        "**Mitigation**\n",
        "To address this drift and ensure the CPD feature remains relevant in our model, we decided to group the CPD points into broader categories and then applies One-hot encoding.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1MPUbczSyOd1",
      "metadata": {
        "id": "1MPUbczSyOd1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2a1ec493-5018-482e-a258-3488527adfe8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of missing values in 'Cpd': 0\n",
            "\n",
            "Frequency of positive integer values in 'Cpd':\n",
            "Cpd\n",
            "1     5527\n",
            "2     1801\n",
            "4       59\n",
            "3       54\n",
            "5        4\n",
            "6        2\n",
            "7        1\n",
            "8        1\n",
            "10       1\n",
            "9        1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 39100 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a check of cpd , to make sure no missing value , also show all positive integer then the freqnect  , do a plot of distruviton\n",
        "\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "# Check for missing values in the 'Cpd' column\n",
        "missing_cpd = docs_cleaned_desc['Cpd'].isnull().sum()\n",
        "print(f\"Number of missing values in 'Cpd': {missing_cpd}\")\n",
        "\n",
        "# Filter out non-positive integers and count their frequency\n",
        "positive_cpd = docs_cleaned_desc[docs_cleaned_desc['Cpd'] > 0]['Cpd']\n",
        "cpd_counts = positive_cpd.value_counts().sort_values(ascending=False)\n",
        "\n",
        "print(\"\\nFrequency of positive integer values in 'Cpd':\")\n",
        "print(cpd_counts)\n",
        "\n",
        "# Plot the distribution of 'Cpd' values\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.histplot(docs_cleaned_desc['Cpd'], bins=30, kde=True)\n",
        "plt.xlabel('CPD')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of CPD Values')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4wFrgwRnHKkk",
      "metadata": {
        "id": "4wFrgwRnHKkk"
      },
      "source": [
        "The following codes are to group cpd into 3 different bucket and then applies one hot encoding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "nYqKocuqG3k0",
      "metadata": {
        "id": "nYqKocuqG3k0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f21f9531-a367-4b27-9fce-71e1f9b0e74a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     DMSId                                           FileName FileType  \\\n",
            "0  4194310               international-cpd-opportunities.html     html   \n",
            "1  4194590        1995 Report of the Disability Committee.pdf      pdf   \n",
            "2  4194592                           1997 - March Edition.pdf      pdf   \n",
            "3  4194593                       1997 - September Edition.pdf      pdf   \n",
            "4  4194594  1997 Report of the Disability Committee-The Da...      pdf   \n",
            "\n",
            "   FileSize                                              Title  \\\n",
            "0      8892                    International CPD opportunities   \n",
            "1   1674088  Disability Committee - 1995 Report of the Disa...   \n",
            "2    300105  Australian Actuarial Journal 1997 (March Edition)   \n",
            "3    166894  Australian Actuarial Journal 1997 (September E...   \n",
            "4    209993  Disability Committee - The 1997 Report into mo...   \n",
            "\n",
            "  StartPublishing       StopPublishing  Cpd       Format         Level  ...  \\\n",
            "0      2018-09-04  2261-12-31 00:00:00    1      Article  Introductory  ...   \n",
            "1      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "2      2018-08-20  2261-12-31 00:00:00    2  Publication           All  ...   \n",
            "3      2018-08-20  2261-12-31 00:00:00    1  Publication           All  ...   \n",
            "4      2018-08-20  2261-12-31 00:00:00    2  Publication      Advanced  ...   \n",
            "\n",
            "  Format_Presentation Format_Text-Based Format_Video/Audio-Based  \\\n",
            "0                   0                 1                        0   \n",
            "1                   0                 1                        0   \n",
            "2                   0                 1                        0   \n",
            "3                   0                 1                        0   \n",
            "4                   0                 1                        0   \n",
            "\n",
            "  Event_Major Event Event_Non-Event Related Event_Other Event  CPD_Category  \\\n",
            "0                 0                       0                 1        Medium   \n",
            "1                 0                       1                 0        Medium   \n",
            "2                 0                       1                 0        Medium   \n",
            "3                 0                       1                 0        Medium   \n",
            "4                 0                       1                 0        Medium   \n",
            "\n",
            "   CPD_High  CPD_Low  CPD_Medium  \n",
            "0         0        0           1  \n",
            "1         0        0           1  \n",
            "2         0        0           1  \n",
            "3         0        0           1  \n",
            "4         0        0           1  \n",
            "\n",
            "[5 rows x 55 columns]\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me code, for cpd, group them into effectively:\n",
        "# Low (0 points)\n",
        "# Medium (1-2 points)\n",
        "# High (3+ points), then applies one hot encoding, then check\n",
        "\n",
        "import pandas as pd\n",
        "# Function to categorize CPD points\n",
        "def categorize_cpd(cpd_points):\n",
        "    if cpd_points == 0:\n",
        "        return 'Low'\n",
        "    elif 1 <= cpd_points <= 2:\n",
        "        return 'Medium'\n",
        "    else:\n",
        "        return 'High'\n",
        "\n",
        "# Apply the function to create the CPD_Category column\n",
        "docs_cleaned_desc['CPD_Category'] = docs_cleaned_desc['Cpd'].apply(categorize_cpd)\n",
        "\n",
        "# One-Hot Encode the CPD_Category feature\n",
        "cpd_ohe = pd.get_dummies(docs_cleaned_desc['CPD_Category'], prefix='CPD', dtype=int)\n",
        "\n",
        "# Add the encoded features back to the combined DataFrame\n",
        "docs_cleaned_desc = pd.concat([docs_cleaned_desc, cpd_ohe], axis=1)\n",
        "\n",
        "# Check the result\n",
        "print(docs_cleaned_desc.head())\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "mj8Qq8HbHEqn",
      "metadata": {
        "id": "mj8Qq8HbHEqn"
      },
      "source": [
        "The check suggest new feature are created correctly."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vPKyRGij5A5a",
      "metadata": {
        "id": "vPKyRGij5A5a"
      },
      "source": [
        "**Summary**  \n",
        "The clustering step successfully incorporated the engineered features, resulting in well-defined clusters that align with the business fields, recency, format, event-related aspects, and CPD points. The output showed clear differentiation between clusters, suggesting that the chosen features effectively captured the variations in asset characteristics. This outcome met expectations, confirming that the feature engineering process enhanced the model's ability to identify meaningful and distinct groupings within the dataset.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "L9VcdJJs8a5N",
      "metadata": {
        "id": "L9VcdJJs8a5N"
      },
      "source": [
        "#### Q2b Step 2 scale the data.\n",
        "\n",
        "**Outline**\n",
        "The scaling step ensures that all features contribute equally to the clustering process by eliminating disparities in scale. Since K-means clustering relies on distance calculations, it’s crucial to standardize features to prevent any single feature from dominating the distance metric. This process involves using the StandardScaler from scikit-learn to transform all numeric features to have a mean of 0 and a standard deviation of 1.\n",
        "\n",
        "Steps Taken:\n",
        "- Feature Selection: Selected relevant features (cluster_features) from docs_cleaned_desc along with the TF-IDF matrix (tfidf_df_all) to create the docs_cleaned_desc_cluster dataset.\n",
        "Scaling with StandardScaler:\n",
        "- Applied StandardScaler to all numerical features to standardize them. Ensured all features are on a comparable scale to improve clustering accuracy.\n",
        "- Verifying Data: Checked the resulting dataset to ensure that all values were numerical and within the expected range (Z-scores typically between -3 and 3)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zlT7KsLE3KU_",
      "metadata": {
        "id": "zlT7KsLE3KU_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "939497f8-091d-4e90-f236-9e968e31d1b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 55 columns):\n",
            " #   Column                     Non-Null Count  Dtype         \n",
            "---  ------                     --------------  -----         \n",
            " 0   DMSId                      7550 non-null   int64         \n",
            " 1   FileName                   7550 non-null   object        \n",
            " 2   FileType                   7550 non-null   object        \n",
            " 3   FileSize                   7550 non-null   int64         \n",
            " 4   Title                      7550 non-null   object        \n",
            " 5   StartPublishing            7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing             7550 non-null   object        \n",
            " 7   Cpd                        7550 non-null   int64         \n",
            " 8   Format                     7550 non-null   object        \n",
            " 9   Level                      7550 non-null   object        \n",
            " 10  Description                7550 non-null   object        \n",
            " 11  Region                     7550 non-null   object        \n",
            " 12  Copyright                  7550 non-null   object        \n",
            " 13  EventAgeGroup              6840 non-null   object        \n",
            " 14  Link                       7550 non-null   object        \n",
            " 15  Tags                       6969 non-null   object        \n",
            " 16  DownloadCount              7550 non-null   int64         \n",
            " 17  DurationOnDMS              7550 non-null   float64       \n",
            " 18  AnnualizedDownloadCount    7550 non-null   float64       \n",
            " 19  YearOfStartPublishing      7550 non-null   int32         \n",
            " 20  Year                       7550 non-null   int32         \n",
            " 21  Tag_only                   6969 non-null   object        \n",
            " 22  Tag_only_common            6969 non-null   object        \n",
            " 23  StopPublishing_Year        6068 non-null   float64       \n",
            " 24  Description_Length         6068 non-null   float64       \n",
            " 25  Number of Downloads        6068 non-null   float64       \n",
            " 26  Tokens                     7550 non-null   object        \n",
            " 27  Cleaned_Description        7550 non-null   object        \n",
            " 28  Bigram_Description         7550 non-null   object        \n",
            " 29  general_insurance          7550 non-null   int64         \n",
            " 30  injury_scheme              7550 non-null   int64         \n",
            " 31  actuary_summit             7550 non-null   int64         \n",
            " 32  life_insurance             7550 non-null   int64         \n",
            " 33  enterprise_management      7550 non-null   int64         \n",
            " 34  biennial_convention        7550 non-null   int64         \n",
            " 35  accident_compensation      7550 non-null   int64         \n",
            " 36  Recency_Numerical          7550 non-null   int64         \n",
            " 37  Recency_Category           7550 non-null   object        \n",
            " 38  Recency_Moderately Recent  7550 non-null   int64         \n",
            " 39  Recency_Old                7550 non-null   int64         \n",
            " 40  Recency_Recent             7550 non-null   int64         \n",
            " 41  Recency_Very Old           7550 non-null   int64         \n",
            " 42  Missing_Description        7550 non-null   int64         \n",
            " 43  Simplified_Format          7550 non-null   object        \n",
            " 44  Format_Other               7550 non-null   int64         \n",
            " 45  Format_Presentation        7550 non-null   int64         \n",
            " 46  Format_Text-Based          7550 non-null   int64         \n",
            " 47  Format_Video/Audio-Based   7550 non-null   int64         \n",
            " 48  Event_Major Event          7550 non-null   int64         \n",
            " 49  Event_Non-Event Related    7550 non-null   int64         \n",
            " 50  Event_Other Event          7550 non-null   int64         \n",
            " 51  CPD_Category               7550 non-null   object        \n",
            " 52  CPD_High                   7550 non-null   int64         \n",
            " 53  CPD_Low                    7550 non-null   int64         \n",
            " 54  CPD_Medium                 7550 non-null   int64         \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(27), object(20)\n",
            "memory usage: 3.1+ MB\n"
          ]
        }
      ],
      "source": [
        "docs_cleaned_desc.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "AtT_ymAthhqv",
      "metadata": {
        "id": "AtT_ymAthhqv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7cfb0547-27bf-4544-d830-dd638482f50a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Missing_Description  Format_Other  Format_Presentation  Format_Text-Based  \\\n",
            "0                    0             0                    0                  1   \n",
            "1                    0             0                    0                  1   \n",
            "2                    0             0                    0                  1   \n",
            "3                    0             0                    0                  1   \n",
            "4                    0             0                    0                  1   \n",
            "\n",
            "   Format_Video/Audio-Based  Event_Major Event  Event_Non-Event Related  \\\n",
            "0                         0                  0                        0   \n",
            "1                         0                  0                        1   \n",
            "2                         0                  0                        1   \n",
            "3                         0                  0                        1   \n",
            "4                         0                  0                        1   \n",
            "\n",
            "   Event_Other Event  general_insurance  injury_scheme  ...  study  \\\n",
            "0                  1                  0              0  ...    0.0   \n",
            "1                  0                  0              0  ...    0.0   \n",
            "2                  0                  0              0  ...    0.0   \n",
            "3                  0                  0              0  ...    0.0   \n",
            "4                  0                  0              0  ...    0.0   \n",
            "\n",
            "   superannuation  support  take  technique  think  two  understand  way  \\\n",
            "0             0.0      0.0   0.0        0.0    0.0  0.0         0.0  0.0   \n",
            "1             0.0      0.0   0.0        0.0    0.0  0.0         0.0  0.0   \n",
            "2             0.0      0.0   0.0        0.0    0.0  0.0         0.0  0.0   \n",
            "3             0.0      0.0   0.0        0.0    0.0  0.0         0.0  0.0   \n",
            "4             0.0      0.0   0.0        0.0    0.0  0.0         0.0  0.0   \n",
            "\n",
            "   within  \n",
            "0     0.0  \n",
            "1     0.0  \n",
            "2     0.0  \n",
            "3     0.0  \n",
            "4     0.0  \n",
            "\n",
            "[5 rows x 122 columns]\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Assuming you have a DataFrame called 'docs_cleaned_desc' with your features,\n",
        "# a list or array called 'bert_embeddings' with the BERT embeddings for each document,\n",
        "# and a DataFrame called 'tfidf_df_all' with your TF-IDF features.\n",
        "\n",
        "# Select the columns you want to use for clustering from 'docs_cleaned_desc'\n",
        "\n",
        "cluster_features = docs_cleaned_desc[[ 'Missing_Description', 'Format_Other', 'Format_Presentation',\n",
        "                                       'Format_Text-Based', 'Format_Video/Audio-Based', 'Event_Major Event',\n",
        "                                       'Event_Non-Event Related', 'Event_Other Event','general_insurance','injury_scheme','actuary_summit',\n",
        "                                      'life_insurance','enterprise_management','biennial_convention','accident_compensation','CPD_High','CPD_Low','CPD_Medium',\n",
        "                                      'Recency_Moderately Recent','Recency_Old','Recency_Recent','Recency_Very Old']]\n",
        "\n",
        "\n",
        "# Assuming 'tfidf_df_all' contains your TF-IDF features,\n",
        "# concatenate it with the selected features from 'docs_cleaned_desc'\n",
        "docs_cleaned_desc_cluster = pd.concat([cluster_features, tfidf_df_all], axis=1)\n",
        "\n",
        "# Check the new clustering DataFrame\n",
        "print(docs_cleaned_desc_cluster.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "BIIxDM2n_sX4",
      "metadata": {
        "id": "BIIxDM2n_sX4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ad9726b-6c63-4c35-95b3-b16c9d2a1959"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Columns: 122 entries, Missing_Description to within\n",
            "dtypes: float64(100), int64(22)\n",
            "memory usage: 7.0 MB\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       Missing_Description  Format_Other  Format_Presentation  \\\n",
              "count          7550.000000   7550.000000          7550.000000   \n",
              "mean              0.101060      0.034040             0.318411   \n",
              "std               0.301428      0.181343             0.465891   \n",
              "min               0.000000      0.000000             0.000000   \n",
              "25%               0.000000      0.000000             0.000000   \n",
              "50%               0.000000      0.000000             0.000000   \n",
              "75%               0.000000      0.000000             1.000000   \n",
              "max               1.000000      1.000000             1.000000   \n",
              "\n",
              "       Format_Text-Based  Format_Video/Audio-Based  Event_Major Event  \\\n",
              "count        7550.000000               7550.000000        7550.000000   \n",
              "mean            0.376954                  0.270596           0.526887   \n",
              "std             0.484655                  0.444297           0.499310   \n",
              "min             0.000000                  0.000000           0.000000   \n",
              "25%             0.000000                  0.000000           0.000000   \n",
              "50%             0.000000                  0.000000           1.000000   \n",
              "75%             1.000000                  1.000000           1.000000   \n",
              "max             1.000000                  1.000000           1.000000   \n",
              "\n",
              "       Event_Non-Event Related  Event_Other Event  general_insurance  \\\n",
              "count              7550.000000        7550.000000        7550.000000   \n",
              "mean                  0.257616           0.215497           0.100132   \n",
              "std                   0.437350           0.411194           0.300196   \n",
              "min                   0.000000           0.000000           0.000000   \n",
              "25%                   0.000000           0.000000           0.000000   \n",
              "50%                   0.000000           0.000000           0.000000   \n",
              "75%                   1.000000           0.000000           0.000000   \n",
              "max                   1.000000           1.000000           1.000000   \n",
              "\n",
              "       injury_scheme  ...        study  superannuation      support  \\\n",
              "count    7550.000000  ...  7550.000000     7550.000000  7550.000000   \n",
              "mean        0.038013  ...     0.009423        0.020018     0.009983   \n",
              "std         0.191241  ...     0.072453        0.116266     0.072858   \n",
              "min         0.000000  ...     0.000000        0.000000     0.000000   \n",
              "25%         0.000000  ...     0.000000        0.000000     0.000000   \n",
              "50%         0.000000  ...     0.000000        0.000000     0.000000   \n",
              "75%         0.000000  ...     0.000000        0.000000     0.000000   \n",
              "max         1.000000  ...     1.000000        1.000000     1.000000   \n",
              "\n",
              "              take    technique        think          two   understand  \\\n",
              "count  7550.000000  7550.000000  7550.000000  7550.000000  7550.000000   \n",
              "mean      0.009373     0.005496     0.009034     0.006075     0.007686   \n",
              "std       0.076036     0.051159     0.072548     0.057961     0.063954   \n",
              "min       0.000000     0.000000     0.000000     0.000000     0.000000   \n",
              "25%       0.000000     0.000000     0.000000     0.000000     0.000000   \n",
              "50%       0.000000     0.000000     0.000000     0.000000     0.000000   \n",
              "75%       0.000000     0.000000     0.000000     0.000000     0.000000   \n",
              "max       1.000000     1.000000     1.000000     1.000000     1.000000   \n",
              "\n",
              "              way       within  \n",
              "count  7550.00000  7550.000000  \n",
              "mean      0.00812     0.006579  \n",
              "std       0.06743     0.055883  \n",
              "min       0.00000     0.000000  \n",
              "25%       0.00000     0.000000  \n",
              "50%       0.00000     0.000000  \n",
              "75%       0.00000     0.000000  \n",
              "max       1.00000     0.836793  \n",
              "\n",
              "[8 rows x 122 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a3eb97ab-467b-4cfc-ac49-2c44c8a478e4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Missing_Description</th>\n",
              "      <th>Format_Other</th>\n",
              "      <th>Format_Presentation</th>\n",
              "      <th>Format_Text-Based</th>\n",
              "      <th>Format_Video/Audio-Based</th>\n",
              "      <th>Event_Major Event</th>\n",
              "      <th>Event_Non-Event Related</th>\n",
              "      <th>Event_Other Event</th>\n",
              "      <th>general_insurance</th>\n",
              "      <th>injury_scheme</th>\n",
              "      <th>...</th>\n",
              "      <th>study</th>\n",
              "      <th>superannuation</th>\n",
              "      <th>support</th>\n",
              "      <th>take</th>\n",
              "      <th>technique</th>\n",
              "      <th>think</th>\n",
              "      <th>two</th>\n",
              "      <th>understand</th>\n",
              "      <th>way</th>\n",
              "      <th>within</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.000000</td>\n",
              "      <td>7550.00000</td>\n",
              "      <td>7550.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>0.101060</td>\n",
              "      <td>0.034040</td>\n",
              "      <td>0.318411</td>\n",
              "      <td>0.376954</td>\n",
              "      <td>0.270596</td>\n",
              "      <td>0.526887</td>\n",
              "      <td>0.257616</td>\n",
              "      <td>0.215497</td>\n",
              "      <td>0.100132</td>\n",
              "      <td>0.038013</td>\n",
              "      <td>...</td>\n",
              "      <td>0.009423</td>\n",
              "      <td>0.020018</td>\n",
              "      <td>0.009983</td>\n",
              "      <td>0.009373</td>\n",
              "      <td>0.005496</td>\n",
              "      <td>0.009034</td>\n",
              "      <td>0.006075</td>\n",
              "      <td>0.007686</td>\n",
              "      <td>0.00812</td>\n",
              "      <td>0.006579</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>0.301428</td>\n",
              "      <td>0.181343</td>\n",
              "      <td>0.465891</td>\n",
              "      <td>0.484655</td>\n",
              "      <td>0.444297</td>\n",
              "      <td>0.499310</td>\n",
              "      <td>0.437350</td>\n",
              "      <td>0.411194</td>\n",
              "      <td>0.300196</td>\n",
              "      <td>0.191241</td>\n",
              "      <td>...</td>\n",
              "      <td>0.072453</td>\n",
              "      <td>0.116266</td>\n",
              "      <td>0.072858</td>\n",
              "      <td>0.076036</td>\n",
              "      <td>0.051159</td>\n",
              "      <td>0.072548</td>\n",
              "      <td>0.057961</td>\n",
              "      <td>0.063954</td>\n",
              "      <td>0.06743</td>\n",
              "      <td>0.055883</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>...</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>1.00000</td>\n",
              "      <td>0.836793</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>8 rows × 122 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a3eb97ab-467b-4cfc-ac49-2c44c8a478e4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a3eb97ab-467b-4cfc-ac49-2c44c8a478e4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a3eb97ab-467b-4cfc-ac49-2c44c8a478e4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-299407e7-fc14-4121-a552-c8c173cb3453\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-299407e7-fc14-4121-a552-c8c173cb3453')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-299407e7-fc14-4121-a552-c8c173cb3453 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe"
            }
          },
          "metadata": {},
          "execution_count": 185
        }
      ],
      "source": [
        "# prompt: do me a suitable check of the cluster feature\n",
        "\n",
        "docs_cleaned_desc_cluster.info()\n",
        "docs_cleaned_desc_cluster.describe()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VB3QTN3GccIf",
      "metadata": {
        "id": "VB3QTN3GccIf"
      },
      "source": [
        "scalling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "-QQoxjez3bed",
      "metadata": {
        "id": "-QQoxjez3bed"
      },
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Initialize the StandardScaler\n",
        "scaler = StandardScaler()\n",
        "\n",
        "# Select only numerical features for scaling\n",
        "numerical_features = docs_cleaned_desc_cluster.select_dtypes(include=['number'])\n",
        "\n",
        "# Fit the scaler on the numerical features and transform the data\n",
        "docs_cleaned_desc_cluster_scaled_numeric = pd.DataFrame(\n",
        "    scaler.fit_transform(numerical_features),\n",
        "    columns=numerical_features.columns,\n",
        "    index=numerical_features.index  # Keep original index\n",
        ")\n",
        "\n",
        "# Concatenate scaled numerical features with the remaining features\n",
        "docs_cleaned_desc_cluster_scaled = pd.concat(\n",
        "    [docs_cleaned_desc_cluster_scaled_numeric, docs_cleaned_desc_cluster.drop(columns=numerical_features.columns)],\n",
        "    axis=1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "jfYP1c2j3nUF",
      "metadata": {
        "id": "jfYP1c2j3nUF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7ddd1688-d12b-44f9-a771-6e09c19e3c8e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Index([], dtype='object')\n"
          ]
        }
      ],
      "source": [
        "# prompt: check forme which variable in docs_cleaned_desc_cluster_scaled is no numeric ?\n",
        "\n",
        "non_numeric_cols = docs_cleaned_desc_cluster_scaled.select_dtypes(exclude=['number']).columns\n",
        "print(non_numeric_cols)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d3n-zbqddAV9",
      "metadata": {
        "id": "d3n-zbqddAV9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "919662ef-6347-4020-899e-a0bbea8b01bd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "       Missing_Description  Format_Other  Format_Presentation  \\\n",
            "count         7.550000e+03  7.550000e+03         7.550000e+03   \n",
            "mean          1.505786e-17  4.517358e-17         5.270251e-17   \n",
            "std           1.000066e+00  1.000066e+00         1.000066e+00   \n",
            "min          -3.352921e-01 -1.877213e-01        -6.834903e-01   \n",
            "25%          -3.352921e-01 -1.877213e-01        -6.834903e-01   \n",
            "50%          -3.352921e-01 -1.877213e-01        -6.834903e-01   \n",
            "75%          -3.352921e-01 -1.877213e-01         1.463079e+00   \n",
            "max           2.982474e+00  5.327047e+00         1.463079e+00   \n",
            "\n",
            "       Format_Text-Based  Format_Video/Audio-Based  Event_Major Event  \\\n",
            "count       7.550000e+03              7.550000e+03       7.550000e+03   \n",
            "mean       -7.528930e-18             -8.281823e-17       1.204629e-16   \n",
            "std         1.000066e+00              1.000066e+00       1.000066e+00   \n",
            "min        -7.778284e-01             -6.090832e-01      -1.055302e+00   \n",
            "25%        -7.778284e-01             -6.090832e-01      -1.055302e+00   \n",
            "50%        -7.778284e-01             -6.090832e-01       9.475963e-01   \n",
            "75%         1.285631e+00              1.641812e+00       9.475963e-01   \n",
            "max         1.285631e+00              1.641812e+00       9.475963e-01   \n",
            "\n",
            "       Event_Non-Event Related  Event_Other Event  general_insurance  \\\n",
            "count              7550.000000       7.550000e+03       7.550000e+03   \n",
            "mean                  0.000000       1.505786e-16      -5.270251e-17   \n",
            "std                   1.000066       1.000066e+00       1.000066e+00   \n",
            "min                  -0.589077      -5.241106e-01      -3.335786e-01   \n",
            "25%                  -0.589077      -5.241106e-01      -3.335786e-01   \n",
            "50%                  -0.589077      -5.241106e-01      -3.335786e-01   \n",
            "75%                   1.697571      -5.241106e-01      -3.335786e-01   \n",
            "max                   1.697571       1.907994e+00       2.997795e+00   \n",
            "\n",
            "       injury_scheme  ...         study  superannuation       support  \\\n",
            "count   7.550000e+03  ...  7.550000e+03    7.550000e+03  7.550000e+03   \n",
            "mean    9.034716e-17  ...  3.011572e-17    7.528930e-18  3.011572e-17   \n",
            "std     1.000066e+00  ...  1.000066e+00    1.000066e+00  1.000066e+00   \n",
            "min    -1.987847e-01  ... -1.300629e-01   -1.721855e-01 -1.370276e-01   \n",
            "25%    -1.987847e-01  ... -1.300629e-01   -1.721855e-01 -1.370276e-01   \n",
            "50%    -1.987847e-01  ... -1.300629e-01   -1.721855e-01 -1.370276e-01   \n",
            "75%    -1.987847e-01  ... -1.300629e-01   -1.721855e-01 -1.370276e-01   \n",
            "max     5.030569e+00  ...  1.367298e+01    8.429319e+00  1.358929e+01   \n",
            "\n",
            "               take     technique         think           two    understand  \\\n",
            "count  7.550000e+03  7.550000e+03  7.550000e+03  7.550000e+03  7.550000e+03   \n",
            "mean   4.517358e-17  3.388018e-17 -1.882232e-17 -1.882232e-17  3.011572e-17   \n",
            "std    1.000066e+00  1.000066e+00  1.000066e+00  1.000066e+00  1.000066e+00   \n",
            "min   -1.232796e-01 -1.074408e-01 -1.245279e-01 -1.048111e-01 -1.201906e-01   \n",
            "25%   -1.232796e-01 -1.074408e-01 -1.245279e-01 -1.048111e-01 -1.201906e-01   \n",
            "50%   -1.232796e-01 -1.074408e-01 -1.245279e-01 -1.048111e-01 -1.201906e-01   \n",
            "75%   -1.232796e-01 -1.074408e-01 -1.245279e-01 -1.048111e-01 -1.201906e-01   \n",
            "max    1.302929e+01  1.944060e+01  1.366037e+01  1.714941e+01  1.551701e+01   \n",
            "\n",
            "                way       within  \n",
            "count  7.550000e+03  7550.000000  \n",
            "mean   5.834920e-17     0.000000  \n",
            "std    1.000066e+00     1.000066  \n",
            "min   -1.204360e-01    -0.117740  \n",
            "25%   -1.204360e-01    -0.117740  \n",
            "50%   -1.204360e-01    -0.117740  \n",
            "75%   -1.204360e-01    -0.117740  \n",
            "max    1.471073e+01    14.857301  \n",
            "\n",
            "[8 rows x 122 columns]\n"
          ]
        }
      ],
      "source": [
        "# Check the summary statistics\n",
        "summary_stats = docs_cleaned_desc_cluster_scaled.describe()\n",
        "print(summary_stats)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "FOuXeopVfoam",
      "metadata": {
        "id": "FOuXeopVfoam"
      },
      "source": [
        "check on Z-scores\n",
        "What to Look For:\n",
        "Z-Scores: Values should generally fall between -3 and 3, with most around 0. This further ensures that scaling has normalized the feature distributions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "siMezQgMffU8",
      "metadata": {
        "id": "siMezQgMffU8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d21d262-1c8c-433a-bb4d-4ba29cab59f1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Missing_Description  Format_Other  Format_Presentation  Format_Text-Based  \\\n",
            "0             -0.33527     -0.187709            -0.683445           1.285545   \n",
            "1             -0.33527     -0.187709            -0.683445           1.285545   \n",
            "2             -0.33527     -0.187709            -0.683445           1.285545   \n",
            "3             -0.33527     -0.187709            -0.683445           1.285545   \n",
            "4             -0.33527     -0.187709            -0.683445           1.285545   \n",
            "\n",
            "   Format_Video/Audio-Based  Event_Major Event  Event_Non-Event Related  \\\n",
            "0                 -0.609043          -1.055232                -0.589038   \n",
            "1                 -0.609043          -1.055232                 1.697459   \n",
            "2                 -0.609043          -1.055232                 1.697459   \n",
            "3                 -0.609043          -1.055232                 1.697459   \n",
            "4                 -0.609043          -1.055232                 1.697459   \n",
            "\n",
            "   Event_Other Event  general_insurance  injury_scheme  ...     study  \\\n",
            "0           1.907868          -0.333556      -0.198772  ... -0.130054   \n",
            "1          -0.524076          -0.333556      -0.198772  ... -0.130054   \n",
            "2          -0.524076          -0.333556      -0.198772  ... -0.130054   \n",
            "3          -0.524076          -0.333556      -0.198772  ... -0.130054   \n",
            "4          -0.524076          -0.333556      -0.198772  ... -0.130054   \n",
            "\n",
            "   superannuation   support      take  technique    think       two  \\\n",
            "0       -0.172174 -0.137018 -0.123271  -0.107434 -0.12452 -0.104804   \n",
            "1       -0.172174 -0.137018 -0.123271  -0.107434 -0.12452 -0.104804   \n",
            "2       -0.172174 -0.137018 -0.123271  -0.107434 -0.12452 -0.104804   \n",
            "3       -0.172174 -0.137018 -0.123271  -0.107434 -0.12452 -0.104804   \n",
            "4       -0.172174 -0.137018 -0.123271  -0.107434 -0.12452 -0.104804   \n",
            "\n",
            "   understand       way    within  \n",
            "0   -0.120183 -0.120428 -0.117732  \n",
            "1   -0.120183 -0.120428 -0.117732  \n",
            "2   -0.120183 -0.120428 -0.117732  \n",
            "3   -0.120183 -0.120428 -0.117732  \n",
            "4   -0.120183 -0.120428 -0.117732  \n",
            "\n",
            "[5 rows x 122 columns]\n"
          ]
        }
      ],
      "source": [
        "# Calculate Z-scores\n",
        "z_scores = (docs_cleaned_desc_cluster - docs_cleaned_desc_cluster.mean()) / docs_cleaned_desc_cluster.std()\n",
        "print(z_scores.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91YxzaFCAVv1",
      "metadata": {
        "id": "91YxzaFCAVv1"
      },
      "source": [
        "Examinate Z scores after scaling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "utuUrczAgJgg",
      "metadata": {
        "id": "utuUrczAgJgg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "45106f0d-ba7f-4e56-8ce2-74c874091056"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Sorted Average Z-Scores:\n",
            "                     Average Z-Score\n",
            "CPD_Medium                  0.970596\n",
            "Recency_Very Old            0.752583\n",
            "Event_Major Event           0.526887\n",
            "Format_Text-Based           0.376954\n",
            "Format_Presentation         0.318411\n",
            "...                              ...\n",
            "individual                  0.005897\n",
            "apply                       0.005817\n",
            "premium                     0.005803\n",
            "technique                   0.005496\n",
            "however                     0.005446\n",
            "\n",
            "[122 rows x 1 columns]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 31672 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# `docs_cleaned_desc_cluster` is your standardized DataFrame after scaling\n",
        "# Calculate the average z-score for each column\n",
        "average_z_scores = docs_cleaned_desc_cluster.mean(axis=0)\n",
        "\n",
        "# Convert to DataFrame for easy viewing\n",
        "average_z_scores_df = pd.DataFrame(average_z_scores, columns=['Average Z-Score'])\n",
        "\n",
        "\n",
        "# Sort the average z-scores to easily identify any unusual features\n",
        "average_z_scores_df_sorted = average_z_scores_df.sort_values(by='Average Z-Score', ascending=False)\n",
        "print(\"\\nSorted Average Z-Scores:\")\n",
        "print(average_z_scores_df_sorted)\n",
        "\n",
        "\n",
        "# Plotting the distribution of the average z-scores\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.histplot(average_z_scores_df_sorted['Average Z-Score'], bins=30, kde=True, color='blue')\n",
        "\n",
        "plt.title('Distribution of Average Z-Scores')\n",
        "plt.xlabel('Average Z-Score')\n",
        "plt.ylabel('Frequency')\n",
        "plt.grid(True)\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lkdGonjhf3QX",
      "metadata": {
        "id": "lkdGonjhf3QX"
      },
      "source": [
        "The histogram of the average Z-scores suggests that most features have an average Z-score close to 0, which is what would be expected after proper scaling. In standardization (Z-score normalization), the goal is to center the data around 0 with a standard deviation of 1. Therefore, a distribution with most average Z-scores near 0 indicates that the scaling process was likely performed correctly.\n",
        "\n",
        "However, there are some features with higher average Z-scores, which might indicate that those features have more influence or variation relative to the others. This could be due to natural variations in the data or because these features have a higher impact in distinguishing the data points.\n",
        "\n",
        "The data looks like it's been processed correctly for clustering. The scaling seems appropriate, and there's no immediate indication of any issues. The negative and positive values are expected after standardization, as they indicate how far each value is from the mean in terms of standard deviations."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "DHf3rFFFiGRn",
      "metadata": {
        "id": "DHf3rFFFiGRn"
      },
      "source": [
        "**Summary**\n",
        "The scaling process successfully standardized the features, ensuring that no single feature disproportionately affects the clustering results. The Z-score analysis confirmed that the majority of features are centered around 0, indicating the scaling process was conducted correctly. This preparation step has made the dataset ready for the clustering model, with all features contributing equally to the distance calculations."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "YwsgWHTQhlWR",
      "metadata": {
        "id": "YwsgWHTQhlWR"
      },
      "source": [
        "####Q2b Step3 PCA Dimensionalty reduciton\n",
        "\n",
        "**Outline**\n",
        "The purpose of this step is to reduce the dimensionality of the dataset using Principal Component Analysis (PCA) to make the clustering process more efficient and focused. High-dimensional data, especially with numerous features from TF-IDF vectorization, can be complex and noisy. PCA helps identify and retain the most significant components that explain the majority of variance in the data, simplifying the clustering task.\n",
        "\n",
        "Steps Undertaken:\n",
        "\n",
        "- Data Imputation: Replaced any missing values in docs_cleaned_desc_cluster_scaled with the mean value using SimpleImputer to prepare the dataset for PCA.\n",
        "- Applying PCA: Applied PCA to transform the scaled data and calculate the explained variance for each principal component.\n",
        "- Determining the Optimal Number of Components: Analyzed the cumulative explained variance to decide on the number of components that retain the most variance (e.g., 95%). We selected up to 70 components based on the cumulative variance plot.\n",
        "- Visualizing PCA: Created a heatmap to understand how features contribute to each principal component, which helps to identify which features are most influential."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_jnT-uMvhrTH",
      "metadata": {
        "id": "_jnT-uMvhrTH"
      },
      "source": [
        "The following code applies PCA, and we selected the mean as the imputer for missing values.\n",
        "\n",
        "Rationale: Using the mean as an imputer is suitable when the missing values are few and the data distribution is approximately normal. This approach helps maintain the overall distribution of the dataset, ensuring that the imputed values do not introduce bias or distort the variance, which is crucial for accurately capturing the principal components in PCA."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "XcKI3OLRhprY",
      "metadata": {
        "id": "XcKI3OLRhprY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3fea9e0d-aa39-4258-ccaf-a5ed3b4270ad"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 37896 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.impute import SimpleImputer\n",
        "from sklearn.decomposition import PCA\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Assuming docs_cleaned_desc_cluster_scaled is your scaled data with NaN values\n",
        "# Create an imputer to replace NaN with the mean of the column\n",
        "imputer = SimpleImputer(strategy='mean')  # You can use 'median' or other strategies too\n",
        "\n",
        "# Fit the imputer on your data and transform it\n",
        "docs_cleaned_desc_cluster_scaled_imputed = imputer.fit_transform(docs_cleaned_desc_cluster_scaled)\n",
        "\n",
        "\n",
        "# Assuming docs_cleaned_desc_cluster_scaled is your scaled data\n",
        "# Apply PCA to the data\n",
        "pca = PCA()\n",
        "pca_result = pca.fit_transform(docs_cleaned_desc_cluster_scaled)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(8, 6))\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center', label='Individual Explained Variance')\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid', label='Cumulative Explained Variance')\n",
        "\n",
        "# Adding titles and labels\n",
        "plt.ylabel('Variance')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "plt.legend(loc='best')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "vyRzoQOzn0vP",
      "metadata": {
        "id": "vyRzoQOzn0vP"
      },
      "source": [
        "we want to start with high vairance, should we will use 20 pca"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cO0x3HFzjloi",
      "metadata": {
        "id": "cO0x3HFzjloi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "19d833b9-d2a8-40e9-f3f9-fba7e558bf80"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x800 with 2 Axes>"
            ],
            "image/png": "[encoded data, 164384 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import pandas as pd\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "# Fit PCA\n",
        "pca = PCA(n_components=70)  # Adjust n_components based on your specific requirements\n",
        "pca_result = pca.fit_transform(docs_cleaned_desc_cluster_scaled)\n",
        "\n",
        "# Create a DataFrame for the PCA components\n",
        "pca_df = pd.DataFrame(data=pca.components_, columns=docs_cleaned_desc_cluster.columns)\n",
        "\n",
        "# Create the heatmap\n",
        "plt.figure(figsize=(12, 8))  # Adjust the figure size as needed\n",
        "sns.heatmap(pca_df, cmap='coolwarm', annot=False, cbar=True)\n",
        "plt.title(\"Heatmap of Principal Components vs Features\")\n",
        "plt.xlabel(\"Features\")\n",
        "plt.ylabel(\"Principal Components\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lVyqPCvzrD-w",
      "metadata": {
        "id": "lVyqPCvzrD-w"
      },
      "source": [
        "Use Insights to Refine Feature Engineering:\n",
        "If certain vectorized terms are not correlating as expected, consider refining your preprocessing steps (e.g., removing or adding certain terms, adjusting stopwords).\n",
        "You might also revisit the categorization of non-text features like Recency, Format, and Event Type, ensuring that they are as aligned as possible with the natural groupings in your data.\n",
        "I have added a few stop words to filter out vecotries deisctiopion that have litte meaning in cuslter praitice field."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-33ZLrkbig16",
      "metadata": {
        "id": "-33ZLrkbig16"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "PCA was applied to effectively reduce the dimensionality of the dataset, retaining the most significant components that capture the majority of the variance. By focusing on 70 principal components, we preserved the key information needed for clustering while eliminating noise and redundancy. This dimensionality reduction makes the subsequent clustering more efficient, accurate, and interpretable. Further insights from the PCA heatmap can guide adjustments in feature engineering to enhance clustering results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9BQuTPJprpHL",
      "metadata": {
        "id": "9BQuTPJprpHL"
      },
      "source": [
        "#### Q2b step 4 initial K mean mdoel\n",
        "\n",
        "**Outline**:\n",
        "\n",
        "The purpose of this step is to build the initial K-means clustering model using the PCA-reduced data. Based on the PCA analysis, we selected 70 principal components to capture approximately 80% of the variance, balancing dimensionality reduction while retaining most of the important information. We chose k=8 as the starting point for the number of clusters to identify diverse patterns and groupings within the data.\n",
        "\n",
        "**Business Purpose Connection**:\n",
        "\n",
        "The goal is to streamline the tagging system for the Institute, and K-means fits this purpose well by offering a straightforward, non-overlapping way to group assets, ultimately making it easier for members to navigate and find relevant content.\n",
        "\n",
        "**Why Use K-Means?**\n",
        "1. Simple and Hard-Coded: K-means is straightforward and doesn't require complex parameters, making it easy to implement and understand.\n",
        "\n",
        "2. No Overlapping: It creates clear, non-overlapping clusters, which helps in distinctly separating the data points.\n",
        "\n",
        "3. Easy to Interpret: The centroids provide a clear representation of each cluster (feature importance), making it easier to explain the results.\n",
        "\n",
        "\n",
        "\n",
        "**Steps Undertaken:**\n",
        "- Dimensionality Reduction: Applied PCA to reduce the dataset to 60 principal components, retaining most of the data's variance and eliminating noise.\n",
        "- Initial Clustering: Used K-means with k=8 to identify initial clusters.\n",
        "- Visualization: Reduced the PCA data further to 2 dimensions for easy - visualization, using scatter plots to observe cluster separations and overlaps.\n",
        "Cluster Distribution Analysis: Examined the distribution of data points across the 8 clusters to understand the balance and dominance of each cluster.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Justification of using Kmean for culstering.\n",
        "\n",
        "We selected K-Means for this clustering task because it aligns well with the business goal of simplifying the CPD Hub's tagging system, providing an efficient and interpretable solution for organizing assets.\n",
        "\n",
        "Business consideration:\n",
        ": K-Means offers a straightforward method to group assets without overlap, which addresses the need for a clear and simple tagging system. By creating distinct clusters based on features like business fields, descriptions, and recency, K-Means helps reduce the complexity of the current system and makes it easier for members to navigate and find relevant content. This approach ensures that the tagging system is manageable and user-friendly.\n",
        "\n",
        "\n",
        "Why K-Means is good for on this paritucla busienss enviomrne.t ?\n",
        "\n",
        "1. K-Means is computationally efficient and easy to implement. It performs well with high-dimensional data, especially after reducing dimensions through PCA. Since the Institute might need to run this model frequently after deployment, having a simple and efficient algorithm ensures that it meets the operational needs of the Institute without requiring complex resources.\n",
        "\n",
        "2. Easy to Interpret: TThe centroids in K-Means represent the average values of the features within each cluster, which can be used to understand key drivers behind each grouping. This makes it easy to explain the results to non-technical stakeholders, providing transparency and building confidence in the model. The simplicity in interpretation allows business users to see how certain features influence the groupings, ensuring the solution is understood and trusted.\n",
        "\n",
        "3. K-Means is highly scalable, making it suitable for large datasets like the CPD Hub, which is continuously growing. As the number of assets doubles over the next few years, the scalability of K-Means provides flexibility, ensuring that the model remains effective in the long term and can continue to handle increasing amounts of data without performance issues."
      ],
      "metadata": {
        "id": "ncg6PE7Qml86"
      },
      "id": "ncg6PE7Qml86"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "wiJhErzqV_kX",
      "metadata": {
        "id": "wiJhErzqV_kX"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8PHNjoymv1WT",
      "metadata": {
        "id": "8PHNjoymv1WT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0427aff1-14d6-4132-a9a3-fcadf9a575cd"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 338048 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 2379, 2: 99, 3: 123, 4: 3085, 5: 824, 6: 769, 7: 108, 8: 163}\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Step 1: Apply PCA to reduce dimensions to 80 to retain most of the variance\n",
        "pca = PCA(n_components=80, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled)\n",
        "\n",
        "# Step 2: Apply K-means clustering on the reduced data\n",
        "kmeans1 = KMeans(n_clusters=8, random_state=42, n_init=10)\n",
        "clusters1 = kmeans1.fit_predict(reduced_data)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters1_labeled = clusters1 + 1\n",
        "\n",
        "# Step 3: Further reduce to 2D for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Step 4: Create a scatter plot using the 2D PCA-reduced data\n",
        "plt.figure(figsize=(10, 7))\n",
        "\n",
        "# Adjust the color palette to include enough distinct colors\n",
        "sns.scatterplot(x=pca_components[:, 0], y=pca_components[:, 1], hue=clusters1_labeled, palette='tab10', s=100)\n",
        "\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "\n",
        "# Adjust the legend to ensure all clusters are displayed and are numbered correctly\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans1.n_clusters + 1)])\n",
        "\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters1_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "l2ZbdGTGsZft",
      "metadata": {
        "id": "l2ZbdGTGsZft"
      },
      "source": [
        "**Commentary:**\n",
        "\n",
        "- Imbalanced Clusters: Clusters 1 (2379) and 4 (3085) are significantly larger, indicating they represent the majority of the dataset. This suggests that many assets share common characteristics captured by these clusters.\n",
        "Clusters 2 (99), 3 (123), 7 (108), and 8 (163) are much smaller, representing more niche or less frequent patterns.\n",
        "\n",
        "- Distinct Separation: Clusters 6 and 7 are clearly separated in the plot, indicating well-defined differences in those groups.\n",
        "\n",
        "- Overlap Issue: Noticeable overlap between clusters 1, 2, 3, and 5 suggests these groups share similar features, which might need refining for better differentiation.\n",
        "\n",
        "- Potential Outliers: Small clusters, such as clusters 2 and 7, could represent unique or outlier data points that warrant further investigation."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "sBT8xh0nUh6s",
      "metadata": {
        "id": "sBT8xh0nUh6s"
      },
      "source": [
        "**Summary:**\n",
        "\n",
        "The initial K-means model, using 70 PCA components and k=8, successfully grouped assets into clusters that align with our goal of simplifying the Institute's tagging system. Clusters 1 and 4 are dominant, indicating common characteristics among most assets, while smaller clusters like 2 and 7 suggest niche or outlier groups. There is some overlap between certain clusters, indicating room for refinement, but the separation achieved offers a solid foundation for further analysis. Overall, this model effectively captures distinct patterns, supporting the business objective of creating clearer groupings.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "utfanV0XAouG",
      "metadata": {
        "id": "utfanV0XAouG"
      },
      "source": [
        "####Q2b Step5 Internal validation\n",
        "\n",
        "**Outline:**\n",
        "The aim of this step is to perform internal validation of the K-means clustering model using multiple metrics (WCSS, Davies-Bouldin Index, Silhouette Score, and Dunn Index) to assess the quality and effectiveness of the clusters.\n",
        "\n",
        "These metrics will help determine the optimal number of clusters (k) and guide potential refinements to feature selection, ensuring well-defined and compact clusters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zHteiuurWKaq",
      "metadata": {
        "id": "zHteiuurWKaq"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "jxniY1qJT5c9",
      "metadata": {
        "id": "jxniY1qJT5c9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c12f1440-bb72-43cb-b968-4d5f710a399d"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1500x1000 with 4 Axes>"
            ],
            "image/png": "[encoded data, 165532 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.cluster import KMeans\n",
        "from sklearn.metrics import davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "import numpy as np\n",
        "\n",
        "# Assuming `reduced_data` is your data after applying PCA to 45 components\n",
        "wcss = []\n",
        "db_indices = []\n",
        "silhouette_scores = []\n",
        "dunn_indices = []\n",
        "K = range(2, 15)  # You can adjust this range as needed\n",
        "\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "for k in K:\n",
        "    kmeans1 = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
        "    clusters1 = kmeans1.fit_predict(reduced_data)\n",
        "\n",
        "    # Calculate metrics\n",
        "    wcss.append(kmeans1.inertia_)\n",
        "    db_indices.append(davies_bouldin_score(reduced_data, clusters1))\n",
        "    silhouette_scores.append(silhouette_score(reduced_data, clusters1))\n",
        "    dunn_indices.append(dunn_index(clusters1, kmeans1.cluster_centers_, reduced_data))\n",
        "\n",
        "# Plotting the Metrics\n",
        "plt.figure(figsize=(15, 10))\n",
        "\n",
        "# WCSS Plot\n",
        "plt.subplot(2, 2, 1)\n",
        "plt.plot(K, wcss, 'bo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('WCSS')\n",
        "plt.title('WCSS for Different Values of k')\n",
        "\n",
        "# Davies-Bouldin Index Plot\n",
        "plt.subplot(2, 2, 2)\n",
        "plt.plot(K, db_indices, 'yo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Davies-Bouldin Index')\n",
        "plt.title('Davies-Bouldin Index for Different Values of k')\n",
        "\n",
        "# Silhouette Score Plot\n",
        "plt.subplot(2, 2, 3)\n",
        "plt.plot(K, silhouette_scores, 'go-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Silhouette Score')\n",
        "plt.title('Silhouette Score for Different Values of k')\n",
        "\n",
        "# Dunn Index Plot\n",
        "plt.subplot(2, 2, 4)\n",
        "plt.plot(K, dunn_indices, 'ro-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Dunn Index')\n",
        "plt.title('Dunn Index for Different Values of k')\n",
        "\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cvM08gipJ0uE",
      "metadata": {
        "id": "cvM08gipJ0uE"
      },
      "source": [
        "**WCSS**\n",
        "- The WCSS decreases consistently as k increases, but there's no distinct \"elbow\" point, indicating that adding more clusters continues to improve the fit without a clear optimal cutoff.\n",
        "- Feature Selection Implication: This lack of an elbow suggests that the features might not have strong natural groupings, indicating potential issues with the current feature selection. It could mean that some features included are not helping to create distinct clusters, or that you have too many irrelevant features diluting the impact of more important ones.\n",
        "\n",
        "- Suggestion from WCSS: consider using dimensionality reduction techniques (e.g., PCA) to ensure that only the most relevant features are retained.\n",
        "\n",
        "\n",
        "**Davies-Bouldin Index**\n",
        "- The Davies-Bouldin Index spikes sharply at k=3 and then remains relatively high with minor fluctuations, which is unusual since a good clustering model should have a steadily decreasing or stable index as k increases.\n",
        "- Feature Selection Implication: The spike suggests that as you increase the number of clusters, they overlap significantly, indicating that your current feature set doesn't provide enough separation between clusters. This overlap could be caused by irrelevant or redundant features, leading to confusion in distinguishing between clusters.\n",
        "\n",
        "- Suggestion: Reassess feature selection process, focusing on removing features that might be contributing noise. You could also examine correlations between features and remove those that are highly correlated, as they might be adding redundant information.\n",
        "\n",
        "\n",
        "**Silhouette Score**\n",
        "- The silhouette score is relatively high at k=2 and k=3, but it drops drastically for k values greater than 3, remaining low and fluctuating for larger k values. This indicates poor separation between clusters beyond k=3.\n",
        "- Feature Selection Implication: This drop suggests that the chosen features don't create clear boundaries between clusters, causing them to be less distinct as more clusters are introduced. It indicates that your current features might not capture the true structure of the data.\n",
        "\n",
        "- Next Step: Investigate feature engineering techniques to enhance the distinctiveness of your data points. Consider creating new features that better capture the underlying structure of the data or refining existing features to improve their clustering power.\n",
        "\n",
        "\n",
        "**Dunn Index**\n",
        "- Sharpness Observation: The Dunn Index drops sharply after k=2 and remains low, indicating poor compactness and separation for all k values greater than 2.\n",
        "- Feature Selection Implication: This pattern suggests that your feature set does not support the formation of compact and well-separated clusters beyond two groups. It might indicate that many of the features you're using contribute to noise rather than clarity in your clustering.\n",
        "\n",
        "- Next Step: Focus on reducing dimensionality by retaining only features that show a strong relationship with the clustering outcome. Consider conducting a clustering-specific feature selection method to identify which features have the greatest impact on forming well-defined clusters.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eNL0cwVV1kw1",
      "metadata": {
        "id": "eNL0cwVV1kw1"
      },
      "source": [
        "**Key Summary:**\n",
        "\n",
        "The internal validation metrics reveal that the current feature set is underperforming. The WCSS plot lacks a distinct \"elbow,\" indicating weak natural groupings, while the Davies-Bouldin Index shows considerable overlap at certain k values, pointing to poor separation. Both the Silhouette and Dunn Indexes suggest weak cluster distinctness beyond k=2.\n",
        "\n",
        "Overall, the initial model didn’t meet expectations, and further feature engineering is necessary to enhance clarity and separation before proceeding with K-means clustering."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Sl9wsIlQMXRx",
      "metadata": {
        "id": "Sl9wsIlQMXRx"
      },
      "source": [
        "#### Q2b Step 6  Redo feature selection and PCA .\n",
        "\n",
        "**Objective:**\n",
        "To improve clustering quality based on internal validation metrics, we need to refine the feature selection and PCA process to enhance cluster separation and reduce noise, as suggested by the Davies-Bouldin (DB) Index and other metrics.\n",
        "\n",
        "**Why This Step Is Necessary:**\n",
        "The previous internal validation indicated issues such as overlapping clusters and a lack of clear separation. This suggests that not all features were contributing meaningfully to cluster formation, and some may have introduced noise. By redoing feature selection and PCA, we aim to enhance the clustering model's performance and create more distinct clusters.\n",
        "\n",
        "Adjustments Made:\n",
        "1. Feature Selection Refinement:\n",
        "- Based on the feedback from the Davies-Bouldin Index and our random forest model findings, we have dropped less relevant features that might be contributing noise.\n",
        "\n",
        "- The revised feature set focuses on features more aligned with clustering relevance, ensuring we only retain the most significant variables:\n",
        "- 'Format_Other', 'Format_Video/Audio-Based', 'Event_Non-Event Related'\n",
        "- 'general_insurance', 'injury_scheme', 'actuary_summit', 'life_insurance'\n",
        "'enterprise_management', 'biennial_convention', 'accident_compensation'\n",
        "- 'CPD_High', 'CPD_Low', 'CPD_Medium'\n",
        "- 'Recency_Moderately Recent', 'Recency_Old', 'Recency_Recent', 'Recency_Very Old'\n",
        "\n",
        "This refined selection aims to eliminate redundant or irrelevant features, ensuring that only features with meaningful clustering contributions are retained.\n",
        "\n",
        "2. TF-IDF Vectorization Adjustment:\n",
        "\n",
        "The max_features parameter for TF-IDF has been further lowered to 50, reducing the dimensionality and ensuring that only the most informative terms are included. This change helps reduce overlapping data, improving the Davies-Bouldin Index and other clustering metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "K7mye-udXUZ8",
      "metadata": {
        "id": "K7mye-udXUZ8"
      },
      "source": [
        "**Feature Selection Refinement:**\n",
        "Based on the feedback from the Davies-Bouldin Index and our random forest model findings, we have dropped less relevant features that might be contributing noise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "kG0Sv5BUYHzh",
      "metadata": {
        "id": "kG0Sv5BUYHzh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2501aa2e-8535-4102-d27a-bee3b3e81ba6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 7550 entries, 0 to 7549\n",
            "Data columns (total 55 columns):\n",
            " #   Column                     Non-Null Count  Dtype         \n",
            "---  ------                     --------------  -----         \n",
            " 0   DMSId                      7550 non-null   int64         \n",
            " 1   FileName                   7550 non-null   object        \n",
            " 2   FileType                   7550 non-null   object        \n",
            " 3   FileSize                   7550 non-null   int64         \n",
            " 4   Title                      7550 non-null   object        \n",
            " 5   StartPublishing            7550 non-null   datetime64[ns]\n",
            " 6   StopPublishing             7550 non-null   object        \n",
            " 7   Cpd                        7550 non-null   int64         \n",
            " 8   Format                     7550 non-null   object        \n",
            " 9   Level                      7550 non-null   object        \n",
            " 10  Description                7550 non-null   object        \n",
            " 11  Region                     7550 non-null   object        \n",
            " 12  Copyright                  7550 non-null   object        \n",
            " 13  EventAgeGroup              6840 non-null   object        \n",
            " 14  Link                       7550 non-null   object        \n",
            " 15  Tags                       6969 non-null   object        \n",
            " 16  DownloadCount              7550 non-null   int64         \n",
            " 17  DurationOnDMS              7550 non-null   float64       \n",
            " 18  AnnualizedDownloadCount    7550 non-null   float64       \n",
            " 19  YearOfStartPublishing      7550 non-null   int32         \n",
            " 20  Year                       7550 non-null   int32         \n",
            " 21  Tag_only                   6969 non-null   object        \n",
            " 22  Tag_only_common            6969 non-null   object        \n",
            " 23  StopPublishing_Year        6068 non-null   float64       \n",
            " 24  Description_Length         6068 non-null   float64       \n",
            " 25  Number of Downloads        6068 non-null   float64       \n",
            " 26  Tokens                     7550 non-null   object        \n",
            " 27  Cleaned_Description        7550 non-null   object        \n",
            " 28  Bigram_Description         7550 non-null   object        \n",
            " 29  general_insurance          7550 non-null   int64         \n",
            " 30  injury_scheme              7550 non-null   int64         \n",
            " 31  actuary_summit             7550 non-null   int64         \n",
            " 32  life_insurance             7550 non-null   int64         \n",
            " 33  enterprise_management      7550 non-null   int64         \n",
            " 34  biennial_convention        7550 non-null   int64         \n",
            " 35  accident_compensation      7550 non-null   int64         \n",
            " 36  Recency_Numerical          7550 non-null   int64         \n",
            " 37  Recency_Category           7550 non-null   object        \n",
            " 38  Recency_Moderately Recent  7550 non-null   int64         \n",
            " 39  Recency_Old                7550 non-null   int64         \n",
            " 40  Recency_Recent             7550 non-null   int64         \n",
            " 41  Recency_Very Old           7550 non-null   int64         \n",
            " 42  Missing_Description        7550 non-null   int64         \n",
            " 43  Simplified_Format          7550 non-null   object        \n",
            " 44  Format_Other               7550 non-null   int64         \n",
            " 45  Format_Presentation        7550 non-null   int64         \n",
            " 46  Format_Text-Based          7550 non-null   int64         \n",
            " 47  Format_Video/Audio-Based   7550 non-null   int64         \n",
            " 48  Event_Major Event          7550 non-null   int64         \n",
            " 49  Event_Non-Event Related    7550 non-null   int64         \n",
            " 50  Event_Other Event          7550 non-null   int64         \n",
            " 51  CPD_Category               7550 non-null   object        \n",
            " 52  CPD_High                   7550 non-null   int64         \n",
            " 53  CPD_Low                    7550 non-null   int64         \n",
            " 54  CPD_Medium                 7550 non-null   int64         \n",
            "dtypes: datetime64[ns](1), float64(5), int32(2), int64(27), object(20)\n",
            "memory usage: 3.1+ MB\n"
          ]
        }
      ],
      "source": [
        "docs_cleaned_desc.info()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6goIRav_MXR2",
      "metadata": {
        "id": "6goIRav_MXR2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb7e3fd2-059c-46a6-bad9-901961909eaa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Format_Other  Format_Video/Audio-Based  Event_Non-Event Related  \\\n",
            "0             0                         0                        0   \n",
            "1             0                         0                        1   \n",
            "2             0                         0                        1   \n",
            "3             0                         0                        1   \n",
            "4             0                         0                        1   \n",
            "\n",
            "   general_insurance  injury_scheme  actuary_summit  life_insurance  \\\n",
            "0                  0              0               0               0   \n",
            "1                  0              0               0               0   \n",
            "2                  0              0               0               0   \n",
            "3                  0              0               0               0   \n",
            "4                  0              0               0               0   \n",
            "\n",
            "   enterprise_management  biennial_convention  accident_compensation  ...  \\\n",
            "0                      0                    0                      0  ...   \n",
            "1                      0                    0                      0  ...   \n",
            "2                      0                    0                      0  ...   \n",
            "3                      0                    0                      0  ...   \n",
            "4                      0                    0                      0  ...   \n",
            "\n",
            "   retirement  role  scheme  service  session  set  strategy  study  \\\n",
            "0         0.0   0.0     0.0      0.0      0.0  0.0       0.0    0.0   \n",
            "1         0.0   0.0     0.0      0.0      0.0  0.0       0.0    0.0   \n",
            "2         0.0   0.0     0.0      0.0      0.0  0.0       0.0    0.0   \n",
            "3         0.0   0.0     0.0      0.0      0.0  0.0       0.0    0.0   \n",
            "4         0.0   0.0     0.0      0.0      0.0  0.0       0.0    0.0   \n",
            "\n",
            "   superannuation  support  \n",
            "0             0.0      0.0  \n",
            "1             0.0      0.0  \n",
            "2             0.0      0.0  \n",
            "3             0.0      0.0  \n",
            "4             0.0      0.0  \n",
            "\n",
            "[5 rows x 67 columns]\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "\n",
        "# Select the columns you want to use for clustering from 'docs_cleaned_desc'\n",
        "\n",
        "cluster_features2 = docs_cleaned_desc[[ 'Format_Other', 'Format_Video/Audio-Based',\n",
        "                                       'Event_Non-Event Related','general_insurance','injury_scheme','actuary_summit',\n",
        "                                      'life_insurance','enterprise_management','biennial_convention','accident_compensation','CPD_High','CPD_Low','CPD_Medium',\n",
        "                                      'Recency_Moderately Recent','Recency_Old','Recency_Recent','Recency_Very Old']]\n",
        "\n",
        "\n",
        "# Assuming 'tfidf_df_all' contains your TF-IDF features,\n",
        "# concatenate it with the selected features from 'docs_cleaned_desc'\n",
        "docs_cleaned_desc_cluster2 = pd.concat([cluster_features2, tfidf_df_all2], axis=1)\n",
        "\n",
        "# Check the new clustering DataFrame\n",
        "print(docs_cleaned_desc_cluster2.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_HFpbIruMXR3",
      "metadata": {
        "id": "_HFpbIruMXR3"
      },
      "source": [
        "scalling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "guL-gigBMXR3",
      "metadata": {
        "id": "guL-gigBMXR3"
      },
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Initialize the StandardScaler\n",
        "scaler = StandardScaler()\n",
        "\n",
        "# Select only numerical features for scaling\n",
        "numerical_features2 = docs_cleaned_desc_cluster2.select_dtypes(include=['number'])\n",
        "\n",
        "# Fit the scaler on the numerical features and transform the data\n",
        "docs_cleaned_desc_cluster_scaled_numeric2 = pd.DataFrame(\n",
        "    scaler.fit_transform(numerical_features2),\n",
        "    columns=numerical_features2.columns,\n",
        "    index=numerical_features2.index  # Keep original index\n",
        ")\n",
        "\n",
        "# Concatenate scaled numerical features with the remaining features\n",
        "docs_cleaned_desc_cluster_scaled2 = pd.concat(\n",
        "    [docs_cleaned_desc_cluster_scaled_numeric2, docs_cleaned_desc_cluster2.drop(columns=numerical_features2.columns)],\n",
        "    axis=1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "jVnUqrLwMXR3",
      "metadata": {
        "id": "jVnUqrLwMXR3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "27ced524-029d-4b43-edc9-448b44d16ec7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Index([], dtype='object')\n"
          ]
        }
      ],
      "source": [
        "# prompt: check forme which variable in docs_cleaned_desc_cluster_scaled is no numeric ?\n",
        "\n",
        "non_numeric_cols2 = docs_cleaned_desc_cluster_scaled2.select_dtypes(exclude=['number']).columns\n",
        "print(non_numeric_cols2)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oQzz_G_7MXR3",
      "metadata": {
        "id": "oQzz_G_7MXR3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cdcac6cb-517f-42f5-f5b3-3c063901e889"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "       Format_Other  Format_Video/Audio-Based  Event_Non-Event Related  \\\n",
            "count  7.550000e+03              7.550000e+03              7550.000000   \n",
            "mean   4.517358e-17             -8.281823e-17                 0.000000   \n",
            "std    1.000066e+00              1.000066e+00                 1.000066   \n",
            "min   -1.877213e-01             -6.090832e-01                -0.589077   \n",
            "25%   -1.877213e-01             -6.090832e-01                -0.589077   \n",
            "50%   -1.877213e-01             -6.090832e-01                -0.589077   \n",
            "75%   -1.877213e-01              1.641812e+00                 1.697571   \n",
            "max    5.327047e+00              1.641812e+00                 1.697571   \n",
            "\n",
            "       general_insurance  injury_scheme  actuary_summit  life_insurance  \\\n",
            "count       7.550000e+03   7.550000e+03    7.550000e+03    7.550000e+03   \n",
            "mean       -5.270251e-17   9.034716e-17   -3.764465e-17    3.388018e-17   \n",
            "std         1.000066e+00   1.000066e+00    1.000066e+00    1.000066e+00   \n",
            "min        -3.335786e-01  -1.987847e-01   -3.184114e-01   -2.055282e-01   \n",
            "25%        -3.335786e-01  -1.987847e-01   -3.184114e-01   -2.055282e-01   \n",
            "50%        -3.335786e-01  -1.987847e-01   -3.184114e-01   -2.055282e-01   \n",
            "75%        -3.335786e-01  -1.987847e-01   -3.184114e-01   -2.055282e-01   \n",
            "max         2.997795e+00   5.030569e+00    3.140591e+00    4.865512e+00   \n",
            "\n",
            "       enterprise_management  biennial_convention  accident_compensation  ...  \\\n",
            "count           7.550000e+03         7.550000e+03           7.550000e+03  ...   \n",
            "mean           -3.011572e-17         6.023144e-17           3.011572e-17  ...   \n",
            "std             1.000066e+00         1.000066e+00           1.000066e+00  ...   \n",
            "min            -1.490106e-01        -2.691527e-01          -1.389463e-01  ...   \n",
            "25%            -1.490106e-01        -2.691527e-01          -1.389463e-01  ...   \n",
            "50%            -1.490106e-01        -2.691527e-01          -1.389463e-01  ...   \n",
            "75%            -1.490106e-01        -2.691527e-01          -1.389463e-01  ...   \n",
            "max             6.710930e+00         3.715363e+00           7.197027e+00  ...   \n",
            "\n",
            "         retirement          role        scheme       service       session  \\\n",
            "count  7.550000e+03  7.550000e+03  7.550000e+03  7.550000e+03  7.550000e+03   \n",
            "mean  -4.893804e-17 -3.011572e-17 -4.140911e-17  1.129339e-17  1.317563e-17   \n",
            "std    1.000066e+00  1.000066e+00  1.000066e+00  1.000066e+00  1.000066e+00   \n",
            "min   -1.219594e-01 -1.555469e-01 -1.848856e-01 -1.287657e-01 -1.729721e-01   \n",
            "25%   -1.219594e-01 -1.555469e-01 -1.848856e-01 -1.287657e-01 -1.729721e-01   \n",
            "50%   -1.219594e-01 -1.555469e-01 -1.848856e-01 -1.287657e-01 -1.729721e-01   \n",
            "75%   -1.219594e-01 -1.555469e-01 -1.848856e-01 -1.287657e-01 -1.729721e-01   \n",
            "max    1.044728e+01  9.420097e+00  6.567484e+00  1.148362e+01  7.156337e+00   \n",
            "\n",
            "                set      strategy         study  superannuation       support  \n",
            "count  7.550000e+03  7.550000e+03  7.550000e+03    7.550000e+03  7.550000e+03  \n",
            "mean  -4.140911e-17  3.011572e-17 -7.528930e-18   -3.764465e-18  1.882232e-17  \n",
            "std    1.000066e+00  1.000066e+00  1.000066e+00    1.000066e+00  1.000066e+00  \n",
            "min   -1.392865e-01 -1.352501e-01 -1.337238e-01   -1.757048e-01 -1.406498e-01  \n",
            "25%   -1.392865e-01 -1.352501e-01 -1.337238e-01   -1.757048e-01 -1.406498e-01  \n",
            "50%   -1.392865e-01 -1.352501e-01 -1.337238e-01   -1.757048e-01 -1.406498e-01  \n",
            "75%   -1.392865e-01 -1.352501e-01 -1.337238e-01   -1.757048e-01 -1.406498e-01  \n",
            "max    1.106292e+01  1.006256e+01  1.129936e+01    7.397389e+00  1.129080e+01  \n",
            "\n",
            "[8 rows x 67 columns]\n"
          ]
        }
      ],
      "source": [
        "# Check the summary statistics\n",
        "summary_stats = docs_cleaned_desc_cluster_scaled2.describe()\n",
        "print(summary_stats)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "lrjwMiPsMXR3",
      "metadata": {
        "id": "lrjwMiPsMXR3"
      },
      "source": [
        "check on Z-scores\n",
        "What to Look For:\n",
        "Z-Scores: Values should generally fall between -3 and 3, with most around 0. This further ensures that scaling has normalized the feature distributions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ieYMGO8GMXR4",
      "metadata": {
        "id": "ieYMGO8GMXR4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3fceebf-d9bd-4e2e-8683-f0b7b98bca7f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "   Format_Other  Format_Video/Audio-Based  Event_Non-Event Related  \\\n",
            "0     -0.187709                 -0.609043                -0.589038   \n",
            "1     -0.187709                 -0.609043                 1.697459   \n",
            "2     -0.187709                 -0.609043                 1.697459   \n",
            "3     -0.187709                 -0.609043                 1.697459   \n",
            "4     -0.187709                 -0.609043                 1.697459   \n",
            "\n",
            "   general_insurance  injury_scheme  actuary_summit  life_insurance  \\\n",
            "0          -0.333556      -0.198772        -0.31839       -0.205515   \n",
            "1          -0.333556      -0.198772        -0.31839       -0.205515   \n",
            "2          -0.333556      -0.198772        -0.31839       -0.205515   \n",
            "3          -0.333556      -0.198772        -0.31839       -0.205515   \n",
            "4          -0.333556      -0.198772        -0.31839       -0.205515   \n",
            "\n",
            "   enterprise_management  biennial_convention  accident_compensation  ...  \\\n",
            "0              -0.149001            -0.269135              -0.138937  ...   \n",
            "1              -0.149001            -0.269135              -0.138937  ...   \n",
            "2              -0.149001            -0.269135              -0.138937  ...   \n",
            "3              -0.149001            -0.269135              -0.138937  ...   \n",
            "4              -0.149001            -0.269135              -0.138937  ...   \n",
            "\n",
            "   retirement      role    scheme   service   session       set  strategy  \\\n",
            "0   -0.121951 -0.155537 -0.184873 -0.128757 -0.172961 -0.139277 -0.135241   \n",
            "1   -0.121951 -0.155537 -0.184873 -0.128757 -0.172961 -0.139277 -0.135241   \n",
            "2   -0.121951 -0.155537 -0.184873 -0.128757 -0.172961 -0.139277 -0.135241   \n",
            "3   -0.121951 -0.155537 -0.184873 -0.128757 -0.172961 -0.139277 -0.135241   \n",
            "4   -0.121951 -0.155537 -0.184873 -0.128757 -0.172961 -0.139277 -0.135241   \n",
            "\n",
            "      study  superannuation   support  \n",
            "0 -0.133715       -0.175693 -0.140641  \n",
            "1 -0.133715       -0.175693 -0.140641  \n",
            "2 -0.133715       -0.175693 -0.140641  \n",
            "3 -0.133715       -0.175693 -0.140641  \n",
            "4 -0.133715       -0.175693 -0.140641  \n",
            "\n",
            "[5 rows x 67 columns]\n"
          ]
        }
      ],
      "source": [
        "# Calculate Z-scores\n",
        "z_scores2 = (docs_cleaned_desc_cluster2 - docs_cleaned_desc_cluster2.mean()) / docs_cleaned_desc_cluster2.std()\n",
        "print(z_scores2.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "tQQfh-ZVMXR4",
      "metadata": {
        "id": "tQQfh-ZVMXR4"
      },
      "source": [
        "Examinate Z scores after scaling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_WUwtYx2MXR4",
      "metadata": {
        "id": "_WUwtYx2MXR4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f770dd4f-5671-46d9-c8ad-281cbb5d5fb0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Sorted Average Z-Scores:\n",
            "                          Average Z-Score\n",
            "injury_scheme                9.034716e-17\n",
            "insurance                    6.399590e-17\n",
            "measure                      6.305479e-17\n",
            "financial_service            6.023144e-17\n",
            "biennial_convention          6.023144e-17\n",
            "...                                   ...\n",
            "general_insurance           -5.270251e-17\n",
            "Format_Video/Audio-Based    -8.281823e-17\n",
            "health                      -9.034716e-17\n",
            "Recency_Old                 -9.034716e-17\n",
            "CPD_Medium                  -3.011572e-16\n",
            "\n",
            "[67 rows x 1 columns]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 43152 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "# Calculate the average z-score for each column in the scaled dataset\n",
        "average_z_scores2 = docs_cleaned_desc_cluster_scaled_numeric2.mean(axis=0)\n",
        "\n",
        "# Convert to DataFrame for easy viewing\n",
        "average_z_scores_df2 = pd.DataFrame(average_z_scores2, columns=['Average Z-Score'])\n",
        "\n",
        "# Sort the average z-scores to easily identify any unusual features\n",
        "average_z_scores_df_sorted2 = average_z_scores_df2.sort_values(by='Average Z-Score', ascending=False)\n",
        "print(\"\\nSorted Average Z-Scores:\")\n",
        "print(average_z_scores_df_sorted2)\n",
        "\n",
        "# Plotting the distribution of the average z-scores\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.histplot(average_z_scores_df_sorted2['Average Z-Score'], bins=30, kde=True, color='blue')\n",
        "\n",
        "plt.title('Distribution of Average Z-Scores')\n",
        "plt.xlabel('Average Z-Score')\n",
        "plt.ylabel('Frequency')\n",
        "plt.grid(True)\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1ckSYxbZMXR4",
      "metadata": {
        "id": "1ckSYxbZMXR4"
      },
      "source": [
        "The distribution of average Z-scores shows that most values are centered around 0, with a slight skew towards the left. This indicates that the scaling process has standardized the data well, and the majority of features have a mean close to zero, which is expected in a properly scaled dataset. There are a few features with more extreme Z-scores, but they are not predominant.\n",
        "\n",
        "**Justification for Using the Mean for PCA:**\n",
        "Despite the slight skew, using the mean as an imputer for PCA is still appropriate because:\n",
        "\n",
        "1. Centering Around Zero: PCA assumes that the data is centered around the mean (0). This distribution confirms that most features meet this criterion, ensuring that the principal components capture the maximum variance effectively.\n",
        "\n",
        "2. Variance Calculation: PCA focuses on variance rather than individual data point distributions. Since most data points are close to the mean, the mean imputation ensures that missing values do not distort the overall variance structure.\n",
        "\n",
        "3. Impact on PCA's Performance: The few extreme Z-scores are unlikely to heavily influence the overall PCA results, as PCA emphasizes directions of maximum variance. Hence, mean imputation provides a stable, reliable center for the analysis."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "X4vqrM1JRFc-",
      "metadata": {
        "id": "X4vqrM1JRFc-"
      },
      "source": [
        "PCA: the follwoing codes is to redo our PCA based on the new selected features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "UDVHuEryRGvZ",
      "metadata": {
        "id": "UDVHuEryRGvZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "955741e6-dc55-44ae-d602-ba382cd906a7"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 37432 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.decomposition import PCA\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.impute import SimpleImputer\n",
        "\n",
        "# Assuming 'docs_cleaned_desc_cluster_scaled2' is your scaled data from the previous step without the \"Missing_Description\" feature\n",
        "\n",
        "# Handle any missing values in the scaled data using mean imputation\n",
        "imputer = SimpleImputer(strategy='mean')\n",
        "docs_cleaned_desc_cluster_scaled2_imputed = imputer.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "# Apply PCA to the imputed data\n",
        "pca = PCA()\n",
        "pca_result = pca.fit_transform(docs_cleaned_desc_cluster_scaled2_imputed)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(8, 6))\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center', label='Individual Explained Variance')\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid', label='Cumulative Explained Variance')\n",
        "plt.ylabel('Variance')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "plt.legend(loc='best')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "eTpwxqF2RLEB",
      "metadata": {
        "id": "eTpwxqF2RLEB"
      },
      "source": [
        "**selecting 50 principal components appears to be a good choice**\n",
        "\n",
        "The cumulative explained variance graph shows that around 50 principal components capture approximately 80% of the variance. This means that by choosing 50 components, you're retaining most of the important information in the data while still achieving dimensionality reduction."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "usxo82jElLtt",
      "metadata": {
        "id": "usxo82jElLtt"
      },
      "source": [
        "**Summary:**\n",
        "\n",
        "To improve clustering performance, we refined the feature selection by retaining only those aligned with the clustering model's purpose, guided by internal validation metrics like the Davies-Bouldin Index and our random forest findings.\n",
        "\n",
        "We also adjusted the TF-IDF vectorization to include only the top 50 terms, reducing noise and overlap. After scaling the data, Z-score analysis confirmed that the data is appropriately standardized. Using PCA, we determined that 50 principal components capture around 80% of the variance, balancing dimensionality reduction with retaining key information. These adjustments aim to enhance cluster separation and overall model clarity for the next K-means clustering iteration."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_2zokAbTRfjO",
      "metadata": {
        "id": "_2zokAbTRfjO"
      },
      "source": [
        "#### Q2b Step7  Redo k mean using new selected feature"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "slZs-wdKR3yw",
      "metadata": {
        "id": "slZs-wdKR3yw"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QFplBE7wJ0Wt",
      "metadata": {
        "id": "QFplBE7wJ0Wt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8f04bb2-d959-4f69-aa83-b9a891225ce7"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 126300 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 4936, 2: 204, 3: 112, 4: 1579, 5: 283, 6: 99, 7: 214, 8: 123}\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Apply PCA on the updated scaled data with docs_cleaned_desc_cluster_scaled2\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "\n",
        "# Apply K-means clustering with n=8 on the PCA-reduced data\n",
        "kmeans2_n8 = KMeans(n_clusters=8, random_state=42, n_init=10)\n",
        "clusters2 = kmeans2_n8.fit_predict(reduced_data)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters2_labeled = clusters2 + 1\n",
        "\n",
        "\n",
        "# Visualizing the cluster using PCA compoenets #\n",
        "# Reduce data to 2D using PCA for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components2 = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Scatter plot to visualize the clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components2[:, 0], y=pca_components2[:, 1], hue=clusters2_labeled, palette='tab10', s=100)\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data for k=8')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans2_n8.n_clusters + 1)])\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "DUafsEAULQvk",
      "metadata": {
        "id": "DUafsEAULQvk"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_pgS7Fh8R1KF",
      "metadata": {
        "id": "_pgS7Fh8R1KF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "78ed095e-9e18-4acf-f33b-17a065c5c300"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1500x1000 with 4 Axes>"
            ],
            "image/png": "[encoded data, 162532 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.cluster import KMeans\n",
        "from sklearn.metrics import davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "import numpy as np\n",
        "\n",
        "# Assuming 'reduced_data' is your data after applying PCA with 60 components (as determined previously)\n",
        "wcss = []\n",
        "db_indices = []\n",
        "silhouette_scores = []\n",
        "dunn_indices = []\n",
        "K = range(2, 15)  # Range of k values to test\n",
        "\n",
        "#PCA\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "# Function to calculate the Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "# Iterating through different values of k\n",
        "for k in K:\n",
        "    kmeans1 = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
        "    clusters1 = kmeans1.fit_predict(reduced_data)\n",
        "\n",
        "    # Calculate metrics for each k\n",
        "    wcss.append(kmeans1.inertia_)\n",
        "    db_indices.append(davies_bouldin_score(reduced_data, clusters1))\n",
        "    silhouette_scores.append(silhouette_score(reduced_data, clusters1))\n",
        "    dunn_indices.append(dunn_index(clusters1, kmeans1.cluster_centers_, reduced_data))\n",
        "\n",
        "# Plotting the Metrics\n",
        "plt.figure(figsize=(15, 10))\n",
        "\n",
        "# WCSS Plot\n",
        "plt.subplot(2, 2, 1)\n",
        "plt.plot(K, wcss, 'bo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('WCSS')\n",
        "plt.title('WCSS for Different Values of k')\n",
        "\n",
        "# Davies-Bouldin Index Plot\n",
        "plt.subplot(2, 2, 2)\n",
        "plt.plot(K, db_indices, 'yo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Davies-Bouldin Index')\n",
        "plt.title('Davies-Bouldin Index for Different Values of k')\n",
        "\n",
        "# Silhouette Score Plot\n",
        "plt.subplot(2, 2, 3)\n",
        "plt.plot(K, silhouette_scores, 'go-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Silhouette Score')\n",
        "plt.title('Silhouette Score for Different Values of k')\n",
        "\n",
        "# Dunn Index Plot\n",
        "plt.subplot(2, 2, 4)\n",
        "plt.plot(K, dunn_indices, 'ro-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Dunn Index')\n",
        "plt.title('Dunn Index for Different Values of k')\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yo4P0dacXrvC",
      "metadata": {
        "id": "yo4P0dacXrvC"
      },
      "source": [
        "**Overall Comparison and Conclusion**:\n",
        "\n",
        "There is noticeable improvement across all four metrics. The stability and consistency of the Davies-Bouldin Index, Silhouette Score, and Dunn Index indicate that the adjustments in feature selection and PCA have led to better-defined and more distinct clusters.\n",
        "\n",
        "\n",
        "**WCSS**\n",
        "- Observation: WCSS steadily decreases with k, with a more noticeable drop at k=4 and k=8.\n",
        "- Assessment: Consistency has improved, but the lack of a clear \"elbow\" suggests more refinement is needed.\n",
        "- Verdict: Marginal improvement; the optimal k is still unclear.\n",
        "\n",
        "**Davies-Bouldin Index**\n",
        "- Observation: A gradual rise after k=2, peaking at k=4, with a more stable trend at higher k values.\n",
        "- Assessment: Improved stability and better cluster definition.\n",
        "- Verdict: Noticeable improvement, indicating clearer clusters.\n",
        "\n",
        "**Silhouette Score**\n",
        "- Observation: Highest at k=2, but more stable between k=4 to k=8, with a rise at k=8.\n",
        "- Assessment: Consistency suggests better-defined clusters.\n",
        "- Verdict: Improvement in clustering quality.\n",
        "\n",
        "**Dunn Index**\n",
        "- Observation: Sharp drop after k=2, but more stable and slightly higher for larger k values.\n",
        "- Assessment: Clusters are now more compact and separated.\n",
        "- Verdict: Slight improvement, with k=2 still the strongest candidate.\n",
        "\n",
        "\n",
        "While the plot suggest k-2 might be the best it does not alight with our bueisns objeuct and our domain lkdege from the tagging ysstme, we will insite to try k-10 as a string point\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "i1nFV6FvmA8G",
      "metadata": {
        "id": "i1nFV6FvmA8G"
      },
      "source": [
        "Summary:\n",
        "\n",
        "The initial validation metrics highlighted poor cluster separation and overlap, indicating the need for refinement in feature selection and PCA. After redoing K-means with the improved feature set, the new model now shows clearer, more distinct clusters, better aligning with the goal of simplifying the tagging system. This improved alignment allows us to proceed with further examination of K-means, testing different cluster numbers and distance metrics to optimize the results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "SCPjDDL4bGzg",
      "metadata": {
        "id": "SCPjDDL4bGzg"
      },
      "source": [
        "#### iteration K culster n=8\n",
        "\n",
        "**Outline**\n",
        "Objective: The aim of this step is to apply K-Means clustering with 8 clusters using the refined feature set and PCA-reduced data. This will help us identify more distinct and meaningful clusters, aligning with the goal of simplifying the Institute’s tagging system.\n",
        "\n",
        "Approach:\n",
        "1. PCA Reduction: Reduced the data to 50 principal components to capture around 80% of the variance.\n",
        "2. Clustering: Applied K-Means with n=8 to identify clusters and analyzed the distribution of data points across these clusters.\n",
        "3. Cluster Quality Evaluation: Used the Calinski-Harabasz Index to evaluate the compactness and separation of clusters, where a higher score indicates better-defined clusters.\n",
        "4. Word Cloud Visualization: Generated word clouds for each cluster to understand the common themes and keywords, enhancing interpretability.\n",
        "\n",
        "Reference:\n",
        "We following codes were generate using ChatGPT4 with the fine adjustment from Gemini\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7qsa_sXNbGzl",
      "metadata": {
        "id": "7qsa_sXNbGzl"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "924OV0JcnW1a",
      "metadata": {
        "id": "924OV0JcnW1a"
      },
      "source": [
        "The following codes are to apply kmean to the new PCA features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "tJLq5iv6bGzl",
      "metadata": {
        "id": "tJLq5iv6bGzl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7076dc90-6271-45d5-ecd5-6a966d8411fd"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 126300 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 4936, 2: 204, 3: 112, 4: 1579, 5: 283, 6: 99, 7: 214, 8: 123}\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Apply PCA on the updated scaled data with docs_cleaned_desc_cluster_scaled2\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "\n",
        "# Apply K-means clustering with n=8 on the PCA-reduced data\n",
        "kmeans2_n8 = KMeans(n_clusters=8, random_state=42, n_init=10)\n",
        "clusters2 = kmeans2_n8.fit_predict(reduced_data)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters2_labeled = clusters2 + 1\n",
        "\n",
        "\n",
        "# Visualizing the cluster using PCA compoenets #\n",
        "# Reduce data to 2D using PCA for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components2 = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Scatter plot to visualize the clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components2[:, 0], y=pca_components2[:, 1], hue=clusters2_labeled, palette='tab10', s=100)\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data for k=8')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans2_n8.n_clusters + 1)])\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OjTg1EVBbGzm",
      "metadata": {
        "id": "OjTg1EVBbGzm"
      },
      "source": [
        "Evaluate Clustering Quality Using the Calinski-Harabasz Index, WCSS, DB, Dunn and Shi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "HwSTza-Am7sC",
      "metadata": {
        "id": "HwSTza-Am7sC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5a187a45-5bed-429c-9720-f5d4c557824c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Calinski-Harabasz Index: 246.69239467141193\n",
            "WCSS: 354413.17802405974\n",
            "Davies-Bouldin Index: 2.2110340121432244\n",
            "Dunn Index: 0.17949356426861482\n",
            "Silhouette Score: 0.16582018014970484\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me codes to evelaute this k mean custering . 1 need 5 index, Calinski-Harabasz Index, Wcss , DB, dunn, Silhouette Score\n",
        "\n",
        "import numpy as np\n",
        "from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "\n",
        "# Calculate Calinski-Harabasz Index\n",
        "ch_score = calinski_harabasz_score(reduced_data, clusters2)\n",
        "print(\"Calinski-Harabasz Index:\", ch_score)\n",
        "\n",
        "# Calculate WCSS\n",
        "wcss = kmeans2_n8.inertia_\n",
        "print(\"WCSS:\", wcss)\n",
        "\n",
        "# Calculate Davies-Bouldin Index\n",
        "db_index = davies_bouldin_score(reduced_data, clusters2)\n",
        "print(\"Davies-Bouldin Index:\", db_index)\n",
        "\n",
        "\n",
        "# Function to calculate the Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "dunn_index_value = dunn_index(clusters2, kmeans2_n8.cluster_centers_, reduced_data)\n",
        "print(\"Dunn Index:\", dunn_index_value)\n",
        "\n",
        "\n",
        "# Calculate Silhouette Score\n",
        "silhouette_avg = silhouette_score(reduced_data, clusters2)\n",
        "print(\"Silhouette Score:\", silhouette_avg)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "25P0okzZn8yK",
      "metadata": {
        "id": "25P0okzZn8yK"
      },
      "source": [
        "We save those result in to result_kmean_n8 for later comparision"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3CRdc7t1ntZ-",
      "metadata": {
        "id": "3CRdc7t1ntZ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0c81c131-b90a-4d96-9835-777d5fed756d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'Calinski-Harabasz Index': 246.69239467141193, 'WCSS': 354413.17802405974, 'Davies-Bouldin Index': 2.2110340121432244, 'Dunn Index': 0.17949356426861482, 'Silhouette Score': 0.16582018014970484}\n"
          ]
        }
      ],
      "source": [
        "# prompt: save the 5 inde scoure for this 1st cuslter in result_kmean_n8 , we will compare this with later other iteration\n",
        "\n",
        "result_kmean_n8 = {\n",
        "    'Calinski-Harabasz Index': ch_score,\n",
        "    'WCSS': wcss,\n",
        "    'Davies-Bouldin Index': db_index,\n",
        "    'Dunn Index': dunn_index_value,\n",
        "    'Silhouette Score': silhouette_avg\n",
        "}\n",
        "\n",
        "print(result_kmean_n8)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qpapck7cbGzm",
      "metadata": {
        "id": "qpapck7cbGzm"
      },
      "source": [
        "The evaluation metrics such as the Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, Dunn Index, and Silhouette Score are most meaningful when compared across different K-means models with varying numbers of clusters (k) or different feature sets and distance metrics.\n",
        "\n",
        "So we will comment on how model performance in later part after we complete the other alernative iterations."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5nOvkhrsbGzm",
      "metadata": {
        "id": "5nOvkhrsbGzm"
      },
      "source": [
        "The follwoing codes generate word cloud on description for the culster"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ZKyx2cKXbGzm",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZKyx2cKXbGzm",
        "outputId": "99588069-7e41-4b6b-a551-bc6cef2b48d4"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 434548 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 446472 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 460724 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 378460 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 335656 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 374820 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 402232 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 406420 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me word cloub on the descroption for the above 8 cluster\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Assuming clusters2 and docs_cleaned_desc are your variables from the K-Means clustering with 8 clusters\n",
        "\n",
        "# Combine descriptions based on the updated cluster assignment from kmeans2_n8\n",
        "clustered_descriptions = {}\n",
        "for i in range(len(clusters2)):\n",
        "    cluster = clusters2[i]\n",
        "    description = docs_cleaned_desc['Description'].iloc[i]\n",
        "    if cluster not in clustered_descriptions:\n",
        "        clustered_descriptions[cluster] = description\n",
        "    else:\n",
        "        clustered_descriptions[cluster] += \" \" + description\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans2_n8\n",
        "for cluster, text in clustered_descriptions.items():\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1}')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7OJfNIm7sAOH",
      "metadata": {
        "id": "7OJfNIm7sAOH"
      },
      "source": [
        "The cluster distribution reveals that Cluster 2 (with 4,411 assets) is significantly larger than the others, encompassing a wide range of traditional actuarial fields like General Insurance, Financial Services, and Summits, based on the word cloud. This suggests that Cluster 2 might be too broad and is capturing various themes that could be separated into more distinct categories.\n",
        "\n",
        "Overall , the clusters represent distinct actuarial business fields:\n",
        "\n",
        "Cluster 1: General Actuarial Practice/Professional Development.\n",
        "Cluster 2: Broad category covering General Insurance (GI), Financial Services, and Injury Schemes.\n",
        "Cluster 3: Risk Management, Data Analytics, and Modeling.\n",
        "Cluster 4: Actuarial Events and Conferences.\n",
        "Cluster 5: International Actuarial Associations (AFIR/ERM, ASTIN, IACA).\n",
        "Cluster 6: Public Policy and Government Relations.\n",
        "Cluster 7: Actuarial Modeling Techniques and Machine Learning.\n",
        "Cluster 8: Health Insurance, Mortality, and Climate Change.\n",
        "\n",
        "Issues:\n",
        "Cluster 2 is too broad, covering several fields, indicating a need for further refinement to improve clarity and focus in groupings.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BDSEOFWTbGzm",
      "metadata": {
        "id": "BDSEOFWTbGzm"
      },
      "source": [
        "**Summary**\n",
        "The K-Means model with k=8 demonstrated improved cluster definition and separation compared to previous iterations. The evaluation metrics indicate that while this configuration is a step forward, further refinement and comparison with other k values and distance metrics are essential for identifying the optimal clustering model. The insights gained from the word clouds also align with the goal of simplifying the Institute’s tagging system, showing that this clustering approach is on the right track."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "NkZiJEFlbSAm",
      "metadata": {
        "id": "NkZiJEFlbSAm"
      },
      "source": [
        "#### iteration K culster n=4\n",
        "\n",
        "Outline:\n",
        "\n",
        "**Objective:**\n",
        "The goal of this iteration is to apply K-Means clustering with 4 clusters using the refined feature set and PCA-reduced data, aiming to identify fewer but more distinct clusters to further streamline the Institute’s tagging system.\n",
        "\n",
        "\n",
        "**Approach:**\n",
        "\n",
        "- PCA Reduction: Reduced the dataset to 50 principal components to retain the majority of variance.\n",
        "- Clustering: Applied K-Means with k=4 to group data into fewer clusters, helping identify broader patterns.\n",
        "- Cluster Evaluation: Used Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, Dunn Index, and Silhouette Score to evaluate clustering quality.\n",
        "- Word Cloud Visualization: Created word clouds to analyze the descriptive themes within each cluster.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6pjfaroTbSAx",
      "metadata": {
        "id": "6pjfaroTbSAx"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "utpXluwfpopa",
      "metadata": {
        "id": "utpXluwfpopa"
      },
      "source": [
        "below is the code to run Kmean for culster = 4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1tQgjDIdbSAy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1tQgjDIdbSAy",
        "outputId": "c86518ab-d3db-48d8-b9aa-5a9615d3cb91"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 119696 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 1690, 2: 222, 3: 5424, 4: 214}\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Step 1: Apply PCA on the updated scaled data with docs_cleaned_desc_cluster_scaled2\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "# Step 2: Apply K-means clustering with n=4 on the PCA-reduced data\n",
        "kmeans2_n4 = KMeans(n_clusters=4, random_state=42, n_init=10)\n",
        "clusters2 = kmeans2_n4.fit_predict(reduced_data)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters2_labeled = clusters2 + 1\n",
        "\n",
        "# Step 3: Visualizing the clusters using PCA components\n",
        "# Reduce data to 2D using PCA for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components2 = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Scatter plot to visualize the clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components2[:, 0], y=pca_components2[:, 1], hue=clusters2_labeled, palette='tab10', s=100)\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data for k=4')  # Updated to k=4\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans2_n4.n_clusters + 1)])  # Updated to kmeans2_n4\n",
        "plt.show()\n",
        "\n",
        "# Step 4: Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Y6e1FSGBbSAy",
      "metadata": {
        "id": "Y6e1FSGBbSAy"
      },
      "source": [
        "Cluster Evaluation: Used Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, Dunn Index, and Silhouette Score to evaluate clustering quality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4mkz0awQbSAy",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4mkz0awQbSAy",
        "outputId": "2986456c-14c8-4df2-a606-4b70396f8920"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Calinski-Harabasz Index (k=4): 362.2509662783553\n",
            "WCSS (k=4): 380729.49851597374\n",
            "Davies-Bouldin Index (k=4): 2.663524315522366\n",
            "Dunn Index (k=4): 0.1689216529203727\n",
            "Silhouette Score (k=4): 0.16176387208509146\n",
            "{'Calinski-Harabasz Index': 362.2509662783553, 'WCSS': 380729.49851597374, 'Davies-Bouldin Index': 2.663524315522366, 'Dunn Index': 0.1689216529203727, 'Silhouette Score': 0.16176387208509146}\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "\n",
        "# Calculate Calinski-Harabasz Index\n",
        "ch_score_n4 = calinski_harabasz_score(reduced_data, clusters2)\n",
        "print(\"Calinski-Harabasz Index (k=4):\", ch_score_n4)\n",
        "\n",
        "# Calculate WCSS\n",
        "wcss_n4 = kmeans2_n4.inertia_\n",
        "print(\"WCSS (k=4):\", wcss_n4)\n",
        "\n",
        "# Calculate Davies-Bouldin Index\n",
        "db_index_n4 = davies_bouldin_score(reduced_data, clusters2)\n",
        "print(\"Davies-Bouldin Index (k=4):\", db_index_n4)\n",
        "\n",
        "# Function to calculate the Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "# Calculate Dunn Index for kmeans2_n4\n",
        "dunn_index_value_n4 = dunn_index(clusters2, kmeans2_n4.cluster_centers_, reduced_data)\n",
        "print(\"Dunn Index (k=4):\", dunn_index_value_n4)\n",
        "\n",
        "# Calculate Silhouette Score\n",
        "silhouette_avg_n4 = silhouette_score(reduced_data, clusters2)\n",
        "print(\"Silhouette Score (k=4):\", silhouette_avg_n4)\n",
        "\n",
        "# Save the results for comparison\n",
        "result_kmean_n4 = {\n",
        "    'Calinski-Harabasz Index': ch_score_n4,\n",
        "    'WCSS': wcss_n4,\n",
        "    'Davies-Bouldin Index': db_index_n4,\n",
        "    'Dunn Index': dunn_index_value_n4,\n",
        "    'Silhouette Score': silhouette_avg_n4\n",
        "}\n",
        "\n",
        "print(result_kmean_n4)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3WunEaCsbSAy",
      "metadata": {
        "id": "3WunEaCsbSAy"
      },
      "source": [
        "Comparison between K-Means (k=4) and Original Euclidean K-Means (k=8):\n",
        "\n",
        "- Calinski-Harabasz Index: Higher for K=4 (358.86 vs. 99.28), indicating K=4 has better cluster separation and compactness.\n",
        "\n",
        "- Davies-Bouldin Index: Lower for K=4 (2.69 vs. 4.46), suggesting K=4 clusters are more distinct and better separated.\n",
        "\n",
        "- Silhouette Score: K=4 is positive (0.17) compared to a negative score for K=8 with Cosine Similarity (-0.0379), indicating K=4 clusters are more cohesive.\n",
        "\n",
        "- Dunn Index: Slightly lower for K=4 (0.166 vs. 0.497), but this difference is minor.\n",
        "\n",
        "- WCSS: K=4 has a lower WCSS (390,613 vs. 919,659), indicating a better overall fit to the data.\n",
        "\n",
        "K=4 outperforms K=8 with Cosine Similarity in almost all metrics, indicating that K=4 produces more distinct, cohesive, and compact clusters, making it the more effective clustering approach basd on the internal validation metrics. However, evaluating how assets are grouped within each cluster is essential to ensure the clusters' relevance and meaningfulness to the business context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oTvSib5ZqAJn",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oTvSib5ZqAJn",
        "outputId": "8d1e5c92-1af3-440d-82ce-b8b83620d5b9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'Calinski-Harabasz Index': 362.2509662783553, 'WCSS': 380729.49851597374, 'Davies-Bouldin Index': 2.663524315522366, 'Dunn Index': 0.1689216529203727, 'Silhouette Score': 0.16176387208509146}\n"
          ]
        }
      ],
      "source": [
        "\n",
        "result_kmean_n4 = {\n",
        "    'Calinski-Harabasz Index': ch_score_n4,  # Updated to ch_score_n4\n",
        "    'WCSS': wcss_n4,                        # Updated to wcss_n4\n",
        "    'Davies-Bouldin Index': db_index_n4,     # Updated to db_index_n4\n",
        "    'Dunn Index': dunn_index_value_n4,       # Updated to dunn_index_value_n4\n",
        "    'Silhouette Score': silhouette_avg_n4    # Updated to silhouette_avg_n4\n",
        "}\n",
        "\n",
        "print(result_kmean_n4)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "GQqc7CNQbSAz",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GQqc7CNQbSAz",
        "outputId": "6524825f-622d-439a-9cb5-9945f7f5c122"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 1690, 2: 222, 3: 5424, 4: 214}\n"
          ]
        }
      ],
      "source": [
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "xBAAoPjCbSAz",
      "metadata": {
        "id": "xBAAoPjCbSAz"
      },
      "source": [
        "Word Cloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "h5mOBXpleS1A",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h5mOBXpleS1A",
        "outputId": "92f4e3bc-723e-4ad5-a4a3-9dc19317fff6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 412376 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 390560 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 425536 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 392756 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me world cloud for above cluster\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Assuming clusters2 and docs_cleaned_desc are your variables from the K-Means clustering with 4 clusters\n",
        "\n",
        "# Combine descriptions based on the updated cluster assignment from kmeans2_n4\n",
        "clustered_descriptions = {}\n",
        "for i in range(len(clusters2)):\n",
        "    cluster = clusters2[i]\n",
        "    description = docs_cleaned_desc['Description'].iloc[i]\n",
        "    if cluster not in clustered_descriptions:\n",
        "        clustered_descriptions[cluster] = description\n",
        "    else:\n",
        "        clustered_descriptions[cluster] += \" \" + description\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans2_n4\n",
        "for cluster, text in clustered_descriptions.items():\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1}')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fyuKdfkkqqsJ",
      "metadata": {
        "id": "fyuKdfkkqqsJ"
      },
      "source": [
        "The word clouds and the cluster distribution reveal that Cluster 1 is overwhelmingly large, with 5,418 assets encompassing multiple traditional actuarial fields such as General Insurance (GI), Life Insurance (LI), and Financial Services. This indicates that the cluster isn't meaningful, as it's grouping a wide range of topics together instead of providing distinct and manageable groupings.\n",
        "\n",
        "\n",
        "A more effective clustering should create smaller, more homogeneous groups that reflect specific themes or fields within the actuarial profession. This issue suggests that using just four clusters is too few to capture the nuanced differences in the data, and a higher k value might be more suitable for obtaining more refined clusters."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MwT1qSjzbSAz",
      "metadata": {
        "id": "MwT1qSjzbSAz"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "The K=4 iteration achieved better numerical performance across key metrics like the Calinski-Harabasz Index, WCSS, and Silhouette Score, indicating strong internal cluster separation and cohesion.\n",
        "\n",
        "However, the word cloud analysis revealed that the clusters lacked meaningful distinction, with one cluster (Cluster 1) encompassing too many diverse actuarial fields.\n",
        "\n",
        "This suggests that while the model performed well from a statistical perspective, it failed to produce practically useful and actionable clusters, essentially making it a high-accuracy but low-utility model. A higher K value might be more appropriate to capture the nuanced differences and generate more insightful, business-relevant clusters."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "zZXhSQ2gbYtV",
      "metadata": {
        "id": "zZXhSQ2gbYtV"
      },
      "source": [
        "#### iteration K mean  Cosine distance K=8\n",
        "\n",
        "**Outline:**\n",
        "Objective: The goal of this iteration is to apply K-Means clustering with 8 clusters using a different distance metric, specifically cosine similarity, to explore whether an alternative approach results in more distinct and meaningful clusters that align with business fields.\n",
        "\n",
        "\n",
        "**Why using Cosine similarity**\n",
        "Cosine similarity measures the angle between two vectors, focusing on the orientation, not magnitude, making it effective for capturing relationships in high-dimensional, sparse data. This contrasts with Euclidean distance, which measures absolute distance. Using cosine similarity allows us to capture the relative patterns between assets rather than their raw values, offering a different perspective for clustering.\n",
        "\n",
        "\n",
        "**Approach:**\n",
        "- PCA Reduction: Reduced the data to 50 principal components to capture the majority of variance while retaining essential information.\n",
        "- Cosine Similarity: Calculated the cosine distance matrix and applied K-Means clustering with k=8 using this alternative distance measure.\n",
        "- Cluster Visualization: Visualized the clusters using PCA components in a scatter plot to identify how the assets were grouped under this approach.\n",
        "- Cluster Evaluation: Used Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, Dunn Index, and Silhouette Score to evaluate clustering quality.\n",
        "- Word Cloud Analysis: Generated word clouds for each cluster to identify the predominant themes and assess alignment with actuarial business fields.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "A8kZPpttbYtb",
      "metadata": {
        "id": "A8kZPpttbYtb"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "_tOq4zXQboy3",
      "metadata": {
        "id": "_tOq4zXQboy3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "654a6684-5fd0-4e05-9ab4-b0e5c350f610"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 129664 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 1777, 2: 541, 3: 1838, 4: 586, 5: 1726, 6: 412, 7: 382, 8: 288}\n",
            "Calinski-Harabasz Index for kmeans2_dist8 (k=8): 102.41529373944385\n"
          ]
        }
      ],
      "source": [
        "# prompt: do the keman using n=8 , but this time use althernative idstance , call the result as kmean2_dist8, make sure the visual displot is inculdded\n",
        "\n",
        "import numpy as np\n",
        "from sklearn.cluster import KMeans\n",
        "from sklearn.metrics import pairwise_distances\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.metrics import calinski_harabasz_score\n",
        "\n",
        "\n",
        "# Apply PCA on the updated scaled data with docs_cleaned_desc_cluster_scaled2\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "\n",
        "# Step 2: Calculate cosine distance matrix\n",
        "cosine_distance_matrix = pairwise_distances(reduced_data, metric='cosine')\n",
        "\n",
        "# Apply K-means clustering with n=8 on the PCA-reduced data using cosine similarity\n",
        "# Change algorithm='full' to algorithm='lloyd' or algorithm='elkan'\n",
        "kmeans2_dist8 = KMeans(n_clusters=8, random_state=42, n_init=10)\n",
        "clusters2_dist8 = kmeans2_dist8.fit_predict(cosine_distance_matrix)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters2_labeled_dist8 = clusters2_dist8 + 1\n",
        "\n",
        "\n",
        "# Visualizing the cluster using PCA components #\n",
        "# Reduce data to 2D using PCA for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components2 = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Scatter plot to visualize the clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components2[:, 0], y=pca_components2[:, 1], hue=clusters2_labeled_dist8, palette='tab10', s=100)\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data for k=8 (Cosine Similarity)')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans2_dist8.n_clusters + 1)])\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled_dist8, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)\n",
        "# Evaluate Clustering Quality Using the Calinski-Harabasz Index\n",
        "\n",
        "# Calculate the Calinski-Harabasz Index for kmeans2_dist8\n",
        "ch_score2_dist8 = calinski_harabasz_score(reduced_data, clusters2_dist8)\n",
        "print(f\"Calinski-Harabasz Index for kmeans2_dist8 (k=8): {ch_score2_dist8}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "yNO2SPX5bYtb",
      "metadata": {
        "id": "yNO2SPX5bYtb"
      },
      "source": [
        "- Cluster Evaluation: Used Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, Dunn Index, and Silhouette Score to evaluate clustering quality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cjfEy55fbYtb",
      "metadata": {
        "id": "cjfEy55fbYtb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b6930e98-2109-46a1-e72b-871ad368ce07"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Calinski-Harabasz Index (k=8 with Cosine Similarity): 102.41529373944385\n",
            "WCSS (k=8 with Cosine Similarity): 922584.5144740561\n",
            "Davies-Bouldin Index (k=8 with Cosine Similarity): 4.477821328179322\n",
            "Dunn Index (k=8 with Cosine Similarity): 0.38408392987315587\n",
            "Silhouette Score (k=8 with Cosine Similarity): -0.037470273531059715\n",
            "{'Calinski-Harabasz Index': 102.41529373944385, 'WCSS': 922584.5144740561, 'Davies-Bouldin Index': 4.477821328179322, 'Dunn Index': 0.38408392987315587, 'Silhouette Score': -0.037470273531059715}\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "\n",
        "# Calculate Calinski-Harabasz Index\n",
        "ch_score_dist8 = calinski_harabasz_score(reduced_data, clusters2_dist8)  # Updated to clusters2_dist8\n",
        "print(\"Calinski-Harabasz Index (k=8 with Cosine Similarity):\", ch_score_dist8)\n",
        "\n",
        "# Calculate WCSS\n",
        "wcss_dist8 = kmeans2_dist8.inertia_  # Updated to kmeans2_dist8\n",
        "print(\"WCSS (k=8 with Cosine Similarity):\", wcss_dist8)\n",
        "\n",
        "# Calculate Davies-Bouldin Index\n",
        "db_index_dist8 = davies_bouldin_score(reduced_data, clusters2_dist8)  # Updated to clusters2_dist8\n",
        "print(\"Davies-Bouldin Index (k=8 with Cosine Similarity):\", db_index_dist8)\n",
        "\n",
        "# Function to calculate the Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "# Calculate Dunn Index for kmeans2_dist8\n",
        "dunn_index_value_dist8 = dunn_index(clusters2_dist8, kmeans2_dist8.cluster_centers_, reduced_data)  # Updated\n",
        "print(\"Dunn Index (k=8 with Cosine Similarity):\", dunn_index_value_dist8)\n",
        "\n",
        "# Calculate Silhouette Score\n",
        "silhouette_avg_dist8 = silhouette_score(reduced_data, clusters2_dist8)  # Updated to clusters2_dist8\n",
        "print(\"Silhouette Score (k=8 with Cosine Similarity):\", silhouette_avg_dist8)\n",
        "\n",
        "# Save the results for comparison\n",
        "result_kmean_dist8 = {\n",
        "    'Calinski-Harabasz Index': ch_score_dist8,  # Updated to ch_score_dist8\n",
        "    'WCSS': wcss_dist8,                        # Updated to wcss_dist8\n",
        "    'Davies-Bouldin Index': db_index_dist8,     # Updated to db_index_dist8\n",
        "    'Dunn Index': dunn_index_value_dist8,       # Updated to dunn_index_value_dist8\n",
        "    'Silhouette Score': silhouette_avg_dist8    # Updated to silhouette_avg_dist8\n",
        "}\n",
        "\n",
        "print(result_kmean_dist8)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "m3kssPhabYtb",
      "metadata": {
        "id": "m3kssPhabYtb"
      },
      "source": [
        "Comparison between Cosine Similarity K-Means (k=8) and Original Euclidean K-Means (k=8):\n",
        "\n",
        "**1. Calinski-Harabasz Index:**\n",
        "- Cosine Similarity: 99.28\n",
        "- Original (Euclidean): Higher value (358.86)\n",
        "\n",
        "Insight: The original K-Means model using Euclidean distance has a much higher Calinski-Harabasz Index. This suggests that the original model forms more compact and well-separated clusters compared to the cosine similarity model. This index indicates that the original model is more effective at creating dense and distinct clusters.\n",
        "\n",
        "\n",
        "**2. WCSS :**\n",
        "- Cosine Similarity: 919,659.61\n",
        "- Original (Euclidean): 364,190.06\n",
        "\n",
        "Insight: The original K-Means model has a significantly lower WCSS, indicating that the data points are closer to their cluster centers in the original model. This suggests that the original model does a better job of minimizing intra-cluster variance, leading to tighter clusters.\n",
        "\n",
        "**3. Davies-Bouldin Index:**\n",
        "- Cosine Similarity: 4.46\n",
        "- Original (Euclidean): 2.69\n",
        "\n",
        "Insight: A lower Davies-Bouldin Index signifies better clustering. The original model has a lower value, indicating that it forms clusters that are more distinct from each other. This implies that the original K-Means model has better-defined cluster boundaries.\n",
        "\n",
        "**4. Dunn Index:**\n",
        "- Cosine Similarity: 0.50\n",
        "- Original (Euclidean): 0.16\n",
        "\n",
        "Insight: A higher Dunn Index indicates better clustering quality in terms of maximizing inter-cluster distance and minimizing intra-cluster distance. The cosine similarity model performs better on this metric, suggesting that it achieves greater separation between clusters compared to the original model. This implies that the clusters are more distinct in the cosine similarity model.\n",
        "\n",
        "\n",
        "**5. Silhouette Score:**\n",
        "- Cosine Similarity: -0.038\n",
        "- Original (Euclidean): 0.175\n",
        "\n",
        "Insight: A positive silhouette score close to 1 indicates well-separated clusters, while a negative score suggests overlapping clusters. The original model has a positive score, indicating that the clusters are more distinct, whereas the cosine similarity model's negative score indicates poor clustering quality with significant overlap among clusters.\n",
        "\n",
        "\n",
        "**Overall Insights:**\n",
        "- The original K-Means model using Euclidean distance performs better on most evaluation metrics (Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, and Silhouette Score). This suggests that the original model is more effective at creating compact, distinct, and well-separated clusters.\n",
        "\n",
        "- The cosine similarity model performed better on the Dunn Index, indicating that it achieved better separation between clusters in terms of relative distances. However, its overall clustering quality is weaker compared to the original model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "C8TVHmBXbYtc",
      "metadata": {
        "id": "C8TVHmBXbYtc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3231fd82-0812-4a11-9273-70bc32d08d39"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {0: 1777, 1: 541, 2: 1838, 3: 586, 4: 1726, 5: 412, 6: 382, 7: 288}\n"
          ]
        }
      ],
      "source": [
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_dist8, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "77Wrw91DbYtc",
      "metadata": {
        "id": "77Wrw91DbYtc"
      },
      "source": [
        "Word cloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "IIqf_0cwdVl_",
      "metadata": {
        "id": "IIqf_0cwdVl_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2ff85b9a-5b8b-40e1-bb18-c2c53f6876da"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 393396 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 418692 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 316116 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 438424 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 391872 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 397164 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 326876 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 397568 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me word clouse for the above new cluster\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Assuming clusters2_dist8 and docs_cleaned_desc are your variables from the K-Means clustering with 8 clusters using cosine similarity\n",
        "\n",
        "# Combine descriptions based on the updated cluster assignment from kmeans2_dist8\n",
        "clustered_descriptions = {}\n",
        "for i in range(len(clusters2_dist8)):\n",
        "    cluster = clusters2_dist8[i]\n",
        "    description = docs_cleaned_desc['Description'].iloc[i]\n",
        "    if cluster not in clustered_descriptions:\n",
        "        clustered_descriptions[cluster] = description\n",
        "    else:\n",
        "        clustered_descriptions[cluster] += \" \" + description\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans2_dist8\n",
        "for cluster, text in clustered_descriptions.items():\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1} (Cosine Similarity)')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MJibf_XBupe0",
      "metadata": {
        "id": "MJibf_XBupe0"
      },
      "source": [
        "The clusters suggest distinct themes:\n",
        "\n",
        "Cluster 1 (1,875 assets): Broad actuarial concepts, focusing on risk, data, and modeling—potentially covering a range of actuarial practices.\n",
        "Cluster 2 (401 assets): Catastrophe, young actuaries, and presentations—likely relevant to early-career actuaries and catastrophe risk.\n",
        "Cluster 3 (2,962 assets): Actuarial conventions, insurance, and reports—likely covering general insurance and professional events.\n",
        "Cluster 4 (408 assets): General insurance seminars—centered on insurance practices and industry discussions.\n",
        "Cluster 5 (573 assets): Actuarial summits and conferences—indicating a focus on professional gatherings.\n",
        "Cluster 6 (495 assets): Institute newsletters, health practices, and committees—mainly Institute activities and professional updates.\n",
        "Cluster 7 (546 assets): Financial services, risk management, and enterprise risk—related to financial industry practices.\n",
        "Cluster 8 (290 assets): Injury and compensation schemes—focused on workers' compensation and injury schemes."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "LH-iDlEwbYtc",
      "metadata": {
        "id": "LH-iDlEwbYtc"
      },
      "source": [
        "**Summary**\n",
        "\n",
        "The K-Means clustering using cosine similarity with k=8 provided a refined grouping of assets into distinct actuarial business fields. Compared to the original Euclidean distance clustering, the use of cosine similarity offered a different perspective, focusing more on the direction of the features rather than their magnitudes.\n",
        "\n",
        "Comparison with Original k=8 Clustering:\n",
        "\n",
        "The original K-Means model using Euclidean distance performs better on most evaluation metrics (Calinski-Harabasz Index, WCSS, Davies-Bouldin Index, and Silhouette Score). This suggests that the original model is more effective at creating compact, distinct, and well-separated clusters.\n",
        "\n",
        "The cosine similarity model performed better on the Dunn Index, indicating that it achieved better separation between clusters in terms of relative distances. However, its overall clustering quality is weaker compared to the original model.\n",
        "\n",
        "Conclusion:\n",
        "The original K-Means model using Euclidean distance generally produces better-defined and more cohesive clusters compared to the cosine similarity model. While the cosine similarity approach offers greater separation (as indicated by the Dunn Index), it fails to form distinct clusters overall, as evidenced by the negative silhouette score and lower Calinski-Harabasz Index. Therefore, the original Euclidean distance model is preferable for clustering this dataset effectively.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "TFrfBNnVbYtc",
      "metadata": {
        "id": "TFrfBNnVbYtc"
      },
      "source": [
        "**Overall summary**  \n",
        "*Provide a **clear** and **concise** overall summary here of your conclusions from this section of the notebook.*"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OviUYhKoWvAp",
      "metadata": {
        "id": "OviUYhKoWvAp"
      },
      "source": [
        "#### iteration  Word 2 Vec\n",
        "\n",
        "This section focuses on using Word2Vec embeddings combined with domain-specific knowledge to improve the clustering process. By leveraging the Word2Vec model, we create document embeddings that capture the semantic meaning of each description.\n",
        "\n",
        "These embeddings allow for more nuanced clustering, as documents with similar meanings are grouped together using K-Means. Additionally, domain knowledge of actuarial terms is incorporated to further refine the clusters, ensuring they align with business relevance."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "39jd8m0uWvAu",
      "metadata": {
        "id": "39jd8m0uWvAu"
      },
      "source": [
        "Domain-Specific Knowledge Integration:\n",
        "\n",
        "Domain-specific terms related to actuarial fields (e.g., general insurance, life insurance, data science) were defined and added to the corpus. This step ensures that the Word2Vec model captures domain-relevant semantics, enhancing the quality of the embeddings."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "VFJ3tUD6WvAv",
      "metadata": {
        "id": "VFJ3tUD6WvAv"
      },
      "outputs": [],
      "source": [
        "# Grouped domain-specific terms\n",
        "domain_terms = {\n",
        "    'general_insurance': ['general insurance', 'reinsurance', 'catastrophe', 'IBNR', 'flood', 'storm', 'bushfire', 'motor', 'property', 'professional indemnity'],\n",
        "    'life_insurance': ['life insurance', 'mortality', 'annuity', 'morbidity', 'disability income', 'death', 'endowment', 'longevity'],\n",
        "    'health_disability': ['health insurance', 'disability insurance', 'NDIS', 'participant', 'care', 'modification', 'medical costs'],\n",
        "    'erm': ['ERM', 'enterprise risk management', 'governance', 'compliance', 'operational risk', 'regulatory risk', 'holistic', 'risk appetite'],\n",
        "    'data_science': ['data science', 'machine learning', 'predictive modeling', 'big data', 'analytics', 'statistical'],\n",
        "    'superannuation_wealth': ['superannuation', 'wealth management', 'retirement', 'pensions', 'investment', 'capital guarantee', 'investment objective', 'diversification', 'risk return', 'passive investment', 'active investment', 'retired']\n",
        "}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rLmM-tuuWvAv",
      "metadata": {
        "id": "rLmM-tuuWvAv"
      },
      "source": [
        "We integrate domain-specific knowledge with the primary dataset to improve the quality of the Word2Vec model training.\n",
        "\n",
        "- Step 1: Get the Tokenized Sentences from the Main Dataset\n",
        "- Step 2: Prepare Domain-Specific Sentences\n",
        "- Step 3: Combine the Primary Dataset with Domain-Specific Sentences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "TSOxNAyFWvAv",
      "metadata": {
        "id": "TSOxNAyFWvAv"
      },
      "outputs": [],
      "source": [
        "# Step 1: Get the tokenized sentences from your main dataset\n",
        "primary_sentences = docs_cleaned_desc['Tokens'].tolist()\n",
        "\n",
        "# Step 2: Prepare domain-specific sentences\n",
        "domain_sentences = []\n",
        "for key, terms in domain_terms.items():\n",
        "    for term in terms:\n",
        "        term_tokens = term.split()\n",
        "        for _ in range(10):  # Adjust the number of repetitions as needed, less than before\n",
        "            domain_sentences.append(term_tokens)\n",
        "\n",
        "# Step 3: Combine the primary dataset with the domain-specific sentences\n",
        "combined_sentences = primary_sentences + domain_sentences\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "kmeapUBcWvAv",
      "metadata": {
        "id": "kmeapUBcWvAv"
      },
      "source": [
        "Creating Document Vectors:\n",
        "\n",
        "For each document, an embedding was generated by averaging the Word2Vec vectors of the words within the document. This step converts the textual data into a numerical form that can be used for clustering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "JTVz4m5kWvAv",
      "metadata": {
        "id": "JTVz4m5kWvAv"
      },
      "outputs": [],
      "source": [
        "from gensim.models import Word2Vec\n",
        "\n",
        "# Train Word2Vec model using the combined corpus\n",
        "model = Word2Vec(combined_sentences, vector_size=20, window=4, min_count=1, workers=4, sg=0)\n",
        "\n",
        "# Create document vectors by averaging word vectors\n",
        "def document_vector(tokens):\n",
        "    word_vecs = [model.wv[word] for word in tokens if word in model.wv]\n",
        "    return np.mean(word_vecs, axis=0) if word_vecs else np.zeros(model.vector_size)\n",
        "\n",
        "docs_cleaned_desc['doc_vector'] = docs_cleaned_desc['Tokens'].apply(document_vector)\n",
        "doc_matrix = np.vstack(docs_cleaned_desc['doc_vector'].values)\n",
        "doc_matrix = np.nan_to_num(doc_matrix)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "uTwO5RT1WvAw",
      "metadata": {
        "id": "uTwO5RT1WvAw"
      },
      "source": [
        "performing Principal Component Analysis (PCA) to understand the variance captured by different principal components from the document vector matrix\n",
        "\n",
        "The key steps includes :\n",
        "1. Apply PCA to the Document Vector Matrix:\n",
        "2. Calculate Explained Variance for Each Principal Component:\n",
        "3. Calculate Cumulative Explained Variance:\n",
        "4. Plotting Explained Variance and Cumulative Explained Variance:\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "T6joDIemWvAw",
      "metadata": {
        "id": "T6joDIemWvAw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b3512f45-27be-4e92-ff30-38a74d9b29de"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1200x600 with 2 Axes>"
            ],
            "image/png": "[encoded data, 49816 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Apply PCA to the document vector matrix\n",
        "pca = PCA()\n",
        "pca.fit(doc_matrix)\n",
        "\n",
        "# Explained variance for each principal component\n",
        "explained_variance = pca.explained_variance_ratio_\n",
        "\n",
        "# Cumulative explained variance\n",
        "cumulative_explained_variance = np.cumsum(explained_variance)\n",
        "\n",
        "# Plotting the explained variance and cumulative explained variance\n",
        "plt.figure(figsize=(12, 6))\n",
        "\n",
        "# Explained Variance Plot\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.bar(range(1, len(explained_variance) + 1), explained_variance, alpha=0.5, align='center')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Explained Variance')\n",
        "plt.title('Explained Variance by Principal Components')\n",
        "\n",
        "# Cumulative Explained Variance Plot\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.step(range(1, len(cumulative_explained_variance) + 1), cumulative_explained_variance, where='mid')\n",
        "plt.xlabel('Number of Principal Components')\n",
        "plt.ylabel('Cumulative Explained Variance')\n",
        "plt.title('Cumulative Explained Variance by PCA')\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "OIkokt38WvAw",
      "metadata": {
        "id": "OIkokt38WvAw"
      },
      "source": [
        "Explained Variance Plot (Left):\n",
        "\n",
        "The first principal component explains the highest amount of variance, over 40%, with the subsequent components contributing significantly less.\n",
        "This suggests that the majority of the information in your dataset can be captured by just a few principal components.\n",
        "Cumulative Explained Variance Plot (Right):\n",
        "\n",
        "The cumulative explained variance quickly approaches 100%, meaning that around 15–20 principal components capture most of the variance in the data.\n",
        "This indicates that you can effectively reduce the dimensionality of your data to 15–20 components while retaining most of the original information.\n",
        "\n",
        "**So, what does it mean**:\n",
        "we should use around 15 to 20 principal components. These components capture most of the variance in the data, ensuring you retain the essential information while reducing dimensionality. This range balances the trade-off between capturing the majority of the variance and avoiding unnecessary complexity.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "p-R6f1PWWvAw",
      "metadata": {
        "id": "p-R6f1PWWvAw"
      },
      "source": [
        "Dimensionality Reduction with PCA:\n",
        "\n",
        "PCA was applied to reduce the dimensionality of the document vectors. This step helps to capture the main patterns in the data while reducing noise, making the subsequent clustering more efficient and interpretable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2Scx1QsIWvAw",
      "metadata": {
        "id": "2Scx1QsIWvAw"
      },
      "outputs": [],
      "source": [
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "\n",
        "# Dimensionality reduction with PCA\n",
        "pca = PCA(n_components=4, random_state=42)\n",
        "reduced_doc_matrix = pca.fit_transform(doc_matrix)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "kxj7MWFPWvAw",
      "metadata": {
        "id": "kxj7MWFPWvAw"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ICmZXCr7WvAw",
      "metadata": {
        "id": "ICmZXCr7WvAw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "494a7f21-8c1f-4389-ed4b-a3f56e407f3c"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1500x1000 with 4 Axes>"
            ],
            "image/png": "[encoded data, 154100 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Metrics lists to store results\n",
        "wcss = []\n",
        "db_indices = []\n",
        "silhouette_scores = []\n",
        "dunn_indices = []\n",
        "K = range(2, 12)  # Range of k to test\n",
        "\n",
        "# Function to calculate Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    # Calculate the minimum inter-cluster distance\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    # Calculate the maximum intra-cluster distance\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    # Dunn Index is the ratio of inter-cluster distance to intra-cluster distance\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "# Loop over the range of k values\n",
        "for k in K:\n",
        "    kmeans3 = KMeans(n_clusters=k, random_state=42, n_init=10)\n",
        "    clusters3 = kmeans3.fit_predict(reduced_doc_matrix)  # Use reduced_doc_matrix instead of reduced_data\n",
        "\n",
        "    # Calculate metrics\n",
        "    wcss.append(kmeans3.inertia_)\n",
        "    db_indices.append(davies_bouldin_score(reduced_doc_matrix, clusters3))\n",
        "    silhouette_scores.append(silhouette_score(reduced_doc_matrix, clusters3))\n",
        "    dunn_indices.append(dunn_index(clusters3, kmeans3.cluster_centers_, reduced_doc_matrix))\n",
        "\n",
        "# Plotting the Metrics\n",
        "plt.figure(figsize=(15, 10))\n",
        "\n",
        "# WCSS Plot\n",
        "plt.subplot(2, 2, 1)\n",
        "plt.plot(K, wcss, 'bo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('WCSS')\n",
        "plt.title('WCSS for Different Values of k')\n",
        "\n",
        "# Davies-Bouldin Index Plot\n",
        "plt.subplot(2, 2, 2)\n",
        "plt.plot(K, db_indices, 'yo-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Davies-Bouldin Index')\n",
        "plt.title('Davies-Bouldin Index for Different Values of k')\n",
        "\n",
        "# Silhouette Score Plot\n",
        "plt.subplot(2, 2, 3)\n",
        "plt.plot(K, silhouette_scores, 'go-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Silhouette Score')\n",
        "plt.title('Silhouette Score for Different Values of k')\n",
        "\n",
        "# Dunn Index Plot\n",
        "plt.subplot(2, 2, 4)\n",
        "plt.plot(K, dunn_indices, 'ro-', markersize=8)\n",
        "plt.xlabel('Number of Clusters (k)')\n",
        "plt.ylabel('Dunn Index')\n",
        "plt.title('Dunn Index for Different Values of k')\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Instability in Higher Clusters:**\n",
        "- As we tested for more clusters, the Davies-Bouldin Index increased, and the Silhouette Score and Dunn Index showed instability, suggesting that the clusters were not well-defined. This instability weakens the clustering performance and makes the model unreliable for producing consistent, meaningful groupings."
      ],
      "metadata": {
        "id": "fI3394E90mxd"
      },
      "id": "fI3394E90mxd"
    },
    {
      "cell_type": "markdown",
      "source": [
        "We based on the best suggest to use K=4 to see how good the cluster could be."
      ],
      "metadata": {
        "id": "64EZhMSD0uZ6"
      },
      "id": "64EZhMSD0uZ6"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uh4_VjdEWvAx",
      "metadata": {
        "id": "uh4_VjdEWvAx"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "mTpyDs3gWvAx",
      "metadata": {
        "id": "mTpyDs3gWvAx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2c627150-5390-4e23-818e-e37d109a44ce"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 281552 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 735, 2: 1855, 3: 610, 4: 2383, 5: 477, 6: 273, 7: 955, 8: 262}\n"
          ]
        }
      ],
      "source": [
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Step 4: Apply K-Means clustering using the selected number of clusters (k=4 as identified earlier)\n",
        "kmeans3 = KMeans(n_clusters=8, random_state=42)\n",
        "clusters3 = kmeans3.fit_predict(reduced_doc_matrix)\n",
        "docs_cleaned_desc['Cluster'] = clusters3\n",
        "\n",
        "# Create labeled clusters (adjust cluster labels to start from 1 for easier interpretation if needed)\n",
        "clusters3_labeled = clusters3 + 1\n",
        "\n",
        "# PCA for visualization (reduce to 2D for plotting)\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components = pca_2d.fit_transform(reduced_doc_matrix)\n",
        "\n",
        "# Scatter plot for clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components[:, 0], y=pca_components[:, 1], hue=clusters3_labeled, palette='viridis', s=100)\n",
        "plt.title('K-Means Clustering Results with Word2Vec Embeddings (CBOW)')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1))\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters3_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "print(\"Cluster Distribution:\", cluster_distribution)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Q5Eq8w_VWvAx",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q5Eq8w_VWvAx",
        "outputId": "62616622-d7ff-4f58-c32d-1fb36eb527ee"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 451628 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 380328 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 382272 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 460180 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 340232 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 419324 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 249376 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 414784 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from wordcloud import WordCloud\n",
        "import seaborn as sns\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "# Assuming clusters3 and kmeans3 are your new variables from the third K-Means run\n",
        "\n",
        "# Combine descriptions based on the updated cluster assignment from kmeans3\n",
        "clustered_descriptions = {}\n",
        "for i in range(len(clusters3)):\n",
        "    cluster = clusters3[i]\n",
        "    description = docs_cleaned_desc['Description'].iloc[i]\n",
        "    if cluster not in clustered_descriptions:\n",
        "        clustered_descriptions[cluster] = description\n",
        "    else:\n",
        "        clustered_descriptions[cluster] += \" \" + description\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans3\n",
        "for cluster, text in clustered_descriptions.items():\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1}')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "iH7h0akiWvAx",
      "metadata": {
        "id": "iH7h0akiWvAx"
      },
      "source": [
        "**Summary**  \n",
        "\n",
        "**Optimal Clustering Suggests Too Few Clusters:**\n",
        "The Word2Vec approach consistently pointed to 4 clusters as the optimal number, based on evaluation metrics like Silhouette Score and Dunn Index. However, from our business understanding, we know that the CPD Hub's content should ideally have more than 4 clusters, indicating that this approach is not capturing the complexity of the data effectively.\n",
        "\n",
        "**Instability in Higher Clusters:**\n",
        "As we tested for more clusters, the Davies-Bouldin Index increased, and the Silhouette Score and Dunn Index showed instability, suggesting that the clusters were not well-defined. This instability weakens the clustering performance and makes the model unreliable for producing consistent, meaningful groupings.\n",
        "\n",
        "**Limitations of Word2Vec for This Task:**\n",
        "Although Word2Vec captured the semantic meaning of documents, it did not lead to stable or well-separated clusters, especially when trying to capture the full complexity of CPD Hub assets. This suggests that the approach is not suitable for clustering in this context.\n",
        "\n",
        "**Overall Conclusion:**\n",
        "While Word2Vec embeddings provided some initial value, the approach pointed to too few clusters, missing the complexity needed for the CPD Hub. The metrics also indicated instability, weakening performance and suggesting that this method is not suitable for creating reliable clusters in this business context."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "gx0PmFJKWvAy",
      "metadata": {
        "id": "gx0PmFJKWvAy"
      },
      "source": [
        "#### **Overall summary**  \n",
        "\n",
        "After testing multiple K-Means models, we determined that an 8-cluster solution is the best option. This decision was made after reducing the maximum number of features in the TF-IDF vectorization, as earlier runs indicated that using too many vectors introduced sparsity, which made it difficult for the model to capture meaningful patterns.\n",
        "\n",
        "Internal validation using Silhouette Score, WCSS, Dunn Index, and Davies-Bouldin plots confirmed that the 8-cluster model provides reasonable and interpretable groupings. The word clouds generated from the asset descriptions support this, showing that most clusters correspond to meaningful, real-world categories. We have selected this model as our final solution and will proceed with manual validation in the next phase."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "nhSpAKNJWacv",
      "metadata": {
        "id": "nhSpAKNJWacv"
      },
      "source": [
        "### Q2c - Examine the clustering outputs\n",
        "#### Purpose:\n",
        "The purpose of this section is to thoroughly examine the clustering outputs through manual validation, identify the key characteristics of each cluster, and establish strong connections to the problem context. The goal is to determine whether the clusters are intuitive and appropriately labeled, while ensuring that each cluster’s insights align with the business problem. This examination will confirm whether the clustering model effectively captures the Institute's asset classification needs.\n",
        "\n",
        "We will use a combination of analytical and graphical techniques to identify the most important features defining each cluster and compare features not used in clustering to see if they offer additional insights.\n",
        "\n",
        "#### References:\n",
        "All the codes below are generated using Gemini"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "peJbJ6SFIpQo",
      "metadata": {
        "id": "peJbJ6SFIpQo"
      },
      "source": [
        "#### Final model selected for  K culster n=8\n",
        "Final Model Selected for K-Means Clustering: n = 8\n",
        "The clustering outputs were generated using K-Means with n=8 clusters\n",
        "\n",
        "The following code is a copy from Q2b.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "PQ6WZAW_IpQu",
      "metadata": {
        "id": "PQ6WZAW_IpQu"
      },
      "outputs": [],
      "source": [
        "def keras_deterministic(seed_value= 0):\n",
        "  # force keras to behave deterministically\n",
        "  # source: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development\n",
        "\n",
        "  # Seed value\n",
        "  # Apparently you may use different seed values at each stage\n",
        "  #seed_value= 0\n",
        "\n",
        "  # 1. Set `PYTHONHASHSEED` environment variable at a fixed value\n",
        "  import os\n",
        "  os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "\n",
        "  # 2. Set `python` built-in pseudo-random generator at a fixed value\n",
        "  import random\n",
        "  random.seed(seed_value)\n",
        "\n",
        "  # 3. Set `numpy` pseudo-random generator at a fixed value\n",
        "  import numpy as np\n",
        "  np.random.seed(seed_value)\n",
        "\n",
        "  # 4. Set the `tensorflow` pseudo-random generator at a fixed value\n",
        "  import tensorflow as tf\n",
        "  #tf.random.set_seed(seed_value)\n",
        "  # for later versions:\n",
        "  tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "  # 5. Configure a new global `tensorflow` session\n",
        "  from keras import backend as K\n",
        "  #session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  #sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)\n",
        "  #K.set_session(sess)\n",
        "  # for later versions:\n",
        "  session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "  sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "  tf.compat.v1.keras.backend.set_session(sess)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5drDat3ZIpQu",
      "metadata": {
        "id": "5drDat3ZIpQu"
      },
      "source": [
        "The following codes are to apply kmean to the new PCA features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "UGEmPkD5IpQv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UGEmPkD5IpQv",
        "outputId": "2d8bbff8-92a4-4b7b-c1a4-70830eab2f60"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 126300 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cluster Distribution: {1: 4936, 2: 204, 3: 112, 4: 1579, 5: 283, 6: 99, 7: 214, 8: 123}\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.cluster import KMeans\n",
        "import seaborn as sns\n",
        "import numpy as np\n",
        "\n",
        "# Apply PCA on the updated scaled data with docs_cleaned_desc_cluster_scaled2\n",
        "pca = PCA(n_components=50, random_state=42)\n",
        "reduced_data = pca.fit_transform(docs_cleaned_desc_cluster_scaled2)\n",
        "\n",
        "\n",
        "# Apply K-means clustering with n=8 on the PCA-reduced data\n",
        "kmeans2_n8 = KMeans(n_clusters=8, random_state=42, n_init=10)\n",
        "clusters2 = kmeans2_n8.fit_predict(reduced_data)\n",
        "\n",
        "# Adjust cluster labels to start from 1 instead of 0\n",
        "clusters2_labeled = clusters2 + 1\n",
        "\n",
        "\n",
        "# Visualizing the cluster using PCA compoenets #\n",
        "# Reduce data to 2D using PCA for visualization\n",
        "pca_2d = PCA(n_components=2, random_state=42)\n",
        "pca_components2 = pca_2d.fit_transform(reduced_data)\n",
        "\n",
        "# Scatter plot to visualize the clusters\n",
        "plt.figure(figsize=(10, 7))\n",
        "sns.scatterplot(x=pca_components2[:, 0], y=pca_components2[:, 1], hue=clusters2_labeled, palette='tab10', s=100)\n",
        "plt.title('K-Means Clustering Results with PCA-Reduced Data for k=8')\n",
        "plt.xlabel('Principal Component 1 (PC1)')\n",
        "plt.ylabel('Principal Component 2 (PC2)')\n",
        "plt.legend(title='Cluster', loc='upper right', bbox_to_anchor=(1.15, 1), labels=[str(i) for i in range(1, kmeans2_n8.n_clusters + 1)])\n",
        "plt.show()\n",
        "\n",
        "# Check the distribution of clusters\n",
        "unique, counts = np.unique(clusters2_labeled, return_counts=True)\n",
        "cluster_distribution = dict(zip(unique, counts))\n",
        "\n",
        "print(\"Cluster Distribution:\", cluster_distribution)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3K7oFo70IpQv",
      "metadata": {
        "id": "3K7oFo70IpQv"
      },
      "source": [
        "Evaluate Clustering Quality Using the Calinski-Harabasz Index, WCSS, DB, Dunn and Shi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WJbwmLlcIpQv",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WJbwmLlcIpQv",
        "outputId": "599b9ba1-f6f2-4822-f6b4-963903a080ae"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Calinski-Harabasz Index: 246.69239467141193\n",
            "WCSS: 354413.17802405974\n",
            "Davies-Bouldin Index: 2.2110340121432244\n",
            "Dunn Index: 0.17949356426861482\n",
            "Silhouette Score: 0.16582018014970484\n"
          ]
        }
      ],
      "source": [
        "# prompt: do me codes to evelaute this k mean custering . 1 need 5 index, Calinski-Harabasz Index, Wcss , DB, dunn, Silhouette Score\n",
        "\n",
        "import numpy as np\n",
        "from sklearn.metrics import calinski_harabasz_score, davies_bouldin_score, silhouette_score\n",
        "from scipy.spatial.distance import cdist\n",
        "\n",
        "# Calculate Calinski-Harabasz Index\n",
        "ch_score = calinski_harabasz_score(reduced_data, clusters2)\n",
        "print(\"Calinski-Harabasz Index:\", ch_score)\n",
        "\n",
        "# Calculate WCSS\n",
        "wcss = kmeans2_n8.inertia_\n",
        "print(\"WCSS:\", wcss)\n",
        "\n",
        "# Calculate Davies-Bouldin Index\n",
        "db_index = davies_bouldin_score(reduced_data, clusters2)\n",
        "print(\"Davies-Bouldin Index:\", db_index)\n",
        "\n",
        "\n",
        "# Function to calculate the Dunn Index\n",
        "def dunn_index(clusters, centroids, data):\n",
        "    inter_cluster_distances = np.min(cdist(centroids, centroids)[np.nonzero(cdist(centroids, centroids))])\n",
        "    intra_cluster_distances = np.max([cdist(data[clusters == i], data[clusters == i]).max() for i in np.unique(clusters)])\n",
        "    return inter_cluster_distances / intra_cluster_distances\n",
        "\n",
        "dunn_index_value = dunn_index(clusters2, kmeans2_n8.cluster_centers_, reduced_data)\n",
        "print(\"Dunn Index:\", dunn_index_value)\n",
        "\n",
        "\n",
        "# Calculate Silhouette Score\n",
        "silhouette_avg = silhouette_score(reduced_data, clusters2)\n",
        "print(\"Silhouette Score:\", silhouette_avg)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "MGeBA0WbIpQw",
      "metadata": {
        "id": "MGeBA0WbIpQw"
      },
      "source": [
        "#### Q2c Step 1 Visualizing Word Clouds - Description\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "q6HmlZk8IpQw",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q6HmlZk8IpQw",
        "outputId": "dc9d508b-37b1-4b88-850f-d0dcfdfecb39"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 449808 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 451736 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 494048 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 402420 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 401744 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 420692 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 458456 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 443528 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me word cloub on the descroption for the above 8 cluster\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Assuming clusters2 and docs_cleaned_desc are your variables from the K-Means clustering with 8 clusters\n",
        "\n",
        "# Combine descriptions based on the updated cluster assignment from kmeans2_n8\n",
        "clustered_descriptions = {}\n",
        "for i in range(len(clusters2)):\n",
        "    cluster = clusters2[i]\n",
        "    description = docs_cleaned_desc['Description'].iloc[i]\n",
        "    if cluster not in clustered_descriptions:\n",
        "        clustered_descriptions[cluster] = description\n",
        "    else:\n",
        "        clustered_descriptions[cluster] += \" \" + description\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans2_n8\n",
        "for cluster, text in clustered_descriptions.items():\n",
        "    wordcloud = WordCloud(width=900, height=500, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1}')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "WrUGpgyInefG",
      "metadata": {
        "id": "WrUGpgyInefG"
      },
      "source": [
        "**Cluster 0 - \"Actuarial Summit and Professional Development\"**\n",
        "The dominant themes in this cluster revolve around words such as \"Actuaries Summit,\" \"concurrent,\" \"superannuation,\" and \"leadership,\" suggesting a strong emphasis on events, conferences, and professional development within the actuarial community. This is expected since summits and similar gatherings are significant avenues for actuaries to share knowledge, explore emerging topics, and engage in discussions that shape the profession. Terms like \"risk management\" and \"professional updates\" indicate a focus on broader industry issues, aligning with the idea that this cluster likely represents high-level, generalist discussions pertinent to ongoing developments in the field.\n",
        "\n",
        "The word cloud also shows that this cluster contains content across multiple actuarial topics, which makes sense given that professional events often cover a wide range of themes. This suggests that the assets in this cluster might cater to a more diverse audience, with insights applicable to various practice areas. The inclusion of terms such as \"leadership\" and \"young actuaries\" implies content that also addresses career development and guidance, especially for those newer to the field."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "VSqnulNOnr-v",
      "metadata": {
        "id": "VSqnulNOnr-v"
      },
      "source": [
        "**Cluster 1 - \"Data Modeling and Insurance\"**\n",
        "Cluster 1 is characterized by terms like \"model,\" \"risk,\" \"insurance,\" \"data,\" and \"actuarial,\" which suggest a heavy emphasis on quantitative aspects of actuarial science, particularly around data modeling, risk assessment, and their applications in insurance. These topics are central to actuarial work, highlighting the cluster's relevance to professionals engaged in data analytics, predictive modeling, and advanced statistical techniques. Given that the term \"model\" is highly prominent, this cluster likely contains assets focusing on how actuaries can leverage data to build models that predict insurance claims, assess financial risks, or develop pricing strategies.\n",
        "\n",
        "Interestingly, this cluster might also encompass materials related to data science applications in actuarial practices, blending traditional actuarial skills with emerging trends in data analytics. This reflects a shift in the profession toward incorporating more sophisticated data-driven approaches, aligning with how the industry is evolving to meet contemporary challenges."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oh_p1osxnxoY",
      "metadata": {
        "id": "oh_p1osxnxoY"
      },
      "source": [
        "**Cluster 2 - \"Accident Compensation and Injury Schemes\"**\n",
        "The word cloud for this cluster is dominated by terms like \"accident compensation,\" \"claim,\" \"workers,\" and \"injury,\" which clearly indicate a strong focus on topics related to compensation schemes, injury management, and insurance practices that support these areas. This cluster likely includes assets addressing the intricacies of workers’ compensation, personal injury claims, and the actuarial techniques used to assess and manage these risks.\n",
        "\n",
        "Given the emphasis on \"workers\" and \"injury,\" it's evident that this cluster is particularly relevant to actuaries working in sectors such as workers' compensation insurance, health insurance, and public policy related to injury and accident schemes. The presence of \"scheme\" suggests that discussions may also cover regulatory frameworks and scheme design, offering insights into the legislative and financial aspects of managing these risks."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "v5Ho4t8on3Go",
      "metadata": {
        "id": "v5Ho4t8on3Go"
      },
      "source": [
        "Cluster 3 - \"Financial Services and Insurance Seminars\"\n",
        "Key terms like \"financial services,\" \"general insurance,\" \"seminar,\" and \"convention\" suggest that this cluster is centered on discussions related to the financial services industry and general insurance practices. Given that \"seminar\" and \"convention\" feature prominently, it's likely that this cluster includes content from professional events, workshops, or conferences focused on these topics.\n",
        "\n",
        "The blend of financial services and general insurance points to an intersection where actuaries work across different lines of business, providing insights into risk management, underwriting, pricing, and regulatory compliance. It’s notable that while general insurance and financial services are distinct fields, they often overlap in actuarial practices, as many principles and challenges are shared between these sectors."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "QQz41bc9n-2h",
      "metadata": {
        "id": "QQz41bc9n-2h"
      },
      "source": [
        "Cluster 4 - \"Risk Analysis and Impact Studies\"\n",
        "Words such as \"risk,\" \"impact,\" \"actuaries,\" \"insurance,\" and \"data\" dominate this cluster, indicating a focus on risk analysis, actuarial research, and studies examining the impact of various factors on insurance and financial outcomes. This could cover a range of topics, from how economic shifts affect insurance portfolios to how demographic changes influence retirement planning. The similarity to Cluster 1 is apparent, but this cluster may delve deeper into qualitative aspects of risk, offering a more holistic perspective on how different variables shape the industry.\n",
        "\n",
        "Given the prominence of terms like \"impact\" and \"will,\" there might be a forward-looking element in this cluster, possibly exploring how future trends or uncertainties could influence actuarial practices."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "HqkNQRx2oBJ4",
      "metadata": {
        "id": "HqkNQRx2oBJ4"
      },
      "source": [
        "Cluster 5 - \"Injury Schemes and Compensation Seminars\"\n",
        "Much like Cluster 2, this cluster is characterized by words such as \"injury,\" \"scheme,\" \"seminar,\" and \"workers compensation,\" but the repetition of these terms suggests that this cluster might have a more focused discussion on the practical implementation and challenges of injury compensation schemes. It could contain assets relating to seminar discussions, case studies, and practical applications of actuarial techniques in this specific field.\n",
        "\n",
        "The overlap with Cluster 2 implies that these clusters might serve similar audiences but offer different perspectives—perhaps Cluster 5 is more about the hands-on, day-to-day aspects of managing injury schemes, while Cluster 2 might address broader policy or strategic issues."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jPbKZyLWoErF",
      "metadata": {
        "id": "jPbKZyLWoErF"
      },
      "source": [
        "Cluster 6 - \"Retirement and Superannuation\"\n",
        "The words \"retirement,\" \"income,\" \"superannuation,\" and \"investment\" define this cluster, pointing to a clear focus on retirement planning, income strategies, and the management of superannuation funds. This cluster is likely highly relevant to actuaries involved in pension schemes, financial planning, and long-term investment strategies. The emphasis on \"income\" and \"investment\" suggests that this content may also delve into strategies for ensuring sustainable income streams in retirement, a critical area for many actuaries advising clients or managing retirement funds."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "F8zgysD7oG1P",
      "metadata": {
        "id": "F8zgysD7oG1P"
      },
      "source": [
        "Cluster 7 - \"International Actuarial Studies and Colloquia\"\n",
        "This cluster is unique, featuring terms such as \"AFIR/ERM,\" \"ASTIN,\" \"IACA,\" and \"colloquia,\" which are less common in day-to-day actuarial discussions but significant in international actuarial circles. This suggests a strong focus on international actuarial research, studies, and professional colloquia, possibly representing content from specialized groups or conferences with a global reach. The presence of older or less familiar terms indicates that this cluster might also capture historical content or assets from international actuarial bodies, providing a rich source of knowledge from a diverse, global perspective.\n",
        "\n",
        "This comprehensive analysis not only highlights the distinct themes within each cluster but also underscores how clustering effectively captures the wide-ranging interests and areas of expertise within the actuarial profession."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "UWyJ_UbKSPI4",
      "metadata": {
        "id": "UWyJ_UbKSPI4"
      },
      "source": [
        "#### Q2c Step2 Visualizing Word Clouds: Title\n",
        "\n",
        "The rationale for visualizing word clouds based on titles is that some assets have missing descriptions, making it difficult to capture their content fully. Titles can compensate for this gap, providing additional context and insights into the themes of these assets. Furthermore, certain assets, especially those that are regularly published (e.g., magazines, reports), often have similar titles, which can help us identify patterns or groupings that may not be immediately apparent from the descriptions alone."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "HQp-5wUTMQPj",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HQp-5wUTMQPj",
        "outputId": "4237ee28-fc1e-41da-bebc-78918f0983c8"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 417608 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 436992 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 429664 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 381908 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 372276 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 351004 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 489012 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 431028 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# prompt: do me a word cloud for title for each of the cluster\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "# Assuming clusters2 and docs_cleaned_desc are your variables from the K-Means clustering with 8 clusters\n",
        "\n",
        "# Combine titles based on the cluster assignment from kmeans2_n8\n",
        "clustered_titles = {}\n",
        "for i in range(len(clusters2)):\n",
        "    cluster = clusters2[i]\n",
        "    title = docs_cleaned_desc['Title'].iloc[i]\n",
        "    if cluster not in clustered_titles:\n",
        "        clustered_titles[cluster] = title\n",
        "    else:\n",
        "        clustered_titles[cluster] += \" \" + title\n",
        "\n",
        "# Generate and display word clouds for each cluster from kmeans2_n8 based on titles\n",
        "for cluster, text in clustered_titles.items():\n",
        "    wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)\n",
        "\n",
        "    # Display the word cloud\n",
        "    plt.figure(figsize=(10, 7))\n",
        "    plt.imshow(wordcloud, interpolation='bilinear')\n",
        "    plt.axis('off')\n",
        "    plt.title(f'Word Cloud for Cluster {cluster + 1} (Based on Titles)')  # Adjust cluster number to start from 1 if needed\n",
        "    plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Da4HErwUNQpB",
      "metadata": {
        "id": "Da4HErwUNQpB"
      },
      "source": [
        "#### Q2c Step 3 Deep dive for each culster\n",
        "\n",
        "**Rationale**\n",
        "Given the complexity of 8 clusters, a detailed cluster by cluster analysis helps us understand the unique characteristics of each cluster. By examining multiple metrics, we gain a comprehensive view of what defines each cluster. This process ensures that we capture the distinct features that set each cluster apart.\n",
        "\n",
        "**Outline of Steps:**\n",
        "1. Filter Data by Cluster: Extract data specific to each cluster to focus our analysis.\n",
        "\n",
        "2. Frequency Analysis: For each variable (e.g., CPD, Format, Level, etc.), identify the most common values. This helps us quickly recognize the dominant features within the cluster.\n",
        "\n",
        "3. CPD Proportions: Calculate the CPD distribution within each cluster to understand the emphasis on CPD-related content.\n",
        "\n",
        "4. Visualizing Key Metrics:**\n",
        "\n",
        " - StartPublishing Histogram: Understand the publication timeline, indicating whether assets are more recent or older.\n",
        " - Tag Distribution (Tag_only_common): Identify the main themes or topics within each cluster.\n",
        "\n",
        "5. Word Clouds:\n",
        " - Descriptions: Highlight the central themes and topics using descriptive words.\n",
        " - Titles: Provide additional context, especially for assets with missing descriptions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "o40b86uyOUwD",
      "metadata": {
        "id": "o40b86uyOUwD"
      },
      "source": [
        "######Culster 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "KcLYJvTZZr-W",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KcLYJvTZZr-W",
        "outputId": "0c4ac3f9-c24e-4a4b-b551-7b618dd3b288"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1    518\n",
            "2    206\n",
            "4      9\n",
            "3      1\n",
            "0      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      218\n",
            "Podcast/Vodcast/Audio    197\n",
            "Article                  114\n",
            "Events                    97\n",
            "Publication               90\n",
            "Video                     10\n",
            "Tool and Resources         5\n",
            "Webinar                    3\n",
            "Other                      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2018-08-21    107\n",
            "2019-02-21     92\n",
            "2019-04-11     46\n",
            "2019-04-10     37\n",
            "2019-03-13     20\n",
            "             ... \n",
            "2020-01-23      1\n",
            "2019-03-31      1\n",
            "2019-03-12      1\n",
            "2019-03-05      1\n",
            "2021-01-20      1\n",
            "Name: count, Length: 162, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      218\n",
            "Podcast/Vodcast/Audio    197\n",
            "Article                  114\n",
            "Events                    97\n",
            "Publication               90\n",
            "Video                     10\n",
            "Tool and Resources         5\n",
            "Webinar                    3\n",
            "Other                      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All             732\n",
            "Advanced          2\n",
            "Introductory      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    734\n",
            "International                   1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute         635\n",
            "Speaker/Author    100\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All                  588\n",
            "YAP                   60\n",
            "Retired Actuaries      6\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "personal effectiveness &amp; leadership    133\n",
            "general insurance                           95\n",
            "superannuation                              74\n",
            "professional updates                        73\n",
            "banking                                     71\n",
            "data science                                54\n",
            "business acumen                             45\n",
            "life insurance                              44\n",
            "risk management                             41\n",
            "health                                      35\n",
            "wealth management                           18\n",
            "professionalism training                     1\n",
            "climate change                               1\n",
            "data analytics                               1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 31040 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-240-ddf1ef6d6348>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_0_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 89048 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 451052 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 447704 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Profile Analysis for Cluster 0\n",
        "\n",
        "# Filter the DataFrame for cluster 0\n",
        "cluster_0_df = docs_cleaned_desc[docs_cleaned_desc['Cluster'] == 0]\n",
        "\n",
        "# Analyze characteristics of cluster 0\n",
        "for column in ['Cpd', 'Format', 'StartPublishing', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType', 'Tag_only_common']:\n",
        "    if column in cluster_0_df.columns:\n",
        "        print(f\"\\nAnalysis for {column}:\")\n",
        "        print(cluster_0_df[column].value_counts())  # Display frequency counts for each category\n",
        "\n",
        "# Analyzing CPD and calculating proportions\n",
        "if 'CPD' in cluster_0_df.columns:\n",
        "    print(\"\\nAnalysis for CPD (with proportions):\")\n",
        "    cpd_counts = cluster_0_df['CPD'].value_counts()\n",
        "    cpd_proportions = cpd_counts / len(cluster_0_df)\n",
        "    print(cpd_proportions)\n",
        "\n",
        "\n",
        "# Plot for StartPublishing in Cluster 0\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create a histogram of 'StartPublishing' for cluster 0\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.hist(cluster_0_df['StartPublishing'], bins=20, color='lightblue', edgecolor='black')\n",
        "plt.xlabel('StartPublishing')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of StartPublishing for Cluster 0')\n",
        "plt.grid(True)  # Add grid for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Plot for Tag_only_common in Cluster 0\n",
        "import seaborn as sns\n",
        "\n",
        "# Plot the distribution of Tag_only_common for cluster 0\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=cluster_0_df, palette='viridis')\n",
        "plt.title('Distribution of Tag_only_common for Cluster 0')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Descriptions in Cluster 0\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "# Combine descriptions for cluster 0\n",
        "text_cluster_0 = \" \".join(cluster_0_df['Description'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 0\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_0)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 0 (Based on Descriptions)')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Titles in Cluster 0\n",
        "\n",
        "# Combine titles for cluster 0\n",
        "text_cluster_0 = \" \".join(cluster_0_df['Title'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 0\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_0)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 0 (Based on Titles)')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "G53fw5LFGeRe",
      "metadata": {
        "id": "G53fw5LFGeRe"
      },
      "source": [
        "**Comment on Cluster 0**\n",
        "Cluster 0 appears to encompass regular actuarial productions, including magazines and digital publications. The topics often center around actuarial summits, programs, and youth activities.\n",
        "\n",
        "From the older tags, we notice that \"professional updates\" and \"personal effectiveness\" are the most related themes. These assets are ongoing, with some being published as recently as 2023 and 2024, although not in high numbers. Therefore, this cluster could be labeled as \"Young Actuary Program, and Professional and Personal Development.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "AOYCYjp3Od1z",
      "metadata": {
        "id": "AOYCYjp3Od1z"
      },
      "source": [
        "#####Cluster 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "oDZ4JcihVL6r",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oDZ4JcihVL6r",
        "outputId": "4ca599f7-b83a-496f-eafd-730480585874"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1    1395\n",
            "2     384\n",
            "0      35\n",
            "4      20\n",
            "3      16\n",
            "5       2\n",
            "7       1\n",
            "8       1\n",
            "6       1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Article                  668\n",
            "Publication              384\n",
            "Presentation Slides      372\n",
            "Podcast/Vodcast/Audio    212\n",
            "Video                    139\n",
            "Tool and Resources        45\n",
            "Events                    19\n",
            "Webinar                   11\n",
            "eLearning                  2\n",
            "Workshops                  2\n",
            "Other                      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2018-08-21    306\n",
            "2023-07-31    147\n",
            "2020-02-21     37\n",
            "2020-02-19     36\n",
            "2020-11-13     32\n",
            "             ... \n",
            "2019-02-12      1\n",
            "2022-09-14      1\n",
            "2022-09-26      1\n",
            "2022-09-27      1\n",
            "2022-09-28      1\n",
            "Name: count, Length: 438, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Article                  668\n",
            "Publication              384\n",
            "Presentation Slides      372\n",
            "Podcast/Vodcast/Audio    212\n",
            "Video                    139\n",
            "Tool and Resources        45\n",
            "Events                    19\n",
            "Webinar                   11\n",
            "eLearning                  2\n",
            "Workshops                  2\n",
            "Other                      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All         1847\n",
            "Advanced       8\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    1850\n",
            "Domestic                         3\n",
            "International                    2\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute         1836\n",
            "Speaker/Author      12\n",
            "Public Domain        5\n",
            "Other                2\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All                  1474\n",
            "Retired Actuaries      24\n",
            "YAP                    17\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "general insurance                          300\n",
            "personal effectiveness &amp; leadership    255\n",
            "data science                               202\n",
            "professional updates                       169\n",
            "health                                     114\n",
            "risk management                            100\n",
            "banking                                     98\n",
            "life insurance                              97\n",
            "superannuation                              96\n",
            "business acumen                             61\n",
            "wealth management                           35\n",
            "professionalism training                     5\n",
            "not applicable                               5\n",
            "data analytics                               3\n",
            "climate change                               3\n",
            "climate change working group                 2\n",
            "other                                        1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 29920 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-241-db5ec26880d1>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_1_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 100136 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 381124 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 415852 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Profile Analysis for Cluster 1\n",
        "\n",
        "# Filter the DataFrame for cluster 1\n",
        "cluster_1_df = docs_cleaned_desc[docs_cleaned_desc['Cluster'] == 1]\n",
        "\n",
        "# Analyze characteristics of cluster 1\n",
        "for column in ['Cpd', 'Format', 'StartPublishing', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType', 'Tag_only_common']:\n",
        "    if column in cluster_1_df.columns:\n",
        "        print(f\"\\nAnalysis for {column}:\")\n",
        "        print(cluster_1_df[column].value_counts())  # Display frequency counts for each category\n",
        "\n",
        "# Analyzing CPD and calculating proportions\n",
        "if 'CPD' in cluster_1_df.columns:\n",
        "    print(\"\\nAnalysis for CPD (with proportions):\")\n",
        "    cpd_counts = cluster_1_df['CPD'].value_counts()\n",
        "    cpd_proportions = cpd_counts / len(cluster_1_df)\n",
        "    print(cpd_proportions)\n",
        "\n",
        "\n",
        "# Plot for StartPublishing in Cluster 1\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create a histogram of 'StartPublishing' for cluster 1\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.hist(cluster_1_df['StartPublishing'], bins=20, color='lightblue', edgecolor='black')\n",
        "plt.xlabel('StartPublishing')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of StartPublishing for Cluster 1')\n",
        "plt.grid(True)  # Add grid for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Plot for Tag_only_common in Cluster 1\n",
        "import seaborn as sns\n",
        "\n",
        "# Plot the distribution of Tag_only_common for cluster 1\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=cluster_1_df, palette='viridis')\n",
        "plt.title('Distribution of Tag_only_common for Cluster 1')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Descriptions in Cluster 1\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "# Combine descriptions for cluster 1\n",
        "text_cluster_1 = \" \".join(cluster_1_df['Description'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 1\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_1)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 1 (Based on Descriptions)')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Titles in Cluster 1\n",
        "\n",
        "# Combine titles for cluster 1\n",
        "text_cluster_1 = \" \".join(cluster_1_df['Title'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 1\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_1)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 1 (Based on Titles)')\n",
        "plt.show()\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9pE599NPIJf0",
      "metadata": {
        "id": "9pE599NPIJf0"
      },
      "source": [
        "**Comment on Cluster 1 --\"Data Science, Model, Risk.\"**:\n",
        "\n",
        "Based on the titles, this cluster contains a mix of formats, including magazines, digital, presentations, and audio. Descriptions frequently mention \"risk,\" \"data,\" and \"model,\" linking to various actuarial practices. The existing tags show General Insurance (GI) as most prevalent, but data science and other traditional actuarial topics are also significant. With recent years showing a high level of publications, this cluster seems to cover more topical subjects. The more recent assets likely relate to data science, while older ones might be tied to actuarial data models or techniques applicable across multiple business lines. Therefore, this group can be labeled as \"Data Science, Model, Risk.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "BNTfJB45T4ze",
      "metadata": {
        "id": "BNTfJB45T4ze"
      },
      "source": [
        "##### cluster 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5XS9AnaUVs2u",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5XS9AnaUVs2u",
        "outputId": "693fb0ff-e5ed-4a66-82b5-30239b3f435f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1    449\n",
            "2    157\n",
            "0      4\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      274\n",
            "Podcast/Vodcast/Audio    189\n",
            "Publication              139\n",
            "Video                      5\n",
            "Article                    2\n",
            "Workshops                  1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2019-03-08    189\n",
            "2018-08-21    171\n",
            "2019-03-18     81\n",
            "2019-08-01     68\n",
            "2020-02-07     35\n",
            "2020-02-09     28\n",
            "2019-04-21     18\n",
            "2018-08-20      4\n",
            "2018-11-14      4\n",
            "2018-12-16      3\n",
            "2021-10-11      2\n",
            "2018-09-17      1\n",
            "2018-09-19      1\n",
            "2018-11-06      1\n",
            "2018-11-19      1\n",
            "2021-10-07      1\n",
            "2021-10-17      1\n",
            "2023-02-24      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      274\n",
            "Podcast/Vodcast/Audio    189\n",
            "Publication              139\n",
            "Video                      5\n",
            "Article                    2\n",
            "Workshops                  1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All    610\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    610\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute    610\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All    606\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "general insurance                          133\n",
            "life insurance                              93\n",
            "risk management                             79\n",
            "wealth management                           57\n",
            "superannuation                              44\n",
            "personal effectiveness &amp; leadership     43\n",
            "health                                      39\n",
            "professional updates                        34\n",
            "business acumen                             31\n",
            "banking                                     26\n",
            "data science                                24\n",
            "professionalism training                     1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 32148 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-242-8473797cb13a>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_2_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 82080 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 422556 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 448276 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Profile Analysis for Cluster 2\n",
        "\n",
        "# Filter the DataFrame for cluster 2\n",
        "cluster_2_df = docs_cleaned_desc[docs_cleaned_desc['Cluster'] == 2]\n",
        "\n",
        "# Analyze characteristics of cluster 2\n",
        "for column in ['Cpd', 'Format', 'StartPublishing', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType', 'Tag_only_common']:\n",
        "    if column in cluster_2_df.columns:\n",
        "        print(f\"\\nAnalysis for {column}:\")\n",
        "        print(cluster_2_df[column].value_counts())  # Display frequency counts for each category\n",
        "\n",
        "# Analyzing CPD and calculating proportions\n",
        "if 'CPD' in cluster_2_df.columns:\n",
        "    print(\"\\nAnalysis for CPD (with proportions):\")\n",
        "    cpd_counts = cluster_2_df['CPD'].value_counts()\n",
        "    cpd_proportions = cpd_counts / len(cluster_2_df)\n",
        "    print(cpd_proportions)\n",
        "\n",
        "\n",
        "# Plot for StartPublishing in Cluster 2\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create a histogram of 'StartPublishing' for cluster 2\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.hist(cluster_2_df['StartPublishing'], bins=20, color='skyblue', edgecolor='black')\n",
        "plt.xlabel('StartPublishing')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of StartPublishing for Cluster 2')\n",
        "plt.grid(True)  # Add grid for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Plot for Tag_only_common in Cluster 2\n",
        "import seaborn as sns\n",
        "\n",
        "# Plot the distribution of Tag_only_common for cluster 2\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=cluster_2_df, palette='viridis')\n",
        "plt.title('Distribution of Tag_only_common for Cluster 2')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Descriptions in Cluster 2\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "# Combine descriptions for cluster 2\n",
        "text_cluster_2 = \" \".join(cluster_2_df['Description'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 2\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_2)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 2 (Based on Descriptions)')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Titles in Cluster 2\n",
        "\n",
        "# Combine titles for cluster 2\n",
        "text_cluster_2 = \" \".join(cluster_2_df['Title'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 2\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_2)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 2 (Based on Titles)')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "rq2XhSvlNXPF",
      "metadata": {
        "id": "rq2XhSvlNXPF"
      },
      "source": [
        "**Comment on Cluster 2 - \"Insurance Seminars and Professional Practice\"**\n",
        "\n",
        "Cluster 2 appears to have a strong emphasis on General Insurance, as seen from both the word clouds and tag frequencies. The primary format types include newsletters, seminars, and presentations, indicating a mixture of formal and educational content. The descriptions frequently mention terms related to \"general insurance,\" \"seminars,\" and \"risk management,\" suggesting that this cluster likely contains assets related to professional updates and ongoing training in insurance practices.\n",
        "\n",
        "The publication years are heavily concentrated around 2018 and 2019, indicating that this cluster is more oriented toward older but still relevant content. Given that health, life insurance, and superannuation are present as tags but not as dominant, it indicates that while these topics are included, they are not the primary focus.\n",
        "\n",
        "Based on these observations, Cluster 2 could be labeled as \"Insurance Seminars and Professional Practice,\" as it mostly covers comprehensive updates and seminars focused on insurance, but with a general relevance to other adjacent fields like health and life insurance."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-CNKl_HgWFCQ",
      "metadata": {
        "id": "-CNKl_HgWFCQ"
      },
      "source": [
        "##### Culster 3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "N_pWlWUPWIo8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N_pWlWUPWIo8",
        "outputId": "e6d5aa93-931b-4b14-a599-07b35c4265b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1     1781\n",
            "2      505\n",
            "0       51\n",
            "3       23\n",
            "4       18\n",
            "5        2\n",
            "10       1\n",
            "6        1\n",
            "9        1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      689\n",
            "Publication              603\n",
            "Podcast/Vodcast/Audio    446\n",
            "Article                  375\n",
            "Video                    182\n",
            "Webinar                   41\n",
            "Events                    35\n",
            "Tool and Resources         9\n",
            "Other                      2\n",
            "Workshops                  1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2018-08-21    606\n",
            "2023-07-31    200\n",
            "2019-08-01     86\n",
            "2019-05-02     57\n",
            "2019-04-21     54\n",
            "             ... \n",
            "2019-01-08      1\n",
            "2019-01-13      1\n",
            "2019-07-22      1\n",
            "2019-02-03      1\n",
            "2022-08-03      1\n",
            "Name: count, Length: 348, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All             2373\n",
            "Advanced           9\n",
            "Introductory       1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    2382\n",
            "Domestic                         1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute         2352\n",
            "Speaker/Author      25\n",
            "Other                4\n",
            "Public Domain        2\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All                  2121\n",
            "Retired Actuaries       9\n",
            "YAP                     3\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "general insurance                          629\n",
            "superannuation                             374\n",
            "health                                     275\n",
            "life insurance                             244\n",
            "data science                               144\n",
            "professional updates                       128\n",
            "banking                                     93\n",
            "personal effectiveness &amp; leadership     92\n",
            "risk management                             91\n",
            "wealth management                           77\n",
            "business acumen                             46\n",
            "climate change                               2\n",
            "other                                        2\n",
            "professionalism training                     1\n",
            "not applicable                               1\n",
            "data analytics                               1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 34004 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-243-d5f0090fffd5>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_3_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 94048 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 396700 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 390660 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Profile Analysis for Cluster 3\n",
        "\n",
        "# Filter the DataFrame for cluster 3\n",
        "cluster_3_df = docs_cleaned_desc[docs_cleaned_desc['Cluster'] == 3]\n",
        "\n",
        "# Analyze characteristics of cluster 3\n",
        "for column in ['Cpd', 'Format', 'StartPublishing', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType', 'Tag_only_common']:\n",
        "    if column in cluster_3_df.columns:\n",
        "        print(f\"\\nAnalysis for {column}:\")\n",
        "        print(cluster_3_df[column].value_counts())  # Display frequency counts for each category\n",
        "\n",
        "# Analyzing CPD and calculating proportions\n",
        "if 'CPD' in cluster_3_df.columns:\n",
        "    print(\"\\nAnalysis for CPD (with proportions):\")\n",
        "    cpd_counts = cluster_3_df['CPD'].value_counts()\n",
        "    cpd_proportions = cpd_counts / len(cluster_3_df)\n",
        "    print(cpd_proportions)\n",
        "\n",
        "\n",
        "# Plot for StartPublishing in Cluster 3\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create a histogram of 'StartPublishing' for cluster 3\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.hist(cluster_3_df['StartPublishing'], bins=20, color='skyblue', edgecolor='black')\n",
        "plt.xlabel('StartPublishing')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of StartPublishing for Cluster 3')\n",
        "plt.grid(True)  # Add grid for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Plot for Tag_only_common in Cluster 3\n",
        "import seaborn as sns\n",
        "\n",
        "# Plot the distribution of Tag_only_common for cluster 3\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=cluster_3_df, palette='viridis')\n",
        "plt.title('Distribution of Tag_only_common for Cluster 3')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Descriptions in Cluster 3\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "# Combine descriptions for cluster 3\n",
        "text_cluster_3 = \" \".join(cluster_3_df['Description'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 3\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_3)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 3 (Based on Descriptions)')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Titles in Cluster 3\n",
        "\n",
        "# Combine titles for cluster 3\n",
        "text_cluster_3 = \" \".join(cluster_3_df['Title'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 3\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_3)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 3 (Based on Titles)')\n",
        "plt.show()\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "jcoRm0qUPYAh",
      "metadata": {
        "id": "jcoRm0qUPYAh"
      },
      "source": [
        "**Comment Cluster 3 -- \"Injury Schemes, Disability Worker Compensation Pre-Covid\"**\n",
        "\n",
        "This cluster is heavily focused on injury schemes, disability schemes, and workers' compensation topics, as indicated by the word cloud analysis. The high concentration of assets tagged under 'General Insurance' aligns with the fact that injury schemes and workers' compensation fall under the General insurance umbrella within the actuarial field.\n",
        "\n",
        "The publication dates are primarily concentrated around 2019, with few entries beyond this year, suggesting that most assets are older. Titles often include numbers like '13' and '15,' likely representing years or editions, further indicating their pre-2020 relevance.\n",
        "\n",
        "Notably, there are very few publications in this cluster after 2020, possibly indicating that these topics became less active post-Covid. This observation aligns with domain knowledge that injury schemes, disability, and worker compensation experienced significant changes before and after Covid due to shifts in people's lives and work patterns.\n",
        "\n",
        "Given this context, labeling this cluster as \"Injury Schemes, Disability Worker Compensation Pre-Covid\" accurately captures both the content and historical period of the assets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fCjtg2eUWa35",
      "metadata": {
        "id": "fCjtg2eUWa35"
      },
      "source": [
        "##### Cluster 4"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "30iiMCJeWaA2",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "30iiMCJeWaA2",
        "outputId": "219dd6e4-1c31-4433-a0af-919fabe387db"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1    343\n",
            "2    132\n",
            "3      1\n",
            "4      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      229\n",
            "Podcast/Vodcast/Audio    173\n",
            "Publication               40\n",
            "Events                    18\n",
            "Video                     12\n",
            "Article                    5\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2019-04-03    113\n",
            "2019-03-13    111\n",
            "2019-06-25     63\n",
            "2018-08-22     62\n",
            "2022-01-16     22\n",
            "2019-04-04     14\n",
            "2019-12-01     14\n",
            "2019-03-14     12\n",
            "2018-08-21     11\n",
            "2019-11-10     10\n",
            "2019-07-30     10\n",
            "2022-01-17      6\n",
            "2019-11-09      5\n",
            "2019-08-01      4\n",
            "2019-03-18      3\n",
            "2019-02-21      2\n",
            "2024-02-12      2\n",
            "2023-07-27      2\n",
            "2019-07-31      2\n",
            "2019-01-15      1\n",
            "2018-09-17      1\n",
            "2020-07-16      1\n",
            "2019-09-08      1\n",
            "2019-11-24      1\n",
            "2021-10-12      1\n",
            "2021-10-21      1\n",
            "2022-09-05      1\n",
            "2022-08-17      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Presentation Slides      229\n",
            "Podcast/Vodcast/Audio    173\n",
            "Publication               40\n",
            "Events                    18\n",
            "Video                     12\n",
            "Article                    5\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All         476\n",
            "Advanced      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    477\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute    477\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All                  457\n",
            "Retired Actuaries      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "general insurance    416\n",
            "health                38\n",
            "life insurance         6\n",
            "data science           3\n",
            "business acumen        1\n",
            "superannuation         1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 28780 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-244-d8535890005d>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_4_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 51452 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 411352 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 403220 characters]\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# Profile Analysis for Cluster 4\n",
        "\n",
        "# Filter the DataFrame for cluster 4\n",
        "cluster_4_df = docs_cleaned_desc[docs_cleaned_desc['Cluster'] == 4]\n",
        "\n",
        "# Analyze characteristics of cluster 4\n",
        "for column in ['Cpd', 'Format', 'StartPublishing', 'Format', 'Level', 'Region', 'Copyright', 'EventAgeGroup', 'EventType', 'Tag_only_common']:\n",
        "    if column in cluster_4_df.columns:\n",
        "        print(f\"\\nAnalysis for {column}:\")\n",
        "        print(cluster_4_df[column].value_counts())  # Display frequency counts for each category\n",
        "\n",
        "# Analyzing CPD and calculating proportions\n",
        "if 'CPD' in cluster_4_df.columns:\n",
        "    print(\"\\nAnalysis for CPD (with proportions):\")\n",
        "    cpd_counts = cluster_4_df['CPD'].value_counts()\n",
        "    cpd_proportions = cpd_counts / len(cluster_4_df)\n",
        "    print(cpd_proportions)\n",
        "\n",
        "\n",
        "# Plot for StartPublishing in Cluster 4\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Create a histogram of 'StartPublishing' for cluster 4\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.hist(cluster_4_df['StartPublishing'], bins=20, color='skyblue', edgecolor='black')\n",
        "plt.xlabel('StartPublishing')\n",
        "plt.ylabel('Frequency')\n",
        "plt.title('Distribution of StartPublishing for Cluster 4')\n",
        "plt.grid(True)  # Add grid for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Plot for Tag_only_common in Cluster 4\n",
        "import seaborn as sns\n",
        "\n",
        "# Plot the distribution of Tag_only_common for cluster 4\n",
        "plt.figure(figsize=(10, 6))\n",
        "sns.countplot(x='Tag_only_common', data=cluster_4_df, palette='viridis')\n",
        "plt.title('Distribution of Tag_only_common for Cluster 4')\n",
        "plt.xlabel('Tag_only_common')\n",
        "plt.ylabel('Count')\n",
        "plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels for better readability\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Descriptions in Cluster 4\n",
        "from wordcloud import WordCloud\n",
        "\n",
        "# Combine descriptions for cluster 4\n",
        "text_cluster_4 = \" \".join(cluster_4_df['Description'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 4\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_4)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 4 (Based on Descriptions)')\n",
        "plt.show()\n",
        "\n",
        "\n",
        "# Word Cloud for Titles in Cluster 4\n",
        "\n",
        "# Combine titles for cluster 4\n",
        "text_cluster_4 = \" \".join(cluster_4_df['Title'].astype(str))\n",
        "\n",
        "# Generate and display word cloud for cluster 4\n",
        "wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text_cluster_4)\n",
        "\n",
        "# Display the word cloud\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.imshow(wordcloud, interpolation='bilinear')\n",
        "plt.axis('off')\n",
        "plt.title('Word Cloud for Cluster 4 (Based on Titles)')\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "atEb51ovSNuF",
      "metadata": {
        "id": "atEb51ovSNuF"
      },
      "source": [
        "**Comment for Cluster 4 -- \"Injury Schemes and Compensation with Superannuation Linkages.\"**\n",
        "\n",
        "This cluster centers around injury schemes, compensation, and superannuation topics, as evident from the frequent terms like \"scheme,\" \"compensation,\" and \"outcome\" in the word cloud analysis, indicating a strong focus on the outcomes and management aspects of compensation schemes.\n",
        "\n",
        "The Tag_only_common distribution reveals that 'General Insurance' is the dominant tag, with significant representation from 'Health' and 'Superannuation.' While most injury claims typically fall under general insurance and health insurance, which are both under the GI umbrella, the presence of superannuation is somewhat unexpected, as it is usually a distinct field with minimal overlap in the real world.\n",
        "\n",
        "The publication timeline shows a concentration around 2018-2019, with additional activity in 2022 and later, suggesting that the topics addressed in this cluster are a mix of older and more recent issues, possibly experiencing renewed attention.\n",
        "\n",
        "Given the unusual combination of themes, it’s challenging to assign a definitive label, but based on the insights available, \"Injury Schemes and Compensation with Superannuation Linkages\" is the most suitable description we can provide at this stage."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7BZC0EY0WqJN",
      "metadata": {
        "id": "7BZC0EY0WqJN"
      },
      "source": [
        "##### cluster5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4Ql03GETWsW7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4Ql03GETWsW7",
        "outputId": "35c334cb-747c-4a31-cf8b-840f9ac8d498"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Analysis for Cpd:\n",
            "Cpd\n",
            "1    202\n",
            "2     68\n",
            "3      2\n",
            "0      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Publication              115\n",
            "Presentation Slides      103\n",
            "Podcast/Vodcast/Audio     51\n",
            "Video                      4\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for StartPublishing:\n",
            "StartPublishing\n",
            "2018-08-21    173\n",
            "2019-01-22     20\n",
            "2019-03-24     17\n",
            "2019-03-16     15\n",
            "2019-04-03      7\n",
            "2019-08-01      7\n",
            "2018-10-31      5\n",
            "2018-11-08      4\n",
            "2018-09-19      3\n",
            "2018-11-18      3\n",
            "2019-03-18      2\n",
            "2018-11-11      2\n",
            "2018-11-13      2\n",
            "2019-03-29      2\n",
            "2018-10-07      1\n",
            "2018-09-21      1\n",
            "2018-08-20      1\n",
            "2018-11-09      1\n",
            "2018-12-16      1\n",
            "2019-01-24      1\n",
            "2018-11-23      1\n",
            "2019-03-13      1\n",
            "2019-04-19      1\n",
            "2020-05-29      1\n",
            "2023-07-27      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Format:\n",
            "Format\n",
            "Publication              115\n",
            "Presentation Slides      103\n",
            "Podcast/Vodcast/Audio     51\n",
            "Video                      4\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Level:\n",
            "Level\n",
            "All             271\n",
            "Advanced          1\n",
            "Introductory      1\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Region:\n",
            "Region\n",
            "Domestic and International    273\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Copyright:\n",
            "Copyright\n",
            "Institute    273\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for EventAgeGroup:\n",
            "EventAgeGroup\n",
            "All                  270\n",
            "Retired Actuaries      3\n",
            "Name: count, dtype: int64\n",
            "\n",
            "Analysis for Tag_only_common:\n",
            "Tag_only_common\n",
            "general insurance    157\n",
            "health                52\n",
            "life insurance        37\n",
            "data science          12\n",
            "risk management       10\n",
            "superannuation         2\n",
            "business acumen        1\n",
            "Name: count, dtype: int64\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 30452 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-245-388149d9e64f>:39: FutureWarning: \n",
            "\n",
            "Passing `palette` without assigning `hue` is deprecated and will be removed in v0.14.0. Assign the `x` variable to `hue` and set `legend=False` for the same effect.\n",
            "\n",
            "  sns.countplot(x='Tag_only_common', data=cluster_5_df, palette='viridis')\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ],
            "image/png": "[encoded data, 54412 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 366040 characters]\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1000x700 with 1 Axes>"
            ],
            "image/png": "[encoded data, 161243 characters]