results_extract_from_logs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage Notes:\n",
    "\n",
    "To use this notebook effectively, please follow these guidelines for organizing your files and directories:\n",
    "\n",
    "- Base Directory: Define a base directory where all relevant subfolders are stored. You should specify this base directory within the notebook. For example, you might set it as `./output/2024-08-01/`.\n",
    "\n",
    "- Subfolders: Within the base directory, there should be multiple subfolders. Each subfolder must contain at least one .log file that the notebook will process. The names of these subfolders can be anything, but they are typically generated by default Hydra configurations in the format `%H-%M-%S`. The notebook script can automatically detect these subfolders, so no need for manual specification.\n",
    "\n",
    "- Log Files: Ensure that each subfolder contains the log files to be processed. The functions defined in the notebook will extract and analyze data from these files.\n",
    "\n",
    "Description of the Output CSV File from this notebook is in end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First import the packages and define the log-processing function we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "def process_log_file(log_file_path):\n",
    "    # Initialize empty lists to store the extracted values\n",
    "    session_names = []\n",
    "    r2_values = []\n",
    "    footprints = []\n",
    "    connection_sparsities = []\n",
    "    activation_sparsities = []\n",
    "    effective_macs = []\n",
    "    effective_acs = []\n",
    "    dense_values = []\n",
    "\n",
    "    # Define the keywords to search in the log file\n",
    "    session_keyword = \"- Constructing model for\"\n",
    "    r2_keyword = \"- r2:\"\n",
    "    footprint_keyword = \"- footprint:\"\n",
    "    connection_sparsity_keyword = \"- connection_sparsity:\"\n",
    "    activation_sparsity_keyword = \"- activation_sparsity:\"\n",
    "    synaptic_operations_keyword = \"- synaptic_operations:\"\n",
    "\n",
    "    # Read the log file and extract the information, ignoring lines that contain \"pretraining\"\n",
    "    with open(log_file_path, 'r') as file:\n",
    "        lines = file.readlines()\n",
    "        for line in lines:\n",
    "            if \"pretraining\" in line:\n",
    "                continue\n",
    "            if session_keyword in line:\n",
    "                session_name = line.split(session_keyword)[-1].strip()\n",
    "                session_names.append(session_name)\n",
    "            elif r2_keyword in line:\n",
    "                r2_value = float(line.split(r2_keyword)[-1].strip())\n",
    "                r2_values.append(r2_value)\n",
    "            elif footprint_keyword in line:\n",
    "                footprint = float(line.split(footprint_keyword)[-1].strip())\n",
    "                footprints.append(footprint)\n",
    "            elif connection_sparsity_keyword in line:\n",
    "                connection_sparsity = float(line.split(connection_sparsity_keyword)[-1].strip())\n",
    "                connection_sparsities.append(connection_sparsity)\n",
    "            elif activation_sparsity_keyword in line:\n",
    "                activation_sparsity = float(line.split(activation_sparsity_keyword)[-1].strip())\n",
    "                activation_sparsities.append(activation_sparsity)\n",
    "            elif synaptic_operations_keyword in line:\n",
    "                operations = eval(line.split(synaptic_operations_keyword)[-1].strip())\n",
    "                effective_macs.append(operations.get('Effective_MACs', 0.0))\n",
    "                effective_acs.append(operations.get('Effective_ACs', 0.0))\n",
    "                dense_values.append(operations.get('Dense', 0.0))\n",
    "\n",
    "    # Normalize the lengths of all lists by appending None to make them equal to the max length\n",
    "    max_length = max(len(session_names), len(r2_values), len(footprints), len(connection_sparsities), \n",
    "                     len(activation_sparsities), len(effective_macs), len(effective_acs), len(dense_values))\n",
    "    \n",
    "    session_names = session_names + [None] * (max_length - len(session_names))\n",
    "    r2_values = r2_values + [None] * (max_length - len(r2_values))\n",
    "    footprints = footprints + [None] * (max_length - len(footprints))\n",
    "    connection_sparsities = connection_sparsities + [None] * (max_length - len(connection_sparsities))\n",
    "    activation_sparsities = activation_sparsities + [None] * (max_length - len(activation_sparsities))\n",
    "    effective_macs = effective_macs + [None] * (max_length - len(effective_macs))\n",
    "    effective_acs = effective_acs + [None] * (max_length - len(effective_acs))\n",
    "    dense_values = dense_values + [None] * (max_length - len(dense_values))\n",
    "\n",
    "    # Create a dictionary to store the normalized data\n",
    "    data = {\n",
    "        'Session Name': session_names,\n",
    "        'r2': r2_values,\n",
    "        'Footprint': footprints,\n",
    "        'Connection Sparsity': connection_sparsities,\n",
    "        'Activation Sparsity': activation_sparsities,\n",
    "        'Effective MACs': effective_macs,\n",
    "        'Effective ACs': effective_acs,\n",
    "        'Dense': dense_values\n",
    "    }\n",
    "\n",
    "    # Create a DataFrame\n",
    "    df = pd.DataFrame(data)\n",
    "    return df\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the `base_path`. After processing, the notebook will generate an output CSV file. This file will contain the aggregated data from all the log files, and will be saved at `output_file_path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./extracted_log_data.xlsx'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Base path containing subfolders\n",
    "base_path = \"./output/\"\n",
    "\n",
    "# Create a Pandas Excel writer using XlsxWriter as the engine.\n",
    "output_file_path = './extracted_log_data_with_averages.xlsx'\n",
    "all_subfolder_dfs = []  # To collect data from all subfolders for final averaging\n",
    "\n",
    "with pd.ExcelWriter(output_file_path, engine='xlsxwriter') as writer:\n",
    "    for subfolder in os.listdir(base_path):\n",
    "        subfolder_path = os.path.join(base_path, subfolder)\n",
    "        if os.path.isdir(subfolder_path):\n",
    "            for file_name in os.listdir(subfolder_path):\n",
    "                if file_name.endswith('.log'):\n",
    "                    log_file_path = os.path.join(subfolder_path, file_name)\n",
    "                    df = process_log_file(log_file_path)\n",
    "                    all_subfolder_dfs.append(df)\n",
    "                    sheet_name = f\"{subfolder}_{file_name.replace('.log', '')}\"\n",
    "                    df.to_excel(writer, sheet_name=sheet_name[:31], index=False)  # Sheet name max length is 31 characters\n",
    "    \n",
    "    # Calculate session-level average across all subfolders\n",
    "    if all_subfolder_dfs:\n",
    "        combined_df = pd.concat(all_subfolder_dfs)\n",
    "        session_level_avg_df = combined_df.groupby('Session Name').mean().reset_index()\n",
    "        session_level_avg_df.to_excel(writer, sheet_name=\"Session_Level_Average\", index=False)\n",
    "    \n",
    "        # Calculate overall average across all sessions and subfolders\n",
    "        overall_avg_df = session_level_avg_df.mean(numeric_only=True).to_frame().transpose()\n",
    "        overall_avg_df.insert(0, 'Session Name', 'Overall Mean')\n",
    "        overall_avg_df.to_excel(writer, sheet_name=\"Overall_mean\", index=False)\n",
    "\n",
    "output_file_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of the Output CSV File\n",
    "The output CSV file generated by this notebook contains detailed extracted information from log files, structured as follows:\n",
    "\n",
    "1. Sheets and Data Structure:\n",
    "- Individual Log Data Sheets: For each log file processed, a separate sheet is created in the Excel file. The sheet names are derived from the subfolder name and the log file name. Each of these sheets contains a detailed breakdown of the data extracted from the corresponding log file.\n",
    "- The output CSV file contains several key metrics extracted from the log files. These include `Session Name`, `r2`, `Footprint`, `Connection Sparsity`, `Activation Sparsity`, `Effective MACs`, `Effective ACs`, and `Dense`. Each of these columns provides specific data points that have been extracted and organized from the log files for further analysis.\n",
    "- Session Level Average Sheet: This sheet contains the average values of `r2`, `Footprint`, `Connection Sparsity`, `Activation Sparsity`, `Effective MACs`, `Effective ACs`, and `Dense` of each sessions over different subfolders (for example, different initializations). \n",
    "- Overall Mean Sheet: This sheet summarizes the overall average values across all sessions and subfolders.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "CST_Regression",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}