wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Wrangling - WeRateDogs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This report discusses the data wrangling efforts taken in preparation for the analysis of the WeRateDogs data. These efforts have been divided into three parts:\n",
    "* Gathering the data\n",
    "* Asssessing the data\n",
    "* Cleaning the data\n",
    "Therefore the discussions will be along these three sub-topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gathering Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data used for this analysis was gotten from three different sources.\n",
    "1. The main data used is the **WeRateDogs twitter archive** that was downloaded manually as a `.csv` file from the Udacity learning platform. This was then read into a pandas dataframe using the `pd.read_csv` method.\n",
    "2. In order to get the breed of the dog for each tweet, the **image prediction data** for the each tweet was downloaded programmatically as a `.tsv` file from where it was hosted using the `requests` library. This data was also read into a pandas dataframe using the same method decribed above.\n",
    "3. To complement the main data gotten, other information about the tweets were gotten from Twitter API using `tweepy` library. The json data was read into a text file named `'tweet_json.txt'` which was then parsed into a list of dictionaries. This list was then converted into a dataframe using the `pd.\n",
    "DataFrame` method.\n",
    "A snapshot of the codes used can be seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load WeRateDogs Archive data from csv file\n",
    "twitter_df = pd.read_csv(\"twitter-archive-enhanced.csv\")\n",
    "# load image prediction data from the url provided in the course\n",
    "url = \"https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv\"\n",
    "response = requests.get(url)\n",
    "with open (url.split('/')[-1], mode = 'wb') as f:\n",
    "    f.write(response.content)\n",
    "\n",
    "# read the file into a pandas dataframe\n",
    "image_df = pd.read_csv(\"image-predictions.tsv\", sep = '\\t')\n",
    "# iterate throught the twitter dataset to get each tweet id\n",
    "for _, item in twitter_df.iterrows():\n",
    "    try:\n",
    "        start = timer()\n",
    "        tweet_id = item['tweet_id']\n",
    "\n",
    "        # get additional data from twitter api\n",
    "        tweet = api.get_status(tweet_id, tweet_mode='extended')\n",
    "\n",
    "        # write data to tweet_json.txt file\n",
    "        file_name = 'tweet_json.txt'\n",
    "        if os.path.exists(file_name):\n",
    "            with open (file_name, mode = 'a') as f:\n",
    "                f.write(json.dumps(tweet._json) + '\\n')\n",
    "        else:\n",
    "            with open (file_name, mode = 'wt') as f:\n",
    "                f.write(json.dumps(tweet._json) + '\\n')\n",
    "        print(tweet_id)\n",
    "        end = timer()\n",
    "        print(end-start)\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "tweets = []\n",
    "\n",
    "# read data from tweet_json.txt file to a list of dictionaries\n",
    "with open('tweet_json.txt','r') as file:\n",
    "    for line in file:\n",
    "        # convert json strings to dictionaries\n",
    "        data = json.loads(line)\n",
    "        # append the dictionaries to the tweets list\n",
    "        tweets.append({k:v for (k,v) in data.items() if k in ('id','retweet_count','favorite_count')})\n",
    "tweepy_df = pd.DataFrame(tweets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assessing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this stage, the three dataframes were assessed visually and programmatically for cleaniness issues. 8 quality issues and 3 tidiness issues were identified in this stage. The cleaniness issues are listed below:\n",
    "##### Quality Issues\n",
    "\n",
    "1. The records with non-null values in the `retweeted_status_id, retweeted_status_user_id` and `retweeted_status_timestamp` columns are retweets and not original tweets.\n",
    "2. The records with non-null values in the `in_reply_to_status_id` and `in_reply_to_user_id` columns are replies and not original tweets.\n",
    "3. The `source` column carries the html anchor tag instead of just the tweet source.\n",
    "4. The `name` column contains wrong names for some of the dogs. An example is the record in index `2352`.\n",
    "5. Some other columns asides the ones listed above are null but instead of `NaN`, we have 'None'. This will make working with null data difficult as we continue our analysis.\n",
    "6. The `timestamp` and `retweeted_status_timestamp` columns have their datatypes as `object` rather than `datetime64[ns]`.\n",
    "7. Only about 10% of the records in the `in_reply_to_status_id, in_reply_to_user_id, retweeted_status_id`, `retweeted_status_user_id` and `retweeted_status_timestamp` columns are not null.\n",
    "8. The case of the image predictions is not consistent. Some are lower case while some others are title case.\n",
    "\n",
    "#### Tidiness Issues\n",
    "1. The `twitter_df` and `tweepy_df` have more rows than the `image_df`. This means some tweets don't have image predictions and might have to be dropped.\n",
    "2. The `retweet_count` and `favorite_count` columns from the `tweepy_df` should be in the `twitter_df` instead.\n",
    "3. The dog type should be contained in a single column in the `twitter_df` instead of being spread across multiple columns.\n",
    "4. The image predictions from the `image_df` for each tweet should be merged with the main `twitter_df` in order to form a master df."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the issues identified in the previous stage, the **define, code and test** technique was used to clean the issues from the data. Please check the `wrangle_act.ipynb` notebook for details on how each was cleaned."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}