TidierText.jl

What is TidierText.jl

TidierText.jl is a 100% Julia implementation of the R tidytext package. The purpose of the package is to make it easy analyze text data using DataFrames.

An extensive guide to tidy text analysis by Julia Silge and David Robinson is available here: https://www.tidytextmining.com/.

Installation

For the development version:

using Pkg
Pkg.add(url="https://github.com/TidierOrg/TidierText.jl")

What functions does TidierText.jl support?

• @bind_tf_idf()
• @unnest_tokens()
• @unnest_regex()
• @unnest_characters()
• @unnest_ngrams()
• get_stopwords()
• tidy()
• nma_words

How does the package work?

Let's load the package and read in the UCLA Fall 2018 course dataset.

using TidierData
using TidierText

using CSV

courses = CSV.read(download("https://vincentarelbundock.github.io/Rdatasets/csv/openintro/ucla_f18.csv"), DataFrame)

What are the course names?

@chain courses begin
  @select(id = rownames, course)
  @slice(1:10)
end

10×2 DataFrame
 Row │ id     course                            
     │ Int64  String                            
─────┼──────────────────────────────────────────
   1 │     1  Leadership Laboratory
   2 │     2  Heritage and Values
   3 │     3  Team and Leadership Fundamentals
   4 │     4  Air Force Leadership Studies
   5 │     5  National Security Affairs/Prepar…
   6 │     6  Introduction to Black Studies
   7 │     7  African American Musical Heritage
   8 │     8  UCLA Centennial Initiative: Arth…
   9 │     9  UCLA Centennial Initiative: Soci…
  10 │    10  Student Research Program

Let's tokenize the course names and convert them to lowercase.

tokens = @chain courses begin
  @select(id = rownames, course)
  @slice(1:10)
  @unnest_tokens(word, course, to_lower = true)
end;

@chain tokens @slice(1:10)

10×2 DataFrame
 Row │ id     word         
     │ Int64  SubStrin…    
─────┼─────────────────────
   1 │     1  leadership
   2 │     1  laboratory
   3 │     2  heritage
   4 │     2  and
   5 │     2  values
   6 │     3  team
   7 │     3  and
   8 │     3  leadership
   9 │     3  fundamentals
  10 │     4  air

Let's add the term frequency, inverse document frequency, and the tf-idf.

@chain tokens begin
  @count(id, word)
  @bind_tf_idf(word, id, n)
  @slice(1:10)
end

10×6 DataFrame
 Row │ id     word          n      tf        idf       tf_idf   
     │ Int64  SubStrin…     Int64  Float64   Float64   Float64  
─────┼──────────────────────────────────────────────────────────
   1 │     1  leadership        1  0.5       1.20397   0.601986
   2 │     1  laboratory        1  0.5       2.30259   1.15129
   3 │     2  heritage          1  0.333333  1.60944   0.536479
   4 │     2  and               1  0.333333  0.916291  0.30543
   5 │     2  values            1  0.333333  2.30259   0.767528
   6 │     3  team              1  0.25      2.30259   0.575646
   7 │     3  and               1  0.25      0.916291  0.229073
   8 │     3  leadership        1  0.25      1.20397   0.300993
   9 │     3  fundamentals      1  0.25      2.30259   0.575646
  10 │     4  air               1  0.25      2.30259   0.575646