i #282 Add GitHub Refresh

Adds GitHub refresh and re-enable functionality for parsing replies, social smells, motifs and more. Signed-off-by: Carlos Paradis <[email protected]> --------- Signed-off-by: Carlos Paradis <[email protected]> Co-authored-by: Sean Sunoo <[email protected]> Co-authored-by: Carlos Paradis <[email protected]> Co-authored-by: Nicholas Beydler <[email protected]>
sailuh · Nov 12, 2024 · 513a3f0 · 513a3f0
1 parent f031afb
commit 513a3f0
Show file tree

Hide file tree

Showing 22 changed files with 1,202 additions and 105 deletions.
diff --git a/NAMESPACE b/NAMESPACE
@@ -117,15 +117,21 @@ export(github_api_page_prev)
 export(github_api_project_commits)
 export(github_api_project_contributors)
 export(github_api_project_issue)
+export(github_api_project_issue_by_date)
 export(github_api_project_issue_events)
+export(github_api_project_issue_or_pr_comment_refresh)
 export(github_api_project_issue_or_pr_comments)
+export(github_api_project_issue_or_pr_comments_by_date)
+export(github_api_project_issue_refresh)
+export(github_api_project_issue_search)
 export(github_api_project_pull_request)
 export(github_api_rate_limit)
 export(github_parse_project_commits)
 export(github_parse_project_issue)
 export(github_parse_project_issue_events)
 export(github_parse_project_issue_or_pr_comments)
 export(github_parse_project_pull_request)
+export(github_parse_search_issues_refresh)
 export(gitlog_to_hdsmj)
 export(graph_to_dsmj)
 export(identity_match)
@@ -230,6 +236,7 @@ importFrom(stringi,stri_c)
 importFrom(stringi,stri_cmp_eq)
 importFrom(stringi,stri_detect_regex)
 importFrom(stringi,stri_match_all)
+importFrom(stringi,stri_match_first)
 importFrom(stringi,stri_match_first_regex)
 importFrom(stringi,stri_replace_all_regex)
 importFrom(stringi,stri_replace_first)

diff --git a/NEWS.md b/NEWS.md
@@ -3,7 +3,8 @@ __kaiaulu 0.0.0.9700 (in development)__
 
 ### NEW FEATURES
 
- * `config.R` now contains a set of getter functions used to centralize the gathering of configuration data and these getter functions are used to refactor configuration file information gathering. For example, loading configuration file information with variable assignment is as follows `git_repo_path <- config_file[["version_control"]][["log"]]` but refactoring with a config.R getter function becomes `git_repo_path <- get_git_repo_path(config_file)`. [#230](https://github.com/sailuh/kaiaulu/issues/230)
+ * The GitHUB API has been expanded to use refresh, along with other functions. `github_api_project_issue_search` has been added that makes the search/issues endpoint API calls. `github_api_project_issue_or_pr_comments_by_date` and `github_api_project_issue_by_date` have been added to download issue data and comments by date ranges. `github_parse_search_issues_refresh` has been added that parses the issue data downloaded from the search endpoint in the refresh_issues folder. `github_api_project_issue_refresh` and `github_api_project_issue_or_pr_comment_refresh` were added to download issue data or comments respectively that have not already been downloaded. `format_created_at_from_file` was added to retrieve the greatest date from a JSON file. See the Reference Docs on GitHub section for more details. [#282](https://github.com/sailuh/kaiaulu/issues/282)
+ * `config.R` now contains a set of getter functions used to centralize the gathering of configuration data and these getter functions are used to refactor configuration file information gathering. For example, loading configuration file information with variable assignment is as follows `git_repo_path <- config_file[["version_control"]][["log"]]` but refactoring with a config.R getter function becomes `git_repo_path <- get_git_repo_path(config_file)`.  [#230](https://github.com/sailuh/kaiaulu/issues/230)
  * `refresh_jira_issues()` had been added. It is a wrapper function for the previous downloader and downloads only issues greater than the greatest key already downloaded. [#275](https://github.com/sailuh/kaiaulu/issues/275)
  * `download_jira_issues()`, `download_jira_issues_by_issue_key()`, and `download_jira_issues_by_date()` has been added. This allows for downloading of Jira issues without the use of JirAgileR and specification of issue Id and created ranges. It also interacts with `parse_jira_latest_date()` to implement a refresh capability. [#275](https://github.com/sailuh/kaiaulu/issues/275)
  * `make_jira_issue()` and `make_jira_issue_tracker()` no longer create fake issues following JirAgileR format, but instead the raw data obtained from JIRA API. This is compatible with the new parser function for JIRA. [#277](https://github.com/sailuh/kaiaulu/issues/277)

diff --git a/R/github.R b/R/github.R
diff --git a/R/identity.R b/R/identity.R
@@ -185,6 +185,7 @@ identity_match <- function(project_log,name_column,assign_identity_function,
 # Various imports
 #' @importFrom stringi stri_replace_last
 #' @importFrom stringi stri_replace_first
+#' @importFrom stringi stri_match_first
 #' @importFrom stringi stri_c
 #' @importFrom stringi stri_cmp_eq
 #' @importFrom stringi stri_replace_last_regex

diff --git a/_pkgdown.yml b/_pkgdown.yml
@@ -135,7 +135,15 @@ reference:
   - github_api_project_issue_events
   - github_parse_project_issue_events
   - github_api_project_issue
+  - github_api_project_issue_refresh
   - github_parse_project_issue
+  - github_api_project_issue_search
+  - github_api_project_issue_by_date
+  - github_api_project_issue_refresh
+  - github_parse_search_issues_refresh
+  - github_api_project_issue_or_pr_comments_by_date
+  - github_api_project_issue_or_pr_comment_refresh
+
 - subtitle: Pull Request
   desc: GitHub's Pull Request
 - contents:

diff --git a/conf/helix.yml b/conf/helix.yml
@@ -34,7 +34,7 @@ project:
 
 version_control:
   # Where is the git log located locally?
-  log: ../../rawdata/helix/git_repo/.git
+  log: ../../rawdata/helix/git_repo/helix/.git
   # From where the git log was downloaded?
   log_url: https://github.com/apache/helix
   # List of branches used for analysis
@@ -51,14 +51,12 @@ mailing_list:
   mod_mbox:
     project_key_1:
       mailing_list: https://lists.apache.org/[email protected]
-      save_folder_path: ../../rawdata/helix/mod_mbox/save_mbox_mail/
-#       # mbox_file_path is for use only with parse_mbox() function. It is the file to parse
-#      mbox_file_path: ../../rawdata/helix/mod_mbox/save_mbox_mail/helix.mbox
-    project_key_2:
-      mailing_list: https://lists.apache.org/[email protected]
-      save_folder_path: ../../rawdata/helix/mod_mbox/save_mbox_mail_2/
-#       # mbox_file_path is for use only with parse_mbox() function. It is the file to parse
-#       mbox_file_path: ../../rawdata/kaiaulu/mod_mbox/save_mbox_mail_2/helix.mbox
+      save_folder_path: ../../rawdata/helix/mod_mbox/
+
+#    project_key_2:
+#      mailing_list: https://lists.apache.org/[email protected]
+#      save_folder_path: ../../rawdata/helix/mod_mbox/save_mbox_mail_2/
+
 #   pipermail:
 #     project_key_1:
 #       mailing_list: https://mta.openssl.org/pipermail/openssl-users/
@@ -78,20 +76,20 @@ issue_tracker:
       domain: https://issues.apache.org/jira
       project_key: HELIX
       # Download using `download_jira_data.Rmd`
-      issues: ../../rawdata/helix/jira/issues/helix
-      issue_comments: ../../rawdata/helix/jira/issue_comments/helix
+      issues: ../../rawdata/helix/jira/issues/
+      issue_comments: ../../rawdata/helix/jira/issue_comments/
   github:
     project_key_1:
       # Obtained from the project's GitHub URL
       owner: apache
       repo: helix
       # Download using `download_github_comments.Rmd`
-      issue_or_pr_comment: ../../rawdata/helix/github/issue_or_pr_comment/apache_helix/
-      issue: ../../rawdata/helix/github/issue/apache_helix/
-      issue_search: ../../rawdata/helix/github/issue_search/apache_helix/
-      issue_event: ../../rawdata/helix/github/issue_event/apache_helix/
-      pull_request: ../../rawdata/helix/github/pull_request/apache_helix/
-      commit: ../../rawdata/helix/github/commit/apache_helix/
+      issue_or_pr_comment: ../../rawdata/helix/github/apache_helix/issue_or_pr_comment/
+      issue: ../../rawdata/helix/github/apache_helix/issue/
+      issue_search: ../../rawdata/helix/github/apache_helix/issue_search/
+      issue_event: ../../rawdata/helix/github/apache_helix/issue_event/
+      pull_request: ../../rawdata/helix/github/apache_helix/pull_request/
+      commit: ../../rawdata/helix/github/apache_helix/commit/
   # bugzilla:
   #   project_key_1:
   #     project_key: helix

diff --git a/man/format_created_at_from_file.Rd b/man/format_created_at_from_file.Rd
diff --git a/man/github_api_iterate_pages.Rd b/man/github_api_iterate_pages.Rd
diff --git a/man/github_api_project_issue_by_date.Rd b/man/github_api_project_issue_by_date.Rd
diff --git a/man/github_api_project_issue_or_pr_comment_refresh.Rd b/man/github_api_project_issue_or_pr_comment_refresh.Rd
diff --git a/man/github_api_project_issue_or_pr_comments.Rd b/man/github_api_project_issue_or_pr_comments.Rd
diff --git a/man/github_api_project_issue_or_pr_comments_by_date.Rd b/man/github_api_project_issue_or_pr_comments_by_date.Rd