Any examples of using pointblank with GitHub actions?

[Aariq]

I'm working on a project with data from a long term experiment, and an end goal would be the ability to add new data as a pull request and have a pointblank agent run an interrogation automatically in the way that tests are typically run on pull requests. I'm not exactly sure how to go about doing this with github actions though. Has anyone written an action like this yet that I could look at?

[rich-iannone]

This is a question I’ve received a few times and I think it’s time I had a look at how to do this. I’m going to cobble together a workflow that operates on PRs and signals failure when certain validation conditions are unmet. Thanks for getting the discussion going on this :)

[Aariq]

Beyond just reporting pass/fail, it would be awesome if you could access the results of interrogate() somehow (as an artifact?). In my particular situation the biggest advantage of pointblank is the report and the ability to get the .csv file of the entries that don't pass.

But maybe at that point it makes more sense to be run on the user end?

[rich-iannone]

Sorry I didn't write back earlier but, yes!, this is something I've been thinking about quite a bit. Some existing validations and a few new ones I have in development would be better served if there was some additional information about the processing. This can sort of be done now with create_agent()'s end_fns argument, but right now it's not well documented. I hope to change that soon.

I think that during the experimentation of using GHA to run a data validation in a PR, I'll come up against this. My goal is to create a few examples that are practical for data quality checks during CI.

[rich-iannone]

Just a small update/bump to let you know that this is still a concern and area of work for me. I've added a few functions that are more suited to automation workflows. These are write_testthat_file() (which writes a testthat file from an agent) and yaml_exec() (a function that executes YAML agents/informants and writes output RDS files to an output folder).

These developments will help but the very next thing for me is to test them out in GHA and see if they work well enough. I'll come back to this thread shortly and let you know how that goes and provide examples if everything looks good.

[Aariq]

This is so cool!

[Aariq]

I saw a great talk by Albert Kim recently about using continuous integration for data QA/QC in ecology that reminded me of this discussion. Maybe their repo can be of some inspiration? https://github.com/SCBI-ForestGEO/Dendrobands

I think maybe the simplest thing would be to have github actions render a .Rmd to a github pages hosted website.

[rich-iannone]

@Aariq It would be really awesome to make an example repo that's just like the repo you linked (except using .Rmd artifacts). Bonus points also if a data dictionary (with an informant) could be shown in the README. It would be more compelling I think if the data changed with every workflow run so an idea might be to use tt_time_slice() (https://rich-iannone.github.io/pointblank/reference/tt_time_slice.html) perhaps with some manipulation of the current date (getting a value between 0 and 1) to subset the data with each run. This'll work if the dataset has a date or date-time column.

Would you like to make such an example repo or would you prefer I start that up?

[Aariq]

I don't have the bandwidth right now to work on that, but if you got it started I'd be interested in contributing or reviewing.

[egouldo]

Hi @Aariq How did you end up going with this implementation? Coincidentally I'm working on using pointblank:: within an adaptation of Kim et al.'s approach and came across your question here!

My plan is to generate a report based on interrogating some pointblank agents within parameterised quarto markdown files. These will be in the GitHub flavoured markdown format. I'll run calls to render the parameterised reports using GitHub Actions. The report containing some suggested fixes and a summary of problems to be manually fixed will then be readable in the GitHub repo on the web. Another idea I had was to create an issue from the rendered GitHub flavoured .md file using the create gh issue from file action.

As for running formal tests with testthat::, the plan was to have these tests run separately, potentially by outputting agents to tests with pointblank::write_testthat_file(). The tests will run in the GitHub Actions workflow after the report is generated. Newly added or updated data will trigger report generation and testing, if testing fails, the user makes changes based on the QA report until all outlined issues are resolved + testing passes. Merging the pull request containing new or updated data will be blocked until the data files pass testing.

Cheers,
Elliot.

[Aariq]

I didn't end up pursuing this further, but it would be great to see a working demo of such a setup. The group I work for has implemented automated QA pipelines from time to time, and this would be a potentially useful way to do things. I like the idea of creating an issue (or PR comment?) with output from the validation!

[egouldo]

Once the government agency I'm working for has permission to release the analysis data I'll be sure to share a working demo :)

Oh, and that's a much better idea to make it a PR comment instead of new issue if the user has already made a PR!