Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate R scripts #40

Open
khaeru opened this issue Dec 17, 2021 · 3 comments · May be fixed by #58
Open

Integrate R scripts #40

khaeru opened this issue Dec 17, 2021 · 3 comments · May be fixed by #58
Labels
enh New features or functionality question Further information is requested

Comments

@khaeru
Copy link
Member

khaeru commented Dec 17, 2021

message_data contains some modules that are partly or entirely in R. For instance:

  • The DLE code is entirely in R.
  • The Nexus (water) code is partly in Python, but with data processing scripts in R.

This issue is to discuss approaches so that this code (a) can be integrated into complete workflows that run unsupervised, and (b) can be more reusable, in a standard way. The implementation should be in message-ix-models.

Some ideas:

  1. Provide a CLI command like mix-models r-script foo/bar/baz that will simply invoke a file at (e.g.) message_data/foo/bar/baz.R, while providing some standard environment variables that the script can use to understand paths to data, etc.
  2. Use rpy2 (with documentation & demo code) in Python code to directly call functions from R code in particular files, and retrieve its output for further processing.
@khaeru
Copy link
Member Author

khaeru commented Dec 17, 2021

@jkikstra @Jihoon @adrivinca @awais307 would appreciate some comment here on things like:

  • how you use R code as part of workflows (e.g. do you run an entire script? Interactively? Call it from somewhere?)
  • Which ones are used most heavily, or more likely to be of interest to others to (re)use.
  • Whether the above ideas would be helpful/usable in your workflows.

@khaeru khaeru added enh New features or functionality question Further information is requested labels Dec 17, 2021
@jkikstra
Copy link
Contributor

jkikstra commented Dec 17, 2021

@khaeru, thanks for creating this issue to start discussions - from a first look i think at least option 2 could be really useful, option 1 maybe too but i don't know enough to give a good judgement there.

Detailed input from my side will have to wait a bit until after I return from holidays (probably will only get to it around 14 January).

In general, I was thinking until data becomes public, in message_data I would want to try to select the most useful Rscripts from DLE packages and integrate them in a DLE workflow that uses rpy2.
For instance, I'm imagining that a "build" command could first run basic message, then use Rscripts to create a DLE scenario based on that, and then do a MESSAGE-DLE run after that.

More to follow in the new year.

@adrivinca
Copy link
Contributor

In the nexus work, we have some R scripts that need to be run to process raw data into data then used by other python scripts.
Since these R scripts need to be run just once -and sometimes link to large spatial data on the P drive- we do not include or call them from any python script. A user could just run them to generate new scenario configurations (SSP, SDGs), but otherwise all the output data of those scripts are already included in the message_data/data folder.

@khaeru khaeru linked a pull request Mar 10, 2022 that will close this issue
5 tasks
@khaeru khaeru linked a pull request Mar 10, 2022 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh New features or functionality question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants