Update write.config.FATES.R _ New Version #3372
Conversation
|
@mdietze @serbinsh @istfer @rosiealice : Hi, after simply testing job.sh with a lot uncomments on the original code, I'm ready to make full modification for this write.config.FATES.R file. However, it requires addresses of At the moment, we (with @huitang-earth) are thinking to use fates container provided by noresm-land-sites-platfom. I'm wondering will this container fit to PEcAn in the future or do you have any other suggestions on where I should expect fates to be implemented in PEcAn, so that I could proceed modifying the write.config.FATES.R file? Thanks for your comments in advance! |
|
Hi! |
|
To build on @rosiealice comment, I'll bring up that I'm pretty sure that NCAR's container effort started in part from own efforts at containerizing FATES from the first time we tried to couple FATES and CLM to PEcAn many years ago (all the way back to when Andy Fox was working remotely with Dave Moore at UA and then during his time at NCAR). On a practical level, regardless of which container you use, I suspect you can't use it "as is" but will instead have to build a PEcAn specific container that takes an existing FATES container and adds in the subset of PEcAn R packages that would be required for the interfaces, as well as the rabbitmq libraries required to pass jobs from the PEcAn containers into a FATES container. But I think doing that would be preferable to our team trying to revive our own outdated FATES container. Even better would be if we can convince one of these FATES container efforts to just install the PEcAn libraries by default. |
|
Thanks @mdietze and @rosiealice for your quick comment. Following @mdietze 's idea, I am wondering if it is possible to find the "outdated FATES container" or a prototype of PEcAn-enabled "FATES" container somewhere, so that we could already follow its folder structure to revise the "write.config.FATES.R"? Another alternative that just came to my mind is: @istfer mentioned to me that SIPNET has been containerized and well-updated with PEcAn. Would it be simpler to build a PEcAn-Enabled FATES container by using the existing SIPNET container with some additional libraries being installed there for running FATES? The required libraries needed for running FATES have been clearly listed in this docker file. These libraries only need to be updated occasionally when there is a major update in CLM-FATES that require newer version of some libraries (e.g., ESMF). |
|
Hi all. Coincidentally we just ran a week-long NGEE-Tropics tutorial the last week using an updated version of the tutorial container developed for the NGEE-Arctic E3SM workshop back in 2022. While the tutorial used ELM, if you trace the chain of forks back @huitang-earth and @Hhh-hyc, you can find a CTSM (CLM 5.0) dockerfile that Shawn Serbin created: https://github.com/FASSt-simulation/simulation_containers/blob/main/docker/ctsm/release-clm5.0.35/Dockerfile. I'm not sure about the maintenance status of that repo. Note that the FATES team doesn't have funding to provide much in the way of containerization development or maintenance outside of what we've done for this past tutorial. As such, we can only really provide support in that regard, although we are definitely willing to provide guidance and troubleshooting with FATES specific setups. My expectation for long-term containerization support for FATES is that it will be largely funded and driven by the various host land model specific container efforts (i.e. E3SM containers, CESM containers). Note, I hadn't been aware that there was also a NorESM container effort (i.e. the land sites platform), which is great to see! |
|
Dear all, Many thanks to your prompt responses and suggestions! We just explored all the suggested options together with @huitang-earth. Just to start with, we are thinking to first create a new Dockerfile for fates in PEcAn that loads image from pecan/models in Dockerhub similar to spinet and basgra. Then, we will add libraries and packages required by FATES (according to the Dockerfile for noresm-landsite-platform) into this base image to create a new pecan-fates image. Compared to the other dockerfiles mentioned by @glemieux, we find the dockerfile for noresm-landsite-platform includes newer versions of libraries (e.g., ESMF version 8.2) that are compatible with the most recent versions of CLM-FATES. In line with these thoughts, I have created and pushed a template Dockerfile and model_info.json to our PR as the next task for me to do. I will try to install CLM-FATES in the "pecan/model" container in the following days. Please let me know if you think this is a proper way to move forward at the moment. Any better ideas or suggestions are always very much welcomed! I will keep updating the PR here, please feel free to comment on our progress at any point |
|
|
||
| ## Load CTSM from https://github.com/NorESMhub/noresm-land-sites-platform/blob/main/docker/api/entrypoint_api.sh | ||
| ## 1. Standard address of /ctsm-api/resources/model in pecan ?? | ||
| RUN if [ ! -d /ctsm-api/resources/model ]; then && \ |
There was a problem hiding this comment.
where to put /ctsm-api/resources/model in pecan?
There was a problem hiding this comment.
Just to clarify that this folder refers to where we should put the model source code in the container. I guess there should be a standard folder for keeping each model's source code in the model container?
| RUN set -e && \ | ||
| ## set home folder for running CTSM-fates. 2. what does 'HOME' should be in pecan? | ||
| USER=root && \ | ||
| HOME=/root && \ |
There was a problem hiding this comment.
what does\HOMEin pecan should be?
| ## Dotcime (https://github.com/NorESMhub/ctsm-api/tree/main/docker/dotcime) SHOULD be kept in the home directory as '.cime' | ||
| RUN COPY docker/dotcime /<home folder>/.cime | ||
|
|
||
| ## 3. Download inputdata in dockerfile? |
There was a problem hiding this comment.
Do we need to download input data in docker file?
| RUN COPY docker/dotcime /<home folder>/.cime | ||
|
|
||
| ## 3. Download inputdata in dockerfile? | ||
| ## 4. Build CTSM-fates here or in write.config file? |
There was a problem hiding this comment.
Do we need to build CTSM-fates in Dockerfile or in write.config.R?
There was a problem hiding this comment.
Since CLM-FATES needs to be re-complied every time when some setting changes, I would vote for putting the "building " phase of the model in template.job similar to what have been done there at line 134.
|
|
||
| ######################################################################## | ||
|
|
||
| ## 5. Is the following part (copied from sipnet) necessary for fates? |
There was a problem hiding this comment.
The following part is copied from sipnet. Are they (e.g., copy binaries) still necessary for fates in pecan?
|
With the new commits, all the packages needed for running CTSM-FATES has been added into the Dockerfile, and the environmental parameters have been set for running the model. However, some folders (e.g., where to keep model source code, model executable file, model input and output etc.) needs to be further discussed before moving forward on building the model. Please check the detailed comments in the pull request. |
Motivation and Context
This pull request is ultimately for running fates in the dockerized container in PEcAn, which is currently running in the container of noresm-land-sites-platform and thus would be updated to adapt PEcAn later.
The goal would be achieved by three steps:
Description
As the first step, lines related to parameters and domain files were uncommented to only basically generate 'template.job' for successfully running fates. Some arguments were pre-provided due to the incapability to use BETY Database locally.
Notably, the update has not been completed yet and opening this PR is just for better interaction.
There was an old PR for this work but should be closed for missing the recent contributions of the community, but some comments may still be useful for proceeding the work, which could be found in Update write.configs.FATES.R #3360.
Review Time Estimate
Types of changes
Checklist: