Synchronization with base environment

[ramirezdiego]

Not a problem right now, but wondering whether we want to make this automatic (is possible) at one point. I guess the solution can be not that simple, given the ROOT and G4 dependencies.

[rynge]

The container is currently built from the development tag of the base_environment:

FROM xenonnt/base-environment:development

To make the MC container reproducible, we should pick a more specific tag to build from. However, base environment is currently only tagging sporadically, so I think it is better to leave it as it now, until you find a hard dependency you need to track.

[ramirezdiego]

@rynge Thanks for your response and the short discussion in today's computing telecon: To continue where we left it: I fear that for us (MC team) the current most important package in the container is WFsim (CC @zhut19 @petergaemers). So I would naively assume one can just re-deploy this container as soon as there are commits on WFsim. There is for sure a better approach (also waiting for what the analysis team decided on tha base_environment), but maybe this one could work for now...? I assume there will be lots of updates and testing in the next weeks and we probably can't wait for the monthly synchronization.

[rynge]

Correct - you can just redeploy your mc container when you want to include a new version of WFsim.

If you want to depend on a more stable base_environment, change this line:

https://github.com/XENONnT/montecarlo_environment/blob/master/Dockerfile#L1

To for example:

FROM xenonnt/base-environment:2020.07

As you can tell, we have not tagged in while. Maybe a new tag is in order - we should check with Evan/Joran and see when tagging would be a good idea.

[ramirezdiego]

Thanks Mats. There are a few things I still don't get completely, so let me see if I can bring them up properly:

1. Under /project2/lgrandi/xenonnt/singularity-images we have the tagged containers (not so many...) and the development versions of both, base and montecarlo containers. From the time stamp of these two, it seems that the montecarlo container is deployed as soon as there's an update in the base containter, is that correct? If yes, I think we can live with it for the moment, but indeed a proper tagging procedure could help.
2. When is the development base environment being updated exactly? After every commit in any of the relevant repositories? For example, is it linked to the updates in wfsim or straxen master branchs?
3. The images under /cvmfs/singularity.opensciencegrid.org/xenonnt/ aren't as updated as the ones in project2. When do these get updated? I guess we would rely on these versions for simulations in the OSG login node.
4. When I load singularity shell https://xenon.isi.edu/images/xenonnt-montecarlo%3Adevelopment.simg in OSG, which of the versions am I looking at, from the two above?

(Indeed tagging versions becomes useful to solve most of the questions above :D )

[rynge]

1. The two environments are currently not linked. The timestamps are probably just from the script that pulls them down and puts them on Midway. We might be able to link the two builds using deployhq, but long term, both of these should be stable and not require quick turnaround builds because one changes.
2. The environments are rebuilt on commits, but only to the environments themselves. There are no external triggers like straxen. Note that just like you don't want things to changed underneath you, base_environment locks some versions it includes (https://github.com/XENONnT/base_environment/blob/master/create-env#L13).
3. They are updated at the same time. There is a flow on the bottom of https://github.com/XENONnT/base_environment which has everything except the pulling down to Midway, which happens nightly.
4. That is the source for the images in /project2/lgrandi/xenonnt/singularity-images - there could be up to 24 hour delay as the scripts only run nightly, but otherwise they should be the same.

Note that this is the approach we have currently, and I'm well aware it can be improved. So let's keep discussing!

[rynge]

I forgot to mention: maybe a more formal versioning will solve this, so maybe that is the discussion we should have.

[ramirezdiego]

Thanks a lot! 👍 I cc @ershockley so that he's aware of the discussion.

1. OK! Agreed.
2. Makes sense, but in that case I would ask a more direct question. Right today there was a wfsim version upgrade by @zhut19. Could you show me the steps to let montecarlo_environment know about it and release a new container? This might then lose the synchronization with the base container, so I am curious about how it's done. We build from base, but then on top of that install the newest wfsim?
3. OK, thanks. But then this means that the true "release" date of our environment is the one that figures under /cvmfs/singularity.opensciencegrid.org/xenonnt/, is that correct? Oct 12 14:08, in this case. Makes sense, since that's when you pushed the initial directory fix.

[rynge]

2. Is wfsim already in the montecarlo_environment? I don't see it in there. If not, we should add the install to create-env and version it like we do with straxen.
3. I would probably just look at the timestamps in https://xenon.isi.edu/images/ or in DockerHub.

[ramirezdiego]

2. wfsim is in the base environment, so a priori there was no need to add it to ours. But indeed we might want to immediately update this environment, following wfsim commits. Do we then add that line to our create-env?
3. 👍

[rynge]

https://github.com/XENONnT/base_environment/blob/master/create-env#L89 - well that should probably be versioned like we do for straxen.

To pick up this change, we can trigger new builds in DeployHQ. Once for base_environment, and when that is done, montecarlo_environment. Do you want to try to version it first, or want me to kick off the builds directly?

If you do not already have a DeployHQ account and want to kick off manual builds yourself, ask Chris for an account.