Systematic errors in upload operations from a specific site

[arrabito]

Dear all,

I've noticed systematic errors when uploading files from a particular site to 3 different SE endpoints, while uploads from other sites succeed.

I don't know if these errors are due to some worker node configuration at the site that initiates the upload, but since I was supposing that all the needed libraries for upload operations (gfal-2, ...) are included in DIRACOS, I was not expecting errors due to specific site configuration. Do you have an explanation for that or could you suggest some test/fix?

Or could it be some DIRAC configuration issue?

Here below the error log:

{'OK': False, 'Errno': 0, 'Message': 'Failed to put file to Storage Element. Protocol not supported ( 93 : Failed to copy file Data/proton_20deg_180deg_run4685___cta-prod5b-paranal_desert-2147m-Paranal-sst-fullmoon.simtel.zst to destination url srm://ccsrm02.in2p3.fr:8443/srm/managerv2?SFN=/pnfs/in2p3.fr/data/vo.cta.in2p3.fr/MC/PROD5b/Paranal/proton/sim_telarray/2732/Data/004xxx/proton_20deg_180deg_run4685___cta-prod5b-paranal_desert-2147m-Paranal-sst-fullmoon.simtel.zst: [93] Could not open destination: Protocol not supported or path/url invalid: gsiftp://ccdcacli394.in2p3.fr:2811//upload/10/a13ec2e3-46f7-4af4-9666-32dace0102ad/proton_20deg_180deg_run4685___cta-prod5b-paranal_desert-2147m-Paranal-sst-fullmoon.simtel.zst)', 'CallStack': [' File "/home/data/tmp/222594.1.grid7/6MEMDmvqMf0n5oN1gnefVqpoABFKDmABFKDmIYFKDmABFKDmLtDi9n/DIRAC_d0xyWNpilot/diracos/bin/cta-prod-managedata", line 8, in <module>\n sys.exit(main())\n', ' File "/home/data/tmp/222594.1.grid7/6MEMDmvqMf0n5oN1gnefVqpoABFKDmABFKDmIYFKDmABFKDmLtDi9n/DIRAC_d0xyWNpilot/diracos/lib/python3.9/site-packages/CTADIRAC/Core/scripts/cta_prod_managedata.py", line 77, in main\n result = prod3dm.putAndRegister(lfn, localfile, fmd_json, package)\n', ' File "/home/data/tmp/222594.1.grid7/6MEMDmvqMf0n5oN1gnefVqpoABFKDmABFKDmIYFKDmABFKDmLtDi9n/DIRAC_d0xyWNpilot/diracos/lib/python3.9/site-packages/CTADIRAC/Core/Workflow/Modules/ProdDataManager.py", line 240, in putAndRegister\n res = self._putAndRegisterToSEList(lfn, localfile, ProductionSEList)\n', ' File "/home/data/tmp/222594.1.grid7/6MEMDmvqMf0n5oN1gnefVqpoABFKDmABFKDmIYFKDmABFKDmLtDi9n/DIRAC_d0xyWNpilot/diracos/lib/python3.9/site-packages/CTADIRAC/Core/Workflow/Modules/ProdDataManager.py", line 281, in _putAndRegisterToSEList\n res = self.dm.putAndRegister(lfn, localfile, SE)\n', ' File "/home/data/tmp/222594.1.grid7/6MEMDmvqMf0n5oN1gnefVqpoABFKDmABFKDmIYFKDmABFKDmLtDi9n/DIRAC_d0xyWNpilot/diracos/lib/python3.9/site-packages/DIRAC/DataManagementSystem/Client/DataManager.py", line 570, in putAndRegister\n return S_ERROR("%s %s" % (errStr, res["Message"]))\n']}

Thank you,

Luisa

[chaen]

Hi,
everything is definitely included in DIRACOS2. What version of DIRAC and DIRACOS do you have installed ? Can you reproduce it by hand ?

[arrabito]

Hi,
I'm using DIRAC version 7.3.12 and as far as I know, pilots download the latest DIRACOS2 version from CVMFS (no idea if there is a version number). I cannot reproduce it manually since I've seen this error only for jobs running at Lancaster.

[chaen]

oh ! Check with the site that they do not define environment variables like GFAL_* or anything else. You can check the env dump of the pilot. We had that problem with a site

[arrabito]

Thanks a lot for this suggestion.
Yes, there are GFAL_* variables set:
GFAL_PLUGIN_DIR=/cvmfs/grid.cern.ch/umd-c7wn-test/usr/lib64/gfal2-plugins/
GFAL_CONFIG_DIR=/cvmfs/grid.cern.ch/umd-c7wn-test/etc/gfal2.d/
I've just asked admins to unset them and let's see if it fix the problem.

[marianne013]

Shouldn't you (or DIRAC) unset these variables automatically if they can cause trouble ? I just had a meeting with Matt about something else, and he saw this pop up on my screen, so we talked about this a bit. He now has a "if CTA then unset" condition on his WNs, but sites tend to be very reluctant to make special cases on a VO by VO basis.

[chaen]

The problem is that it is not just the GFAL2 variable, but they also set insane things like the LD_LIBRARY_PATH. So we can unset the gfal2 variable, and you will bump into even sneakier issues.
If you are interested, we bumped into that issue: https://ggus.eu/index.php?mode=ticket_info&ticket_id=153532#update#69

[arrabito]

Just to let you know that unsetting the GFAL variables fixed the issue.
Thanks.

[mdoidge]

Hello, (the site admin here).
I did not realise that setting things like the GFAL* envs or LD_LIBRARY_PATH was so out there to be labelled as insane! We set this (and other variables) at the site for good reason, we pull the grid WN tools from cvmfs rather then a regular rpm install. The motivation on this was to minimise our rpm footprint on what is a shared cluster.
If this approach is now likely to cause issues we will have to rethink. This has somewhat blindsided us though.
Thanks,
Matt

[chaen]

Hey Matt,

Don't worry, we have been doing this insane thing too until recently 😆 That's actually why we did not notice the problem until we stopped using them :-)
I totally understand the reason of doing what you do, and it makes sense for VOs not shipping any tool. But it will inevitably break something for VOs that setup their own environment.

I am a bit surprised though that it blindsided you. I had understood that after our ticket, the several UK sites stopped setting this environment by default.

In any case, in the case of LHCb, we added in our client source script some unsetting of the variables that are likely to cause issues. But this is clearly not maintainable, and it is much easier to not set things in the first instance if not asked.

# Unset variables which are likely to cause issues if set by the site
unset GFAL_CONFIG_DIR GFAL_PLUGIN_DIR GLEXEC_LOCATION GLITE_LOCATION_VAR
unset GLITE_LOCATION GLOBUS_LOCATION JAVA_HOME LCG_LOCATION
unset LD_LIBRARY_PATH LESSOPEN MANPATH PERL5LIB PYTHONPATH SRM_PATH

[mdoidge]

Hello,
I think because LHCB (and probably other DIRAC user groups) avoided problems at our site due to the "hacks" they've had to make, and I didn't file this away under the correct section in my head.
The fix at our site is fairly straightforward - we already have an "if grid-user" clause at the start of our ENV script to distinguish between grid and local users, so AIUI I just need to remove DIRAC using VOs from this list.

Thanks,
Matt

[marianne013]

If it's any consolation it passed me by as well. Anything that happens at the T1 usually gets written of as another T1 snafu, unlikely to apply to the T2s.