ERROR: DataString is getting too long

[arrabito]

Dear all,

In our last production we randomly get this kind of error:

2022-11-03 08:42:55 UTC dirac-jobexec/Script INFO: Command is: /scratch/condor/dir_1381/DIRAC_jk2Ubypilot/19480794/dirac_prod6_muon_run --start_run 0 --run 00000016 --with-half-moon --degraded-mirror-ref MST-NectarCam Paranal muon North 20
2022-11-03 11:08:11 UTC dirac-jobexec/Subprocess ERROR: DataString is getting too long (52428442):  from process 12890 instead of 0.
Got signal 18 from process 12571 instead of 11802.
Got signal 18 from process 12963 instead of 11866.
Got signal 18 from process 12571 instead of 11946.
Got signal 18 from process 12571 instead of 12059.
Got signal 18 from process 12571 instead of 12150.
Got signal 18 from process 12963 instead of 12199.
....

We are not able to reproduce this error when running locally and we think it's related to DIRAC.

Could you please tell me if you have an explanation for this kind of issue?

Thank you,

Luisa

[fstagni]

This, IIUC, means that the application is printing more than 5MB of output every 0.01 seconds. That's a lot of logging! A quick fix would be that your application is less verbose.
Otherwise we can increase DIRAC's buffer limit (requires a release) but I am not sure about the possible consequences.

[andresailer]

As an in joke to today's BILD: I spent already too much time thinking about getting that error six years ago :/ #2876

Limit is set here, I think,

DIRAC/src/DIRAC/Workflow/Modules/Script.py

Line 40 in f20e8f4

self.bufferLimit = 52428800

This can be made configurable with some thinking (but fine for me if you crank that up without thinking about it.)

[marianne013]

I think we always told our users that this was a user error, and any logging should go to a file and not the console.

[fstagni]

#6559

[marianne013]

You are just encouraging them !!!

[fstagni]

Freedom for the people!

[fstagni]

So, this issue is in fact already written down in #5575. I mark this comment as "solution" to this thread.