Jobs keep being rescheduled

[arnkress]

Hi,
I'm assisting a user to run jobs on the biomed VO. I'm using DIRAC version v7r3p15, installed by our admin.

My problem is that the jobs (here a simple "hello world") are repeatedly marked as being rescheduled, and never executed.

Source                Status    MinorStatus             ApplicationStatus              DateTime
============================================================================================================
JobManager            Received  Job accepted            Unknown                        2022-01-19 16:33:41
JobPath               Checking  JobSanity               Unknown                        2022-01-19 16:33:41
JobSanity             Checking  JobScheduling           Unknown                        2022-01-19 16:33:41
JobScheduling         Waiting   Pilot Agent Submission  Unknown                        2022-01-19 16:33:41
Matcher               Matched   Assigned                Unknown                        2022-01-19 16:47:43
[email protected]   Matched   Job Received by Agent   Unknown                        2022-01-19 16:47:43
[email protected]   Matched   Submitting To CE        Unknown                        2022-01-19 16:47:57
JobPath               Checking  JobSanity               Unknown                        2022-01-19 18:52:23
JobSanity             Checking  JobScheduling           Unknown                        2022-01-19 18:52:23
JobScheduling         Waiting   Pilot Agent Submission  Unknown                        2022-01-19 18:52:23
Matcher               Matched   Assigned                Unknown                        2022-01-19 18:52:37
[email protected]  Matched   Job Received by Agent   Unknown                        2022-01-19 18:52:37
[email protected]  Matched   Submitting To CE        Unknown                        2022-01-19 18:52:37
JobPath               Checking  JobSanity               Unknown                        2022-01-19 20:57:23
JobSanity             Checking  JobScheduling           Unknown                        2022-01-19 20:57:23
JobScheduling         Checking  JobScheduling           On Hold: after rescheduling 2  2022-01-19 20:57:23
JobScheduling         Waiting   Pilot Agent Submission  Unknown                        2022-01-19 21:03:23
Matcher               Matched   Assigned                Unknown                        2022-01-19 21:04:42
[email protected]  Matched   Job Received by Agent   Unknown                        2022-01-19 21:04:42
[email protected]  Matched   Submitting To CE        Unknown                        2022-01-19 21:05:01
JobPath               Checking  JobSanity               Unknown                        2022-01-19 23:07:23
JobSanity             Checking  JobScheduling           Unknown                        2022-01-19 23:07:23
JobScheduling         Checking  JobScheduling           On Hold: after rescheduling 3  2022-01-19 23:07:23
JobScheduling         Waiting   Pilot Agent Submission  Unknown                        2022-01-19 23:17:25
Matcher               Matched   Assigned                Unknown                        2022-01-19 23:46:06
[email protected]   Matched   Job Received by Agent   Unknown                        2022-01-19 23:46:06
[email protected]   Matched   Submitting To CE        Unknown                        2022-01-19 23:46:29

After I while, it reaches the maximum number of reschedulings:

$ dirac-wms-job-status 118363518
JobID=118363518 Status=Failed; ApplicationStatus=Unknown; MinorStatus=Maximum of reschedulings reached; Site=EGI.RAL.uk;

Do you have any idea how to investigate this problem ?

[fstagni]

Hi,
this problem was spotted for version 7.3.14 and it's fixed since then. When you say "using v7r3p15" version, do you mean client, server, or pilot version?
BioMed VO I believe runs on EGI/FG service, so cc @atsareg

[arnkress]

Thank you for your answer. I'm actually using the client version v7r3p15, on a server provided by France Grilles (SCIGNE platform).

[atsareg]

The server is at v7r3p15 also since this week. Can you please try again

[atsareg]

And yes, the pilots are at v7r3p15 as well

[atsareg]

The problem was tracked down to the faulty user's proxy which prevented the jobWrapper to contact DIRAC services, including the state update. The job was rescheduled by the StalledJobAgent. After fixing the proxy, the problem is solved.