-
Notifications
You must be signed in to change notification settings - Fork 176
/
release.notes
12600 lines (8831 loc) · 425 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v9.0.0a31]
*WorkloadManagement
FIX: (#7827) SandboxStore not able to assign a sandbox to a job
FIX: (#7825) JobWrapper checks existence of executable in jobIDPath
FIX: (#7821) Getting pilot reference from job parameters
*Resources
FIX: (#7819) AREXCE returns an error if a queue is not found in the ARC instance configuration
[v9.0.0a30]
*DataManagementSystem
NEW: (#7817) experimental token support for FTS transfers
FIX: (#7781) dirac_dms_find_lfns: Check if requested path exists and returns error if not. Prevents users hitting #7487
FIX: (#7780) Return correct error for non-existing directory. Partial fix for #7487
NEW: (#7764) added checksum comparison in the fc/se consistency tool
NEW: (#7756) FileCatalogHandler: add function export_getFileDetails to get the (user) metadata for a list of LFNs
FIX: (#7719) Remove `def findDirIDsByMetadata(self, metaDict, dPath, credDict):` method from `MultiVODirectoryMetadata` (derived) class which caused an extra VO suffix added when searching. The method is meant to be used _internally_ only on keys which are already expanded in a MultiVO case. Add a user-level def `findDirectoriesByMetadata(self, queryDict, path, credDict)` to the derived class thus adding a VO suffix for a directory search. Fixes #7687.
CHANGE: (#7694) dirac-dms-replicate-and-register-request: Make request chunk size configurable; default behaviour unchanged.
*FrameworkSystem
NEW: (#7817) allow to bypass cache when retrieving tokens
FIX: (#7813) SSHCE, Try python3 before unversioned python
Fix a typo in the key Addler -> Adler of the return dict of putAndRegister
*TransformationSystem
FIX: (#7806) RequestTaskAgent only considers requests in final states, and consider files in intermediate state as problematic (https://github.com/DIRACGrid/DIRAC/issues/7116)
NEW: (#7806) RequestTaskAgent uses getBulkRequestStatus instead of getRequestStatus
RMS: (#7806)
NEW: (#7806) implement getRequestStatus
NEW: (#7697) InputDataAgent: new Option MultiVO, which makes the FileCatalog Query use the author of the DN, rather than the Host, to resolve MultiVO metadata correctly. Fixes #7681
*Resources
FIX: (#7803) added a 30s gfal2 timeout for downloading the SRR
FIX: (#7790) SSHComputingElement fix: added check of result
FIX: (#7726) Update CloudCE cloudinit.template for EL8+
CHANGE: (#7715) HTCondorCE: UseSSLSubmission: use the generated proxy file for everything, no longer need to have certificate of user present on the server.
FIX: (#7713) AREX submission issue not properly handled
NEW: (#7695) better error message when no matching protocol between 2 SE
CHANGE: (#7689) remove BOINCCE
*ResourceStatusSystem
FIX: (#7801) use always a from address (from Operations ResourceStatus/Config/FromAddress ) when sending email notifications, to avoid "spoofing" domains restrictions
NEW: (#7783) add a DIRAC to GOCDB service type conversion
CHANGE: (#7774) do not consider Endpoint for StorageOccupancy
FIX: (#7765) Do not use tinezone aware datetime
FIX: (#7755) delete the Occupancy cache only for older entries
*WorkloadManagement
FIX: (#7797) Allow jobs to be KILLED from more states
FIX: (#7797) KILLED is a final job state
FIX: (#7716) PilotWrapper - check for the presence of the -l pilot option
*ConfigurationSystem
CHANGE: (#7796) VOMS2CSAgent: if a nickname is set, this nickname will always be used and no new accounts are going to be created if a DN changes or a user is in multiple VOs
NEW: (#7796) VOMS2CSAgent: New option "ForceNickname", if this option is enabled no dirac user is created if no nickname attribute is set for a user
CHANGE: (#7796) IAMService: use logger and return errors for users so that the VOMS2CSAgent can notify admins about issues
NEW: (#7742) Configuration-system-shell: added reload and sort commands
*WorkloadManagementSystem
CHANGE: (#7792) SiteDirector will always bundle the proxy
CHANGE: (#7762) removed JobDB's SiteMask and Logging
FIX: (#7751) Proper killing of jobs when not matched, running or stalled
FIX: (#7707) JobDB: update LastUpdateTime when the job is matched
NEW: (#7699) added possibility to specify UserEnvVariable (pilot option) at CE level
*Workflow
FIX: (#7786) Avoid incorrect error strings in Workflow execute
*Core
NEW: (#7775) Support aggregating by date in MySQL.getCounters
FIX: (#7750) add more safeguard when processing result queue in ProcessPool
CHANGE: (#7720) register the DictCache destructor as an atexit handler
NEW: (#7700) fetch nickname attribute from IAM
CHANGE: (#7696) drop DIRAC_MYSQL_CONNECTION_GRACE_TIME and stalled connections reuse
*Subsystem
CHANGE: (#7733) For PilotWrapper tests, use the artifacts (created in Pilot repo)
*Diracx
NEW: (#7711) populate diracx section from Iam
*Doc
FIX: (#7688) Move the explanation how to enable tokens to a place that is shown in ReadTheDocs in the end.
*Interfaces
FIX: (#7684) Document BadJobParameterError
[v9.0.0a29]
*WorkloadManagementSystem
FIX: (#7649) added log headers to InputDataResolution modules
CHANGE: (#7629) add jobGroup to job parameters
FIX: (#7584) ServerUtils: prevent getPilotAgentsDB from returning None
FIX: (#7576) Fix potential circular import in WorkflowReader. Mostly seen in the creation of the documentation.
FIX: (#7787) added a 30s gfal2 timeout for downloading the SRR
*TransformationSystem
FIX: (#7741) RequestTaskAgent only considers requests in final states, and consider files in intermediate state as problematic (https://github.com/DIRACGrid/DIRAC/issues/7116)
NEW: (#7741) RequestTaskAgent uses getBulkRequestStatus instead of getRequestStatus
RMS: (#7741)
NEW: (#7741) implement getRequestStatus
[v8.0.52]
*ResourceStatusSystem
FIX: (#7800) use always a from address (from Operations ResourceStatus/Config/FromAddress ) when sending email notifications, to avoid "spoofing" domains restrictions
*WorkloadManagement
NEW: (#7643) Support Pydantic 2
FIX: (#7621) remove random shuffle in PilotWrapper
CHANGE: (#7609) Perform bulk lookup of job parameters from elasticsearch
CHANGE: (#7608) Make RemoteRunner more resilient to CE issues
FIX: (#7594) JobMonitoring.getJobParameters should pass jobID as an int to ElasticJobParametersDB
FIX: (#7590) AREX "out" and "err" need to exist before file integrity check
*Accounting
FIX: (#7640) AccountingDB only generate condition if needed
*Resources
NEW: (#7638) HTCondorCE: Added UseSSLSubmission option. Allows one to use a configured DN at given Sites for job submission, instead of proxies or tokens. Only at participating CEs and conditions apply.
*Core
FIX: (#7634) Avoid printing out clear text password in SQLAlchemy
FIX: (#7591) File.secureOpenForWrite: fix exception when opening in binary mode, fixes #7581
*DataManagementSystem
NEW: (#7633) Add a protocol parameter to the getReplicas method family
NEW: (#7619) prepare for FTS 3.13 release with breaking API
NEW: (#7617) DataManager.putAndRegister rejects too long filename
*Deployment
FIX: (#7628) fix the path of the CVMFS sync_packages.sh script
*All
FIX: (#7616) fix pylint 3.2.0 warnings
*MonitoringSystem
FIX: (#7584) ServerUtils: prevent getMonitoringDB from returning None
[v9.0.0a28]
*WorkloadManagementSystem
FIX: (#7574) serverUtils: jit imports
FIX: (#7534) Pilots submitted by SiteDirector won't add the pythonVersion flag
FIX: (#7521) Fix memory reporting
FIX: (#7510) SandboxStore: add VO if needed
*WorkloadManagement
FIX: (#7571) support file:/... as a location for the pilot files
FIX: (#7564) make sure CVMFS_locations is a list
CHANGE: (#7553) Remove files from the RemoteRunner execution
FIX: (#7552) JobCleaningAgent: fix exception in deleteJobsByStatus caused by mismatching job ID types
NEW: (#7529) introduce JobWrapperOfflineTemplate for uses in systems without external connectivity
CHANGE: (#7460) introduce JobWrapper.preprocess, process and postprocess
*test
NEW: (#7570) added pilot workflow tests to integration_tests
*Core
FIX: (#7569) Support M2Crypto 0.40.0+
CHANGE: (#7566) Replace the default PFN type ROOT_All with ROOT
FIX: (#7524) Depend on packaging
*environment.yml
NEW: (#7555) add cwltool
*Resources
FIX: (#7545) TimeLeft utility was unable to get values from the cfg
FIX: (#7532) support the case where HTCondor kills the jobs
*Test
FIX: (#7540) Redirect the output of popen in a file to fix #7473
FIX: (#7539) extra_module default to empty list
*FrameworkSystem
CHANGE: (#7511) ProxyDB: removed tables ProxyDB_Proxies and ProxyDB_Tokens
[v9.0.0a27]
*Core
FIX: (#7505) plotting TypeLoader works with editable installation
NEW: (#7453) Introduce an RPC stub equivalent for DiracX
*Test
CHANGE: (#7502) use pytest-rerun instead of flaky
*WorkloadManagementSystem
CHANGE: (#7498) removed GridEnv
FIX: (#7497) If the SoftwareDistModule is set in the Operations Section, add it to the Job JDL to restore previous behaviour
NEW: (#7453) FutureJobStateUpdate.setJobStatusBulk return a DiracX RPC stub
*WorkloadManagement
FIX: (#7493) check the VO from the task queues before submitting pilots
FIX: (#7488) JobAgent.setupProxy takes owner instead of ownerDN
*RequestManagementSystem
CHANGE: (#7453) adapt ForwardDISET to DiracX stub
*FrameworkSystem
CHANGE: (#7442) removed NotificationDN
[v9.0.0a26]
*FrameworkSystem
FIX: (#7491) dirac-proxy-info without the dirac group
*WorkloadManagement
FIX: (#7490) missing result in return statement
*tests
NEW: (#7484) add the state key in the test environment to fix diracx execution
[v9.0.0a25]
*Accounting
FIX: (#7486) Fix errors during insert into ac_in_* tables.
*Core
FIX: (#7483) Fix DISET calls with proxy to be used passed as an argument
*RequestManagementSystem
FIX: (#7482) RequestTask - download no-VOMS proxy if the owner group does not define VOMSRole
[v9.0.0a24]
*WorkloadManagement
FIX: (#7480) JobCleaningAgent - select random jobs for deletion rather than head and tail jobs.
FIX: (#7475) JobID type in PushJobAgent
*Interfaces
CHANGE: (#7472) dfind - more explicit failure report
*DataManagementSystem
NEW: (#7471) add tools for consistency checks
*tests
FIX: (#7470) check running containers in integration tests script
*WorkloadManagementSystem
CHANGE: (#7464) PilotAgentsDB: removed OutputReady and Broker fields
[v9.0.0a23]
*WorkloadManagement
FIX: (#7458) jobID type issue in JobAgent
*ConfigurationSystem
FIX: (#7454) getQueue() overriding the CE tags
*FrameworkSystem
FIX: (#7451) dirac-proxy-init printInfo without the dirac group
[v9.0.0a22]
*WorkloadManagement
FIX: (#7448) get pilot logging info with a token from an AREXCE
*WorkloadManagementSystem
FIX: (#7447) Each job has its own JobReport in JobAgent
FIX: (#7447) JobAgent exits when all the jobs have been processed
FIX: (#7446) StatesAccountingAgent: skip the first iteration in order to avoid double commit after a restart
CHANGE: (#7439) SandboxStore: remove external SE feature
FIX: (#7436) JobDB: fix mismatch of string and integer jobIDs. In some cases API calls would fail because JObDB.getJobParameters and JobDB.getJobsAttributes return dictionaries with integer keys, while the function was called with a string jobID. This fixes for example the StalledJobAgent being unable to reschedule matched jobs.
NEW: (#7425) TornadoPilotLoggingHandler modify the handler to accept VO name sent by a pilot. Required in a case where the VO cannot be guessed from a proxy. This change requires https://github.com/DIRACGrid/Pilot/pull/230
NEW: (#7421) SandboxDB: add VO field
CHANGE: (#7414) move the content of SubmissionPolicy in SiteDirector
*FrameworkSystem
FIX: (#7443) correctly set the duration of tokens in cache
*RequestManagementSystem
FIX: (#7441) make sure OwnerDN is defined before trying to access its value
*MonitoringSystem
CHANGE: (#7432) moved to weekly indices for agent and service monitoring
[v9.0.0a21]
*Test
CHANGE: (#7417) write the diracx CsSync config in the CS instead of taking the yaml file from diracx repo
*FrameworkSystem
CHANGE: (#7413) removed Proxies persistency flag
*Core
FIX: (#7412) Adding VOMS extensions without having environment variables set
NEW: (#7412) Add DIRAC_DISABLE_GCONFIG_REFRESH environment variable to prevent gConfig being accidentally used
FIX: (#7409) Use proxy lifetime for tokens from legacy proxy exchange (https://github.com/DIRACGrid/diracx/issues/130)
*WorkloadManagement
FIX: (#7409) Add DiracX to payload proxies used by compute elements (#7402)
FIX: (#7406) SiteDirector should not interact with CEs if there is 0 pilot to submit
*WorkloadManagementSystem
CHANGE: (#7407) JobDB simplifications
CHANGE: (#7405) Removed Private Pilot functionality
[v9.0.0a20]
*WorkloadManagement
FIX: (#7399) JobAgent rescheduling wrong jobs
FIX: (#7387) JobAgent interaction with JobMonitoringClient
*WorkloadManagementSystem
CHANGE: (#7396) PilotAgentsDB: move from OwnerGroup to VO
FIX: (#7379) The callback for the Stager was failing, because of a type mismatch in the jobID used to retrieve the status. Jobs never came out of Staging.
NEW: (#7375) pilotWrapper: using CVMFS_locations for discovering the pilot files
*FrameworkSystem
FIX: (#7391) send notifications for expiring proxies
Documentation on how to deploy a third party tool (fluent-bit) to grab, format and send Dirac current logs to ElasticSearch and/or splitted logs files
*Resources
FIX: (#7376) AREXCE should break when a valid delegation ID is found
*Core
FIX: (#7374) Converting p12 files with filenames containing special characters
[v9.0.0a19]
[v9.0.0a18]
[v9.0.0a17]
[v9.0.0a16]
[v9.0.0a14]
[v9.0.0a14]
[v9.0.0a15]
[v9.0.0a14]
[v9.0.0a13]
[v9.0.0a12]
[v9.0.0a11]
[v9.0.0a10]
[v9.0.0a9]
[v9.0.0a8]
[v9.0.0a7]
*Resources
FIX: (#7366) make sure the WLCG accounting file is json
[v9.0.0a6]
*WorkloadManagementSystem
FIX: (#7364) added SiteDirector option for CVMFS_locations
[v9.0.0a5]
*FrameworkSystem
FIX: (#7360) wrong service uptime calculation
*Resources
FIX: (#7357) get batch system details from local cfg instead of environment variables
FIX: (#7349) remove valid argument from setToken
*docs
NEW: (#7356) added doc for PreInstalledPilotEnv
*Core
NEW: (#7344) Introduce DIRAC_MYSQL_CONNECTION_GRACE_TIME to specify the grace time of the MySQL connection pool
*TransformationSystem
NEW: (#7337) add an index on Status and Type for the TransformationDB.Transformations table
*WorkloadManagementSystem
FIX: (#7332) basic JobMonitoring client fixes
FIX: (#7331) JobAgents will set jobStatus=Failed/Payload failed if and only if the job was previously Running
*CORE
FIX: (#7332) allow for proxyLocation to work with host certificate
[v9.0.0a4]
*docs
FIX: (#7330) just use `pip install DIRAC` for DIRAC client install
FIX: (#7330) updated picture for py3 stack
*Interfaces
FIX: (#7328) Improve performance of job delete/kill/reschedule API
FIX: (#7322) Lower log level for printing version in submitJob
*WorkloadManagementSystem
FIX: (#7327) Allow "ANY" in the valid sites list
NEW: (#7318) initial FutureClient for JobStateUpdate
FIX: (#7316) ElasticJobParametersDB: do not configure the IndexPrefix name
[v9.0.0a3]
*WorkloadManagement
FIX: (#7305) Use copy+remove in PilotSync agent to avoid SELinux problems
FIX: (#7288) WMSUtilities supports VO-specific token tags
*WorkloadManagementSystem
FIX: (#7301) correctly handle UTC in JobLoggingDB float timestamp
*Core
FIX: (#7297) convert exception object to string representation in ElasticSearchDB
*DataManagementSystem
CHANGE: (#7297) default proxy lifetime for FTS is 36h and force the redelegation when 12h are left
*FrameworkSystem
FIX: (#7284) disabledVOs not available when doing dirac-configure
*DiracX
NEW: (#7261) DiracX credentials are now included in proxy PEM files created by DIRAC
NEW: (#7261) DiracX is now mandatory 🎉
[v9.0.0a2]
*WorkloadManagementSystem
FIX: (#7282) JobStatus: allow WAITING -> KILLED
FIX: (#7263) SiteDirector acts for a VO
*FrameworkSystem
FIX: (#7280) show status SysAdminCLI KeyError
*Resources
FIX: (#7279) getting more details about failed/aborted pilots from HTCondor
*Core
FIX: (#7273) createClient: also look at HandlerMixin classes to find `export_`ed functions. Fixes the client Documentation creation, fixes #7265
*WorkloadManagement
NEW: (#7271) SiteDirector - added preinstalledEnv and preinstalledEnvPrefix pilot options
[v9.0.0a1]
*Core
NEW: (#7258) Support using DiracX as a backend for RPC calls
*WorkloadManagementSystem
CHANGE: (#7257) Refactor PilotLoggingAgent: download proxies at initialisation. Renew a proxy if it is about to expire.
*Resources
NEW: (#7256) Add support for loading external providers in CloudCE
NEW: (#7256) Add OpenNebula6 (XMLRPC) provider for CloudCE/libcloud
*Subsystem
FIX: (#7253) prevent AREXCE from creationg delegations each time a CE got a small and temporary issue
*docs
NEW: (#7252) add documentation about setting up DIRAC to submit pilots with tokens
*WorkloadManagement
CHANGE: (#7250) pilot submission with tokens in a multi-VO context
[v8.1.0a23]
*TransformationSystem
NEW: (#7239) TransformationCleaningAgent: Use the RequestManagementSystem to asynchronously remove files, rather than serially in the agent run
[v8.1.0a22]
*WorkloadManagementSystem
FIX: (#7238) Output sandbox are assigned when using DiracX
CHANGE: (#7237) Removed interaction with MJF
FIX: (#7226) SiteDirector adds wnVO Pilot options, if needed
CHANGE: (#7215) add VO info in the table Jobs of JobDB
*WorkloadManagement
FIX: (#7234) AREX does not return an error if delegation not found
FIX: (#7230) Group to vo conversion in TaskQueueDB
*ConfigurationSystem
CHANGE: (#7225) explicitly state the VO in each default group upon installation
*DiracX
CHANGE: (#7222) Get a access/refresh token pair when generating a proxy
NEW: (#7222) Support using diracx as backend for the SandboxStoreHandler
[v8.1.0a21]
*Core
FIX: (#7216) replace ColorBar.draw_all() with figure.draw_without_rendering()
*WorkloadManagement
FIX: (#7213) emove OwnerDN from SiteDirector & PilotStatus
*WorkloadManagementSystem
CHANGE: (#7190) add VO info in TaskQueueDB
FIX: (#7187) code simplification for SandboxStoreClient
[v8.1.0a20]
*Test
FIX: (#7206) manually start the extra services when running the integration test
NEW: (#7143) allow integration tests to run against DiracX
*TransformationSystem
CHANGE: (#7204) TransformationCleaningAgent removes files by chunk
FIX: (#7200) Drop TS permission check for all read-only functions
*WorkloadManagement
FIX: (#7201) JobMonitoringClient uses the new DiracClient class
CHANGE: (#7194) Job.setNumberOfProcessors raises an exception if the function arguments aren't valid.
Please follow the template:
*WorkloadManagementSystem
FIX: (#7196) do not try to insert nan values in AccountingDB
NEW: (#7143) call the diracx Job Monitoring endpoint
NEW: (#7105) Enhance dirac-admin-get-pilot-output to download remote pilot logs. Currently only from a SE.
*FrameworkSystem
FIX: (#7192) replace create_access_token with create_token
*ConfigurationSystem
CHANGE: (#7143) do not look for the shifter under the Default or <Setup> section
*Core
NEW: (#7143) get a token for a proxy
[v8.1.0a19]
*ConfigurationSystem
CHANGE: (#7182) VOMS2CS agent only sends a mail in case of change in the CS
*WorkloadManagementSystem
FIX: (#7180) passing pilotDN instead of owner in getPilotProxyFromVOMSGroup()
[v8.1.0a18]
*Production
CHANGE: (#7162) Move from DN to Username
*WorkloadManagementSystem
CHANGE: (#7157) PilotAgentsDB: move DN in favor of username
[v8.1.0a17]
*TransformationSystem
NEW: (#7170) Implement a finer grained permissions model for Transformations.
*DataManagementSystem
FIX: (#7168) better error reporting when an existing LFN has no replicas (at least it doesn't crash)
*WorkloadManagement
CHANGE: (#7167) Stop using PID namespaces with SingularityCE
*Resources
FIX: (#7160) default providerType if not found in IdProviderFactory
*WorkloadManagementSystem
FIX: (#7155) Adding Owner as JDL requirement
CHANGE: (#7151) factorize some utilities of JobSubmit into functions
[v8.1.0a16]
*FrameworkSystem
FIX: (#7149) Do not overwrite the tornado port if already set
FIX: (#7117) replaced deprecated importlib read_text function to fix failure in CI tests
*Resources
FIX: (#7147) MultiVO proxy renewal in AREX
FIX: (#7146) Propagate environment variables from host to container.
FIX: (#7134) Don't print client_secret in oauth2 debug
FEAT: (#7120) allow to pass credentials when downloading the SRR file
CHANGE: (#7103) ARC and ARC6 are now deprecated
CHANGE/FIX: (#7102) proxy renewal logic in AREX
FIX: (#7102) submitting tokens with AREX
FIX: (#7102) correctly report aborted pilots with AREX
*TransformationSystem
FIX: (#7144) dirac_production_runjoblocal: updated list of Pilot modules
CHANGE: (#7124) use userName instead of DN
*Core
FIX: (#7142) look also for DIRAC.rootPath/etc/dirac.cfg before printing a warning of not found cfg
*docs
change: Replace the use of sphinx-panels with sphinx-design, this unblocks the use of latest sphinx versions (panels was requiring sphinx < 5)
FIX: (#7133) formatting item list
*WorkloadManagement
NEW: (#7132) Introducing 'Scouting' status in job state transitions
*ResourceStatusSystem
FIX: (#7120) Add the Error status in the scripts
*WorkloadManagementSystem
FIX: (#7101) JobManagerHandler: get the OwnerDN only when needed.
[v8.1.0a15]
*Resources
FIX: (#7100) concatenate tags instead of overriding them in ComputingElement.initializeParameters()
NEW: (#7079) audienceName attribute in CEs
NEW: (#7065) add a test for the PoolCE to highlight the fact that submission failures cannot be handled by the caller with the return value
*WorkloadManagementSystem
FIX: (#7098) Exceptions in StalledJobAgent
CHANGE: (#7097) modifying access specifiers of JobPolicy from private to protected
FIX: (#7081) disable activityMonitoring for JobAgent
NEW: (#7079) audience support in SiteDirector
CHANGE: (#6973) validating the JDL job format using a pydantic model
*FrameworkSystem
FIX: (#7094) Support Authlib v1.2.1
*FrameworkSyste
FIX: (#7079) fix TokenManager(Client/Handler/Utilities)
*WorkloadManagent
NEW: (#7068) New TornadoSandboxStoreHandler implementation
*Core
NEW: (#7068) TornadoClient - added sendFile() method
FIX: (#7067) Close DISET selector when we no longer need it
*WorkloadManagement
FIX: (#7065) management of the job status in JobAgent
NEW: (#7065) add a test for the JobAgent to make sure the status of the submissions are correctly handled
*docs
NEW: (#7065) documentation about ComputingElement
*tests
NEW: (#7024) added script for comdirac hackathon certification tests
[v8.1.0a14]
*WorkloadManagementSystem
NEW: (#7059) new script dirac-admin-add-pilot
*dashboard
NEW: (#7039) add missing dashboards of issue #6742 also to the LHCb grafana organisation. The dashboards can be checked/reviewed at https://monit-grafana.cern.ch/dashboards/f/qwdBd3KVk/dev-dirac-certification
Please follow the template:
*docs
FIX: (#7023) added remaining info from COMDIRAC Wiki. Addresses part of #6746
[v8.1.0a13]
*Core
FIX: (#7058) print a warning message when no CFG file is found
FIX: (#6976) retry upon DNS lookup error
FIX: (#6955) HandlerManager - create separate class for each service handler
FIX: (#6955) BaseRequestHandler - fullComponentName can be only defined in the configuration
*docs
FIX: (#7057) added a note about removal of elastic job parameters
CHANGE: (#6969) more details in the "supercomputers" section
*Resources
CHANGE: (#7055) use double slash in Echo xroot URLs
FIX: (#7042) reduce memory consumption of AREXCE.getJobOutput()
CHANGE: (#7036) remove CREAM CE
FIX: (#7028) SSHCE getPilotOutput remove stamp
FIX: (#7000) ARCLogLevel
FIX: (#6999) remove executables definition from AREXCE.submitJob()
NEW: (#6969) cleanJob() in AREX
CHANGE: (#6969) executables from inputs are passed as executables in the XRSL in ARC and AREX
*test
FIX: (#7053) Workflow tests need pilot.cfg to work
*WorkloadManagementSystem
FIX: (#7052) Implementation of DefaultLocalCEType option
FIX: (#7046) fix(StalledJobAgent): after a job was Killed, we have to force the Failed.
CHANGE: (#7033) TokenManagerHandler provides client access token if username is not provided.
CHANGE: (#7031) Using JobManager instead of JobDB for rescheduling stalled jobs
CHANGE: (#7019) removed old format of ElasticJobParametersDB
NEW: (#7017) add the possibility of adding generic options for running the pilots
FIX: (#6980) JobStatus: add staging to checking transition, needed by StorageManagementSystem (Stager)
NEW: (#6969) tests for RemoteRunner
CHANGE: (#6969) RemoteRunner can deal with MP applications, clean job outputs and process more applications
FIX: (#6969) Modify the DIRACVersion in the PushJobAgent
*WorkloadManagement
NEW: (#7042) md5 checksum comparison in RemoteRunner
NEW: (#7041) Add /Resources/Computing/DefaultLocalCEType option
FIX: (#7007) return specific exit code in RemoteRunner in case of failure
FIX: (#7006) Getting the value of /LocalSite/RemoteExecution
*RequestManagementSystem
FIX: (#7013) fix the monitoring of Operations not yet entered into the DB
*FrameworkSystem
CHANGE: (#6967) enable/disableLogsFromExternalLibs only used from LocalConfiguration
FIX: (#6967) remove a deprecated method
CHANGE: (#6967) gLogger date format
NEW: (#6967) type hinting in gLogger
*Interfaces
FIX: (#6961) Removed dinit, sessions are now instantiated automatically. Also removed DCOMMANDS_PPID.
PR for adding Dirac side code related to https://github.com/DIRACGrid/WebAppDIRAC/pull/729/commits
CHANGE: (#6956) Add getTokensByUserID method and make it compliant with ProxyManager code
*Integration tests
FIX: (#6954) Make error nicer when the IAM docker instance is not working properly
*dashboards
NEW: (#6911) added json export of Grafana demo dashboard
[v8.1.0a12]
*WorkloadManagementSystem
FIX: (#6953) Watchdog fix: always report correct types
FIX: (#6931) make singletons of VirtualMachineDB and PilotAgentsDB
PR for fixing dead link in doc (monitoring part)
FIX: (#6949) change urls for WMS and Logs dashboards
*tests
CHANGE: (#6948) Integration tests: upped Opensearch version to 2.1.0
NEW: (#6937) integration tests for IdProviders
*Core
CHANGE: (#6947) remove support for MySQL 5.7
FIX: (#6944) protection for opensearch_dsl being merged in opensearchpy
FIX: (#6924) cast method names to string before sending monitoring
*Resources
NEW: (#6937) unit tests for IdProviders
FIX: (#6927) HTCondorCEComputingElement - set _CONDOR_AUTH_SSL_CLIENT_CADIR before executing condor commands
*Tornado
FIX: (#6933) do not use activityMonitor in TornadoREST
*FrameworkSystem
FIX: (#6922) stdoutJson logger prints all the necessary fields
FIX: (#6922) MicrosecondJsonFormatter does not dump DIRAC specific fields (`timeStampIsShown`, `colorIsShown`, etc)
[v8.1.0a11]
*Resources
FIX: (#6913) better handle exceptions in AREXCE
CHANGE: (#6902) (AREX)ComputingElement - use proxy only if it is set up
FIX: (#6883) add workaround for https timeout when dowloading file
*WorkloadManagementSystem
FIX: (#6913) config helper error message
FIX: (#6907) SiteDirector: failure in submitting Accounting or Monitoring report only prints an error
*MonitoringSystem
FIX: (#6904) while updating ES documents, wait and retry on ConflictError
CHANGE: (#6875) introduce gMonitoringDB global instance
*Interfaces
FIX: (#6903) make -F (file) option work for drm command. This fixes #6896
*WorkloadManagement
NEW: (#6902) Enable PilotManager to manage pilots with tokens
*Core
FIX: (#6901) Tornado PeriodicCallbacks are actually called
FEAT: (#6901) Tornado Handlers report to ActivityMonitor
FEAT: (#6901) Activity Monitoring reports useful information
FIX: (#6874) TornadoREST - fix the error in resolution of exposed methods
Thank you for writing the text to appear in the release notes. It will show up
exactly as it appears between the two bold lines
Please follow the template:
*Production System
CHANGE: (#6898) change column types in ProductionDB from VARCHAR(255) to LONGTEXT
*FrameworkSystem
CHANGE: (#6897) Default log level is INFO
NEW: (#6897) Define global configuration options for log level
*ConfigurationSystem
FIX: (#6888) Fix the connection to ES database when using certificates
*DataManagementSystem
CHANGE: (#6884) refactor (DFC): rewrite the file/replica deletion stored procedure as they were in 5.6
CHANGE: (#6880) FTS3DB uses `in` instead of `not in` for some query
CHANGE: (#6880) FTS3DB removal of operations uses python datetime instead of mysql builtin
[v8.1.0a11]
*Resources
FIX: (#6913) better handle exceptions in AREXCE
CHANGE: (#6902) (AREX)ComputingElement - use proxy only if it is set up
FIX: (#6883) add workaround for https timeout when dowloading file
*WorkloadManagementSystem
FIX: (#6913) config helper error message
FIX: (#6907) SiteDirector: failure in submitting Accounting or Monitoring report only prints an error
*MonitoringSystem
FIX: (#6904) while updating ES documents, wait and retry on ConflictError
CHANGE: (#6875) introduce gMonitoringDB global instance
*Interfaces
FIX: (#6903) make -F (file) option work for drm command. This fixes #6896
*WorkloadManagement
NEW: (#6902) Enable PilotManager to manage pilots with tokens
*Core
FIX: (#6901) Tornado PeriodicCallbacks are actually called
FEAT: (#6901) Tornado Handlers report to ActivityMonitor
FEAT: (#6901) Activity Monitoring reports useful information
FIX: (#6874) TornadoREST - fix the error in resolution of exposed methods
Thank you for writing the text to appear in the release notes. It will show up
exactly as it appears between the two bold lines
Please follow the template:
*Production System
CHANGE: (#6898) change column types in ProductionDB from VARCHAR(255) to LONGTEXT
*FrameworkSystem
CHANGE: (#6897) Default log level is INFO
NEW: (#6897) Define global configuration options for log level
*ConfigurationSystem
FIX: (#6888) Fix the connection to ES database when using certificates
*DataManagementSystem
CHANGE: (#6884) refactor (DFC): rewrite the file/replica deletion stored procedure as they were in 5.6
CHANGE: (#6880) FTS3DB uses `in` instead of `not in` for some query
CHANGE: (#6880) FTS3DB removal of operations uses python datetime instead of mysql builtin