Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao · 2023-06-12T15:51:27Z

PR description:

This PR introduces a new type of DQM MonitorElement, TH2Poly, for HGCal DQM in the future. This feature allows a display of polygonal histograms on the CMS DQM GUI. As a demonstration, a wafer map can be displayed like the screenshot here [1].

TH2Poly is a 2D histogram class inherited from TH2. Polygonal bins, defined by TGraph, can be loaded using the AddBin() method. After setting up the polygonal bins, a TH2Poly object can store information through Fill() or SetBinContent().

A workflow for creating polygonal histograms looks like this:
DQM Service -> DQM EDAnalyzer -> CMS DQM GUI

An implementation of TH2Poly in DQM Service and MonitorElement is necessary to display the polygonal histograms. It involves updates on two repositories: cmssw and dqmgui_prod. A pull request is created in the dqmgui_prod repository [2] with a relevant issue reported in this link [3], which is about setting up a CMS DQM GUI with the new feature.

PR validation:

The workflow and the implementation have been tested: (a) From this feature branch, monitor elements of TH2Poly can be stored in a DQM root file [4]. (b) The DQM root file can be uploaded to a CMS DQM GUI, which is built following the steps noted in this issue [3]. Polygonal maps can be displayed on the DQM GUI, as demonstrated in [1].

[1] https://ykao.web.cern.ch/ykao/raw_data_handling/hgcal_dqm_gui/screenshot_demo_th2poly_wafermap.png
[2] cms-DQM/dqmgui_prod#14
[3] cms-DQM/dqmgui_prod#13
[4] A DQM root file containing demo polygonal maps: /afs/cern.ch/work/y/ykao/public/example_HGCAL_DQM/DQM_V0001_HGCAL_R000123469.root

cmsbuild · 2023-06-12T15:59:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35885

This PR adds an extra 44KB to repository

cmsbuild · 2023-06-12T15:59:32Z

A new Pull Request was created by @ywkao for master.

It involves the following packages:

DQMServices/Core (dqm)
DataFormats/Histograms (dqm, core)

@smuzaffar, @Dr15Jones, @makortel, @nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@missirol, @barvic, @rovere this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

makortel

I think in addition the following places need to be amended accordingly

cmssw/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc

Lines 250 to 253 in e18c96d

    
           static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) { 
        
             switch (iTypeIndex) { 
        
               case kIntIndex: 
        
                 return new IntTreeHelper(iTree, iFullNameBufferPtr);

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Lines 465 to 468 in e18c96d

    
             m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope)); 
        
             m_treeReaders[kTProfile2DIndex].reset( 
        
                 new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope)); 
        
           }

How about adding some tests to make sure all the I/O components work properly? (I don't remember if there are already any such tests that could be easily amended)

makortel · 2023-06-12T19:41:55Z

DQMServices/Core/interface/DQMStore.h

+                                 double lowY,
+                                 double highY,
+                                 FUNC onbooking = NOOP()) {
+        return bookME(name, MonitorElementData::Kind::TH2F, [=]() {


Shouldn't this be

Suggested change

return bookME(name, MonitorElementData::Kind::TH2F, [=]() {

return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

?

Indeed, this should be corrected. I will update it in a commit accordingly.

makortel · 2023-06-12T19:43:04Z

DQMServices/Core/src/MonitorElement.cc

+  /// set polygon bin (TH2Poly)
+  void MonitorElement::addBin(TGraph *graph) {
+    auto access = this->accessMut();
+    static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph);


What if the ROOT object is not TH2Poly? The other similar functions throw an exception via the incompatible() function.

Indeed, incompatible() function is necessary. This block of code will be modified as follows:

- static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + if (kind() == Kind::TH2Poly) { + static_cast<TH2Poly *>(accessRootObject(access, __PRETTY_FUNCTION__, 2))->AddBin(graph); + } else { + incompatible(__PRETTY_FUNCTION__); + }

makortel · 2023-06-12T19:44:31Z

DataFormats/Histograms/interface/MonitorElementCollection.h

@@ -140,7 +140,8 @@ struct MonitorElementData {
    TH2I = 0x23,
    TH3F = 0x30,
    TPROFILE = 0x40,
-    TPROFILE2D = 0x41
+    TPROFILE2D = 0x41,
+    TH2Poly = 0x60


Why such a large gap between the previous value (0x41)?

I guess I kept thinking of hexagons for a wafer map and then put a number starting with six. We can assign 0x24 for TH2Poly if it is feasible.

+ TH2Poly = 0x24, TH3F = 0x30, TPROFILE = 0x40, - TPROFILE2D = 0x41, - TH2Poly = 0x60 + TPROFILE2D = 0x41

emanueleusai · 2023-06-13T03:13:04Z

type hgcal

emanueleusai · 2023-06-13T03:13:13Z

please test

cmsbuild · 2023-06-13T07:05:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/33112/summary.html
COMMIT: 5fa5e4e
CMSSW: CMSSW_13_2_X_2023-06-12-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41932/33112/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 15 lines to the logs
Reco comparison results: 59 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3190910
DQMHistoTests: Total failures: 2477
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3188411
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-06-13T16:49:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/35914

This PR adds an extra 40KB to repository

cmsbuild · 2023-06-13T16:50:06Z

Pull request #41932 was updated. @smuzaffar, @Dr15Jones, @makortel, @nothingface0, @emanueleusai, @cmsbuild, @pmandrik, @syuvivida, @tjavaid, @micsucmed, @rvenditti can you please check and sign again.

ywkao · 2023-06-13T17:15:37Z

I think in addition the following places need to be amended accordingly

cmssw/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc

Lines 250 to 253 in e18c96d

static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) {

switch (iTypeIndex) {

case kIntIndex:

return new IntTreeHelper(iTree, iFullNameBufferPtr);

cmssw/DQMServices/FwkIO/plugins/DQMRootSource.cc

Lines 465 to 468 in e18c96d

m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope));

m_treeReaders[kTProfile2DIndex].reset(

new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope));

}

How about adding some tests to make sure all the I/O components work properly? (I don't remember if there are already any such tests that could be easily amended)

Okay, I will modify the lines and test it. Would PR #37665 be a good reference for the implementation of TH2Poly?

Also, I would appreciate any instructions from experts on how to test all the I/O components.

makortel · 2023-06-13T20:54:56Z

Would PR #37665 be a good reference for the implementation of TH2Poly?

That PR indeeds looks like a good example.

cmsbuild · 2025-01-23T09:24:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43405

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/Components/plugins/DQMStoreStats.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/Core/src/DQMNet.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/FwkIO/plugins/DQMRootOutputModule.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372
- File DQMServices/FwkIO/plugins/DQMRootSource.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372

cmsbuild · 2025-01-23T09:25:14Z

Pull request #41932 was updated. @Dr15Jones, @antoniovagnerini, @cmsbuild, @makortel, @rseidita, @smuzaffar can you please check and sign again.

makortel · 2025-01-23T14:45:25Z

@cmsbuild, please test

cmsbuild · 2025-01-23T16:58:54Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/43931/summary.html
COMMIT: c0af007
CMSSW: CMSSW_15_0_X_2025-01-23-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41932/43931/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3819085
DQMHistoTests: Total failures: 112
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3818953
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

- Add proper dimension checks based on histogram type (TH1/TH2/TH3) - Match dimension handling pattern used in MonitorElement::Reset() method - Fix FastTimerService fatal error related to incorrect dimension access

cmsbuild · 2025-01-29T05:12:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41932/43462

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/Components/plugins/DQMStoreStats.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/Components/plugins/MEtoEDMConverter.cc modified in PR(s): Rename BranchDescription to ProductDescription #47190
- File DQMServices/Core/src/DQMNet.cc modified in PR(s): [LLVM Analyzer][DQM] Fix dead assignments #46247
- File DQMServices/FwkIO/plugins/DQMRootOutputModule.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372, Rename BranchDescription to ProductDescription #47190
- File DQMServices/FwkIO/plugins/DQMRootSource.cc modified in PR(s): [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372

cmsbuild · 2025-01-29T05:12:50Z

Pull request #41932 was updated. @Dr15Jones, @antoniovagnerini, @cmsbuild, @makortel, @rseidita, @smuzaffar can you please check and sign again.

makortel · 2025-01-30T18:38:24Z

@cmsbuild, please test

cmsbuild · 2025-01-30T21:08:52Z

+1

Size: This PR adds an extra 16KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-03d614/44082/summary.html
COMMIT: 7755770
CMSSW: CMSSW_15_0_X_2025-01-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41932/44082/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 4016770
DQMHistoTests: Total failures: 71
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4016679
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 218 log files, 189 edm output root files, 50 DQM output files
TriggerResults: no differences found

cmsbuild added this to the CMSSW_13_2_X milestone Jun 12, 2023

cmsbuild added dqm-pending core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jun 12, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2023

makortel reviewed Jun 12, 2023

View reviewed changes

cmsbuild added tests-started hgcal and removed tests-pending labels Jun 13, 2023

cmsbuild added tests-approved tests-pending code-checks-pending and removed tests-started tests-approved code-checks-approved labels Jun 13, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 13, 2023

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Jan 23, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 23, 2025

cmsbuild added tests-started and removed tests-pending labels Jan 23, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 23, 2025

cmsbuild mentioned this pull request Jan 27, 2025

Rename BranchDescription to ProductDescription #47190

Merged

Fix MonitorElement::getNcells() dimension handling

7755770

- Add proper dimension checks based on histogram type (TH1/TH2/TH3) - Match dimension handling pattern used in MonitorElement::Reset() method - Fix FastTimerService fatal error related to incorrect dimension access

cmsbuild added tests-pending code-checks-pending and removed tests-approved code-checks-approved labels Jan 29, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 29, 2025

cmsbuild added tests-started and removed tests-pending labels Jan 30, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement TH2Poly in DQM Services for HGCal DQM #41932

Implement TH2Poly in DQM Services for HGCal DQM #41932

ywkao commented Jun 12, 2023

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 •

edited

Loading

makortel left a comment

makortel Jun 12, 2023

ywkao Jun 13, 2023

makortel Jun 12, 2023

ywkao Jun 13, 2023 •

edited

Loading

makortel Jun 12, 2023

ywkao Jun 13, 2023

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

ywkao commented Jun 13, 2023

makortel commented Jun 13, 2023

cmsbuild commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

makortel commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

cmsbuild commented Jan 29, 2025

cmsbuild commented Jan 29, 2025

makortel commented Jan 30, 2025

cmsbuild commented Jan 30, 2025

	static TreeHelperBase* makeHelper(unsigned int iTypeIndex, TTree* iTree, std::string* iFullNameBufferPtr) {
	switch (iTypeIndex) {
	case kIntIndex:
	return new IntTreeHelper(iTree, iFullNameBufferPtr);

	m_treeReaders[kTProfileIndex].reset(new TreeObjectReader<TProfile>(MonitorElementData::Kind::TPROFILE, m_rescope));
	m_treeReaders[kTProfile2DIndex].reset(
	new TreeObjectReader<TProfile2D>(MonitorElementData::Kind::TPROFILE2D, m_rescope));
	}

	return bookME(name, MonitorElementData::Kind::TH2F, [=]() {
	return bookME(name, MonitorElementData::Kind::TH2Poly, [=]() {

Implement TH2Poly in DQM Services for HGCal DQM #41932

Are you sure you want to change the base?

Implement TH2Poly in DQM Services for HGCal DQM #41932

Conversation

ywkao commented Jun 12, 2023

PR description:

PR validation:

cmsbuild commented Jun 12, 2023

cmsbuild commented Jun 12, 2023 • edited Loading

makortel left a comment

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

makortel Jun 12, 2023

Choose a reason for hiding this comment

ywkao Jun 13, 2023

Choose a reason for hiding this comment

emanueleusai commented Jun 13, 2023

emanueleusai commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

Comparison Summary

cmsbuild commented Jun 13, 2023

cmsbuild commented Jun 13, 2023

ywkao commented Jun 13, 2023

makortel commented Jun 13, 2023

cmsbuild commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

makortel commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

Comparison Summary

cmsbuild commented Jan 29, 2025

cmsbuild commented Jan 29, 2025

makortel commented Jan 30, 2025

cmsbuild commented Jan 30, 2025

Comparison Summary

cmsbuild commented Jun 12, 2023 •

edited

Loading

ywkao Jun 13, 2023 •

edited

Loading