Modules not auto-generating a cfi file in cfi-python currently present in the HLT menu (Run 3 and Phase 2)

[mmusich]

Disclaimer: this issue is mostly for documentation purposes.
TSG has started to investigate possible replacements of the current Configuration DB model for storing HLT menus in view of the Phase 2 upgrade.
One of the possibilities that was discussed, is to fetch the list of configurable parameters uniquely from the auto-generated configuration fragment files that the fillDescriptions mechanism provides in cmssw.
At the moment, this is unfortunately not possible as not all the plugins used at HLT have a viable fillDescriptions implementation that is both :

1. providing default values for all the top level parameters.
2. generating a cfi file in the cfi-python folder (to be parsed instead of other cfi files in the python directory), see addDefault vs addWithDefaultLabel.

In the recent past I have opened a series of PRs to bring the list of outstanding modules down: #47017, #47045, #47079, #47107, #47136, #47191 and #47224, but few remain to be addressed, especially those that either:

• have complicated class inheritance structure (e.g. using algorithm factories)
• take branching decisions (ifValues and similar) to populate the defaults of certain parameters depending on the value of others (see also discussion at add fillDescriptions to several plugins used at HLT (4/N) #47107 (comment) and following).

As of any IB more recent that CMSSW_15_0_X_2025-02-05-1100, the following script can be used to identify such modules in the Run 3 menu:

• BTagProbabilityToDiscriminator
• CSCChannelMapperESProducer
• CSCIndexerESProducer
• CaloTopologyBuilder
• CaloTowersCreator
• CandIPProducer
• CandSecondaryVertexProducer
• DTRecHitProducer
• DTRecSegment4DProducer
• DetIdAssociatorESProducer
• ESRecHitProducer
• EcalSeverityLevelESProducer
• EmptyESSource
• GsfMaterialEffectsESProducer
• HcalChannelPropertiesEP
• HcalTimeSlewEP
• JetTagProducer
• MuonIdProducer
• PropagatorWithMaterialESProducer
• RPCRecHitProducer
• SecondaryVertexProducer
• SeedGeneratorFromRegionHitsEDProducer
• SiPixelTemplateDBObjectESProducer
• SiStripBackPlaneCorrectionDepESProducer
• SiStripLorentzAngleDepESProducer
• SoftLepton
• StripCPEESProducer
• TSGFromL2Muon
• TrackIPProducer
• TrajectoryCleanerESProducer

While the following script can be used to identify such modules in the Phase 2 menu:

• CSCChannelMapperESProducer
• CSCIndexerESProducer
• CaloTowersCreator
• CandIPProducer
• CandSecondaryVertexProducer
• DTRecHitProducer
• DTRecSegment4DProducer
• DetIdAssociatorESProducer
• EcalSeverityLevelESProducer
• EmptyESSource
• GsfMaterialEffectsESProducer
• HBHEPhase1Reconstructor
• HGCalRecHitProducer
• HcalChannelPropertiesEP
• MuonIdProducer
• PFCandidateFwdPtrProducer
• PropagatorWithMaterialESProducer
• RPCRecHitProducer
• TrajectoryCleanerESProducer

I would appreciate if there was some help from contacts from the reconstruction group to address (or discuss) the remaining ones (especially the ones associated to the Phase 2 menu, which is our ultimate goal).

Cc:
@cms-sw/hlt-l2 @fwyzard @Sam-Harper

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @mmusich.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

So the new way fillDescriptions works is it generates a file in cfipython/*/<SubSystem>/<Package>/ directory named <ModuleType>.py which contains a function named which can be used to generate an instance of the proper cms. type for the module which declares the parameter names/types for all the parameters.

If a addDefaultWithLabel is used, then fillDescriptions also writes a _cfi.py file which loads the .py python package and uses the new function to generate the class instance.

Therefore it is not necessary to add addDefault or addDefaultWithLabel to all fillDescriptions to do what you want. Instead, you can use the .py files.

[mmusich]

Instead, you can use the .py files.

(Maybe) fine. Though several modules in the list above are still lacking any configuration validation, which I guess it means there are no parameters in such .py files.

[fwyzard]

So the new way fillDescriptions works is it generates a file in cfipython/*/<SubSystem>/<Package>/ directory named <ModuleType>.py which contains a function named which can be used to generate an instance of the proper cms. type for the module which declares the parameter names/types for all the parameters.

just curious, what happens if a module has multiple possible configurations ?

[Dr15Jones]

just curious, what happens if a module has multiple possible configurations ?

The different _cfi.py files still import the package but then only change the values of parameters which differ from the 'default'.

E.g.

     from Foo.Bar.modules import FiiProducer

     fii = FiiProducer( value = 3.2)

[makortel]

assign hlt,alca,geometry,reconstruction,upgrade

[cmsbuild]

New categories assigned: hlt,alca,geometry,reconstruction,upgrade

@atpathak,@bsunanda,@civanch,@consuegs,@Dr15Jones,@jfernan2,@kpedro88,@makortel,@mandrenguyen,@Martin-Grunewald,@mdhildreth,@mmusich,@Moanwar,@perrotta,@srimanob,@subirsarkar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

EmptyESSource

EmptyESSource has fillDescriptions() that calls addDefault(), and thus the EmptyESSource.py file exists and can be used as described above.

[mmusich]

and thus the EmptyESSource.py file exists and can be used as described above.

Out of curiosity, I fail to see why having this class is a superior approach than the plain regular cfi-based PSet (which would be useful to have in release by the way). Can someone clarify? @rovere FYI

[fwyzard]

IMHO the new syntax is a little bit nicer.

With a full configuration:

myProducer = cms.Producer("MyProducer",
  source = cms.InputTag("myData")
}

With a cfi-based approach:

from Subsystem.Package.myProducer_cfi import myProducer as _myProducer
myProducer = _myProducer.clone(
  source = "myData"
)

With the new approach:

from Subsystem.Package.MyProducer import MyProducer
myProducer = MyProducer(
  source = "myData"
)

[Dr15Jones]

• The new syntax does not require one to specify a module label. This is very handy for modules like the PoolOutputModule.
• One can declare that a value be given for a parameter without being forced to give a default. If the parameter is missing in the configuration the code will fail in the python instead of having to wait until it is parsed by the C++. Again, handy for PoolOutputModule.
• One can declare that some values are optional. Then one can set the value without having to explicitly set the cms. type or whether it is tracked or untracked.

[fwyzard]

• The new syntax does not require one to specify a module label.

How does that work ?

from IOPool.Output.PoolOutputModule import *
process.??? = PoolOutputModule(
   outputCommands = 'keep *'
)

[Dr15Jones]

How does that work ?

I meant in the package IOPool.Output.PoolOutputModule (I actually prefer using the IOPool.Output.modules package myself) one does not need to specify a module label. This is in contrast to a _cfi.py file where a module label (i.e. a named variable) is required.

When using IOPool.Output.PoolOutputModule to add a PoolOutputModule to a cms.Process, you do need to assign a module label like we have always done. In fact if someone did

process.load("IOPool.Output.PoolOutputModule")

nothing would be added to the process.

Sorry for the confusion.

[fwyzard]

I actually prefer using the IOPool.Output.modules package myself

Ah, yes, I see:

from Subsystem.Package.modules import *
myProducer = MyProducer(
  source = "myData"
)

And thanks for the clarification :-)

[mmusich]

@cms-sw/alca-l2 @cms-sw/reconstruction-l2 @cms-sw/upgrade-l2 @cms-sw/geometry-l2

can you clarify if you can help with the modules in the list at #47275 (comment) that don't fall in the category as described at #47275 (comment) (i.e. the ones that don't implement a fillDescriptions at all)?

[jfernan2]

@24LopezR @rbhattacharya04 as Muon Reco contacts, could you take care of the Muon related modules, please? Thanks

[jfernan2]

@abdoulline @igv4321 as HCAL Reco contacts, could you take care of the HCAL related modules, please? Thanks

[jfernan2]

@jhakala as ECAL Reco contact, could you take care of the ECAL related modules, please? Thanks

[jfernan2]

@felicepantaleo @rovere as HGCAL Reco contacts, could you take care of the HGCAL related modules, please? Thanks

[jfernan2]

@mondalspandan @SWuchterl as BTAG conveners, could you take care of the BTAG related modules, please? Thanks

[jfernan2]

@kskovpen @mmasciov as Tracking conveners, could you take care of the Tracking related modules, please? Thanks

[jfernan2]

@jlidrych as StripTracker reco contact, could you take care of the Sistrip related modules, please? Thanks

[jhakala]

@jhakala as ECAL Reco contact, could you take care of the ECAL related modules, please? Thanks

Hi @jfernan2 , I am no longer ECAL reco contact. Please refer ECAL reco requests to @ReyerBand or @thomreis .