Skip to content

LST: set T5 occupancy threshold at 100K#51333

Open
slava77 wants to merge 1 commit into
cms-sw:masterfrom
slava77:CMSSW_20_1_X-lst019-a-T5-occupancy
Open

LST: set T5 occupancy threshold at 100K#51333
slava77 wants to merge 1 commit into
cms-sw:masterfrom
slava77:CMSSW_20_1_X-lst019-a-T5-occupancy

Conversation

@slava77

@slava77 slava77 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

This is an interim measure to address the very extreme tail event(s) such as seen the phase-2 170pre2 relval QCD 2TeV jet events in #51245 (comment)

a somewhat rounded upper limit of 100K is placed on the maximum number of T5s (per module [accounting unit]).

  • Not seen in 200 ttbar PU200 events (the max here is close to/below 50K).
  • happens in about 3% of QCD 2 TeV dijet events (200 events initially made from 16_1_X relval)

The slow event seen in the relval (tarball used to reproduce)

  • original pre2: was killed after it took over 44 hours in the production job. I reproduced it locally (the event didn't complete as of this message; more than 24 hours since the start; running on CPU).
  • After the fix that long event completes in about 40 mins on CPU backend. Reducing the T5 cap to 50K reduces the processing time to just about 30 mins; to avoid reproducibility issues due to going over the cap in ttbar the cap is kept at 100K.

Tests on 1K events show minimal differences in physics (tracking running on GPU, some non-reproducibility expected)

  • ttbar events image
  • QCD 2TeV events image

(all HLT MTV plots for QCD and ttbar)

@cmsbuild

cmsbuild commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

cms-bot internal usage

@cmsbuild

Copy link
Copy Markdown
Contributor

@cmsbuild

Copy link
Copy Markdown
Contributor

A new Pull Request was created by @slava77 for master.

It involves the following packages:

  • RecoTracker/LSTCore (reconstruction)

@Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @VinInn, @VourMa, @dgulhan, @elusian, @felicepantaleo, @gpetruc, @mmasciov, @mmusich, @mtosi, @rovere this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@rovere

rovere commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

@slava77 thanks for looking into this in such a short time.
Just for my education: the 40min processing time is under which conditions? CPU or GPU?

@mmusich

mmusich commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

test parameters:

  • enable_tests = gpu, hlt_p2_timing
  • workflows_gpu = 34434.7503
  • workflows = ph2_hlt
  • relvals_opt = -w upgrade,standard
  • relvals_opt_gpu = -w upgrade,standard

@mmusich

mmusich commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

@cmsbuild, please test

@slava77

slava77 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Just for my education: the 40min processing time is under which conditions? CPU or GPU?

CPU; I updated the PR description

@mmusich

mmusich commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

the timing tests failures look related to some kind of glitch the bot

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-28ab2e/54318/hlt-p2-timing.log

@cmsbuild

Copy link
Copy Markdown
Contributor

-1

Failed Tests: HLTP2Timing RelVals-AMD_MI300X
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-28ab2e/54318/summary.html
COMMIT: 1894345
CMSSW: CMSSW_20_1_X_2026-06-26-1100/el9_amd64_gcc13
Additional Tests: GPU,HLT_P2_TIMING,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51333/54318/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_MI300X

  • 34434.75134434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka.log

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 61
  • DQMHistoTests: Total histograms compared: 4106100
  • DQMHistoTests: Total failures: 0
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 4106082
  • DQMHistoTests: Total skipped: 18
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 60 files compared)
  • Checked 257 log files, 213 edm output root files, 61 DQM output files
  • TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 144 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 19396
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 147611
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 1 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 80 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 17467
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 149540
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 2 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 76 differences found in the comparisons
  • DQMHistoTests: Total files compared: 7
  • DQMHistoTests: Total histograms compared: 167007
  • DQMHistoTests: Total failures: 15084
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 151923
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
  • Checked 25 log files, 20 edm output root files, 7 DQM output files
  • TriggerResults: found differences in 2 / 6 workflows

Max Memory Comparisons exceeding threshold NVIDIA_H100

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...
  • Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff -237.3 exceeds +/- 30.0 MiB

@slava77

slava77 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

34434.75134434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka.log

with A fatal system signal has occurred: external termination request ; not sure if the stack trace with CLUEAlgoAlpaka is suggestive of anything. This shouldn't be related to this PR though.

@dan131riley

Copy link
Copy Markdown
Contributor

34434.75134434.751_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121_HLT75e33TimingAlpaka.log

with A fatal system signal has occurred: external termination request ; not sure if the stack trace with CLUEAlgoAlpaka is suggestive of anything. This shouldn't be related to this PR though.

For the sake of completeness of the record, the relevant bits of the stack trace are below. Threads 6 and 7 are in hsaKmtWaitOnMultipleEvents_ExtCtx waiting on device ioctl operations. Thread 2 in edm::impl::WaitingThread::threadLoop is in hip::hipEventSynchronize waiting on a lock in hip::MemoryPool::ReleaseFreedMemory. Thread 1 is in HGCalSoARecHitsLayerClustersProducer waiting on a lock allocating a memory buffer in makeClustersCMSSW. Threads 1 and 2 are likely waiting on the same lock, the mystery is who is holding the lock? Threads 6 and 7 should not be holding a memory allocation lock, so it appears the lock has been lost.

Sitting on a lock in Device::NullStream (on thread 1) suggests that something has gone wrong very early in the rocm/hip device stream setup.

Edited stack trace:

Begin processing the 1st record. Run 1, Event 1, LumiSection 1 on stream 0 at 26-Jun-2026 21:20:35.227 CEST

A fatal system signal has occurred: external termination request
The following is the call stack containing the origin of the signal.

Fri Jun 26 23:49:49 CEST 2026

Thread 7 (Thread 0x146664000640 (LWP 1361278) "cmsRun"):
#0  0x000014669bcfae3b in ioctl () from /lib64/libc.so.6
#1  0x00001466843322b0 in hsakmt_ioctl () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#2  0x0000146684327f53 in hsaKmtWaitOnMultipleEvents_ExtCtx () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#3  0x0000146684286e7b in rocr::core::Runtime::AsyncEventsLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#4  0x00001466842e34dd in rocr::os::ThreadTrampoline(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#5  0x000014669bc813f9 in start_thread () from /lib64/libc.so.6
#6  0x000014669bd065d0 in clone3 () from /lib64/libc.so.6

Thread 6 (Thread 0x146663a00640 (LWP 1361279) "cmsRun"):
#0  0x000014669bcfae3b in ioctl () from /lib64/libc.so.6
#1  0x00001466843322b0 in hsakmt_ioctl () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#2  0x0000146684327f53 in hsaKmtWaitOnMultipleEvents_ExtCtx () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#3  0x00001466842ab999 in rocr::core::Signal::WaitAnyExceptions(unsigned int, hsa_signal_s const*, hsa_signal_condition_t const*, long const*, long*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#4  0x0000146684287248 in rocr::core::Runtime::AsyncEventsLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#5  0x00001466842e34dd in rocr::os::ThreadTrampoline(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libhsa-runtime64.so.1
#6  0x000014669bc813f9 in start_thread () from /lib64/libc.so.6
#7  0x000014669bd065d0 in clone3 () from /lib64/libc.so.6

Thread 2 (Thread 0x145d3ec00640 (LWP 1361297) "edm async pool"):
#0  0x000014669bc7e5a0 in __lll_lock_wait () from /lib64/libc.so.6
#1  0x000014669bc8491d in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#2  0x000014668e46979a in hip::MemoryPool::ReleaseFreedMemory() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#3  0x000014668e283359 in hip::Device::ReleaseFreedMemory() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#4  0x000014668e296cdd in hip::hipEventSynchronize(ihipEvent_t*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#5  0x0000145e25ca45e6 in std::_Function_handler<void (), edm::impl::WaitingThread::run<alpaka_rocm_async::detail::EDMetadataAcquireSentry::asyncWait()::{lambda()#1}, alpaka_rocm_async::detail::EDMetadataAcquireSentry::asyncWait()::{lambda()#2}>(edm::WaitingTaskWithArenaHolder, alpaka_rocm_async::detail::EDMetadataAcquireSentry::asyncWait()::{lambda()#1}&&, alpaka_rocm_async::detail::EDMetadataAcquireSentry::asyncWait()::{lambda()#2}&&, std::shared_ptr<edm::impl::WaitingThread>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/libHeterogeneousCoreAlpakaCoreROCmAsync.so
#6  0x000014669c92d333 in edm::impl::WaitingThread::threadLoop() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/libFWCoreConcurrency.so
#7  0x000014669bee1c94 in std::execute_native_thread_routine (__p=0x145d4ab80ce0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:104
#8  0x000014669bc813f9 in start_thread () from /lib64/libc.so.6
#9  0x000014669bd065d0 in clone3 () from /lib64/libc.so.6

Thread 1 (Thread 0x14669b8c6300 (LWP 1361237) "cmsRun"):
#0  0x000014669bcf92cf in poll () from /lib64/libc.so.6
#1  0x0000146692f48c5e in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#2  0x0000146692f48e63 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014669bc7e59e in __lll_lock_wait () from /lib64/libc.so.6
#5  0x000014669bc8491d in pthread_mutex_lock@@GLIBC_2.2.5 () from /lib64/libc.so.6
#6  0x000014668e28d17a in hip::Device::NullStream(bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#7  0x000014668e2925a8 in std::_Function_handler<amd::HostQueue& (), hip::MemoryPool::MemoryPool(hip::Device*, hipMemPoolProps const*, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#8  0x000014668e5d2661 in amd::VmHeap::CommitMemory(void*, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#9  0x000014668e5d28f9 in amd::VmHeap::MapPhysMemory(unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#10 0x000014668e5d2acf in amd::VmHeap::AllocBlock(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#11 0x000014668e5d3661 in amd::VmHeap::Alloc(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#12 0x000014668e5d3918 in amd::VmHeapArray::Alloc(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#13 0x000014668e46bf59 in hip::MemoryPool::AllocateMemory(unsigned long, hip::Stream*, void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#14 0x000014668e456b75 in hip::hipMallocAsync(void**, unsigned long, ihipStream_t*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw-patch/CMSSW_20_1_X_2026-06-26-1100/external/el9_amd64_gcc13/lib/libamdhip64.so.7
#15 0x0000145e0fe93f04 in alpaka::BufUniformCudaHipRt<alpaka::ApiHipRt, TilesAlpaka<alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1ul>, unsigned int>, HGCalSiliconTilesConstants>, std::integral_constant<unsigned long, 1ul>, unsigned int> alpaka::trait::AsyncBufAlloc<TilesAlpaka<alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1ul>, unsigned int>, HGCalSiliconTilesConstants>, std::integral_constant<unsigned long, 1ul>, unsigned int, alpaka::DevUniformCudaHipRt<alpaka::ApiHipRt>, void>::allocAsyncBuf<alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false> >(alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false>, alpaka::Vec<std::integral_constant<unsigned long, 1ul>, unsigned int> const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#16 0x0000145e0fe93bca in auto alpaka::allocAsyncBuf<TilesAlpaka<alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1ul>, unsigned int>, HGCalSiliconTilesConstants>, unsigned int, alpaka::Vec<std::integral_constant<unsigned long, 1ul>, unsigned int>, alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false> >(alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false>, alpaka::Vec<std::integral_constant<unsigned long, 1ul>, unsigned int> const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#17 0x0000145e0fe918ae in CLUEAlgoAlpaka<alpaka::AccGpuUniformCudaHipRt<alpaka::ApiHipRt, std::integral_constant<unsigned long, 1ul>, unsigned int>, alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false>, HGCalSiliconTilesConstants, 96>::makeClustersCMSSW(unsigned int, float const*, float const*, int const*, float const*, float const*, unsigned int const*, float*, float*, unsigned int*, int*, unsigned char*, unsigned int*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#18 0x0000145e0fe9149b in alpaka_rocm_async::HGCalLayerClustersAlgoWrapper::run(alpaka::uniform_cuda_hip::detail::QueueUniformCudaHipRt<alpaka::ApiHipRt, false>&, unsigned int, float, float, float, HGCalSoARecHitsLayout<128ul, false>::ConstViewTemplateFreeParams<128ul, false, true, true>, HGCalSoARecHitsExtraLayout<128ul, false>::ViewTemplateFreeParams<128ul, false, true, true>) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#19 0x0000145e0fec6f96 in alpaka_rocm_async::HGCalSoARecHitsLayerClustersProducer::produce(alpaka_rocm_async::device::Event&, alpaka_rocm_async::device::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#20 0x0000145e0fec4fd7 in alpaka_rocm_async::stream::EDProducer<>::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/pluginRecoLocalCaloHGCalRecProducersPluginsPortableROCmAsync.so
#21 0x000014669cc61ab2 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02947/el9_amd64_gcc13/cms/cmssw/CMSSW_20_1_X_2026-06-24-1100/lib/el9_amd64_gcc13/libFWCoreFramework.so

Current Modules:

Module: HGCalSoARecHitsLayerClustersProducer@alpaka:hltHgcalSoARecHitsLayerClustersProducer (crashed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants