Problem with muon simulations from VirtualWall

Hello,

I am trying to run some muon simulations using REST v2.2.11 (master branch) with the only modification of adding Sign in · GitLab to fix virtualWall generator

I am launching 1000 events from a virtualWall and using verbose mode. Most muons leave 0 deposited energy but some do

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 161

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 162

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 163

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 164

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 165

Primary generation...
Event Deposited energy in sensitive volume:  129677
End of event 166

Primary generation...

On some attempts to run the simulation it gets randomly stuck on some event for several minutes, I’m guessing its a particularly computationally intensive event, sometimes I just relaunch it to save time.

When the simulation finishes sometimes I get a segmentation break such as:

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 987

Primary generation...
Event Deposited energy in sensitive volume:  52804.1
End of event 988

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 989

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 990

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 991

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 992

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 993

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 994

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 995

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 996

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 997

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 998

Primary generation...
Event Deposited energy in sensitive volume:  0
End of event 999

======================== run summary ======================
1000 Events simulated
===========================================================
Graphics systems deleted.
Visualization Manager deleting...
Writing geometry...
GDML: initializating variables
GDML: replacing expressions in GDML
GDML: creating temporary file
Info in <TGeoManager::Import>: Reading geometry from file: /home/lobis/gitlab/IAXOD0-REST//geometry//Argon/IAXOD0_Setup_Vac.gdml_
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is World. Master volume is World
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 1, max placements = 22
Info in <TGeoManager::CloseGeometry>: 23 nodes/ 23 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f459b985ffa in __GI___waitpid (pid=49023, stat_loc=stat_loc
entry=0x7ffd67dacc80, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00007f459b90d0ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f45ab2b6477 in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x55c925a73640) at /home/lobis/apps/ROOT/root-6.16.00/core/unix/src/TUnixSystem.cxx:2119
#3  TUnixSystem::StackTrace (this=0x55c925a73640) at /home/lobis/apps/ROOT/root-6.16.00/core/unix/src/TUnixSystem.cxx:2413
#4  0x00007f45ab2b8bc4 in TUnixSystem::DispatchSignals (this=0x55c925a73640, sig=kSigSegmentationViolation) at /home/lobis/apps/ROOT/root-6.16.00/core/unix/src/TUnixSystem.cxx:3644
#5  <signal handler called>
#6  0x000055c94815e6e8 in ?? ()
#7  0x000055c9257170bd in main ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6  0x000055c94815e6e8 in ?? ()
#7  0x000055c9257170bd in main ()
===========================================================

Other attempts results in a valid .root file but after calling restViewG4Event on the file there are no events to browse (it starts at some random event index such as 151512315) even though on the log there are some events with energy deposited in sensitive volume.

I have had this problem with the v2.2.11_dev version and though it was something related to some commit on the dev branch but I guess either there is something wrong on the master branch or (probably) I am doing something wrong with my simulation.

My RML file is the following:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<!-- Constructing XML-like REST metadata input file
     File : config.rml
     First concept author J. Galan (26-Apr-2015)
     Modified for IAXOD0 background model by G. Luzon & E. Ruiz-Chóliz (March-2016)

     Any commented text will just not end up in the configBuffer inside TRestMetadata 
     We can comment out pieces of configuration using XML commenting style.

     By default REST units are mm, keV and degrees
-->
<restG4>

    <globals>
        <variable name="EXPERIMENT" value="IAXOD0" overwrite="false"/>
        <variable name="VERSION" value="2.11" overwrite="false"/>
        <variable name="IAXOD0_PATH" value="/home/lobis/gitlab/IAXOD0-REST/" overwrite="true"/>
        <variable name="IAXOD0_DATAPATH" value="${IAXOD0_PATH}/simulations_data/cosmic_muons" overwrite="false"/>
        <variable name="IAXOD0_G4PATH" value="${IAXOD0_PATH}/restG4_simulations/restG4/" overwrite="true"/>
        <variable name="IAXOD0_GEOPATH" value="${IAXOD0_PATH}/geometry/" overwrite="true"/>
        <variable name="IAXOD0_GEODIR" value="${IAXOD0_GEOPATH}/Argon" overwrite="true"/>   <!-- options "Argon" or "Xenon" -->
        <variable name="IAXOD0_GAS" value="Argon-Isobutane 2Pct 10-10E3V/cm"
                  overwrite="true"/>  <!-- options "Argon-Isobutane 2Pct 10-10E3V/cm", "Xenon-Isobutane 2Pct 10-10E3V/cm" or any other gas in gases.rml file-->
        <!-- Default for G4 simulations (defined in .sh script)-->

        <variable name="REST_N_EVENTS" value="1" overwrite="false"/>

        <variable name="REST_STEP_IN_MM" value="1" overwrite="false"/>

        <parameter name="mainDataPath" value="${IAXOD0_DATAPATH}"/>
        <parameter name="gasDataPath" value="${REST_PATH}/data/gasFiles/"/>
        <parameter name="verboseLevel" value="info"/>
        %options are : silent, warning, info, debug
    </globals>

    <TRestRun name="Background Model" title="Simulations ${EXPERIMENT}. Version ${VERSION}.">
        <parameter name="experiment" value="${EXPERIMENT}"/>
        <parameter name="runType" value="simulation"/>
        <parameter name="runNumber" value="auto"/>
        <parameter name="runTag" value="Muons_ArIso"/>
        <parameter name="runDescription" value="We launch Muons from a virtual Wall"/>
        <parameter name="user" value="${USER}"/>
        <parameter name="verboseLevel" value="0"/>
        <parameter name="overwrite" value="off"/>
        <parameter name="outputFile" value="Run_[fRunType]_[fRunTag]_[fRunNumber]_lobis_v[VERSION].root"/>
    </TRestRun>

    <TRestG4Metadata name="restG4 Simulation run" title="Muons_ArIso">

        <parameter name="geometryPath" value="${IAXOD0_GEODIR}"/>
        <parameter name="gdml_file" value="IAXOD0_Setup_Vac.gdml"/>
        <parameter name="maxTargetStepSize" value="100" units="um"/>
        <parameter name="subEventTimeDelay" value="100" units="us"/>

        <!-- A member to store the component activity (and mass) are now present in G4Metadata -->
        <parameter name="activity" value="100"/>
        <parameter name="mass" value="1.0"/>

        <!-- The number of events to be simulated is now defined in TRestG4Metadata -->
        <parameter name="Nevents" value="${REST_N_EVENTS}"/>

        <generator type="virtualWall" position="(0,450,25)mm" size="1000mm"
                   rotation="(90,0,0)">
            <source particle="mu-">
                <energyDist type="TH1D" file="${REST_PATH}/data/distributions/Muons.root"
                            spctName="cosmicmuon"/>
                <angularDist type="TH1D" file="${REST_PATH}/data/distributions/CosmicAngles.root"
                             spctName="Theta2"/>
            </source>
        </generator>

        <storage sensitiveVolume="gas">
            <parameter name="energyRange" value="(0,5)" units="GeV"/>
            <activeVolume name="gas" chance="1"/>            <!-- Volume 0 -->
            <activeVolume name="vetoFront" chance="1"/>        <!-- Volume 1 -->
            <activeVolume name="vetoBack" chance="1"/>        <!-- Volume 2 -->
            <activeVolume name="vetoTop" chance="1"/>        <!-- Volume 3 -->
            <activeVolume name="vetoBottom" chance="1"/>        <!-- Volume 4 -->
            <activeVolume name="vetoEast" chance="1"/>        <!-- Volume 5 -->
            <activeVolume name="vetoWest" chance="1"/>        <!-- Volume 6 -->
            <!-- <activeVolume name="LeadTop" chance="1"/> -->        <!-- Volume 7 -->
            <activeVolume name="LeadShielding" chance="1"/>        <!-- Volume 8 -->
            <activeVolume name="mylarCat" chance="1"/>        <!-- Volume 9 -->
            <activeVolume name="CuCathodePattern" chance="1"/>    <!-- Volume 10 -->
            <!--<activeVolume name="PEShielding" chance="1" />	-->    <!--Added by Cristina 2/5/19-->
        </storage>

    </TRestG4Metadata>

    <TRestPhysicsLists name="default" title="First physics list implementation." verboseLevel="debug">

        <parameter name="cutForGamma" value="1" units="um"/>
        <parameter name="cutForElectron" value="1" units="mm"/>
        <parameter name="cutForPositron" value="1" units="mm"/>
        <parameter name="cutForMuon" value="0.001" units="um"/>
        <parameter name="cutForNeutron" value="1" units="mm"/>
        <parameter name="minEnergyRangeProductionCuts" value="1" units="keV"/>
        <parameter name="maxEnergyRangeProductionCuts" value="1" units="GeV"/>

        <!-- EM Physics lists -->
        <!-- <physicsList name="G4EmLivermorePhysics"> </physicsList> -->
        <!-- <physicsList name="G4EmPenelopePhysics"> </physicsList> -->
        <physicsList name="G4EmStandardPhysics_option3"></physicsList>

        <!-- Decay physics lists -->
        <physicsList name="G4DecayPhysics"></physicsList>
        <physicsList name="G4RadioactiveDecayPhysics"></physicsList>
        <physicsList name="G4RadioactiveDecay">
            <option name="ICM" value="true"/>
            <option name="ARM" value="false"/>
        </physicsList>

        <!-- Hadron physics lists -->
        <physicsList name="G4HadronElasticPhysicsHP"></physicsList>
        <physicsList name="G4IonBinaryCascadePhysics"></physicsList>
        <physicsList name="G4HadronPhysicsQGSP_BIC_HP"></physicsList>
        <physicsList name="G4NeutronTrackingCut"></physicsList>
        <physicsList name="G4EmExtraPhysics"></physicsList>

    </TRestPhysicsLists>

    <TRestGas nameref="${IAXOD0_GAS}" file="${REST_PATH}/data/definitions/gases.rml"></TRestGas>

    <!--<TRestReadout nameref="IAXOD0_Readout_v2" file="$/home/cristina/GitLab/IAXOD0-REST/G4sims/geometry/IAXOD0_Readout_v2_45deg.root"> </TRestReadout> -->

</restG4>

launched from:

# PROJECT_DIR points to the directory where this script is stored
# export PROJECT_DIR="$(dirname "$(realpath "$0")")"
PROJECT_DIR="$HOME/gitlab/IAXOD0-REST"
echo "$PROJECT_DIR"
# this variable points to the root directory where we will store the results of the simulations 
export IAXOD0_PATH="$PROJECT_DIR"

simulation_type="cosmic_muons"

export REST_N_EVENTS=300

# data will be saved to mainDataPath, we create this directory in case it doesn't exist
export mainDataPath="$IAXOD0_PATH/simulations_data/$simulation_type"

mkdir -p $mainDataPath

restG4 $PROJECT_DIR/restG4_simulations/restG4/argon/MuonsFromVirtualWall.rml

Also it is worth noting that I do a simulation using a 1keV muon from the origin of the detector (0,0,0) and it seems to work correctly and I can browse the events.

I have experienced this break before. I have never been able to identify the source of it. However, the problem sometimes disappeared when I re-compiled restG4, or launched a second time restG4. It seems not 100% reproducible. Once solved the problem, I could launch many jobs without this last minute seg.fault. It would be good to understand the source, is annoying.

Yes It has also happened to me some other times in the past, maybe its worthwhile to spend some time trying to locate the segfault with a debugger.

I have done some tests setting the vetoFront as the sensitiveVolume and I found some events that leave energy on the sensitive volume. (which previously was not happening).

However after exploring the output root file I find the 3 events that were registered in the analysis tree but I don’t find the event tree.

I crated an issue (Sign in · GitLab) with this problem.

The bug must have been introduced in a commit to v2.2.11_dev branch but I am not sure how to locate it since I don’t fully understand how information to event tree is saved.