Random crashes in raw to signal analysis

REST version : v2.3.6
REST commit : 9baf6336

Hello,
I am experiencing crashes (segmentation violation) on random CAST runs, when running the raw to signal analysis. The runs are analyzed up to ~80% before crashing. If I open the root file, there are only the Event and Analysis Tree, but all the other metadata and channel activity histograms are missing (see attachment).
I am analyzing runs from 10418 to 10455. Crashes occurs with runs R10428, R10438. The same problems occurs with REST version 2.3.5 (not clean state, commit b8d9147b) with the addition of R10422 (that, on the contrary, ends successfully with REST 3.5.6).
With REST version 2.3.5, clean state, commit f83c73da everything works.
In attachment there are the rml files used for the analysis.

run.xml (1.4 KB)
rawToSignal.rml (2.4 KB)
processes.rml (21.0 KB)
globals.xml (1.6 KB)
output_root_file

Where are found the aqs files?

I downloaded them from Sultan:
/storage/cast/SR2019/labData/Argon

I placed your files at the /tmp/ directory at sultan. Then, just modified the mainDataPath to be at /tmp and copied the readouts.root file into /tmp/ also and modified it at the corresponding RML.

Then I source v2.3.6 official source /programas/REST/stable/v2.3.6/thisREST.sh and it seems to work normally.

jgalan@sultan:/tmp$ restManager --c RawToSignal.rml --f /storage/cast/SR2019/labData/Argon/R10428_Background_23hr_Vm_340_Vd_113_Pr_1.4_Gain_0x1_Shape_0xD_Clock_0x02-000.aqs 
Warning in <UnknownClass::SetDisplay>: DISPLAY not set, setting it to 185.228.230.138:0.0
Starting pid: 14821
                            
              [==                                  Launching TRestManager...                                   ==]              
                            
              [==                                    Creating TRestManager                                     ==]              
              [==                                            path:.                                            ==]              

-- Info : thread 0: validating process chain
              TRestProcessRunner: 1 + 5 processes loaded, 1 threads prepared!              



              ====================================================================================================              
              ||                             Process : TRestRawVetoAnalysisProcess                              ||              
              ||                     Name: veto  Title: defaultTitle  VerboseLevel: warning                     ||              
              ||                        -----------------------------------------------                         ||              
              ||                                                                                                ||              
              ||                        Analysis tree observables added by this process                         ||              
              ||                        +++++++++++++++++++++++++++++++++++++++++++++++                         ||              
              ||                                    ++ veto_MaxPeakAmplitude                                    ||              
              ||                                        ++ veto_PeakTime                                        ||              
              ||                        +++++++++++++++++++++++++++++++++++++++++++++++                         ||              
              ||                                                                                                ||              
              ||                                      Veto signal ID: 4322                                      ||              
              ||                                                                                                ||              
              ====================================================================================================              
                            

              ====================================================================================================              
              ||                            Process : TRestRawSignalAnalysisProcess                             ||              
              ||                            Name: sAna  Title:   VerboseLevel: info                             ||              
              ||                        -----------------------------------------------                         ||              
              ||                                                                                                ||              
              ||                        Analysis tree observables added by this process                         ||              
              ||                        +++++++++++++++++++++++++++++++++++++++++++++++                         ||              
              ||                                 ++ sAna_AmplitudeIntegralRatio                                 ||              
              ||                                     ++ sAna_AmplitudeRatio                                     ||              
              ||                                     ++ sAna_AveragePeakTime                                    ||              
              ||                                      ++ sAna_BaseLineMean                                      ||              
              ||                                    ++ sAna_BaseLineSigmaMean                                   ||              
              ||                                     ++ sAna_EventTimeDelay                                     ||              
              ||                                      ++ sAna_FullIntegral                                      ||              
              ||                                     ++ sAna_HoursFromStart                                     ||              
              ||                                     ++ sAna_IntegralBalance                                    ||              
              ||                                    ++ sAna_MaxPeakAmplitude                                    ||              
              ||                                       ++ sAna_MaxPeakTime                                      ||              
              ||                                    ++ sAna_MaxPeakTimeDelay                                    ||              
              ||                                      ++ sAna_MeanRate_InHz                                     ||              
              ||                                      ++ sAna_MinEventValue                                     ||              
              ||                                    ++ sAna_MinPeakAmplitude                                    ||              
              ||                                       ++ sAna_MinPeakTime                                      ||              
              ||                                   ++ sAna_NumberOfGoodSignals                                  ||              
              ||                                     ++ sAna_NumberOfSignals                                    ||              
              ||                                  ++ sAna_PeakAmplitudeIntegral                                 ||              
              ||                                     ++ sAna_RateOfChangeAvg                                    ||              
              ||                                      ++ sAna_RiseSlopeAvg                                      ||              
              ||                                       ++ sAna_RiseTimeAvg                                      ||              
              ||                                   ++ sAna_SaturatedChannelID                                   ||              
              ||                                    ++ sAna_SecondsFromStart                                    ||              
              ||                                      ++ sAna_SlopeIntegral                                     ||              
              ||                                    ++ sAna_ThresholdIntegral                                   ||              
              ||                                     ++ sAna_TimeBinsLength                                     ||              
              ||                                    ++ sAna_TripleMaxIntegral                                   ||              
              ||                                      ++ sAna_baseline_map                                      ||              
              ||                                    ++ sAna_baselinesigma_map                                   ||              
              ||                                    ++ sAna_max_amplitude_map                                   ||              
              ||                                   ++ sAna_pointsoverthres_map                                  ||              
              ||                                      ++ sAna_risetime_map                                      ||              
              ||                                    ++ sAna_thr_integral_map                                    ||              
              ||                        +++++++++++++++++++++++++++++++++++++++++++++++                         ||              
              ||                                                                                                ||              
              ||                                 Baseline range : ( 20 , 200 )                                  ||              
              ||                                Integral range : ( 200 , 500 )                                  ||              
              ||                                  Point Threshold : 2.2 sigmas                                  ||              
              ||                                  Signal threshold : 1 sigmas                                   ||              
              ||                              Number of points over threshold : 7                               ||              
              ||                                                                                                ||              
              ====================================================================================================              
                            


              ====================================================================================================              
              ||                      Process : TRestDetectorSignalChannelActivityProcess                       ||              
              ||            Name: chActivity  Title: Channel activity process  VerboseLevel: warning            ||              
              ||                        -----------------------------------------------                         ||              
              ||                                                                                                ||              
              ||                               Low signal threshold activity : 25                               ||              
              ||                              High signal threshold activity : 50                               ||              
              ||                             Number of daq histogram channels : 300                             ||              
              ||                                    Start daq channel : 4320                                    ||              
              ||                                     End daq channel : 4620                                     ||              
              ||                           Number of readout histogram channels : 250                           ||              
              ||                                   Start readout channel : 0                                    ||              
              ||                                   End readout channel : 250                                    ||              
              ||                                                                                                ||              
              ====================================================================================================              
                            
              ====================================================================================================              
              [==                          TRestProcessRunner: Starting the Process..                          ==]              
              Waiting for processes to finish ...              
-- Info : TRestThread : Writing temp file. Thread id : 0
              [==                          TRestProcessRunner: 45311 processed events                          ==]              
              Configuring output file, merging thread files together              
              [==                     DataBase Entry Added! Run Number: 10428, File ID: 0                      ==]              
              [==                                     TRestRun Created ...                                     ==]              
              [==                                        - Path : /tmp/                                        ==]              
              [==           - Filename : R10428_00000_RawToSignal_Background_23hr_jgalan_2.3.6.root            ==]              
              [==                                            Done!                                             ==]              
jgalan@sultan:/tmp$ restRoot R10428_00000_RawToSignal_Background_23hr_jgalan_2.3.6.root 
Warning in <UnknownClass::SetDisplay>: DISPLAY not set, setting it to 185.228.230.138:0.0

= REST Version: v2.3.6
= Loading libraries ...
 - libRestDetector.so
 - libRestRaw.so
 - libRestConnectors.so
 - libRestGeant4.so
 - libRestFramework.so
 - libRestTrack.so


Attaching file R10428_00000_RawToSignal_Background_23hr_jgalan_2.3.6.root as run0...
Attaching event TRestDetectorSignalEvent as ev0...

Attaching metadata structures...
- md0_RawSignals (TRestProcessRunner)
- md0_virtualDAQ (TRestRawMultiFEMINOSToSignalProcess)
- md0_rawChActivity (TRestRawSignalChannelActivityProcess)
- md0_veto (TRestRawVetoAnalysisProcess)
- md0_sAna (TRestRawSignalAnalysisProcess)
- md0_zS (TRestRawZeroSuppresionProcess)
- md0_chActivity (TRestDetectorSignalChannelActivityProcess)
- md0_iaxo_readout (TRestDetectorReadout)
Error in <TGClient::TGClient>: only one instance of TGClient allowed
root [0] .ls
TFile**		R10428_00000_RawToSignal_Background_23hr_jgalan_2.3.6.root	
 TFile*		R10428_00000_RawToSignal_Background_23hr_jgalan_2.3.6.root	
  OBJ: TTree	EventTree	TRestDetectorSignalEventTree : 0 at: 0x564c59f43950
  OBJ: TRestAnalysisTree	AnalysisTree	REST Process Analysis Tree : 0 at: 0x564c59f3f6b0
  OBJ: TH1D	daqChannelActivityRaw	daqChannelActivityRaw : 0 at: 0x564c5a179730
  OBJ: TH1D	daqChannelActivity	daqChannelActivity : 0 at: 0x564c5a182b20
  OBJ: TH1D	rChannelActivity	readoutChannelActivity : 0 at: 0x564c5a186830
  OBJ: TH1D	rChannelActivity_1	readoutChannelActivity : 0 at: 0x564c5a187480
  OBJ: TH1D	rChannelActivity_2	readoutChannelActivity : 0 at: 0x564c5a1880d0
  OBJ: TH1D	rChannelActivity_3	readoutChannelActivity : 0 at: 0x564c5a1a5f70
  OBJ: TH1D	rChannelActivity_M	readoutChannelActivity : 0 at: 0x564c5a1a6f60
  OBJ: TH1D	rChannelActivity_1H	readoutChannelActivity : 0 at: 0x564c5a1a7b40
  OBJ: TH1D	rChannelActivity_2H	readoutChannelActivity : 0 at: 0x564c5a182f80
  OBJ: TH1D	rChannelActivity_3H	readoutChannelActivity : 0 at: 0x564c5a183c30
  OBJ: TH1D	rChannelActivity_MH	readoutChannelActivity : 0 at: 0x564c5a184c90
  KEY: TTree	EventTree;67	TRestDetectorSignalEventTree
  KEY: TRestAnalysisTree	AnalysisTree;2	REST Process Analysis Tree
  KEY: TRestRun	IAXOD0-2021;2	IAXOD0 2021 data taking
  KEY: TRestProcessRunner	RawSignals;1	Raw processing and analysis
  KEY: TRestRawMultiFEMINOSToSignalProcess	virtualDAQ;1	defaultTitle
  KEY: TRestRawSignalChannelActivityProcess	rawChActivity;1	defaultTitle
  KEY: TRestRawVetoAnalysisProcess	veto;1	defaultTitle
  KEY: TRestRawSignalAnalysisProcess	sAna;1	
  KEY: TRestRawZeroSuppresionProcess	zS;1	
  KEY: TRestDetectorSignalChannelActivityProcess	chActivity;1	Channel activity process
  KEY: TH1D	daqChannelActivityRaw;1	daqChannelActivityRaw
  KEY: TH1D	daqChannelActivity;1	daqChannelActivity
  KEY: TH1D	rChannelActivity;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_1;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_2;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_3;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_M;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_1H;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_2H;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_3H;1	readoutChannelActivity
  KEY: TH1D	rChannelActivity_MH;1	readoutChannelActivity
  KEY: TRestDetectorReadout	iaxo_readout;1	IAXO-D0 readout 0.5 mm-Pitch 120+120 channels
root [1] 

What is the output you get from rest-config --welcome?

For the 2.3.6 version is:

Commit : 9baf6336 (2021-06-16 09:05:37 +0200)
Branch/Version : master/v2.3.6
Compilation date : 2021-06-17 08:14
Official release : No
Clean state : Yes

Hi again,

compiling the latest version, I cannot reproduce the error either.

What it happens to you if you go to sultan, /tmp directory and you execute the following with the version that crashes?

cd /tmp/
restManager --c RawToSignal.rml --f /storage/cast/SR2019/labData/Argon/R10428_Background_23hr_Vm_340_Vd_113_Pr_1.4_Gain_0x1_Shape_0xD_Clock_0x02-000.aqs

Please, copy/paste full output.

Then, if you switch to stable, what it happens?

cd /tmp/
source /programas/REST/stable/v2.3.6/thisREST.sh
restManager --c RawToSignal.rml --f /storage/cast/SR2019/labData/Argon/R10428_Background_23hr_Vm_340_Vd_113_Pr_1.4_Gain_0x1_Shape_0xD_Clock_0x02-000.aqs

Hi again,
in Sultan with this version it works:

Commit : 4c9defdd (2021-06-21 21:12:05 +0200)
Branch/Version : master/v2.3.6
Compilation date : 2021-06-22 11:23
Official release : No
Clean state : Yes

Actually, I do not know how to go to REST 2.3.6, commit 9baf6336, in Sultan…

Ok, one needs to compile specific commits, it is not available as such.

I compiled it and I was able to reproduce the error.

              ====================================================================================================              
              [==                          TRestProcessRunner: Starting the Process..                          ==]              
40911 Events (8.1MB/s), 0.2 min ETA, (Pause: "p") 90.4%[===============================================================>-------]
 *** Break *** segmentation violation
40927 Events (8.6MB/s), 0.2 min ETA, (Pause: "p") 90.4%[===============================================================>-------]


===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================

Thread 2 (Thread 0x7f87d707e700 (LWP 18648)):
#0  0x00007f8822f8104a in __GI___waitpid (pid=20243, stat_loc=stat_loc
entry=0x7f87d7075a00, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00007f8822f080ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f88296b5917 in TUnixSystem::StackTrace() () from /programas/root/root-6.20/lib/libCore.so
#3  0x00007f88296b8064 in TUnixSystem::DispatchSignals(ESignals) () from /programas/root/root-6.20/lib/libCore.so
#4  <signal handler called>
#5  0x00007f87d803650f in TRestRawSignal::IsACDSaturation(int, int) () from /programas/REST/dev_javi/lib/libRestRaw.so
#6  0x00007f87d803b790 in TRestRawSignalAnalysisProcess::ProcessEvent(TRestEvent*) () from /programas/REST/dev_javi/lib/libRestRaw.so
#7  0x00007f8829c56f44 in TRestThread::ProcessEvent() () from /programas/REST/dev_javi/lib/libRestFramework.so
#8  0x00007f8829c582d8 in TRestThread::StartProcess() () from /programas/REST/dev_javi/lib/libRestFramework.so
#9  0x00007f8823a59e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f882326f4a4 in start_thread (arg=0x7f87d707e700) at pthread_create.c:456
#11 0x00007f8822fb1d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f882a0fab80 (LWP 18473)):
#0  0x00007f8822f8128d in nanosleep () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f8822faadc4 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32
#2  0x00007f8829c36f48 in TRestProcessRunner::RunProcess() () from /programas/REST/dev_javi/lib/libRestFramework.so
#3  0x00007f8829bf7c4c in TRestManager::ReadConfig(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, TiXmlElement*) () from /programas/REST/dev_javi/lib/libRestFramework.so
#4  0x00007f8829bf8c23 in TRestManager::InitFromConfigFile() () from /programas/REST/dev_javi/lib/libRestFramework.so
#5  0x00007f8829c0b3cc in TRestMetadata::LoadConfigFromElement(TiXmlElement*, TiXmlElement*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >) () from /programas/REST/dev_javi/lib/libRestFramework.so
#6  0x00007f8829c147bb in TRestMetadata::LoadConfigFromFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () from /programas/REST/dev_javi/lib/libRestFramework.so
#7  0x000055d86889af59 in main ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f87d803650f in TRestRawSignal::IsACDSaturation(int, int) () from /programas/REST/dev_javi/lib/libRestRaw.so
#6  0x00007f87d803b790 in TRestRawSignalAnalysisProcess::ProcessEvent(TRestEvent*) () from /programas/REST/dev_javi/lib/libRestRaw.so
#7  0x00007f8829c56f44 in TRestThread::ProcessEvent() () from /programas/REST/dev_javi/lib/libRestFramework.so
#8  0x00007f8829c582d8 in TRestThread::StartProcess() () from /programas/REST/dev_javi/lib/libRestFramework.so
#9  0x00007f8823a59e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f882326f4a4 in start_thread (arg=0x7f87d707e700) at pthread_create.c:456
#11 0x00007f8822fb1d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

===========================================================


jgalan@sultan:/tmp$ s), 0.2 min ETA, (Pause: "p") 90.4%[===============================================================>-------]

Still, the error is not appearing at the latest development, or at the official tag v2.3.6. So, it might be connected with a WIP develoment.

Ok, thanks. Indeed with the “clean” versions the crash seems not to appear.