Problem launching restManager with many files

RESTFramework
1

REST version : development/v2.2.18
REST commit : bfac8b48 (2020-07-22 18:38:07 +0200)

Hi!

I was trying to get some plots from a couple of Runs when I encountered a problem. As usual, I ran the command restManager --c config.rml --f "/path/R01399_0*Hits*", but, as shown in the picture, it breaks when it reaches 510 subruns.

Imagen1
Imagen11371×727 570 KB

I tried passing fewer files, and it works as one would expect, so I’m guessing it’s got to be related to the number of files, since it’s the first time I’ve come across this kind of problem (I had never analysed so many files at once). Perhaps there’s some problem with the paralellisation that’s done under the hood?

I’m sorry I can’t provide more information, I’ve tried some other things but I’m not able to pinpoint the problem.

Cheers,
Óscar

2

It is a limitation of linux, that one process cannot open more than 1024 files. You can increase the limit by editing /etc/security/limits.conf:

*                soft     nofile     2048

this increases the limit to 2048.

You can also break the limit temporary. In the bash, type:

ulimit -HSn 2048

ps.
The limitation is 1024 while one can only open 500+ files. Maybe each input file is opened twice in REST?

1 Like
3

Yes, that’s the problem. The number of open files system limitation.

However, this could be bypassed if inside the REST code we find a way to open and close the TRestRun pointer in the same loop. It happened to me with TRestMetadataPlot, since I had to open of the order of 10k files. Somehow I found the way to fill the graphs opening/closing each TRestRun and then do whatever.

This problems are present also in TRestAnalysisPlot, and I am trying to find a solution for that. Not sure if there is an easy solution for the restManager processing case.

The system modification is a temporal solution. We never know when we will pass the limit again. On top of that, we might move to a cluster where we need to face those limitations.

Another temporal solution is to generate a shell script that writes the restManager commands to be executed. I placed such script at:

1 Like
4

The problem for TRestAnalysisPlot was solved in this commit

Just by assuring TRestRun *r = new TRestRun is created and deleted inside the loop.

for( int n = 0; ...)
{
      TRestRun *run = new TRestRun("filename");

      ....
      ....

      delete run;

}