EOF encountered in .pkl log files when simulating more than one events

[deepchatterjeeligo]

@reedessick Simulating multiple events using simulate.py (i.e. -N >1) while simultaneously running lvalertTest_listenMP is creating an EOF error with some pkl files : (pickle.load() fails)

Could be due to multiple processes trying to access the same file is causing a conflict.

Example: (simulating 7 gstlal events in 3 s)
deep@debian:~/github_forked/lvalertTest$ simulate.py -N 7 -r 3.0 -g /home/deep/FAKE_DB/ -i "H1,L1" -o /home/deep/OUT_DIR/ -s /home/deep/github_forked/lvalertTest/etc/gstlal.ini -v

Gives the following error:

Traceback (most recent call last):
File "/home/deep/github_forked/lvalertTest/bin/simulate.py", line 191, in
action.execute() ### actually perform the action
File "/home/deep/github_forked/lvalertTest/lib/schedule.py", line 90, in execute
return self.foo( *self.args, **self.kwargs )
File "/home/deep/github_forked/lvalertTest/lib/schedule.py", line 259, in writeLog
httpResponse = gdb.writeLog( self.graceDBevent.get_graceid(), self.message, filename=self.filename, tagname=self.tagname )
File "/home/deep/github_forked/lvalertTest/lib/ligoTest/gracedb/rest.py", line 451, in writeLog
jsonD, lvalert = self.__log__(graceid, message, filename=filename, tagname=tagname )
File "/home/deep/github_forked/lvalertTest/lib/ligoTest/gracedb/rest.py", line 406, in __log__
ind = self.__path2len__(self.__logsPath__(graceid))
File "/home/deep/github_forked/lvalertTest/lib/ligoTest/gracedb/rest.py", line 194, in __path2len__
return len(self.__extract__(path))
File "/home/deep/github_forked/lvalertTest/lib/ligoTest/gracedb/rest.py", line 213, in __extract__
ans = pickle.load(file_obj)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 880, in load_eof
raise EOFError
EOFError
continue? (yes/no) : no

[reedessick]

if this hypothesis is correct, this could be solved by adding a timeout parameter to FakeDb.extract and/or FakeDB.append along with locking behavior to guarantee that only a single process can operate on a file at a time.

[reedessick]

A proposed locking procedure, which should make all file manipulations conducted through the FakeDb API atomic, is as follows:

We lock by

mkdir GraceID.lock
write pid, hostname >> GraceID.lock/pid

and unlock by

rm GraceId.lock/pid
rmdir GraceID.lock

This should be encapsulated into 2 function calls that bracket all other manipulations, eg:

def logs()
self.lock(gid)
...
self.unlock(gid)
return ans

within each call to lock, set up a time-out loop that will retry for you automatically. Both timeout and cadence should be attributes of FakeDb that can be modified (at instantiation time?) as needed.

Need to write a big warning about possible abuse of file-system through mkdir and rmdir repeatedly and quickly. Also, possible issues with NFS caching if we're accessing the FakeDb directory from different machines.