BatchJobs hiccups that we may be able to work around

[HenrikBengtsson]

I'm going to use this issue/thread to keep recording issues that seem to be related to the BatchJobs package per se, but that we might be able to work around in this package (until a proper BatchJobs solution is found).

[HenrikBengtsson]

Here's the first one. On a TORQUE/PBS cluster with a shared file system:

Sample 20 ('Patient10,A08297_vs_A06930') of 132 ...
...
Error in value[[3L]](cond) :
  Error syncing registry (Error in sqliteSendQuery(con, statement, bind.data): rsqlite_query_send: could not execute1: database is locked
)
Calls: resolve ... tryCatchList -> tryCatchOne -> <Anonymous> -> stopf
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted

UPDATE 2016-08-31: The "database is locked" problem have been reported by other BatchJobs users and is discussed in for instance tudo-r/BatchJobs#58. There are also some suggestions for tweaking the settings (e.g. job.delay) to work around this problem. See also Issue #67.

[HenrikBengtsson]

Another one:

Error in dbDoQueries(reg, extractSubList(msgs, "msg"), flags = "rw", max.retries,  :
  Error while etablishing the connection: Error in .local(drv, ...) : Could not connect to database:
unable to open database file

UPDATE 2016-05-16: This is an error you get if you remove the registry files, e.g.

> unlink("multicore_ncpus_2-files", recursive=TRUE)
> reg
Job registry:  multicore_ncpus_2
Error in dbDoQuery(reg, query) :
  Error while etablishing the connection: Error in .local(drv, ...) : Could not connect to database:
unable to open database file

[alexvorobiev]

Do expired jobs belong to the list?
Every now and then I am getting

task 1 failed - "BatchJobExpiration: Job of registry 'BatchJobs_1291466732' expired:

on a server with high CPU utilization (using SLURM). Is there any way to work around those?

[HenrikBengtsson]

Possibly - that would go into the field of automatically restarting jobs, which would be possible but sounds like a task for BatchJobs itself.

Now, BatchJobs has frozen its development in favor of upcoming batchtools. I think the latter would be a good place to bring up a discussion on this.