Webserver waits for task to finish in result()

[LourensVeen]

In the webserver (webserver/__init__.py), I noticed that result() waits for the corresponding Celery task to finish. What happens if you launch a long-running operation, and then request the result? That would block a server thread I think. Now what if you request it multiple times? Does that DOS the server?

I think it would be better to return a status message stating that the result is not yet available, and to try again in the future, instead of waiting. I had a look through the HTTP error codes, and 404 is the correct one to use here, as it means that the resource is unavailable at the moment (because it hasn't been calculated yet) but may become available in the future.

It should be possible for the user to distinguish between a 404 because the requested task does not exist, and a 404 because the result is not yet available. This could be done through a human-readable error message, or we could add a task status request that returns 404 if the task doesn't exist, but a "Processing" status if it does but isn't done yet.

[larsmans]

I think you're right. This is important if we want to run a public demo.

I do think we should keep the old behavior and add a flag to change to the behavior you suggest.

[LourensVeen]

Keeping the old behavior wouldn't fix the security issue though. Fixing that requires removing any functionality that can block server threads for a long time.

[larsmans]

I know. But the REST server isn't really meant to face the public web. This is a backend tool.

[LourensVeen]

Okay, right. So it's scalability more than security.

So the scenario then is that you have a bunch of people in a room following a tutorial and all running longish jobs on a single server, and waiting for the result at the same time, and the whole thing becoming unresponsive. And then you can tell them to add an extra flag.

Still, I could also see someone starting a task from the REPL, setting it to wait for the result, opening a new window with a new REPL and doing the same for another task, and then once you have five tasks in flight, it doesn't work anymore, even if there's enough capacity on your cluster.