ThreadPool() is not supported on AWS Lambda using Python 3.7 and above

[mdw123]

We use docspring for filling out documents in our event-driven python system that runs within AWS Lambdas.

Recently, when upgrading from Python 3.6, your component started failing with the following callstack:

File "/var/task/api/external/docspring_api.py", line 10, in __init__     self._client = docspring.Client()   
File "/var/task/docspring/api/pdf_api.py", line 32, in __init__     api_client = ApiClient()   
File "/var/task/docspring/api_client.py", line 68, in __init__     self.pool = ThreadPool()   
File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 925, in __init__    Pool.__init__(self, processes, initializer, initargs)   
File "/var/lang/lib/python3.8/multiprocessing/pool.py", line 196, in __init__     self._change_notifier = self._ctx.SimpleQueue()   
File "/var/lang/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue     return SimpleQueue(ctx=self.get_context())   
File "/var/lang/lib/python3.8/multiprocessing/queues.py", line 336, in __init__     self._rlock = ctx.Lock()   
File "/var/lang/lib/python3.8/multiprocessing/context.py", line 68, in Lock     return Lock(ctx=self.get_context())   File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)   
File "/var/lang/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__     sl = self._semlock = _multiprocessing.SemLock( OSError: [Errno 38] Function not implemented

This occurs because the lambda execution environment does not support shared memory.

More: https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

[ndbroadbent]

Hello, sorry to hear about that! I’m not too familiar with the internals of how our python API library works, or why it needs to use shared memory. We generate this client library using OpenAPI generator. It might be possible that this issue is fixed in a later version, but I’m not sure, and it can take a lot of effort to update since we have a few customizations that need to be merged in carefully. The other difficulty is that this might just be a limitation with AWS functions, so not too sure if we promise anything unfortunately. But I will definitely put this on our todo list and see if there’s something we can do. In the meantime, it might be easier if you could please make the HTTP requests to call the API directly.

[scraimer]

I found that someone else ran into a similar issue, and solved it by using their own thread-safe (not multiprocessing-safe) lock.

Since the one built in to Python is meant to be multiprocessing-safe, and requires /dev/shm (which doesn't exist in Lambda).

ecederstrand/exchangelib@ea45d19