Context
InheritableThread is a thread class that pyspark provides to support sync between JVM and python threads.
The ThreadPool class used in this package is python's original multiprocessing.pool.ThreadPool.
What goods/bads will it bring? Dunno, I'll start looking around.
Proposal
Without touching this package's source code, a user can already nicely make their own ThreadPool that uses their own Thread class. If this package were to adopt the pyspark Thread class, then the steps may be:
- Create a custom
DummyProcess that inherits pyspark.InheritableThread.
- Create a custom
ThreadPool that uses the custom DummyProcess.
- Use the custom
ThreadPool in this package's SparkDistributedBackend._get_pool method.
Context
InheritableThread is a thread class that pyspark provides to support sync between JVM and python threads.
The
ThreadPoolclass used in this package is python's originalmultiprocessing.pool.ThreadPool.What goods/bads will it bring? Dunno, I'll start looking around.
Proposal
Without touching this package's source code, a user can already nicely make their own
ThreadPoolthat uses their ownThreadclass. If this package were to adopt the pyspark Thread class, then the steps may be:DummyProcessthat inheritspyspark.InheritableThread.ThreadPoolthat uses the customDummyProcess.ThreadPoolin this package'sSparkDistributedBackend._get_poolmethod.