draugr.multiprocessing_utilities.pooled_queue_processor.PooledQueueProcessor¶
- class draugr.multiprocessing_utilities.pooled_queue_processor.PooledQueueProcessor(func, args: Iterable = (), kwargs=None, max_queue_size=100, n_proc=None, max_tasks_per_child=None, fill_at_construction=True, blocking=True)[source]¶
Bases:
object
This is a workaround of Pythons extremely slow interprocess communication pipes. The ideal solution would be to use a multiprocessing.queue, but it apparently communication is band limited. This solution has processes complete tasks (batches) and a thread add the results to a queue.queue.
- __init__(func, args: Iterable = (), kwargs=None, max_queue_size=100, n_proc=None, max_tasks_per_child=None, fill_at_construction=True, blocking=True)[source]¶
Methods
__init__
(func[, args, kwargs, ...])close
()close pool
fill
()fill queue
get
()- return
fill queue if not full
put
(res)- param res
raise_error
(excptn)- param excptn
terminate pool
Attributes
return: :rtype: