draugr.multiprocessing_utilities.pooled_queue_processor.PooledQueueProcessor

class draugr.multiprocessing_utilities.pooled_queue_processor.PooledQueueProcessor(func, args: Iterable = (), kwargs=None, max_queue_size=100, n_proc=None, max_tasks_per_child=None, fill_at_construction=True, blocking=True)[source]

Bases: object

This is a workaround of Pythons extremely slow interprocess communication pipes. The ideal solution would be to use a multiprocessing.queue, but it apparently communication is band limited. This solution has processes complete tasks (batches) and a thread add the results to a queue.queue.

__init__(func, args: Iterable = (), kwargs=None, max_queue_size=100, n_proc=None, max_tasks_per_child=None, fill_at_construction=True, blocking=True)[source]

Methods

__init__(func[, args, kwargs, ...])

close()

close pool

fill()

fill queue

get()

return

maybe_fill()

fill queue if not full

put(res)

param res

raise_error(excptn)

param excptn

terminate()

terminate pool

Attributes

queue_size

return: :rtype:

close() None[source]

close pool

fill() None[source]

fill queue

get() Any[source]
Returns

maybe_fill() None[source]

fill queue if not full

put(res) None[source]
Parameters

res

property queue_size: int

return: :rtype:

raise_error(excptn) None[source]
Parameters

excptn

terminate() None[source]

terminate pool