pypz.executors.pipeline.executor module

class pypz.executors.pipeline.executor.PipelineExecutor(pipeline: Pipeline)

Bases: object

This class implements the feature to execute an entire pipeline i.e., each operator individually via an OperatorExecutor. To increase the performance of the execution, each OperatorExecutor will be launched in a separate thread.

Warning

This class has the sole purpose to allow you to test pipelines locally, it should not be used for production workloads, since python is not very optimal for concurrent executions. That is the reason to limit the operator count.

Parameters:

pipeline – the pipeline instance to be executed

__futures: set[Future]

Creating the OperatorExecutor objects. Notice that none of the OperatorExecutors may handle interrupts, since this will be handled on PipelineExecutor level.

_max_operator_count: int = 32

This value limits the number of operators to be executed parallely. An exception will be thrown, if exceeded.

interrupt(signal_number, current_stack)

This method is called upon receiving a system signal e.g., SIGINT. We are interrupting each OperatorExecutor by invoking interrupt() only if it is still running. Notice that we cancel futures i.e., should an OperatorExecutor not yet be scheduled, it will prevent to be scheduled.

shutdown()

This method shuts down the ThreadPoolExecutor. Notice that it blocks until all OperatorExecutor has finished. Notice as well that we don’t cancel futures, since it will be handled upon handling the interrupt signals.

start(exec_mode: ExecutionMode = ExecutionMode.Standard)

This method triggers the execution by creating a ThreadPoolExecutor and submitting the OperatorExecutor’s corresponding method.

Parameters:

exec_mode – pypz.executors.commons.ExecutionMode