Executor

In pypz each operator is self-contained, which means that there is no external dependencies for its runtime (like an executor infrastructure) i.e., you can execute it for example on your local machine or in a Kubernetes cluster as well. This requires an embedded OperatorExecutor, which controls the execution flow of the entire operator incl. plugins.

The executor has the following main components:

Execution Context

The ExecutionContext is responsible to maintain all the necessary information during execution. It can be forwarded to the states of the state machine as well.

The execution context has the following responsibilities:

  • check, if all required parameters are set

  • maintain exit code

  • register plugins along their types

  • traverse plugin dependency graph

  • store information about execution mode

State machine

digraph state_machine { size="10,20"; ratio="auto"; ranksep="2"; rankdir="TB"; margin="0.0,0.5"; graph [fontname="Verdana", fontsize="28"]; start [shape=circle, label="", style=filled, color=black, width=0.2, height=0.2]; end [shape=circle, label="", width=0.2, height=0.2, style=filled, color=black, fillcolor=black, peripheries=2]; entry [shape=rect, label="StateEntry", style=rounded, width=2, height=0.6]; killed [shape=rect, label="StateKilled", style=rounded, width=2, height=0.6]; service_start [shape=rect, label="StateServiceStart", style=rounded, width=2, height=0.6]; service_shutdown [shape=rect, label="StateServiceShutdown", style=rounded, width=2, height=0.6]; resource_creation [shape=rect, label="StateResourceCreation", style=rounded, width=2, height=0.6]; resource_deletion [shape=rect, label="StateResourceDeletion", style=rounded, width=2, height=0.6]; operation_init [shape=rect, label="StateOperationInit", style=rounded, width=2, height=0.6]; operation_running [shape=rect, label="StateOperationRunning", style=rounded, width=2, height=0.6]; operation_shutdown [shape=rect, label="StateOperationShutdown", style=rounded, width=2, height=0.6]; start -> entry; entry -> service_start [label="SignalServiceStart"]; entry -> killed [label="SignalShutdown"]; service_start -> resource_creation [label="SignalResourcesCreation"]; service_start -> resource_deletion [label="SignalResourcesDeletion"]; service_start -> service_shutdown [label="SignalError"]; service_start -> service_shutdown [label="SignalShutdown"]; resource_creation -> operation_init [label="SignalOperationInit"]; resource_creation -> resource_deletion [label="SignalError"]; resource_creation -> service_shutdown [label="SignalServicesStop"]; resource_creation -> resource_deletion [label="SignalShutdown"]; operation_init -> operation_running [label="SignalOperationStart"]; operation_init -> operation_shutdown [label="SignalError"]; operation_init -> operation_shutdown [label="SignalShutdown"]; operation_running -> operation_shutdown [label="SignalOperationStop"]; operation_running -> operation_shutdown [label="SignalError"]; operation_running -> operation_shutdown [label="SignalShutdown"]; operation_shutdown -> resource_deletion [label="SignalResourcesDeletion"]; operation_shutdown -> service_shutdown [label="SignalServicesStop"]; operation_shutdown -> resource_deletion [label="SignalError"]; resource_deletion -> service_shutdown [label="SignalServicesStop"]; resource_deletion -> service_shutdown [label="SignalError"]; service_shutdown -> killed [label="SignalKill"]; service_shutdown -> killed [label="SignalError"]; killed -> end; }

You can find more information incl. invoked plugin methods in the corresponding states module.

Execution modes

As you might see on the state machine diagram, there are different routes from start to end. Which route the state machine will take depends not just on the execution results of the states, but on the specified execution mode as well.

Imagine the case, where your one or more operators in your pipeline crashed. Resources might not have been released in this case. Since pypz does not know anything about your resources, only you do, pypz cannot provide an integrated feature to clean up your resources. However, what pypz can provide is an execution mode, where only resources will be deleted instead of running the actual business logic.

Check pypz.executors.commons.ExecutionMode for more details.