pypz.core.specs.operator module

class pypz.core.specs.operator.Operator(name: str = None, *args, **kwargs)

Bases: Instance[Plugin], InstanceGroup, RegisteredInterface, ABC

This class represents the operator instance specs. This class shall be used to integrate your processing logic with pypz. An operator spec can contain plugins as nested instance.

Parameters:

name – name of the instance, if not provided, it will be attempted to deduce from the variable’s name

class Logger(logger_plugins: set[LoggerPlugin])

Bases: ContextLoggerInterface

This is a wrapper class for the logging functionality. It wraps all the implementation of the LoggerPlugin and by invoking either method it will invoke the corresponding method of each LoggerPlugin.

Note

The logger instance is provided to all plugins, so the plugins can call the methods of this logger. However, if a LoggerPlugin would invoke the logger methods, it would cause an infinite recursion. This is prevented so that in every logger call, the call trace will be analyzed and if any instance of a LoggerPlugin is found, then a RecursionError will be thrown.

Parameters:

logger_plugins – logger plugin instances collected in the operator instance

_abc_impl = <_abc._abc_data object>

_debug(event: str | None = None, context_stack: list[str] = None, *args: Any, **kw: Any) → Any

_error(event: str | None = None, context_stack: list[str] = None, *args: Any, **kw: Any) → Any

_info(event: str | None = None, context_stack: list[str] = None, *args: Any, **kw: Any) → Any

_warning(event: str | None = None, context_stack: list[str] = None, *args: Any, **kw: Any) → Any

__context: Instance

Reference to the context aka parent object. Derived automatically in the override implementation of __setattr__.

__depends_on: set

Set of other instances that is this instance depending on. Note however that the type of the dependencies are checked dynamically in runtime, since dependencies can only be defined on the same type of instance.

__expected_parameters: dict[str, ExpectedParameter]

Map of expected parameters defined as descriptor of the class. Key is the name of the parameter, value is the parameter descriptor. Used to check, if an expected (described) parameter value shall be set upon instance parameter setting.

__full_name: str

Full name of the instance which uniquely identifies it up to its topmost context. For example, if an instance A has a parent context B, which has a parent context C, then the full name of A is ‘C.B.A’. This value is calculated and the result is stored after the first calculation to avoid recalculation every time.

__logger: ContextLogger

Context logger. For more information refer to Operator.Logger class

__logger_plugins: set[LoggerPlugin]

Collection of logger addons to store for context logging

__nested_instance_type: Type[NestedInstanceType] | None

Stores the specified type of the nested instances. This is required to be able to discover those instances

__nested_instances: dict[str, NestedInstanceType]

This dictionary holds all the instances that are nested in the context of this instance object

__parameters: InstanceParameters

The interpreted instance parameters i.e., cascading and templates are interpreted

__replicas: list[Operator]

List of replica instances

__replicate()

__replication_group_index: int

Specifies the group index, if the operator is replicated.

__replication_origin: Operator | None

Reference to the original instance, which was the base for the replication

__simple_name: str

Name of the instance, which is represented by the object created from the implementation class. If not provided and there is a parent context, then the parent context will use the name of the variable.

__spec_classes: set

Set of specs classes that are in the class hierarchy i.e., which specs classes are contributing to the implementation of this class

__spec_name: str

The name of the spec constructed of module and qualified class name. Notice that we separate the module name and the class name so that we can identify them at loading by name.

_abc_impl = <_abc._abc_data object>

abstract _on_init() → bool

This method shall implement the logic to initialize the operation. It will be called after services are started and resources are created.

Returns:

• True, if finished

• False, if more iteration required (to not block the execution)

abstract _on_running() → bool | None

This method shall implement the actual business logic. It will be called after the _on_init has successfully finished

Returns:

• True, if finished

• False, if more iteration required (to not block the execution)

• None, if automatically to be determined based on all inputs i.e., if no more input record, then finish

abstract _on_shutdown() → bool

This method shall implement the logic to shut down the operation. It will be called, after the _on_running has successfully finished.

Returns:

• True, if finished

• False, if more iteration required (to not block the execution)

_operator_image_name: str | None

PARAMETER - Name of the docker image, where the operator can be found. This is optional for the execution, but is required at deployment.

_replication_factor: int

PARAMETER - The replication factor specifies, how many replicas shall be created along the original instance.

static create_from_dto(instance_dto: OperatorInstanceDTO, *args, **kwargs) → Operator

Creates an instance object from the DTO representation. It is capable to retrieve and load specified classes and to update created instances according to the DTO.

Parameters:

instance_dto – instance DTO

Returns:

instance object specified by the DTO

static create_from_string(source, *args, **kwargs) → Operator

Helper method to provide the functionality to create an instance from a json model specified either as string or as dict.

Parameters:

source – model as string

Returns:

instance object specified by the DTO

get_context() → Pipeline

get_dto() → OperatorInstanceDTO

Converts the instance information into the corresponding Data Transfer Object (DTO)

Returns:

DTO from instance

get_group_index() → int | None

get_group_name() → str | None

If an Operator instance is replicated, then the replication group name is the actual instance name of the original instance.

Returns:

the replication group name

get_group_principal() → Instance | None

get_group_size() → int

get_logger() → ContextLogger

get_operator_image_name()

get_replica(replica_id: int) → Operator

Returns the replica instance by id. The id is the actual place in the replica list, which is ensured during the replica creation.

Parameters:

replica_id – replica id

Returns:

replica specified by the id

get_replicas() → list[Operator]

Returns:

replica list

get_replication_factor()

is_principal() → bool

update(source: OperatorInstanceDTO | dict | str) → None

Overridden to allow connection and replica updates.