Parameters

Operators and plugins may define parameters to allow the users to tune their functionalities. As matter of fact, parameters can be set for each Instance via the following method:

pypz.core.specs.instance.Instance.set_parameter()

In the following section it will be explained:

how an instance can define expected parameters
how parameters can be cascaded through multiple level of nested instances
how template interpolation is working

Expected Parameters

Your instance implementation can expect parameters either as required or as optional. The ExpectedParameter descriptor class is used for this purpose.

Example:

class TestImpl(Instance):
    required_param = ExpectedParameter(required=True, parameter_type=str)
    optional_param = ExpectedParameter(required=False, parameter_type=str)

    def __init__(self):
        self.required_param = None
        self.optional_param = "defaultValue"

This is equivalent to:

class TestImpl(Instance):
    required_param = RequiredParameter(str)
    optional_param = OptionalParameter(str)

    def __init__(self):
        self.required_param = None
        self.optional_param = "defaultValue"

As you can see, the expected parameters shall be defined both as class and instance variable, where the class variable has the type of the descriptor class and the instance variable the actual parameter value.

Note

Only the execution context will check required parameters automatically. You can invoke instance methods outside of the executor without providing required parameters. However, you can always check manually, if all the required parameters have been set by the following method…

pypz.core.specs.instance.Instance.get_expected_parameters()

… by looping through all the expected parameters, where required = True and check, if there is already a parameter set with that name via:

pypz.core.specs.instance.Instance.has_parameter()

Note

Note that although you can define an initial value for the required parameters, a value setting will still be required.

This descriptor class provides the following features:

ensures type conformity
provides alternative name for parameters, which can differ from the variable’s name
provides callback, if the value has been updated
bound to Instance's parameter store, hence if you define a parameter this way, you can access as well via instance.get_parameter() method

Cascading Parameters

Imagine the case, where you have a pipeline with 5 operators, each with 1 input port and 1 output port plugin. To specify the parameter ‘channelLocation’ (see Data Transfer) you would need to set the same value for 10 different instances. To avoid such tedious thing, pypz provides a feature to cascade parameters from higher contexts. Sticking to the example, you can set the parameter ‘channelLocation’ on pipeline level for all existing plugins in the pipeline like this:

pipeline.set_parameter(">>channelLocation", "valid_url")

There are two types of cascading parameters:

including, if the parameter name is prefixed by “#”, then the parameter will not just be cascaded, but will be applied to the instance, where it was set
excluding, if the parameter name is prefixed by “>”, then the parameter will be cascaded, but it will not be applied to the instance, where it was set

The number of prefixes determines, how many levels will a parameter be cascaded. The example above is a two-level, excluding cascading parameter i.e., it will be cascaded from pipeline to plugin level without applying it to either pipeline or operator level.

Examples:

# Set on pipeline and operator level
pipeline.set_parameter("#param", "value")

# Set on pipeline, operator and plugin level
pipeline.set_parameter("##param", "value")

# Set on operator and plugin level
operator.set_parameter("#param", "value")

# Set on plugin level
operator.set_parameter(">param", "value")

# Set on operator and plugin level
pipeline.set_parameter(">#param", "value")

# Set on pipeline and plugin level
pipeline.set_parameter("#>param", "value")

Template Parameters

Imagine the case, where your operator has to access secured resources and it needs the appropriate credentials. Obviously, there are options like using some credential store solutions e.g., key-vaults or using third-party libs. However, pypz provides a template interpolation feature for parameters.

Based on the time, when the templates are interpolated, there are two different templating syntax.

Instance Time Templates

Syntax: ${}

# Set on pipeline and operator level
instance.set_parameter("secret", "${env:SECRET}")

In this case the template is resolved as soon as the parameter is set on the instance via the set_parameter() method. This can be used, if you deploy a pipeline from your system, where you can set the value for the environment variable.

Warning

Since the template is interpreted already on the instance object the secret will be visible in the serialized configuration! If you want to prevent this, then you should use execution time templates described in the next section.

Execution Time Templates

Syntax: $()

# Set on pipeline and operator level
instance.set_parameter("secret", "$(env:SECRET)")

In this case the template is resolved by the executor itself i.e., on the machine, where the executor is started. This requires the capability to control the environment variables on that machine. Unlike in case of the instance time template parameters, the value of the execution time template is not visible in the serialized configuration.

Note

Note that currently only environment variables can be resolved. It is planned to extend this feature to resolve files and remote locations as well in the future.