netunicorn.base.pipeline.Pipeline

class Pipeline(tasks=(), early_stopping=True, report_results=True, environment_definition=None)

Bases: ExecutionGraph

Pipeline is a class that takes a tuple of Tasks and executes them in order. Each element in the tuple should be either a Task or a tuple of Tasks. Pipeline will execute elements in order and return the combined result of all tasks. If element is a tuple of tasks, these tasks would be executed in parallel.

The result of pipeline execution would be one of the following:

- Success: if all tasks succeed, then Success is returned

- Failure: if any task fails, then Failure is returned

Returning object will always contain a Result object for each task executed.

If early_stopping is set to True, then any task after first failed wouldn’t be executed.

Parameters:

• tasks (Collection[Union[Task, TaskDispatcher, Collection[Union[Task, TaskDispatcher]]]], default: ()) – tasks (ordered by stages) to be executed

• early_stopping (bool, default: True) – whether to stop executing tasks after first failure

• report_results (bool, default: True) – whether executor should connect core services to report pipeline results in the end

• environment_definition (Optional[EnvironmentDefinition], default: None) – environment definition for the pipeline

Attributes

last_stage	Current last stage of the pipeline.
name	Execution Graph name.
early_stopping	Whether to stop executing tasks after a first failure.
report_results	Whether executor should connect core services to report execution results in the end.
environment_definition	Environment definition for the execution graph.
graph	Graph of tasks and their order.
override_graph_validation	Disable graph validation.

Methods

copy	Return a copy of the pipeline.
draw	Draw execution graph using networkx library.
is_execution_graph_valid	Validates execution graph according to the ExecutionGraph rules.
then	Add a task or list of tasks as a separate stage to the end of the pipeline.

last_stage: Union[str, int]

Current last stage of the pipeline.

then(element)

Add a task or list of tasks as a separate stage to the end of the pipeline.

Parameters:

element (Union[Task, TaskDispatcher, Collection[Union[Task, TaskDispatcher]]]) – a task or tuple of tasks to be added

Returns:

Pipeline – self

copy()

Return a copy of the pipeline.

Returns:

Pipeline – a copy of the pipeline

draw(nx_layout_function=<function shell_layout>)

Draw execution graph using networkx library.

Parameters:

nx_layout_function (default: <function shell_layout at 0x7fdfbd01c280>) – networkx layout function to use for drawing (from networkx.drawing.layout)

Returns:

None – None

static is_execution_graph_valid(obj)

Validates execution graph according to the ExecutionGraph rules.

Returns:

bool – True if execution graph is valid, raises an exception otherwise

Parameters:

obj (ExecutionGraph)

name: str

Execution Graph name.

early_stopping: bool

Whether to stop executing tasks after a first failure.

report_results: bool

Whether executor should connect core services to report execution results in the end.

environment_definition: EnvironmentDefinition

Environment definition for the execution graph.

graph

Graph of tasks and their order.

override_graph_validation

Disable graph validation. Executor and other components will not validate the graph before execution.