netunicorn.base.execution_graph.ExecutionGraph

class ExecutionGraph(early_stopping=True, report_results=True, environment_definition=None)

Bases: object

ExecutionGraph is a class that allows you to flexibly define a graph of tasks and their order. It has the next rules:

1. Execution graph is a directed graph.

2. Execution graph always starts with a node “root” (this node is automatically added in the new graph during initialization).

3. Any node of type TaskDispather would be dispatched to Task. Any node of type Task would be executed. Any nodes of other types would not be executed but would be treated as synchronization points.

4. Number of components in the graph must be 1, the graph should be weakly connected, and all nodes should be accessible from the root node.

5. Any edge have either “strong” or “weak” type. If no type provided, it’s a “strong” edge.

Type is provided as an attribute “type” with value “strong” or “weak”. Executor will not treat incoming “weak” edges as requirements for the execution of the task, but will traverse them to start executing next tasks. “Weak” edges are required for cycle dependencies to avoid deadlocks. You can consider weak links as links for defining the execution flow, and strong links to define both execution flow and prerequisites.

6. Any edge can have attribute “counter” with integer value. This value would be used to determine how many times this edge should be traversed. If this attribute is not present, then edge would be traversed infinitely. You can use this attribute to implement finite loops in the graph.

7. Any edge can have attribute “traverse_on” with either string value “success”, “failure”, or “any”. This attribute controls whether executor will traverse this edge on success or failure of the task. If this attribute is not present, the executor will obey the usual rules considering the “early_stopping” parameter (trat all edges as “success” for “early_stopping=True” or “any” for “early_stopping=False”). If this attribute is present, then the executor will traverse this edge only if the task was executed successfully (“success”), failed (“failure”), or in any case (“any”). The next rules apply:

If “early_stopping” is True, all edges are treated as having “traverse_on” attribute set to “success”. Setting the “traverse_on” attribute to “failure” would make the executor to traverse this edge in case of the task failure and will not break the execution of the graph. Also, setting the “traverse_on” attribute to “any” would make executor to traverse this edge in case of the task failure or success and will not break the execution of the graph. If “early_stopping” is False, all edges are treated as having “traverse_on” attribute set to “any”. Setting this attribute to “success” or “failure” would make the executor to traverse this edge only in case of the task success or failure respectively.

Parameters:

• early_stopping (bool, default: True) – If True, then the execution of the graph would be stopped if any task fails. If False, then the execution of the graph would be continued even if some tasks fail. In both cases, “traverse_on” attribute of the edges could be used to control the traversal of the graph.

• report_results (bool, default: True) – If True, then the executor would connect core services to report execution results in the end.

• environment_definition (Optional[EnvironmentDefinition], default: None) – environment definition for the execution graph

Attributes

name	Execution Graph name.
early_stopping	Whether to stop executing tasks after a first failure.
report_results	Whether executor should connect core services to report execution results in the end.
environment_definition	Environment definition for the execution graph.
graph	Graph of tasks and their order.
override_graph_validation	Disable graph validation.

Methods

draw	Draw execution graph using networkx library.
is_execution_graph_valid	Validates execution graph according to the ExecutionGraph rules.

name: str

Execution Graph name.

early_stopping: bool

Whether to stop executing tasks after a first failure.

report_results: bool

Whether executor should connect core services to report execution results in the end.

environment_definition: EnvironmentDefinition

Environment definition for the execution graph.

graph

Graph of tasks and their order.

override_graph_validation

Disable graph validation. Executor and other components will not validate the graph before execution.

static is_execution_graph_valid(obj)

Validates execution graph according to the ExecutionGraph rules.

Returns:

bool – True if execution graph is valid, raises an exception otherwise

Parameters:

obj (ExecutionGraph)

draw(nx_layout_function=<function shell_layout>)

Draw execution graph using networkx library.

Parameters:

nx_layout_function (default: <function shell_layout at 0x7fdfbd01c280>) – networkx layout function to use for drawing (from networkx.drawing.layout)

Returns:

None – None