The pieces / weirdnet-pipelines

weirdnet-pipelines

DAGs of application-defined sources, steps and sinks with validation, ordering and a runner.

GitHub crates.io docs.rs weirdnet-pipelines = "0.1"

What it is

Pipelines as graphs of nodes whose meaning belongs to the application. The runner reaches the application only through Host; lookups, permissions, approvals and storage stay in the application.

What is in it

  • Kind: the application’s node type. Graph<K>: nodes (ID=KIND, +TEE) and edges (A>B[:NAME]).
  • Graph::order (cycles are errors naming their nodes) and Graph::check (duplicates, missing nodes, self-loops, no source or sink, and more).
  • execute: each node once, in order, with fan-out, fan-in and tees; a failing node skips the nodes below it.
  • frame: JSON column frames from rows, CSV, JSON or NDJSON, with union, select, filter and Markdown output.

Where it fits

  • Extracted from Muninn.
  • License: BSD-3-Clause.