Skip to content
Back to

Workflow orchestration

On the Flow canvas, you design visual pipelines that move data from source to outcome. The workflow engine runs them reactively on the server: when inputs or configuration change, affected nodes re-run while independent branches execute in parallel. You reuse work through templates and subgraphs, and every run can leave audit and lineage metadata for compliance review. See Compliance evidence for data lineage and Data for connectors, storage, and file formats.

Visual pipeline model

Section titled “Visual pipeline model”

A workflow is a directed graph of nodes connected by edges. Each node performs one step and passes typed dataset ports downstream.

Execution engine

Section titled “Execution engine”

When you trigger a run, the engine:

  • Reacts to change - hash-driven scheduling re-runs nodes when inputs or configuration change
  • Runs in parallel - independent branches execute concurrently where the graph allows
  • Executes server-side - large files, PDF parsing, connector fetch, and PyAirbyte jobs run on the workflow engine
  • Persists run-scoped data - uploads, Arrow datasets, exports, and audit sidecars survive pod restarts within your workspace boundary
Validations

Apply schema, business-rule, and tolerance checks as deterministic steps; failures route to owners with full context attached.

Transformations

Reshape, normalise, and map payloads between source formats and target schemas with versioned transformation logic.

Compute

Run grouping, filtering, aggregations, enrichment, and other compute steps on governed datasets inside the workflow boundary.

Node editor

Section titled “Node editor”

When you open a node for editing, the editor tabs include JSON (after Data, or after the primary tab when there is no Data, and before Ports / Settings). The tab shows the full workflow-config object for that node. Valid edits apply when the editor loses focus; invalid JSON or changes to id / type keep your draft and show an error. Marker stroke nodes stay canvas-only and do not include this tab.

Canvas preferences

Section titled “Canvas preferences”

Open the bottom omni toolbar Preferences control to adjust canvas display. Among fit view, lock, layout direction (LR / TB), and full vs data-only mode, you can switch node corner radius with an icon-only split control (rounded or square). The choice is stored on the workflow display config and applies to standard workflow node cards.

Node families

Section titled “Node families”

The sidebar under Workflows → Nodes documents each node type. Representative families:

Operations and validations

Section titled “Operations and validations”

The Operations node exposes column-level transforms (strings, dates, decimals, conditionals, lookups). You combine visual operation stacks in the UI or use natural-language compile (Governed AI) to generate operations from plain English.

The sidebar catalogues Transformations, Validations, and Conditions you attach to Operations nodes. Use them to enforce data quality before data leaves a pipeline branch.

Exception management

Section titled “Exception management”

Configure On error on operations and nodes to raise error or raise exception. Exceptions persist for the workflow and appear on the Exception Management board. See Exception management.

Document processing

Section titled “Document processing”

PDF and office documents uploaded to File Input are parsed server-side via Docling on the platform cluster (shared document-parse service for all workspaces). Optional AI structuring enriches results for review in the UI.

Subgraphs and templates

Section titled “Subgraphs and templates”
  • Save workflows as templates and share them within teams or the community catalog
  • Encapsulate reusable fragments as subgraphs and graft them onto larger pipelines
Section titled “Related documentation”
  • Security - platform security controls and SOC 2 summary
  • Compliance evidence - atomic data lineage and workflow file store
  • Data - File Input, PyAirbyte connectors, Arrow storage, REST triggers, exports
  • Governed AI - AI transforms and natural-language operation compile
  • Observability - run telemetry; canvas port audit items vs security audit ledger