Operational telemetry
Operational telemetry is how your team runs Fontana day to day: search logs, trace errors, watch dashboards, and confirm services are up. Fontana centralises this at the platform layer so every workspace does not need its own observability stack.
Your operators use HyperDX on the platform cluster as the primary OTLP destination for logs, traces, and bundled dashboards. Gatus runs in-cluster (not inside the Flow app) and records service health history for platform and tenant workloads.
What gets collected
Section titled “What gets collected”| Source | Examples | Typical use |
|---|---|---|
| Platform workloads | Flow, workflow engine, Convex backend | Pod logs, errors, run lifecycle |
| Flow browser | UI errors and product telemetry | Front-end incidents and usage signals |
| Convex | Structured execution and console events | Backend behaviour and log stream topics |
| Workflow engine | Run lifecycle, capacity samples | Pipeline performance and failures |
| Connectors | Connector job output in the engine pod | Ingress and egress troubleshooting |
| Node metrics | CPU, memory, load on the host | Capacity and saturation |
Signals from each workspace are routed through observability-api in that workspace cluster before export to HyperDX, so tenancy boundaries stay intact.
HyperDX dashboards
Section titled “HyperDX dashboards”During platform provisioning, your deployment team typically:
- Onboards a HyperDX team and OTLP credentials for the environment
- Loads bundled dashboards for platform and tenant services
- Points Convex log stream and browser telemetry webhooks at the workspace observability ingress
Ask your administrator for the HyperDX UI URL on your environment. On local k3d workspaces, run fontana resources <tenant> for the dedicated observe-<tenant> hostname (for example https://observe-demo.localhost/).
Authentication: the HyperDX UI requires a native login before showing any data. Provisioning registers the admin account through the HyperDX API during fontana apply, so no open welcome page is exposed after install. Admin credentials are stored in your workspace Vault and surfaced to operators by fontana secrets <tenant> under the Operator surfaces block.
HyperDX alerting
Section titled “HyperDX alerting”Fontana distinguishes event delivery (automatic) from notification routing (operator choice):
| Step | Automatic on fontana apply? | What you get |
|---|---|---|
| OTLP ingestion key | Yes (when HyperDX provisions cleanly) | Secret observability-keys; pods send logs to HyperDX |
| Bundled dashboards | Yes | Fontana — Logs, Fontana — Infrastructure, Fontana — WORM audit (integrity scan outcomes) |
| Integrity scan OTLP events | Yes (daily CronJob) | One log per scan with ServiceName:fontana-worm-integrity-scan and fontana.integrity_scan.outcome |
| Kubernetes audit security alerts | Yes | Bundled K8s audit dashboard plus alert rules for authentication failures, secret access, destructive verbs, and ClusterRoleBinding changes; each alert delivers a webhook to observability-api, which appends a WORM evidence record |
| Email / Slack / PagerDuty alert | No (one-time setup) | HyperDX alert rule on a dashboard tile or saved search; you pick the channel |
To add notifications for failed integrity scans: open Fontana — WORM audit in HyperDX, use the Integrity scan failures tile menu → Create alert, choose threshold above 0, and attach your Slack or email channel. Fontana cannot provision that channel without your webhook or recipient list.
If apply logs team exists but Secret observability-keys has no stored credentials, HyperDX is in a stuck state: restore the Secret from backup or reset the HyperDX data volume and re-run fontana apply before expecting scan events or alerts to work.
Gatus service health
Section titled “Gatus service health”Gatus probes HTTP endpoints for Flow, workflow engine, Convex, Vault, Postgres, Docling, and related services. It provides a health history page separate from HyperDX log search.
Gatus records availability and uptime for platform and tenant HTTP endpoints. Your administrator can share the in-cluster URL and port for your box.
Authentication: the Gatus status page enforces HTTP basic auth. Credentials are generated at deploy time, stored in your workspace Vault, and surfaced by fontana secrets <tenant> under the Operator surfaces block. In-cluster health probes are unaffected by the UI login.
HyperDX mirror vs WORM system of record
Section titled “HyperDX mirror vs WORM system of record”HyperDX is the operational search plane: fast log query, dashboards, and alerting during incidents. When security audit events are mirrored into HyperDX, they appear alongside pod logs and browser telemetry for convenience.
ImmuDB is the compliance system of record for platform security audit. HyperDX retention TTLs, index rebuilds, or search deletes do not change the append-only WORM ledger. For diligence, cite ImmuDB integrity procedures and Immutable audit trail.
See Data retention for retention boundaries across workflow data, HyperDX, and ImmuDB.
Node log and metrics collector (otel-collector)
Section titled “Node log and metrics collector (otel-collector)”Each workspace cluster runs an OpenTelemetry collector as a DaemonSet on every node. It ships container logs and node CPU, memory, and load metrics to that cluster’s HyperDX OTLP receiver.
| Property | Posture |
|---|---|
| Scope | Per workspace cluster only; no cross-tenant telemetry surface |
| Host access | Runs as root with read-only mounts of host log directories and host filesystem paths needed to read /proc and /sys for node metrics |
| Kubernetes API | Pod metadata is derived from log file paths on the node |
| Egress | Allowed only to the in-cluster HyperDX OTLP endpoint (plus DNS), enforced by NetworkPolicy |
| Redaction | A transform processor runs in every log pipeline before export, replacing bearer tokens, JWTs, API keys, and Vault tokens with [REDACTED] |
Within a single-tenant cluster, the node already runs only that workspace’s pods, so host-level log collection does not read another tenant’s workloads. This is an operational telemetry control, distinct from WORM audit evidence.
Workflow canvas signals (operational context)
Section titled “Workflow canvas signals (operational context)”While a workflow runs, the Flow canvas shows per-node status, row counts, and port audit items (validation warnings, transform notes, user edits). These help operators and builders debug a run in context.
That canvas audit is workflow data quality and provenance on the workflow file store. It is not operational telemetry in HyperDX and not the WORM security ledger. Port audit items and lineage sidecars answer how a value was derived; ImmuDB answers who acted on the platform. See Storage and processing, Data lineage, and Security audit (WORM).
Related documentation
Section titled “Related documentation”- Observability overview - operational vs security audit planes
- Security audit (WORM) - ImmuDB compliance ledger (distinct from HyperDX search)
- Data retention - HyperDX TTL vs WORM retention
- Deployment - HyperDX onboarding during install and upgrade