Security audit (WORM)

Security audit (WORM) is Fontana’s compliance-grade event history: who did what, when, and with what outcome, stored so later edits cannot silently rewrite the past. WORM means write once, read many: new records append; existing entries are not updated or removed through normal application paths.

Each workspace has its own ImmuDB ledger. observability-api in that workspace is the only writer to the ledger. Application components do not call ImmuDB directly; they emit events that observability-api normalises, redacts, and appends.

HyperDX may mirror the same events for operator search. For diligence and evidence, treat ImmuDB as the system of record, not HyperDX. HyperDX is a search plane with its own retention TTL; expiry or reindexing there does not alter the append-only WORM ledger. See Data retention and Operational telemetry.

What is WORM?

Write once, read many (WORM) storage appends new audit records and detects tampering through cryptographic chaining. Fontana uses ImmuDB per workspace for security audit; operational log search in HyperDX is a separate plane.

What gets recorded

Security audit events share a normalised schema: workspace, timestamp, actor, action, resource, outcome, and correlation id. Sensitive material is redacted before storage (no API keys, bearer tokens, or full JWT bodies).

Category	Examples	Typical source
Identity and access	Sign-in, MFA, role and provisioning changes	Zitadel (automated poll)
Platform and cluster	Failed auth, secret access, privileged API calls	Kubernetes API audit (bundled HyperDX alert rules fire on these events and append a WORM evidence record per alert)
Application and admin	RBAC, BYOK, admin configuration	Convex structured admin logs (auditable: true) on the log stream
Workflow and AI (security-tagged)	Privileged runs, sensitive tool or connector access	Application audit producers (via Convex)

Identity events from Zitadel (login, MFA, federation, deprovision) reach the ledger automatically without custom Flow code.

Fail closed

If append to the WORM ledger fails, ingest paths return errors rather than dropping security events quietly. Your deployment team monitors observability-api and ImmuDB health as part of platform operations.

Review and integrity

Each workspace runs a daily integrity scan CronJob (immu-audit-scan, schedule 0 3 * * * UTC):

1. Scan (init container) — immuclient login + immuclient scan against the ImmuDB ledger.
2. Evidence (observability-api container) — writes a JSON manifest and raw scan log to the workspace evidence volume (worm-audit-evidence PVC at /evidence on Kubernetes), optionally uploads both files to S3 Object Lock when your platform team configures wormAudit.s3, and posts a structured OTLP log to HyperDX.

Artifact	Location	Purpose
immuclient-scan-<scanId>.log	Evidence volume	Raw scan output for auditors
manifest-<scanId>.json	Evidence volume	Tenant, outcome, SHA-256 of log, S3 keys, HyperDX reported flag
OTLP log	HyperDX (ServiceName:fontana-worm-integrity-scan)	Operator search and optional alert rules

HyperDX alerting: Fontana automatically emits a success or failure log on every scan when HyperDX is provisioned (observability-keys Secret present). The bundled Fontana — WORM audit dashboard (injected on fontana apply) shows recent scan outcomes. Email, Slack, or PagerDuty notifications are a one-time HyperDX alert rule you attach to the dashboard tile or saved search (Fontana does not know your notification channel until you configure it). See Operational telemetry § HyperDX alerting.

If the scan fails, or HyperDX OTLP delivery fails when configured, the CronJob exits non-zero so Kubernetes retains a failed Job for investigation (fail-closed).

Operators and auditors also use:

• HyperDX for interactive search during incidents (security audit mirror at ServiceName:fontana-worm-audit)
• In-cluster REST (http://immu-audit:8080) for smoke tests and break-glass verification (see operator runbook)

Read-only auditor access can be provisioned separately from the writer credential used by automated append.

Verify integrity scan (operators)

After deploy, trigger a one-off Job from the CronJob and confirm evidence plus HyperDX delivery:

export KUBECONFIG="$FONTANA_HOME/kubeconfig/tenant-<id>.yaml"   # e.g. tenant-demo
NS=fontana
JOB="immu-audit-scan-manual-$(date +%s)"
kubectl create job -n "$NS" "$JOB" --from=cronjob/immu-audit-scan
kubectl wait -n "$NS" --for=condition=complete "job/$JOB" --timeout=300s
kubectl logs -n "$NS" "job/$JOB" -c immuclient-scan
kubectl logs -n "$NS" "job/$JOB" -c evidence

Expect the evidence container log line outcome=success. In HyperDX, open the Fontana — WORM audit dashboard or search ServiceName:fontana-worm-integrity-scan. If fontana apply logged observability-keys missing, fix HyperDX provisioning before testing OTLP (see runbook below).

HyperDX must be provisioned

Integrity scan evidence requires Secret observability-keys (ingestion API key). When HyperDX MongoDB already has a team but the Secret was lost, fontana apply warns and skips re-onboarding. Restore the Secret from backup or reset the HyperDX PVC and re-run apply. Until then, the evidence Job fails on OTLP delivery by design.

Names that sound like audit but are not WORM

Name	What it is
Audit logs → Chat in Flow	LLM debug inspection for authorised operators; stored in Convex
Port audit items on the canvas	Data-quality warnings and errors on workflow outputs; workflow file store, not WORM
Data lineage (.lineage.json sidecars)	Calculation provenance; workflow file store, not WORM
Workflow document history	Edit history in Convex tables; mutable application data

For calculation provenance, see Data lineage and Compliance evidence. For retention boundaries, see Data retention.

See Immutable audit trail for schema detail, producer map, and workspace isolation.

Related documentation

• Observability overview - operational telemetry vs security audit
• Operational telemetry - HyperDX and Gatus
• Immutable audit trail - full WORM architecture and producer map
• Identity and access - Zitadel and identity events in the ledger