Monitoring, logs and alerts

What this part does

This part answers one question for the whole system: is omnium keeping its promises right now, and if not, who knows about it? Sections 3 to 9 each promised alerts "through the central alert service". This section builds that service and everything that feeds it.

The user's original ask was short: "we should have a centralized alerts system". The design turns that into eight questions.

What we have today

We have most of the parts, but none of them tells a person. This was checked in the code on 7 Oct 2026.

How it works

Two kinds of input feed one service, and one service feeds three channels.

1. Code raises an alert in one line. When something fails, the code calls alerts.raise_(session, rule, subject, detail). That adds one row to alert_signal inside the same transaction as the failure. If the transaction rolls back, the signal goes too.
2. A checker looks at our numbers every 15 seconds. It runs a short list of named queries against tables we already keep (delivery state, import runs, heartbeats). Each breach adds a "raise" signal. Each subject that is healthy again adds a "clear" signal.
3. Every background process beats. Each one writes a process_beat row every 10 seconds. The checker reads those rows to find silent and stuck workers.
4. The alert service folds signals into alerts. It reads new signals with the same reader the outbox uses, groups them by key, and updates the alert table. There is at most one open alert per key, and the database enforces it. Repeats only raise a count.
5. It decides what to send. It applies mutes and root alerts, then the timing rules (30 s to group, 5 minutes between updates, 4 hours between reminders), then the route for the alert's team.
6. It sends, and records the send. Critical goes to Slack and email. Warning goes to Slack. Info stays in the console. Each Slack alert is one message that is edited in place as the alert changes.
7. People act in the console. They acknowledge ("I have it"), mute with a reason, or resolve with a note. Each is a workflow command, so it is recorded with who and why.
8. Someone watches the watcher. The alert service beats to AWS CloudWatch. If it goes silent for 60 s, AWS raises an alarm. Until that is set up (Section 14), the console shows a red banner.

Event rules and condition rules

Some problems are moments. Others are states. Both end as one alert.

The life of one alert

Group, hide, mute

These three ideas come from Prometheus Alertmanager. We build them inside our alert service.

• Group. One message for many alerts of one rule, such as "3 NDTV files behind". The list is in the console.
• Hide. While a root alert such as "database unreachable" is open, the alerts it causes downstream are kept but not sent. They point at the root through alert.root_alert_id.
• Mute. A person mutes a rule or a subject for a set time, with a reason, for example during planned work. If the alert is still open when the mute ends, it is sent again.

Timing

These match Alertmanager's own defaults. They are fast enough to act on and calm enough not to be ignored.

Heartbeats: silent and stuck

A stopped worker looks exactly like a quiet day. Heartbeats tell the two apart.

• Silent. Every background loop writes its own process_beat row every 10 s: the command service with its sport workers and its import process, the delivery-worker's build and send loops, the stats-worker's stats and timed-job loops, and the alert service. Three missed beats (30 s) raise worker-silent. Because each loop beats on its own, a stuck import is caught even while scoring beside it is fine.
• Stuck. A process can be alive and still do nothing, for example waiting on a hung network call. The beat comes from the work loop and carries progress_at, the time of the last finished item. Work waiting with no progress for 60 s raises worker-stuck, even while the process still beats.
• Database time. Beats and ages are compared with the database clock, never the process clock. A process with a wrong clock cannot fool the check.

One trace id through everything

A trace id is a short id that is stamped on every log line and row that belongs to one change. The bridge accepts a standard W3C traceparent header, or makes one. The id is saved on the command and the outbox row, and carried by the delivery-worker as it builds and sends. Every log line is JSON with the same fields: trace id, command id, subject, workflow, client, outcome and time taken.

One search for a fixture, a command or a trace id then shows the whole path of a change. The alert carries the trace id too, so "show logs" on an alert opens exactly the right lines.

The product numbers

We alert on what clients and scorers feel, not on CPU graphs. CPU and memory are for finding a cause, not for waking someone.

Speed and freshness alerts look at two windows at once: the last hour and the last 5 minutes. Both must be bad. One slow spike does not alert anyone, and a fixed problem stops alerting about 5 minutes after it is fixed, not an hour later. This is the method from the Google SRE workbook chapter "Alerting on SLOs".

A worked example

Low-level design

The design keeps all alert state in Postgres. A command pays one added row, and only when something is wrong. Folding, checking and sending run in a small separate service.

What the code looks like today

These blocks are real code from the repo on 7 Oct 2026, trimmed with .... They show why "none of them tells a person" is true.

The 5 Prometheus rules, with nowhere to go.

# deploy/observability/prometheus/prometheus.yml.tmpl:6-11
global:
  scrape_interval: 15s
  scrape_timeout: 10s

rule_files:
  - /etc/prometheus/alerts.yml

# deploy/observability/prometheus/alerts.yml (one of 5 rules)
  - name: omnium-liveness
    rules:
      - alert: OmniumTargetDown
        expr: up{job=~"omnium-.*"} == 0
        for: 2m
        labels:
          severity: critical

The config loads the rules, so Prometheus works out when they fire. But there is no alerting: block and no Alertmanager, so a firing rule shows only on the Prometheus web page. The 5 rules are IngestPublishLagHigh (over 5 s), IngestPublishBacklog (oldest unpublished over 10 s), OmniumTargetDown, OmniumWorkerUnhealthy and PublishPipelineStalled.

# deploy/observability/prometheus/Dockerfile:20-25
# Retention is a CLI flag (not a config-file field). tsdb is EPHEMERAL (no EFS);
# history resets on task restart ...
CMD ["--config.file=/etc/prometheus/prometheus.yml", \
     "--storage.tsdb.path=/prometheus", \
     "--storage.tsdb.retention.time=15d", \

Prometheus keeps its data inside the container. A restart loses all history.

The Slack alerter for dead letters only logs.

# packages/core/src/omnium_core/publishing/delivery/deadletter.py:137-161 (trimmed)
class SlackDeadLetterAlerter:
    ...
    async def alert(self, entry: DeadLetterEntry) -> None:
        log.warning("slack alert (seam, not sent): %s", self.format_message(entry))

A dead letter is a delivery that failed every retry and was given up. Even when delivery_alerter is set to slack, this class writes a log line and sends nothing. The default is logging anyway (packages/core/src/omnium_core/settings.py:193). Dead letters themselves are kept only in the log by default (delivery_deadletter_backend = "logging", settings.py:189).

A real Slack sender exists, but it is off and nothing calls it.

# packages/core/src/omnium_core/notify/slack.py:64-87 (trimmed)
async def send_slack(text: str, *, settings: Settings | None = None, ...) -> bool:
    """POST ``text`` to Slack asynchronously. Returns True only on 2xx; never raises."""
    cfg = _resolve_settings(settings)
    if not _enabled(cfg):
        return False
    body = _payload(text, username=username or cfg.slack_username, blocks=blocks)
    ...
        resp = await http.post(cfg.slack_webhook_url, json=body)

It posts to a Slack incoming webhook, a fixed URL that can post but never edit a message. slack_enabled defaults to False and slack_webhook_url to empty (settings.py:169-170). A search of the code finds no caller outside its own package and tests. The agreed design changes it into a bot-token client that posts once and then edits the same message (decision N6).

The JSON log formatter, in the public API only.

# packages/api/src/omnium_api/observability/logging.py:28-46 (trimmed)
class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload: dict[str, object] = {
            "ts": self.formatTime(record, "%Y-%m-%dT%H:%M:%S%z"),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        request_id = request_id_var.get()
        if request_id:
            payload["request_id"] = request_id
        for key, value in record.__dict__.items():
            if key not in _STD_ATTRS and not key.startswith("_"):
                payload[key] = value
        ...
        return json.dumps(payload, default=str)

request_id_var is a context variable: a value Python keeps per request, so every log line in that request can read it without passing it around. The middleware sets it at packages/api/src/omnium_api/observability/middleware.py:112. The agreed design moves this formatter into core and adds trace id, command id and subject (decision N7).

The workers log plain text.

# packages/worker/src/omnium_worker/__main__.py:196
# (the same line is in scheduler/__main__.py:157 and stats/__main__.py:215)
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(message)s")

The worker also has a small JSON helper, log_event (packages/worker/src/omnium_worker/observability.py:329-338), used for a few lines such as the delivery heartbeat. So worker logs today are a mix of plain text and JSON, with no shared fields.

The trace id starts fresh at the outbox.

# packages/core/src/omnium_core/jobs_outbox.py:62-70
    event = DomainEvent(
        kind=kind,
        ...
        trace_id=trace_id or uuid7().hex,
        payload=payload or {},
    )

domain_event already has a trace_id column (packages/contract/src/omnium_contract/models/scheduler.py:45). But the scoring pipeline calls jobs_outbox.emit without one (packages/core/src/omnium_core/scoring/pipeline.py:725-731), so a new id is made here. The request id from the API never reaches it.

Queue numbers are never set outside tests.

# packages/worker/src/omnium_worker/observability.py:226-229
def set_queue_snapshot(snapshot: QueueSnapshot) -> None:
    """Publish what a queue looks like now. Called from the loop, not the scrape."""
    with _queues_lock:
        _queues[snapshot.name] = snapshot

The docstring says the loop calls it. A search finds callers only in packages/worker/tests/test_observability.py. So omnium_queue_depth and omnium_queue_oldest_seconds are always empty, and their Grafana panels are blank.

The only heartbeat is a log line.

# packages/worker/src/omnium_worker/delivery.py:501-503
            if now - last_heartbeat >= heartbeat_seconds:
                observability.log_event("delivery.heartbeat", pending=runner.pending_count())
                last_heartbeat = now

It runs every 30 s (HEARTBEAT_SECONDS = 30.0, line 124). Nothing reads it. The scheduler and the import runner have no heartbeat at all.

The new tables

Agreed, to build Agreed design, not in the code yet. All seven tables are new.

-- Every part writes here, in its own transaction. Rows are only ever added.
CREATE TABLE alert_signal (
  id        bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  xid       xid8 NOT NULL DEFAULT pg_current_xact_id(),  -- late-commit-safe reader (Section 4, L9)
  rule      text NOT NULL,                  -- 'delivery-behind'
  subject   text NOT NULL,                  -- 'assignment:<id>', shown as "NDTV S3"
  kind      text NOT NULL CHECK (kind IN ('raise', 'clear')),
  detail    jsonb NOT NULL DEFAULT '{}',
  trace_id  text,
  raised_at timestamptz NOT NULL DEFAULT now()
);

-- One row per problem. At most one open row per key.
CREATE TABLE alert (
  id              bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  key             text NOT NULL,            -- rule || ':' || subject
  rule            text NOT NULL REFERENCES alert_rule (code),
  subject         text NOT NULL,
  severity        text NOT NULL CHECK (severity IN ('critical', 'warning', 'info')),
  state           text NOT NULL CHECK (state IN ('open', 'acknowledged', 'resolved')),
  count           integer NOT NULL DEFAULT 1,
  first_at        timestamptz NOT NULL,
  last_at         timestamptz NOT NULL,
  acknowledged_by uuid, acknowledged_at timestamptz,
  resolved_by     uuid,                     -- NULL with resolved_at set = resolved by itself
  resolved_at     timestamptz, resolve_note text,
  root_alert_id   bigint REFERENCES alert (id),   -- hidden under this alert (A4)
  detail          jsonb NOT NULL DEFAULT '{}',
  trace_id        text
);
CREATE UNIQUE INDEX alert_one_open ON alert (key) WHERE state <> 'resolved';
CREATE INDEX alert_inbox ON alert (state, severity, last_at DESC);

CREATE TABLE alert_event (alert_id bigint REFERENCES alert, at timestamptz, what text, by_user uuid, note text);
CREATE TABLE alert_send  (id bigint PRIMARY KEY, alert_id bigint REFERENCES alert, channel text, target text,
                          status text, attempt int, receipt text, error text, sent_at timestamptz);
CREATE TABLE alert_rule  (code text PRIMARY KEY, kind text, severity text, team text, threshold jsonb,
                          runbook text NOT NULL, enabled boolean NOT NULL DEFAULT true);
CREATE TABLE alert_route (team text PRIMARY KEY, slack_channel text, emails text[]);
CREATE TABLE alert_mute  (id bigint PRIMARY KEY, rule text, subject text, until timestamptz NOT NULL,
                          reason text NOT NULL, by_user uuid NOT NULL);

-- One row per running process, updated every 10 s by its own work loop.
CREATE TABLE process_beat (
  process_id  text PRIMARY KEY,             -- 'delivery:task-3f2a'
  kind        text NOT NULL,                -- one loop: command, sport-worker, build, send, stats, timed-jobs, imports, alerts
  version     text NOT NULL,                -- image tag and code_sha
  started_at  timestamptz NOT NULL,
  beat_at     timestamptz NOT NULL,         -- database time, never the process clock
  progress_at timestamptz,                  -- last item finished
  waiting     integer NOT NULL DEFAULT 0,   -- items queued for this process
  working_on  text
);

What each table is for:

Three details matter most.

• alert_signal.xid holds the id of the transaction that wrote the row. The reader uses it to never skip a row that committed late, the same way the outbox does (Section 4, L9). See the database page.
• alert_one_open is a partial unique index: a unique rule that applies only to rows matching its WHERE. Here it means "one row per key that is not resolved". The database itself makes sure one problem is one alert, even if two copies of the service ran at once.
• process_beat.beat_at is set with now() in the database, never from the process clock.

Raising an alert

Agreed, to build Agreed design, not in the code yet. New file packages/core/src/omnium_core/alerts/api.py.

async def raise_(session: AsyncSession, rule: str, subject: str, detail: dict | None = None) -> None:
    """Add one signal in the caller's transaction. Never updates a row, never waits on a lock."""
    rules.require(rule)                                   # unknown code -> error in tests (A5)
    await session.execute(INSERT_SIGNAL, {"rule": rule, "subject": subject, "kind": "raise",
                                          "detail": detail or {}, "trace_id": trace.current_id()})
    await session.execute(NOTIFY_ALERT)                   # SELECT pg_notify('alert', '')

async def raise_alone(sessions: Sessions, rule: str, subject: str, detail: dict | None = None) -> None:
    """For a failure whose own transaction rolled back, such as a command refused after 5 tries."""
    async with sessions.short() as s:
        await raise_(s, rule, subject, detail)

In plain words:

• rules.require(rule) checks the rule code exists in the registry. A typo fails in tests, not in production.
• The insert only adds a row. It never updates one, so it never waits for a lock.
• trace.current_id() copies the current trace id onto the signal, so the alert can show "show logs".
• pg_notify('alert', '') wakes the alert service at once. This is the same wake-up the delivery-worker's build uses (Section 8).
• raise_alone is for the rare case where the failure is the rollback, such as "command refused after 5 tries". It opens its own short transaction so the alert survives.

A condition rule

Agreed, to build Agreed design, not in the code yet. New file packages/core/src/omnium_core/alerts/checks.py.

@condition("delivery-behind")
async def delivery_behind(s: AsyncSession, t: Threshold) -> list[Breach]:
    rows = await s.execute(sa.text("""
        SELECT s.assignment_id::text AS a,
               max(extract(epoch FROM now() - d.changed_at)) AS behind_s, max(s.failures) AS failures
          FROM delivery_state s
          JOIN feed_document d ON d.doc_key = s.doc_key     -- changed_at: Section 9, E1
         WHERE (d.seq > s.last_seq AND d.changed_at < now() - make_interval(secs => :behind))
            OR s.failures >= :fails
         GROUP BY s.assignment_id"""),
        {"behind": t["behind_s"], "fails": t["failures"]})
    return [Breach(subject=f"assignment:{r.a}", detail={"behind_s": r.behind_s, "failures": r.failures})
            for r in rows]

In plain words:

• A condition rule is a named query in our code. Nobody types SQL into a screen.
• The thresholds (behind_s 60, failures 3) come from alert_rule.threshold, edited in the admin panel.
• An assignment is behind when its file has a newer seq than the client has, and that change is older than the threshold. It is also in trouble after 3 failures in a row.
• feed_document.changed_at is a new column from Section 9 (decision E1). Today the table has updated_at only (packages/contract/src/omnium_contract/models/feeds.py:101). The delivery_state columns last_seq and failures exist today (packages/contract/src/omnium_contract/models/delivery.py:228, 248).
• Every check runs with a 2 s statement limit on its own pool of 2 connections. A slow check is cancelled and cannot load the database.

Heartbeats

Agreed, to build Agreed design, not in the code yet. New file packages/core/src/omnium_core/observability/beat.py.

class Beat:
    """Called by the work loop itself, so a stuck loop stops making progress even if the process lives."""
    def progress(self, working_on: str | None = None, waiting: int = 0) -> None: ...
    async def run(self) -> None:                         # every BEAT_EVERY_S
        # INSERT ... ON CONFLICT (process_id) DO UPDATE SET beat_at = now(), progress_at = ..., waiting = ...
        ...

The work loop calls progress() after each item it finishes. run() writes the row every 10 s. If the loop is stuck, progress_at stops moving while beat_at still moves. That is exactly the "stuck" case.

The alert service loop

Agreed, to build Agreed design, not in the code yet. New service packages/worker/src/omnium_worker/alerts/__main__.py.

It is its own small service, so a crash in the delivery-worker or the stats-worker cannot take alerts down with it. Only one copy works at a time: it holds a Postgres advisory lock (a named lock the app takes with pg_try_advisory_lock), and a second copy waits.

1. Wake. On pg_notify('alert', ''), with a safety read every 1 s.
2. Fold. Read new signals with the outbox reader. Group them by key. Upsert into alert: open a new one, add to the count, reopen within 15 minutes, or resolve on a clear. Write alert_event rows. One statement per batch, measured at 64 ms for 47,709 signals.
3. Check, every 15 s. Run each condition rule. Breaching subjects become raise signals. Open alerts whose subject is healthy again get a clear.
4. Send. For each alert that changed, apply mutes and roots, then timing, then the route. Record the send as "sending", call Slack or email, then record the receipt or the error.
5. Beat. Write its own process_beat row, and send one CloudWatch metric every 10 s. The alarm treats missing data as a breach.
6. No database. If its own database calls fail for 30 s, it sends "database unreachable" straight to Slack and email from memory, at most once per 15 minutes.

The first rules

These come from the promises in Sections 3 to 9. All are seeded into alert_rule with a runbook (what to do).

unexpected-error replaces an outside error-tracking product. Crashes are grouped by error type and place and carry the trace id (decision A12).

Settings

Agreed, to build Agreed design, not in the code yet.

Thresholds per rule (1 s lock wait, 60 s behind, and so on) live in alert_rule.threshold, edited in the admin panel. Who is told about what changes without a deploy. A new client, sport or team is a settings change, not a code change.

Retention

Every run table gets a cleanup job. A nightly job deletes old rows in batches of 10,000 by their time index, table by table, under its own time limit, so there are no long locks.

Today, integration runs, stat jobs and dead letters have no cleanup. The outbox has a 30-day prune (packages/core/src/omnium_core/jobs_outbox.py:75) that nothing calls. Delivery attempts are pruned at 14 days (delivery_history_days, packages/core/src/omnium_core/settings.py:244). CloudWatch log retention is set in the separate infrastructure repo, which was not checked.

Files

The ops.* commands are written like any other workflow command. See how workflow code is written.

Order of work

1. Migration: the new tables, the first rules with their runbooks, and one team route with its Slack channel left empty until it is named.
2. Shared JSON logging in core, switched on in every service. Nothing else changes.
3. Beats from every loop, and the Health page reading them.
4. The alert service on staging with sending off, console only, for one week, to measure the noise. Then Slack on. Then email.
5. Replace the Slack stub and keep dead letters in the database.
6. The trace id from the bridge header through command, outbox, build and send.
7. The nightly cleanup.
8. Remove the 5 Prometheus rules once their checks run; move Prometheus to lasting storage.

Parked AWS steps wait for Section 14 (parked 7 Oct): the CloudWatch alarm on the alert service's beat (A8), email through SES (N6), the Slack token in Secrets Manager, and lasting storage for Prometheus (N8). Nothing is done on AWS without asking first.

Until Section 14: the console inbox and Health page work fully. Slack works with its bot token typed into the admin panel, encrypted like client credentials (Section 9, D9). Email waits for SES. Nothing outside watches the alert service yet, which is a known gap. As a stop-gap, every console screen shows a red banner when the alert service has not beaten for 60 s.

Tests that prove it

Screens

Staff see every alert and the health of every part in the console, in plain words, with the next step written on the alert. All of this is Agreed, to build and not built.

A condition alert has no Resolve button, because it resolves itself when the cause clears.

A scorer never sees system alerts. They see only their own command answers (Section 4), such as "The system failed on this command. Our team has been alerted. Reference 7f3a." The reference is the alert key, so support can find it in one search.

When things go wrong

The rule for every case: an alert can be late, but it is never lost and never silent. When the database itself is the problem, the alert service tells people without using the database.

Decisions

All 22 decisions were agreed on 7 Oct 2026.

Built today, or still to build

Numbers

All measured on a local laptop (Postgres 17.9, database bench_s4, Python 3.12) on 7 Oct 2026, from the Section 11 design doc.

A normal command writes one log line and two numbers, about 12 µs (from the doc's sum of the costs above). Against the 1.48 ms median transaction measured in Section 4, that is under 1%. A command that raises an alert pays 0.2 ms more.

Numbers from the code: Prometheus scrapes every 15 s (prometheus.yml.tmpl:7) and keeps 15 days (Dockerfile:25); the delivery heartbeat is every 30 s (delivery.py:124); FTP sends have a 30 s time limit (settings.py:224).

Read next

• Feeds and delivery: delivery_state, dead letters, and the files the delivery-behind rule watches.
• The database: the late-commit-safe reader that alert_signal uses, and how cleanups avoid long locks.
• Lessons from the Games: the problems found by people looking that this section answers.