Life of a score

What this page follows

One change, through every part, in the order it happens. If you read only one page after The big picture, read this one. Each stop links to the page that explains that part in full.

How to read the tags after each Today line:

How it works

The 13 stops, in one line each:

1. The tap. The scorer app saves the goal on the device, in its outbox, with a key made once.
2. The request. The bridge sends one HTTP command to omnium, with the key and the last event the screen saw.
3. The engine. Lock the match row, check the key and the version, run the sport's rules, save everything in one transaction.
4. The answer. "Accepted, event 57", sent only after the commit.
5. The wake-up. The commit fires a Postgres NOTIFY; one listener per server wakes.
6. Live screens. Scorer B and the console get the new value pushed. Targets: 0.5 s and 1 s.
7. The delivery-worker builds. Finds the files that contain this match, builds them from the main database, and saves a new version only if the content changed.
8. Redis and Google. The new file goes into Redis, write-if-newer. Google's next pull gets it.
9. Client queues. Each client destination sends the file on its own queue, and records proof of delivery.
10. Stats. The stats queue works out the heavier numbers: top scorers, the group table.
11. Fans. The public API answers from Redis, or from the read-only copy, never older than 5 s.
12. The alert service. Watches every stop, and tells one person once if any stop fails.
13. The trace id. One id on every log line and row, from the tap to the file.

A worked example: one goal, stop by stop

Each stop has the agreed design first, then the Today line.

1. The tap: the outbox gives it a key

19:42:10.000. Scorer A taps Goal, picks India, the scorer and the assist, and types 63. The desk shows 1–0 at once, with a small "sending" mark. The agreed target for click to screen is under 0.1 s (V6).

Before anything goes over the network, the shared outbox in the bridge package writes one row to IndexedDB, the browser's own small database. It survives a crash, a reload and a new deploy.

// The outbox row for this goal. Shape from the design doc, Section 7 (new; not in the code).
{
  key: "7f3a2c1e-…",            // crypto.randomUUID(), made once, sent on every retry (C2)
  subject: "fixture:01a1d4e2-…", // one send loop per match, oldest first (C12)
  code: "football.goal",
  input: { entry: "IND", kind: "open_play", scorer: "IND-09", assist: "IND-17", minute: 63 },
  saw: 56,                       // the last play event this screen showed (L17)
  createdAt: "2026-10-07T14:12:10.000Z",
  tries: 0,
  state: "waiting"
}

The key is the most important value on this page. Every retry sends the same key, so the server can tell "this goal again" from "a second goal".

Today Partly built: there is no outbox. The bridge makes a new key on every call unless the caller passes one (packages-ts/omnium-bridge/src/app.ts:332), and nothing resends by itself. A person who taps again after a timeout sends a second command. No football desk exists among today's desks (searched boxing-scoring-ui/src for football, 7 Oct); the football.goal command does exist (packages/flows/src/omnium_flows/football/commands.py:57-68).

2. The request: one command through the bridge

The desk runs inside an iframe of the admin panel. It posts a command message to the panel, and the panel sends the HTTP request with the person's sign-in. The app never sees a token or omnium's address.

POST /workflows/football/subjects/fixture/01a1d4e2-…/commands
traceparent: 00-4bf92f35…-…-01

{
  "code": "football.goal",
  "key": "7f3a2c1e-…",
  "input": { "entry": "IND", "kind": "open_play", "scorer": "IND-09", "assist": "IND-17", "minute": 63 },
  "saw": 56,
  "sentAt": "2026-10-07T14:12:10.000Z",
  "source": "desk-1"
}

Network time from India: about 200 ms for the round trip to US East (Section 1), or 20 to 60 ms to an AWS region in India (Section 4). Both are estimates from the design docs. Which region prod uses was not checked for this page. The clock below uses the 200 ms case, so the request arrives at about 19:42:10.100.

Today Partly built: the route is built: send_command at packages/admin/src/omnium_admin/routers/bridge.py:213-224, body CommandIn at bridge.py:183 with code, input, an optional key, dry_run and source. A scoring workflow goes to _score, which calls pipeline.command (bridge.py:262-297). There is no saw, no sentAt and no traceparent.

3. The engine: key, lock, version, rules, save

The command service (its own ECS service, L15) runs the one road. For an events match, the middle of the road is the step: the saved state plus this one goal gives the new state. This is the whole transaction, statement by statement. It joins the command road from Section 4, the ball transaction from Section 5 and the wake-up from Section 8.

-- 0. Fast path, before any lock: a finished retry is answered at once (L3).
SELECT outcome, answer, payload_hash FROM command WHERE command_id = '7f3a2c1e-…';   -- no row

BEGIN;                                                    -- READ COMMITTED; limits come from the pool (L16)
-- 1. The match lock and its log number, in one read (L1).
SELECT last_seq FROM fixture WHERE id = $match FOR NO KEY UPDATE;                  -- 56
-- 2. The key again, under the lock. This is the check that counts (L3).
SELECT outcome, answer, payload_hash FROM command WHERE command_id = '7f3a2c1e-…';   -- no row
-- 3. The saved state: one row, never the whole log (S1, M1).
SELECT last_seq, code_sha, snapshot FROM scoreboard WHERE fixture_id = $match;
--    saw 56, last_seq 56: nothing missed. code_sha matches the loaded worker (S6).

-- In the football worker process, no SQL: check(state, event), then step(state, event).

-- 4. The writes. Seq 57 is the new log number.
INSERT INTO timeline_item (fixture_id, seq, unit, event_type, payload, sent_at, received_at, ...)
     VALUES ($match, 57, 'event', 'football.goal', $input, $sent_at, now(), ...);
UPDATE scoreboard SET snapshot = $state, last_seq = 57, code_sha = $sha, state_hash = $h, updated_at = now()
 WHERE fixture_id = $match;
INSERT INTO stat_value (fixture_id, subject_id, subject_role, segment, occurred_on, "values", last_seq, ...)
     VALUES (...)                                          -- only the lines this goal changed (M3)
ON CONFLICT (fixture_id, subject_id, subject_role, segment, occurred_on)
  DO UPDATE SET "values" = EXCLUDED."values", last_seq = EXCLUDED.last_seq;
INSERT INTO published_document ... ;                       -- the match document clients read
INSERT INTO domain_event (..., trace_id) VALUES (..., '4bf92f35…') RETURNING id;   -- the outbox note
SELECT pg_notify('omnium_change', $outbox_id::text);       -- delivered only if we commit (B1)
INSERT INTO command (command_id, sender, subject_id, payload_hash, outcome, answer, seq, ...)
     VALUES ('7f3a2c1e-…', 'desk-1', $match, $hash, 'accepted', $answer, 57, ...);
UPDATE fixture SET last_seq = 57 WHERE id = $match;
COMMIT;                                                    -- synchronous_commit = on

What each part does, in plain words:

• The lock is the match's own row. FOR NO KEY UPDATE makes a second command on this match wait, but lets plain reads and inserts of child rows through. Another match never waits. Scorer B's goal, if B taps too, waits here for a few milliseconds.
• The key is checked twice. Once without a lock, to answer most retries at once (0.09 ms at the median, 10 million rows, laptop). Once under the lock, so two copies of one command can never both run.
• The version check is the play level. The sender saw event 56, and the match is at 56. If scorer B's goal had landed first as 57, this command would roll back and answer conflict, with event 57 inside (L17).
• The rules run on the saved state. half_is_open today reads one flag from the state. In the design, the sport's check(state, event) does the same in a worker process that can be killed at a limit (L10, M6).
• One step, not a replay. step(state, event) returns the new state, the lines it changed and any emits. The state stays under 16 KB (M2).
• Everything commits together. The log row, the state, the lines, the published document, the outbox note, the wake-up and the command row. Either all of them are saved, or none.

The worker call is one request and one reply:

{"id": "7f3a", "op": "check_and_step", "sport": "football", "state": {}, "events": [{}]}
{"id": "7f3a", "ok": true, "refusal": null, "step": {"state": {}, "lines": [], "emits": []}, "cpu_ms": 0.08}

The message shape is from the design doc; the bodies are left empty here, and cpu_ms is the design doc's own example value.

Two trips, not a dozen (M9). As written above, the transaction is about a dozen statements; Sections 4 and 5 counted about 9 trips for their shorter versions. The agreed form sends it in 2 trips: one statement that locks and reads the key, the state and the versions together, and one call to a database function that does every write and the commit. On a laptop, the Section 4 command transaction took 1.48 ms at the median and 3.98 ms at the 99th percentile. On prod, about 2 to 3 ms is expected (estimate). The target for one event is under 5 ms of server time at any point in a match (S9).

The goal commits at about 19:42:10.103 on the example clock.

Today Partly built: the live path is pipeline.command (packages/core/src/omnium_core/scoring/pipeline.py:845). It sets two SET LOCAL limits (pipeline.py:841-842), takes an advisory lock on the first 8 bytes of the fixture id (pipeline.py:869), looks for the key in timeline_item (pipeline.py:879, ledger.py:299), reads the whole match log (pipeline.py:897), appends the row, then refolds the whole log again (pipeline.py:686). Football's state action reads the whole log (flows/football/actions.py:34). It writes the scoreboard, the published document (pipeline.py:965) and the outbox note (pipeline.py:971-979). There is no command table, no version check and no NOTIFY. Live scoring desks were not on prod during the Games.

4. The answer: only after the commit

19:42:10.200. The answer reaches the desk. The sending mark turns into a tick for 2 s, then the field goes back to idle. The outbox deletes the row. The agreed target for save to tick is under 1 s (V6).

{ "outcome": "accepted", "seq": 57 }

The design fixes the outcome field and its five values, and an accepted answer carries the new seq. The rest of the body is not fixed yet. The other four answers, and what the desk does with each:

If the answer is lost in a stadium tunnel, the desk resends with key 7f3a2c1e-…. The quick check finds the command row and answers duplicate, with the first answer. One goal, not two.

Today Partly built: the answer can go out before the commit. The commit runs in the teardown of the FastAPI dependency get_write_session (packages/admin/src/omnium_admin/deps.py:28), which FastAPI 0.139 runs after the response is sent (checked on Commands, 7 Oct). Answers have two shapes: HTTP 200 with accepted: false, or an HTTP error from _fail (bridge.py:77-91).

5. The wake-up: NOTIFY, then one read per server

19:42:10.103 plus 0.2 ms. The commit delivers the pg_notify('omnium_change', …). Every server container that keeps a listening connection wakes. Measured on a laptop on 6 Oct: 0.20 ms at the median, 0.53 ms at the 99th percentile, 500 tries.

# Agreed design, not in the code yet: admin/src/omnium_admin/live/listener.py (new). Trimmed.
await conn.add_listener("omnium_change", lambda *_: wake.set())
await on_change()                                  # catch up from the last place first
while True:
    try:
        await asyncio.wait_for(wake.wait(), timeout=5.0)   # safety read every 5 s
    except asyncio.TimeoutError:
        pass
    wake.clear()
    await on_change()                              # one read for any number of wake-ups

The wake-up is only a doorbell. The outbox row is the truth. On a wake-up, each container reads new domain_event rows once, for all its screens, with the late-commit-safe reader: it reads only rows from transactions older than the oldest one still running, so a slow transaction that commits late is never skipped (L9).

The same doorbell wakes the delivery-worker (stop 7), the stats queue (stop 10) and, through its own channel, the alert service (stop 12).

Today Agreed, to build: there is no pg_notify or LISTEN anywhere in packages/*/src. Each open screen polls domain_event once a second with id > cursor (bridge.py:766, bridge.py:924), which can skip a late commit.

6. Live screens: scorer B and the console

The container's shared watches see that the outbox row touches fixture 01a1d4e2-…. The watch "board of this match" is worked out once and its new value is pushed to every screen that watches it. The schedule watch lists this match among its records, so only that one row is worked out and pushed to the console (G6).

event: value   id: <outbox place>   data: {"watchId": "w7", "version": 57, "value": {...}}

Scorer B's screen held the version before and receives the next one (57 here is an example version): no gap, so it applies the value. The goal glows for a second. If a version were skipped, the screen would show "Catching up" and re-read that one record (B5). Scorer B's next tap carries "saw 57", so it can never be saved on top of a view that missed this goal.

Targets (B8): under 0.5 s from the save to scorer B, under 1 s to any other staff screen. These are estimates from the measured parts, to be proven by the Section 12 tests.

Today Partly built: it works, more slowly. The screen's stream finds the row on its next 1 s poll (_POLL_SECONDS, bridge.py:766) and sends only "the area fixture changed". The host waits 250 ms (packages-ts/omnium-bridge/src/host.ts:196), then re-reads every watch that depends on fixture, one after another. From the code's constants: 0.25 to 1.25 s plus the re-reads, which were not measured.

7. The delivery-worker builds: only the touched files, only if changed

The delivery-worker is woken by the same doorbell. It reads the outbox with the same safe reader and asks which files contain this match:

SELECT doc_key FROM feed_dependency
 WHERE record_kind = 'fixture' AND record_id = $match;

In this example two files come back: the result details file and the calendar of this Games. Which files hold a live score depends on each feed's definition; the two here are examples. The medal table, team medals and the rest are not touched.

Then, per file:

1. Wait a moment. A 1 s timer per file gathers a burst. It never runs more than 3 s from the first change (D2), so a steady stream of goals cannot hold a file back.
2. Build from the main database. The delivery-worker calls the feed resolvers directly, in its own process, never through the public API and never from the read-only copy (D1). A copy could be a few milliseconds behind and miss this very goal.
3. Hash the body, without its time stamp. Same as the stored content_hash: stop, nothing is saved or sent. Different: save (D3, E1).
4. Save in one transaction: the body, the new content_hash, changed_at = the goal's commit time (never "now"), the next seq, and the file's feed_dependency rows (E2). Say the new versions are 9120 and 9121.
5. After the commit, put both new versions on every destination queue that wants them, Redis included.

Cost, for scale: hashing the 4.3 MB calendar took 1.74 ms (measured, design doc). Building that calendar took 1.3 s (a comment in the code, api app.py:157-165). The result file is smaller; its build time was not measured. The agreed budget is a build under 3 s and a send under 5 s, inside the 10 s target (D6).

Today Partly built: after the commit, dirty.flush (deps.py:32) fires the invalidation listeners, and the feed listener in the admin service arms a 1 s debounce (feed_debounce_seconds, packages/core/src/omnium_core/settings.py:143) with no maximum wait. The pass then rebuilds all 7 files of each changed Games (publishing/feed.py:351), each one over HTTP through the public API (feed.py:158-194), and each takes a new version number whether it changed or not (feed.py:295).

8. Redis and Google: write-if-newer, then pull

Redis is one more destination, with its own queue, retries and proof (E8). Postgres is written first and Redis right after, because the two cannot share a transaction (D11). The Redis job runs one Lua script:

-- Agreed design, not in the code yet. KEYS[1] = feed:{doc_key}; ARGV = version, etag, gzip body
local cur = redis.call('HGET', KEYS[1], 'v')
if cur and tonumber(cur) >= tonumber(ARGV[1]) then return 0 end
redis.call('HSET', KEYS[1], 'v', ARGV[1], 'etag', ARGV[2], 'gz', ARGV[3])
return 1

It writes version 9120 only if Redis holds something older, so a retry or a second copy can never put an old file back. Measured: 0.30 ms at the median for a 3.3 KB file (laptop, Redis 7 in Docker, 7 Oct).

Google's next pull. Google calls /getfeeds with its secret key (D14) and the ETag of the file it holds. The API server checks the key in memory, with no database read (D13). It asks Redis only for the version: 0.27 to 0.30 ms. The version is newer than its copy in memory, so it fetches the gzip body once, keeps it, and answers 200 with the new ETag. On Google's poll after that, the ETag matches: 304, no body (E11).

Section 1 sets the target for API and push clients at 2 s from the save, for 95 in 100 updates. With the 1 s file timer, the Redis copy is ready a little over 1 s after the save on the example clock (estimate). How often Google polls was not checked.

Today Partly built: every /getfeeds hit reads Postgres four times before any cache (packages/api/src/omnium_api/routers/getfeeds.py:159-175). The Redis cache is a 3 s timer, never refreshed by a rebuild, and off when no REDIS_URL is set. /getfeeds needs no key (packages/api/src/omnium_api/auth.py:42).

9. Client queues: each destination on its own

Each client destination is one row in delivery_assignment, and gets its own queue and task (D4, E3). Versions 9120 and 9121 go on each queue that wants them:

Every transport has a 10 s connect and a 60 s send limit (E4). A queue keeps only the newest waiting version of each file: if a second goal makes version 9135 while 9120 still waits for DailyHunt, 9120 is dropped and 9135 is sent.

After each send, the proof goes into delivery_state in one statement (E5). The columns below are the agreed ones; the statement is a sketch written for this page:

-- Sketch from E5 (columns from Section 9; the design doc gives no SQL for it).
UPDATE delivery_state
   SET last_seq = 9120, last_delivered_at = now(), status = 'sent', failures = 0,
       sent_hash = $hash, sent_bytes = $bytes, receipt = $s3_etag,
       age_ms = (extract(epoch FROM now() - $changed_at) * 1000)::int
 WHERE assignment_id = $ndtv_s3 AND doc_key = $doc_key;

age_ms is the time from the goal's save to the confirmed send: the number the 10 s target is measured on. An operator who is asked "did NDTV get the goal?" reads this row on the client page.

Today Partly built: the delivery worker asks the database who is owed a file every 5 s (delivery_poll_seconds, settings.py:238). A file is owed when its seq is newer (publishing/delivery/dispatcher.py:142), and every rebuild makes a new seq, so all 7 files go to every destination. Sends run in one shared round of 8 that waits for the slowest (publishing/delivery/runner.py:214-224), and SFTP has no time limit (transports.py:436-437). One hanging host stalls every client.

10. Stats: the heavier numbers, right after

The goal's own transaction already wrote the lines it changed: the scorer's and the assist's lines in stat_value (S10, M3). The heavier stats wait for nobody's answer. The stats queue reads the same outbox row with the late-commit-safe reader (D7) and queues the match's stat runs, such as football's football.top_scorers.node and football.table.node. The stats engine (packages/stats) runs them on its Procrastinate queue in Postgres, in lanes, so a long backfill never sits in front of a live match.

If a stat job fails after its tries, the alert service raises stat-job-failed, naming the stat and the match.

How long this takes was not measured; the design sets no separate target for it.

Today Partly built: the scheduler's tail reads domain_event rows with id above its bookmark (packages/stats/src/omnium_stats/triggers.py:65-71), listening to command.accepted, fixture.completed and fixture.reopened (triggers.py:30). It waits 2 s after a pass that found nothing (IDLE_SECONDS, packages/scheduler/src/omnium_scheduler/__main__.py:39). A late commit can be skipped; a 15-minute sweep repairs it. A failed job is tried 3 times (packages/stats/src/omnium_stats/worker.py:87), then writes a stat.failed event that only the Runs screen shows.

11. Fans: Redis first, then the read-only copy

A fan's app asks the public API for the score. The public API reads Redis first (D8). On a miss it reads the read-only copy through Sessions.read(), which uses the copy only while it is at most 5 s behind (DB1, DB3). The goal reached the copy about a millisecond after the commit: 0.90 ms at the median under 5,700 writes a second (laptop, 7 Oct). On AWS, copy lag is not measured yet.

Staff screens, scorers, the delivery-worker and alert checks never read the copy (DB2). A live match's file that misses Redis is read from the main database, never the copy.

Today Agreed, to build: there is one database and no copy. If REPLICA_DATABASE_URL were set, the public API would send all its reads to it (packages/api/src/omnium_api/app.py:50-60), and the feed builder reads through the public API, so client files could go out without the goal. That is why DB5 switches a copy on only after the delivery-worker builds from the main database directly.

12. The alert service: watching every stop

Nobody watches a dashboard to know this goal arrived. The alert service watches for them. Code adds one alert_signal row in the same transaction as a failure (alerts.raise_), and a checker looks at our own numbers every 15 s. Signals fold into one open alert per problem, sent once to Slack, email or the console (A1 to A3).

In the DailyHunt case at stop 9: the checker finds the destination 60 s behind, opens one delivery-behind alert, waits 30 s to group, and sends one message: "DailyHunt SFTP: 2 files behind". When the host recovers, the alert resolves itself.

Today Agreed, to build: alerts reach nobody. Prometheus has 5 rules and no receiver (deploy/observability/prometheus/prometheus.yml.tmpl:10-11), and the dead-letter Slack alerter only writes a log line (packages/core/src/omnium_core/publishing/delivery/deadletter.py:161).

13. The trace id: one search, the whole path

The bridge accepted traceparent with trace id 4bf92f35… at stop 2. That id is saved on the command row and the domain_event row, carried by the delivery-worker when it builds (each feed_document keeps the last 20 trace ids it was built from) and when it sends (delivery_attempt), and stamped on every JSON log line (A9, N7).

The design names the fields every line carries: trace id, command id, subject, workflow, client, outcome and time taken. It does not fix their key names, so the keys below are illustrative:

{"svc": "commands",  "trace_id": "4bf92f35…", "command_id": "7f3a2c1e-…", "subject": "fixture:01a1d4e2-…", "workflow": "football", "outcome": "accepted", "ms": 2.4}
{"svc": "delivery",  "step": "build", "trace_id": "4bf92f35…", "subject": "fixture:01a1d4e2-…", "outcome": "built", "ms": 210}
{"svc": "delivery",  "trace_id": "4bf92f35…", "client": "NDTV S3", "outcome": "sent", "ms": 640}
{"svc": "delivery",  "trace_id": "4bf92f35…", "client": "DailyHunt SFTP", "outcome": "timeout", "ms": 60000}

One search for 4bf92f35 shows the goal's whole path. An alert carries its trace id too, so "show logs" on the DailyHunt alert opens exactly these lines. Writing one JSON log line cost 6.9 µs on a laptop (7 Oct).

Today Partly built: domain_event has a trace_id column, but the id is made fresh at the outbox (packages/core/src/omnium_core/jobs_outbox.py:68) because the scoring pipeline passes none (pipeline.py:971-979). The stats engine carries that id into its runs (triggers.py:87); the bridge, the file builders and delivery do not. Only the public API writes JSON logs throughout, with its own request id; the workers write mostly plain text.

The timeline

From the tap to the client's file, on the example clock. Every time says where it comes from.

A second story: an import after a staff fix

The same match. Full time is 20:35, and the desk's events say India 2, Japan 1. At 20:41 an operator checks the official match sheet: Japan scored a late goal the desk missed. The real result is 2–2. Two minutes later a Scout run arrives from the results site, which still says 2–1.

Today Partly built:

• The console's results.edit_side writes the score and pins result on that side (packages/core/src/omnium_core/workflows/writers/fixtures.py:312-318).
• The import first compares each row with its own last read, not with the database (packages/core/src/omnium_core/integrations/service.py:835-851). If the site's row has not changed since the last run, it is not sent at all.
• If it is sent (say the site's status changed), _update_side finds result pinned, keeps 2–2 and adds 1 to kept_by_hand (packages/core/src/omnium_core/ingest/fixtures.py:1490-1513).
• Scout's 2–1 is kept only in integration_row.raw. There is no compare, no alert and no review item, and no writer reads results_official, so an import can change an official score unless a person pinned it.
• The import holds an advisory lock on the whole event until its job commits (packages/core/src/omnium_core/workflows/engine.py:116), so every console edit on that Games waits for it.

When things go wrong

Every case ends with the goal saved once, and every reader getting it, or a person told. This is the agreed design.

Decisions

The decisions this page leans on, by stop. All are Agreed. Each page under "Where to read more" has the full list.

Built today, or still to build

Checked in the code on 7 Oct 2026, branch feat/result-feeds, commit 76ef1e2.

Numbers

Where to read more