radar architecture

Radar — Web App Architecture & Build Plan

Status: For build · Rev: v2 (granular) · Owner: Office of the CTO / MocX Scope: Architect the new Radar (formerly Hermes) fleet-intelligence web app — a real-time, performant, “God-view” command center for ~10k electric tricycles (kekes) in Lagos, designed to scale to 100k. Inputs: the Hermes Mapping Architecture Brief (MapLibre + deck.gl + Valhalla, “decouple the look from the truth”) and a verified read of three backends: hermes-java-api, vcu-consumer, rechaj-backend-api. Companion: see radar-kickoff.md for the Claude Code session plan that drives this build.

This is the engineering plan: principles → topology → real-time data plane → frontend rendering → accuracy engine → data stores → performance playbook → capacity → a granular, staged delivery plan (§13) → testing, rollout, team, timeline, and appendices (data contracts, glossary, ADRs).

Where we are today (grounded read)
Architectural principles
Target system topology
The real-time data plane
Frontend architecture (Track A)
Accuracy engine (Track B)
Radar BFF
Data stores
Telemetry pipeline changes
Performance engineering playbook
Scalability & capacity
Observability, SLOs, cost
Security & RBAC
Stages of work — granular
Risks, trade-offs & open questions
Stack & credentials
Testing & quality gates
Rollout, feature flags & migration safety
Team shape & RACI
Timeline & critical path

Appendix A — Wire protocol (FrameCodec) byte spec
Appendix B — Glossary
Appendix C — Decision log (ADR index)

0. Where we are today (grounded read of the three backends)

A principal-grade plan starts from the truth of the current systems, not a greenfield fantasy.

System	Stack	What it does	Hard constraints found
hermes-java-api	Spring Boot 3.4 / Java 21	Read-mostly BFF. Reads a primary Postgres + TimescaleDB (`vcu_telemetry_data`). Redis caches (`hermes:telemetry:cache`, `hermes:active_riders:cache`). `VcuActivityStatus` heartbeat (GREEN<10m / YELLOW / ORANGE / RED / BLACK≥24h). `VcuLocation` (movement + km-today), `VcuLocationHistory` (geofence visit ledger), SoC history via `time_bucket`.	Poll-only — no WebSocket/SSE. Jobs `fixedDelay=10s` (telemetry cache), `30s` (geofence), `5m` (riders), daily cron. No per-role authorization (`anyRequest().authenticated()`). Single-replica scheduling (no ShedLock).
vcu-consumer	NestJS	MQTT (CloudAMQP) → normalize 3 payload dialects → TimescaleDB `vcu_telemetry_data`. Firmware emits ~every 10s moving / 1h heartbeat.	Pure ETL — no derived metrics, alarms, geofence, or Redis publish. No continuous aggregates, no `device_id` index, no retention/compression. Remote-control path is dead code (`publishCommand()` never connects). Firmware parses ~70 signals, publishes a subset, consumer stores fewer (faults, temps, lock dropped).
rechaj-backend-api	Spring Boot, ~2,900 files	System of record: wallets/overdraft, battery-as-collateral loans, SRRR revenue, payouts, swap/retrofit ops, KYC, disputes. Redis pub/sub event bus + inbox-event outbox; Quartz jobs; payments (Paystack/Monnify/OPay).	Events designed for downstream dashboards (`SWAP_REVENUE_EVENT`, `SWAP_SUPPORT_*`). No Kafka. No geofence/polygon model. Unlock exists; explicit immobilize/lock endpoint not found.

Three load-bearing gaps the architecture must close:

No real-time push in any read path → we need a WebSocket spine.
No map-matching → Valhalla (Track B) is net-new in vcu-consumer.
Telemetry store not built for time-series reads at scale (no aggregates/index/retention) and drops the richest signals (faults, temps, lock).

Scope & non-goals. In scope: the desktop command-center web app (primary, 1440+, fluid to large displays) + its real-time + accuracy + data backends. The 390px Field-Technician companion is in product scope but reuses the existing mobile flow and the same BFF/real-time spine — it is not re-architected here (treat it as a thin client of Stage 5/6 APIs). Explicit non-goals: 3D terrain and an orbital globe (Lagos is flat; a dense urban fleet is a street-level problem — use 2.5D tilt + building extrusions instead); a generic GIS/editing tool; replacing rechaj-backend as the system of record (Radar reads it, never owns its data).

1. Architectural principles

Decouple the look from the truth. Display (MapLibre + deck.gl) never touches a raw coordinate; accuracy (Valhalla, server-side) snaps GPS to roads before anything is drawn.
Push, don’t poll. Live state is broadcast at a fixed cadence; the client interpolates to 60 fps. Polling is reserved for cold reads (profiles, history, reports).
The client is a renderer, not a database. It holds only what’s in/near the viewport, as compact typed arrays.
Cost must not scale linearly with the fleet. No paid per-ping APIs in the hot path. Self-hosted Valhalla = free internal call. Fan-out is viewport-scoped.
Binary on the wire, columnar in memory. Positions move as packed ArrayBuffers parsed off-thread into deck.gl binary attributes — zero per-vehicle JS allocation on the hot path.
Backpressure everywhere. Every hop can shed/coalesce; the map degrades (LOD, clustering) rather than freezing.
One BFF, role-aware. A single Radar gateway composes hermes + core + telemetry, enforces RBAC, and shapes payloads. The browser never fans out to five services.

2. Target system topology

┌─────────────────────────────────────────────────────────────────────────────┐
│  BROWSER — Radar (the "Look", Track A)                                        │
│  React shell (imperative, non-reactive hot path)                              │
│   ├─ MapLibre GL canvas  ── Carto Dark Matter / MapTiler dark style           │
│   ├─ deck.gl (interleaved): ScatterplotLayer(vehicles, binary) · TripsLayer   │
│   │     · IconLayer(stations/techs) · HexagonLayer/H3 (demand) · PathLayer    │
│   ├─ Web Workers: protocol decode, Supercluster, dead-reckoning tick          │
│   └─ Data plane client: WebSocket (live) + REST/GraphQL (cold)                │
└───────────────▲───────────────────────────────────────────────▲──────────────┘
        live (binary deltas, viewport-scoped)              cold reads (JSON)
                │                                                  │
┌───────────────┴──────────────────────┐        ┌──────────────────┴─────────────┐
│  RADAR REALTIME GATEWAY (new, Go/Node)│        │  RADAR BFF (extend hermes-java) │
│   · WebSocket fan-out                 │        │   · composes hermes + core +    │
│   · viewport → H3 cell subscriptions  │        │     telemetry; RBAC enforced    │
│   · snapshot + delta encoder (binary) │        │   · money/risk, health, ops     │
│   · presence / heartbeat (Activity)   │        │     aggregations; caching       │
│   · backpressure + coalescing         │        └──────────────────▲─────────────┘
└───────────────▲───────────────────────┘                     reads │
        subscribes to   │  Redis pub/sub (positions + events)        │
┌───────────────┴───────────────────────────────────────────────────┴───────────┐
│  ACCURACY ENGINE + STATE (the "Truth", Track B) — your infra (EKS)              │
│   vcu-consumer (NestJS)  ──►  Valhalla (Meili map-match, ★ new hop)             │
│        ├──────────────► TimescaleDB (raw + matched, aggregates, retention)      │
│        └──────────────► Redis: live geo-state (hash + H3 cell sets) + pub/sub   │
│   rechaj-backend-api ──► Redis pub/sub (wallet/loan/swap/payout events)         │
│   Valhalla graph ◄── OSM Nigeria (Geofabrik) + ★ fleet traces (data moat)       │
└─────────────────────────────────────────────────────────────────────────────────┘

Why a separate realtime gateway, not SSE on hermes-java-api? High-connection fan-out with binary framing + backpressure is a specialized workload. A tiny Go/Node service that only subscribes to Redis and pushes binary frames scales horizontally and keeps the JVM BFF stateless/cacheable.

3. The real-time data plane (the centerpiece)

Design goals: a 10k-vehicle live map that feels 60 fps, costs O(viewers × visible-vehicles) not O(viewers × fleet), and survives a 100k fleet.

3.1 Ingest → match → state

vcu-consumer keeps writing raw pings to Timescale (audit). New: per ping it (a) calls Valhalla to snap to road (Stage 3), (b) writes the matched row to Timescale, (c) updates Redis live state, (d) publishes a compact event.
Redis live state: veh:{id} hash (lat, lng, heading, speed, soc, status, ts, cell) + per-H3-cell membership set cell:{h3} → {ids}. H3 res ~8–9 (≈0.1–0.7 km²) for viewport bucketing. Optional GEOADD for radius queries.
Publish: PUBLISH cell:{h3} <binary delta> so the gateway fans out per cell without scanning the fleet.

3.2 Viewport-scoped subscriptions (the cost lever)

Client sends viewport bbox + zoom (debounced ~150 ms on pan/zoom end).
Gateway maps bbox → visible H3 cells (+1 margin ring), subscribes to exactly those Redis channels, unsubscribes from cells scrolled out.
One cluster view ≈ 200–800 vehicles, not 10k. Below a zoom threshold the client switches to server-side clustered counts per cell — “see the whole city” is cheap.

3.3 Wire protocol — binary, columnar, snapshot + delta

Full byte spec in Appendix A. Summary:

Snapshot on subscribe: columnar ArrayBuffer for all in-view vehicles (fixed-width columns → typed-array views, no JSON).
Deltas at 1–2 Hz: only changed vehicles + removed-ids, same layout.
Events (swap, fault, KaaG, money/risk) on a separate low-volume JSON channel.
Encoding: hand-rolled typed-array columns first (fastest to parse); FlatBuffers is the upgrade if schemas churn. permessage-deflate on.

3.4 Client-side smoothing — dead reckoning + path interpolation

Screen 60 fps; data 1–2 Hz. A Web Worker dead-reckons each vehicle forward along heading@speed — better, along the matched road polyline — into the deck.gl binary buffer.
On each tick, positions are reconciled (eased to truth over ~200 ms) so corrections don’t snap. Smooth motion from cheap data.

3.5 Backpressure & coalescing

One coalesced frame per client per tick window (never per-ping fan-out).
Per-connection send queue + high-water mark; slow client → drop intermediate deltas, resend a fresh snapshot (last-write-wins).
Valhalla calls bounded by a worker pool; on backlog, fall back to “draw raw, mark low-confidence.”

3.6 Fan-out math (sanity check)

10k vehicles, ping/10s ⇒ ~1k matched updates/s. Trivial for Redis.
600-vehicle viewport @2 Hz, ~16 B/row ⇒ ~19 KB/s steady; 200 operators ⇒ ~3.8 MB/s aggregate. One small gateway; shard by cell-range at 10×.

4. Frontend architecture (Track A — the look)

4.1 Rendering stack

MapLibre GL JS (v≥3, WebGL2) owns the basemap (Carto Dark Matter, or bespoke MapTiler — MapLibre-compatible). 2.5D tilt + fill-extrusion buildings. No 3D terrain, no globe (Lagos is flat; globe is a screenshot).
deck.gl interleaved via @deck.gl/mapbox MapboxOverlay so deck layers respect map depth/3D and share one WebGL context.

4.2 Layer catalog (and how each scales)

Layer	deck.gl	Data path	Scaling technique
Vehicles	`ScatterplotLayer`/`IconLayer` (binary attrs)	live WS deltas → worker → buffer	server-side viewport cull + zoom LOD
Live trails	`TripsLayer`	rolling N-sec matched path	time-windowed; in-view only; length by zoom
Demand/density	`H3HexagonLayer`/`HexagonLayer`	server aggregate counts	aggregated server-side; never raw at city zoom
Stations/retrofit	`IconLayer` (pins)	cold REST + low-rate events	small fixed set
Routes/dispatch	`PathLayer`/`TripsLayer`	Valhalla geometry	single active route
Risk/heat	`ScatterplotLayer` (color by score)	money/risk on the vehicle row	reuse vehicle buffer; recolor only

4.3 LOD ladder

z ≤ 11 (city): server-clustered counts (H3 pillars/numbers), no individual vehicles.
z 12–14 (cluster): vehicles as dots, no trails, status color.
z ≥ 15 (street): dots + short matched trails + hover labels; full detail on click.
LOD change switches the subscription type (clustered vs individual), not just the draw — data volume drops with zoom.

4.4 State management & the React boundary

Hard rule: React never re-renders on a position tick. Live buffer lives outside React (worker-owned ArrayBuffer); ticks call deck.setProps/redraw() imperatively.
React owns only chrome (open panel, selected id, filters, role). Selection resolves id → cold profile fetch; the buffer keeps moving underneath.

4.5 Web Workers

Decode worker (binary → shared buffer, transferable, zero-copy) · Cluster worker (Supercluster/H3) · Tick worker (dead-reckoning loop).

4.6 Performance budget

Frame ≤16.6 ms (60 fps) during pan/zoom with 1k visible vehicles + trails.
Time-to-first-map ≤1.5 s; time-to-live-dots ≤2.5 s (mid laptop).
No main-thread long task >50 ms on the hot path. Memory <300 MB steady (LRU-evict off-screen rows).

5. Accuracy engine (Track B — the truth)

Valhalla self-hosted on EKS; routing tiles from OSM Nigeria (Geofabrik), refreshed on schedule.
Online matching in the consumer hot path: feed the last K pings (a short trail) to /trace_attributes; take snapped point + edge + confidence. Cache the per-vehicle trail in Redis.
Batch matching nightly: re-match full-day traces for replay/analytics (higher quality).
Confidence policy: high → snap; low → raw + flag (never lie). A per-cluster match-rate metric exposes where OSM is wrong.
Fallback (HERE/TomTom) only for spot-validation + genuinely blank OSM, server-side, rate-limited. No paid per-ping calls.
Data moat: accumulate matched traces → derive informal/unmapped roads → feed the Valhalla graph (and selectively OSM). Map accuracy becomes a byproduct you own.

6. Radar BFF (cold-read + aggregation gateway)

Recommended: extend hermes-java-api for reads (reuses its Timescale/Redis wiring); add the realtime gateway separately for push.
Responsibilities: compose hermes + rechaj-backend + telemetry into client-shaped payloads (asset profile, money/risk board, station burn-down, fault feed); enforce RBAC (close anyRequest().authenticated() — gate by UserProfileRoleType); cache hot aggregates in Redis with short TTLs.
API style: REST for simple reads; GraphQL for asset-profile composition (one round trip vs five). Cursor-paginate lists.

7. Data stores

7.1 TimescaleDB (`vcu_telemetry_data`) — must be hardened

The biggest backend-perf risk: today no device_id index, no continuous aggregates, no retention/compression → the replay scans raw rows.

Index (device_id, time DESC).
Continuous aggregates 1-min + 1-hour per device (soc/voltage/current/speed/temp avg/min/max).
Compression chunks >7 d (native columnar). Retention drop/downsample raw beyond N days; keep aggregates long-term.
Widen schema for the dropped signals (faults/DTCs, MOSFET/motor temps, lock state, 4G/GPS signal).

7.2 Redis

Live geo-state (per-vehicle hash + per-H3-cell set + optional GEO) — gateway’s snapshot source.
Pub/sub per-cell position channels (new, from vcu-consumer) + existing rechaj-backend event channels forwarded as the events stream.
Presence/heartbeat last-seen → VcuActivityStatus without a DB hit. Cluster Redis at 10×.

7.3 Postgres (canonical)

Riders, kekes, stations, retrofit, clusters, users/roles — read via hermes/BFF; add the role-enforcement read path + any new surfaced columns.

8. Telemetry pipeline changes (vcu-consumer + firmware)

Insert the Valhalla map-match hop (§3.1, §5).
Publish matched positions to Redis (per-cell channels + live hash) — the bridge to the gateway.
Stop dropping signals: persist faults/DTCs, MOSFET/motor temps, lock state, 4G/GPS. (Firmware already publishes 6 MCU alarm flags and parses a 28-flag fault register — one uncomment to publish.)
Build continuous aggregates (§7.1).
Fix the command-down path: the consumer’s publishCommand() is dead; the firmware listens for MECHON/MECHOFF/INVON/INVOFF/AUTO on sub/<device_id>. Wire a real, audited command endpoint so immobilize is end-to-end.

9. Performance engineering playbook

Binary columnar transport (§3.3, App. A) — no JSON hot path; typed-array views; transferable ArrayBuffers.
deck.gl binary attributes — data:{length, attributes:{...}}; never JS-object arrays for 10k points.
Server-side viewport culling + LOD — client never holds the whole fleet; zoom changes data volume.
Server-side clustering (H3/Supercluster) at city zoom — pillars, not points.
Dead-reckoning interpolation — 60 fps from 1–2 Hz; reconcile-ease on ticks.
Imperative render path — React out of the hot loop; deck.gl setProps only.
Coalesced 1–2 Hz broadcast + last-write-wins delta dropping.
Map-match once, server-side — clients draw clean geometry.
Timescale aggregates + index + compression — replay/analytics read rollups.
CDN-tile the basemap — only telemetry is dynamic.
Connection hygiene — one WS/client; heartbeats; resume-with-snapshot; backoff.
Workers for all parsing/clustering/ticking — main thread renders + input only.

10. Scalability & capacity

Fleet 10k→100k: ingest ~1k→~10k matched/s (fine for Redis + a Valhalla worker pool); storage handled by compression + retention. The constant cost is the realtime gateway → scale horizontally by H3 cell-range sharding (each instance owns a band of cells; a routing layer connects clients to the instance(s) owning their viewport).
Operators: fan-out is O(visible vehicles); dense-cell hotspots handled by per-cell pre-rendered frames copied to subscribers.
Valhalla stateless → scale replicas; tile rebuilds offline.
Region: single (eu-west-1); gateway + Redis colocate with the consumer to keep match→broadcast < 10 ms.

11. Observability, SLOs, cost

SLOs: live position freshness P95 < 3 s (ping→screen); 60 fps interaction P95; BFF profile read P95 < 300 ms; match success > 95% in core clusters.
Metrics: ingest rate, match latency/confidence, broadcast lag, per-client queue depth & drops, WS connections, browser RUM frame time, Timescale chunk/compression stats.
Tracing: ping→match→publish→gateway→client ack, sampled.
Cost: basemap (Carto free / MapTiler metered, domain-locked); zero per-ping vendor cost (Valhalla self-hosted); gateway + Valhalla on existing EKS. Map-matching is a fixed infra cost, not a per-fleet line item.

12. Security & RBAC

Close the authorization gap: Radar gates endpoints, fields, map layers by UserProfileRoleType (Field Support / Swap Ops / Board-Exec / Cluster Manager). Role views become real, not cosmetic.
WS auth: short-lived signed token on connect (from BFF login), re-validated on resume; gateway authorizes subscriptions by role + cluster scope.
Command audit: every immobilize/unlock writes an audit row (extend VcuUnlockAuditLog); high-friction confirm; rate-limit.
Frontend keys: browser keys (MapTiler) domain-restricted; snap-to-road keys strictly server-side.

13. Stages of work — granular

Eight stages, aligned to the brief (Glow / Truth / Intelligence) but decomposed to task level. Stages 0–2 ship the “wow” with zero hard accuracy work. Task IDs are S.n; role tags: [FE] frontend · [GW] realtime gateway/infra · [BE] Java/Node backend · [DATA] Timescale/data · [FW] firmware · [QA] test. Effort is rough eng-weeks.

Stage 0 — Foundations & spike · `~1 wk` · [FE]+[GW]

Objective: prove the rendering ceiling and lock the four irreversible tech choices before any feature work.

0.1 [FE] Scaffold radar-web: Vite + React 18 + TypeScript, ESLint/Prettier, Vitest; install maplibre-gl@≥3, deck.gl, @deck.gl/mapbox, @deck.gl/aggregation-layers, h3-js. CI: typecheck + lint + build + bundle-size budget.
0.2 [FE] Render-ceiling spike: render 10k → 100k synthetic vehicles via ScatterplotLayer with binary attributes (Float32Array positions, Uint8Array colors). Add Stats.js + a “load N” dev panel + Long-Tasks PerformanceObserver. Target: 60 fps @10k, ≥30 fps @100k on a mid laptop.
0.3 [FE] Basemap spike: wire Carto Dark Matter style; enable pitch/bearing (2.5D); add fill-extrusion buildings; A/B a bespoke MapTiler dark style. Capture screenshots; ADR-001 basemap.
0.4 [GW] Gateway-language spike: micro-bench Go (nhooyr/websocket or gobwas/ws) vs Node (uWebSockets.js) — 5k idle connections + 2 Hz binary broadcast of a 600-row frame; measure CPU, RSS, p99 send latency. ADR-002 gateway lang.
0.5 [GW] H3 resolution spike: bucket a sample of real Lagos positions at res 7/8/9; measure vehicles-per-cell distribution at each zoom; pick the res that keeps a cluster viewport ≈ 200–800 vehicles. ADR-003 H3 res.
0.6 [FE] Protocol spike: prototype typed-array columnar encode/decode in a worker vs a FlatBuffers schema; measure decode time for a 1k-row frame. ADR-004 wire format.
0.7 [FE] Perf harness: reusable FPS/long-task/RUM overlay component used in every later stage.

Acceptance: 10k dots @60fps & 100k @≥30fps verified; basemap reads as command-center; ADR-001…004 merged. Dependencies: none. Risk: none material — this de-risks everything.

Stage 1 — The Glow (Phase 1) · `~2 wk` · [FE]

Objective: reproduce the God-View visual system on the real MapLibre + deck.gl stack, fed by a mock live source, so the demo ships independent of any backend.

1.1 [FE] <RadarMap> shell: MapLibre + MapboxOverlay interleaved deck.gl; resize observer; pitch/bearing/zoom controls; design-system theme tokens; light/dark parity.
1.2 [FE] Vehicle layer: ScatterplotLayer/IconLayer on binary attrs; status→color (active/warning/offline/immob/known) + risk-band tint; keke vs technician icon; selected-ring.
1.3 [FE] TripsLayer trails: rolling N-second path per vehicle; trail length scales with zoom; mint/teal glow per the brief.
1.4 [FE] Mock live source worker: replays a recorded/seeded trace at 1–2 Hz, emitting the exact FrameCodec format (App. A) so the Stage-2 swap is a one-line source change. This is the contract boundary — get it right here.
1.5 [FE] Dead-reckoning interpolator (tick worker): advance along heading@speed; reconcile-and-ease toward each server frame; pluggable to follow a polyline (Stage 3).
1.6 [FE] React boundary: live buffer outside React; imperative deck.setProps. CI guard: a test asserting a pan/zoom/position-tick causes zero chrome re-renders (render-counter).
1.7 [FE] LOD ladder (§4.3): z-driven dot/trail/label visibility + (mock) clustered-count layer for city zoom.
1.8 [FE] Selection → asset popup + profile chrome (reuse the design-system components already built); cold-read served by a mock.
1.9 [FE] Stations / retrofit / technician IconLayers from mock cold data; the merged “Locations” layer.
1.10 [FE] States: loading, empty-in-view, offline/last-known banner, error boundary.

Acceptance: the current God-View visuals reproduced on the real stack with smooth 1–2 Hz mock motion; render-count guard green; no backend dependency. Dependencies: 0.1, App. A. Risk: React/hot-path coupling → enforced by 1.6.

Stage 2 — Real-time spine · `~2–3 wk` · [GW]+[BE]+[FE] (backend track, parallel with Stage 1)

Objective: stream real fleet positions live to the map, viewport-scoped and binary.

2.1 [GW] Gateway scaffold (chosen lang): WS server, /healthz, Prometheus metrics, structured logs, graceful drain.
2.2 [GW] Redis live-state schema + writer lib: veh:{id} hash, cell:{h3} set, TTL/last-seen; documented in App. A.
2.3 [BE/Node] vcu-consumer publish: on each ping compute H3 cell (h3-js/native), upsert veh:{id}, diff cell membership (move between cell sets), PUBLISH cell:{h3} compact delta. (Raw positions first; matched in Stage 3.)
2.4 [GW] Subscription manager: client viewport bbox+zoom → visible H3 cells (+1 ring); Redis SUBSCRIBE/UNSUBSCRIBE diffing per connection; debounce churn on fast pans.
2.5 [GW] FrameCodec impl (encoder) per App. A: snapshot builder (read cell members → columnar buffer) + delta builder (per-tick changed rows + removed-ids) + versioned header.
2.6 [GW] Coalescing broadcast loop: tick window (500 ms–1 s) accumulates per-cell changes → one frame per client; last-write-wins.
2.7 [GW] Backpressure: per-conn bounded send queue; high-water → drop deltas + flag resync → send fresh snapshot; export queue-depth/drop metrics.
2.8 [GW] Presence/heartbeat: last-seen → VcuActivityStatus; sweep to RED/BLACK; emit status-change events.
2.9 [FE] Data-plane client: connect (with auth token), send viewport (debounced 150 ms), decode frames in the decode worker into the buffer, swap the 1.4 mock for the socket, reconnect-with-snapshot + exponential backoff + jitter.
2.10 [GW] Subscription auth: validate the BFF-issued WS token; authorize cell subscriptions by role + cluster scope.
2.11 [QA] Load test: 200 simulated operators + a replayed fleet; measure freshness P95, gateway CPU/RSS, egress; 1-hour soak; chaos (kill a Redis replica, drop a client).

Acceptance: real positions stream live, viewport-scoped; freshness P95 < 3 s; 200 operators sustained 1 h; reconnect seamless; backpressure verified under a slow-client test. Dependencies: 0.4/0.5 (ADRs), 1.4 (FrameCodec), 2.2. Risk: dense-cell hotspots → per-cell pre-rendered frame shared across subscribers (2.6).

Stage 3 — The Truth / map-matching (Phase 2) · `~3 wk` · [GW]+[BE]+[FE]

Objective: vehicles ride the real road network; geofencing/ETAs unlocked; zero paid hot-path calls.

3.1 [GW] Valhalla on EKS: container + Helm chart; build routing tiles from the OSM Nigeria .osm.pbf (Geofabrik); PVC for tiles; /status health; HPA.
3.2 [GW] Tile-refresh job (scheduled, off-peak) + a documented rebuild/rollback runbook.
3.3 [BE/Node] Online match in vcu-consumer: keep last-K-ping trail per vehicle in Redis; call Valhalla /trace_attributes; extract snapped lat/lng, edge id, confidence.
3.4 [BE] Confidence policy: confidence ≥ θ → snap (use matched); < θ → keep raw + lowConfidence flag; emit per-cluster match-rate metric.
3.5 [BE/DATA] Persist matched row to Timescale (+confidence, edge id); publish matched position (replaces raw on the wire from 2.3).
3.6 [FE] Interpolator upgrade: dead-reckon along the matched polyline (the edge geometry from Valhalla) instead of straight heading — visibly smoother and road-true.
3.7 [GW] Valhalla client worker pool + bounded queue; on backlog, bypass match → publish raw with lowConfidence (backpressure-safe).
3.8 [BE] Fallback adapter (HERE or TomTom) behind a backend endpoint, rate-limited, server-key only; used for spot-validation + blank-OSM cells, never per-ping.
3.9 [BE] Nightly batch re-match of full-day traces (high quality) feeding the Activity replay + analytics.
3.10 [QA] Match-rate dashboard; assert > 95% in core clusters; an automated “off-road jitter” visual regression on a known trace.

Acceptance: vehicles align to roads (no off-road jitter); match-rate > 95% core; zero paid calls in the hot path; fallback only on genuine miss. Dependencies: Stage 2 publish path. Risk: Lagos OSM gaps → low-confidence-as-raw + the trace moat (Stage 7) closes it.

Stage 4 — Data stores hardening · `~1–2 wk` · [DATA] (parallel; unblocks replay/analytics)

Objective: make vcu_telemetry_data fast and bounded; store the dropped signals.

4.1 [DATA] Add composite index (device_id, time DESC) (CREATE INDEX CONCURRENTLY).
4.2 [DATA] Continuous aggregates: cagg_telemetry_1m + cagg_telemetry_1h per device_id (soc/voltage/current/speed/temp avg/min/max) + refresh policies.
4.3 [DATA] Compression policy on chunks > 7 d (segment-by device_id); retention/downsample raw beyond N days.
4.4 [DATA/Node] Widen schema: faults jsonb, mosfet_temp, motor_temp, lock_state, sig_4g, gps_strength; vcu-consumer writes them (consume the firmware fields already on the wire).
4.5 [BE] Re-point Activity replay + SoH curves to the aggregates (raw only at street zoom / short windows).
4.6 [DATA] Migration runbook: chunk-aware, off-peak, reversible; before/after query-latency benchmark on a full-day window.

Acceptance: full-day replay reads aggregates in < 300 ms; storage growth flattened (compression ratio recorded); new signals queryable. Dependencies: none hard (can start immediately). Risk: live-hypertable migration → off-peak, chunk-aware, tested on a staging copy.

Stage 5 — BFF, RBAC & cold reads · `~2 wk` · [BE]+[FE]+[QA]

Objective: every prototyped surface (the Money/Risk, Health, Ops features already designed) backed by a real, role-gated endpoint.

5.1 [BE] BFF endpoints (extend hermes): GET /asset/{id} (compose keke + wallet + loan + telemetry + faults), GET /money-risk/board, GET /station/{id}/burndown, GET /faults/feed, GET /fleet/pulse.
5.2 [BE] RBAC layer: gate endpoints/fields by UserProfileRoleType; the WS token carries role + cluster scope; centralized policy (not per-controller ifs).
5.3 [BE] GraphQL (or a batched REST /asset/{id}?include=) for profile composition — one round trip; cursor pagination on all lists.
5.4 [BE] Redis-cache hot aggregates (short TTL) + event-driven invalidation (subscribe to rechaj-backend events).
5.5 [BE] Events stream: forward rechaj-backend Redis events (wallet/loan/swap/payout) + telemetry faults to the gateway’s events channel; the client renders them in Alerts & faults / Risk board.
5.6 [FE] Replace every mock cold-read with the real endpoints; loading/error/empty per surface; optimistic where safe.
5.7 [QA] Contract tests (schema), an RBAC matrix test (each role sees exactly its slice — no field leakage), profile read P95 < 300 ms.

Acceptance: all designed surfaces live on real data, role-gated; RBAC matrix green; profile P95 < 300 ms. Dependencies: 2 (events), 4 (telemetry reads). Risk: rechaj-backend coupling → read via its events/inbox, avoid deep cross-service joins.

Stage 6 — Command & control · `~1 wk` · [BE]+[FE]+[FW?]

Objective: make immobilize/unlock/KaaG end-to-end, audited and safe (today it’s dead code).

6.1 [BE] Command endpoint → MQTT publish to sub/{deviceId}: MECHOFF/MECHON (immobilize/reactivate), INVON/INVOFF/AUTO (KaaG). Replace/repair the dead publishCommand() (ensure the MQTT client actually connects).
6.2 [BE] Audit + reliability: extend VcuUnlockAuditLog (actor, reason, ts, result); idempotency keys; require VCU ack; surface ack/timeout.
6.3 [BE] Guardrails: RBAC (command perms by role), high-friction confirm token, rate-limit, offline-queue (command holds if VCU is dark, executes on reconnect — matches the prototyped “pending sync”).
6.4 [BE/stream] Theft auto-flag: lock engaged + GPS motion → critical security event onto the events stream.
6.5 [FE] Command UI: confirm modal (reason + acknowledgement), pending/acked/failed states, the audit-trail view; theft banner + one-click immobilize (already prototyped).
6.6 [QA] End-to-end on a bench VCU; safety review sign-off before production enable; feature-flagged rollout.

Acceptance: an operator immobilizes a keke from Radar end-to-end with an audit row + VCU ack; theft auto-flag fires; offline-queue verified. Dependencies: 2, 5; firmware command-topic confirmed (resolve the pub/ vs vcu/pub/ namespace question). Risk: safety → RBAC + confirm + audit + rate-limit + bench sign-off.

Stage 7 — The Intelligence (Phase 3) · `~3 wk` · [BE]+[DATA]+[FE]

Objective: the predictive god view; the map measurably improves itself.

7.1 [BE/DATA] Demand aggregates per H3 cell × time bucket (from matched positions) → served to the client.
7.2 [FE] Demand heat-pillars (H3HexagonLayer, extruded by demand); the city-zoom LOD renders these.
7.3 [BE] Stranding / predicted-offline scoring (heuristic v1, transparent): fuse overdraft + loan delinquency + SoC + depleted-stock at home station + activity gap + telemetry decline → score + reasons (the model we already prototyped); surface as map flags + alerts.
7.4 [DATA] Trace-accumulation pipeline: aggregate matched traces → candidate geometry for unmapped/informal roads → feed the Valhalla graph; selective OSM contribution policy (the moat).
7.5 [FE] 3D building-extrusion polish; lighting/tilt tuning; the “cyberpunk command center” finish.
7.6 [BE] Real Impact/board analytics on aggregates: CO₂ (wire the currently-hardcoded 0.0), rider fuel savings, SRRR, payout health.
7.7 [QA] Model-eval harness: precision/recall of predicted-offline vs actual next-day outcomes; A/B heuristics.

Acceptance: predictive layers live; demand pillars render at city zoom; match-rate in operating clusters improves over time from the moat; board view reads real financials. Dependencies: 3, 4, 5. Risk: model quality → ship transparent heuristics first, iterate with the eval harness.

Sequencing & critical path — see §19.

14. Risks, trade-offs & open questions

Realtime gateway is new infra — the one genuinely new service. Keep it tiny/single-purpose (Redis sub → binary WS push), stateless except connection state.
Online map-matching latency in the hot path — short trail + worker pool + raw-fallback under backpressure; batch re-match for quality.
Single-replica hermes scheduling (no ShedLock) — must be fixed before scaling the BFF horizontally.
MQTT topic-namespace mismatch (firmware pub/<id> vs consumer vcu/pub/#) — confirm the broker rewrite before wiring publish/command.
Firmware change dependency — storing faults/temps/lock and the command path may need a firmware publish change + OTA; sequence with the hardware team.
Open ADRs: Go vs Node gateway (lean Go); GraphQL vs batched REST; H3 res 8 vs 9; how aggressively to contribute traces to public OSM; new radar-web repo vs a module of an existing frontend.

15. Stack & credentials

Component	Role	Cost	Key
MapLibre GL JS (v≥3)	Renderer	free (npm)	—
deck.gl	GPU viz	free (npm)	—
Carto Dark Matter	Basemap style	free, no key	—
MapTiler Cloud (alt)	Bespoke dark style	freemium	`MAPTILER_API_KEY` (domain-locked)
Valhalla (self-host)	Map-matching	infra only	`VALHALLA_URL` (internal)
OSM Nigeria (Geofabrik)	Road graph	free download	—
HERE / TomTom	Fallback only	freemium	`HERE_API_KEY` / `TOMTOM_API_KEY` (server-side)
Radar realtime gateway	WS fan-out	infra (new)	internal
Redis	live state + pub/sub	existing	internal
TimescaleDB	telemetry store	existing	internal

Hot-path rule, restated: the only thing that scales with the fleet is self-hosted compute. No metered vendor call is ever in the live path.

16. Testing & quality gates

Unit: FrameCodec encode/decode (round-trip + fuzz); H3 cell math; dead-reckoning + reconciliation; RBAC policy.
Contract: BFF response schemas (consumer-driven); the WS FrameCodec spec (App. A) shared and version-checked between gateway and client in CI.
Performance (CI-gated): the render-count guard (1.6); a headless deck.gl frame-time benchmark; bundle-size budget; a k6/custom WS load harness (Stage 2.11) run nightly with thresholds (freshness P95 < 3 s, gateway p99 send < 50 ms).
RBAC matrix: automated test that each role sees exactly its endpoints/fields/layers — zero leakage.
Visual regression: map screenshots per zoom/LOD (mock data, deterministic seed) — catches layer/style drift, like the design app’s preview checks.
Resilience/chaos: kill a Redis replica, drop a slow client, backlog Valhalla — assert graceful degradation, not freeze.
E2E: Playwright happy-paths (select asset → profile; immobilize confirm flow against a stub broker; replay scrub).
Accessibility: the chrome (panels, search, command modals, role switcher) meets the design-system a11y bar — keyboard-operable, focus-visible, ARIA roles/labels, WCAG-AA contrast on the dark canvas. The map canvas itself is exempt from pixel-a11y, but every map action has a keyboard/list equivalent (search-to-select, the at-risk/fault lists are click-to-locate). Carried forward from the current prototype’s a11y pass.

17. Rollout, feature flags & migration safety

Feature flags (the existing featureflag module / a client flag service): every stage behind a flag — radar.map.v2, radar.realtime, radar.mapmatch, radar.commands, radar.intelligence. Dark-launch to internal ops first, then a pilot cluster (e.g., Yaba), then fleet-wide.
Strangler pattern: the new Radar runs alongside the current Hermes app; route a cohort of operators to it; the old app is the fallback until parity + soak.
Data migrations (Stage 4) are additive and reversible: CREATE INDEX CONCURRENTLY, new nullable columns, caggs built in the background; no destructive change on the hot table during business hours.
Command enablement (Stage 6) ships disabled; enabled per-role after the bench safety review; offline-queue + audit are prerequisites, not follow-ups.
Kill switch: the gateway can fall back to “last snapshot, no live” and the client to a poll-based read of hermes if the realtime path is unhealthy — never a blank map.

18. Team shape & RACI

Indicative; one engineer can wear two hats on a small team.

Workstream	Lead role	Stages	Skills
Frontend / rendering	Senior FE	0,1,2(client),3(interp),5,7	React, WebGL/deck.gl, workers, perf
Realtime gateway	Backend/infra	0(spike),2,3(Valhalla),6	Go/Node, WS, Redis, EKS
Telemetry & data	Data/BE	2(publish),3(match),4,7	NestJS, Timescale, H3, Valhalla
Platform BFF	Java BE	5,6	Spring, RBAC, GraphQL
QA / perf	QA	every stage exit gate	load testing, Playwright, RUM
Firmware liaison	(hardware team)	4(signals),6(commands)	VCU firmware, MQTT, OTA

RACI highlights: CTO office = Accountable for architecture + the hot-path-cost rule; FE lead = Responsible for the render budget; data lead = Responsible for the Timescale migration; QA = Responsible for stage exit gates; firmware team = Consulted on signals/commands.

19. Timeline & critical path

Wk:        1   2   3   4   5   6   7   8   9   10  11  12  13
Stage 0  [██]
Stage 1      [████████]                 (demo-able after wk3)
Stage 2      [████████████]   ← parallel backend track
Stage 4  [██████]              ← start immediately, no deps
Stage 3              [████████████]      (needs S2 publish)
Stage 5                      [████████]  (needs S2,S4)
Stage 6                              [████]
Stage 7                              [████████████]

Two independent tracks from day one: the Glow (FE: 0→1) ships a demo by ~wk 3 with no backend; the Truth (GW/DATA: 0→2→3, plus 4 immediately) proceeds in parallel.
Hard dependencies: 3 needs 2’s publish; 5 needs 2 (events) + 4 (telemetry); 6 needs 2+5; 7 needs 3+4+5.
First shippable value: end of Stage 1 (the look). First real-time value: end of Stage 2. First “honest map”: end of Stage 3.
Calendar ~12–13 weeks to full Phase-3 with the indicative team; the demo lands in week 3.

Appendix A — Wire protocol (FrameCodec) byte spec

Little-endian. One WebSocket binary message = one frame. Shared between gateway (encoder) and client (decoder); version-checked in CI.

Header (8 bytes)

Off	Type	Field
0	u8	`version` (=1)
1	u8	`frameType` (0=snapshot, 1=delta)
2	u16	`cellEpoch` (subscription generation, for resync)
4	u32	`rowCount`

Vehicle row (16 bytes, columnar — i.e., arrays-of-columns, not array-of-structs) For rowCount rows, columns are laid out contiguously (all ids, then all lats, …) so each column maps to one typed-array view:

Column	Type	Bytes	Notes
`id`	u32	4	vehicle id
`lat`	f32	4	matched latitude
`lng`	f32	4	matched longitude
`heading`	u16	2	degrees ×100 → /100
`speed`	u8	1	km/h
`soc`	u8	1	%
`statusFlags`	u8	1	bit0–2 status enum, bit3 lowConfidence, bit4 kaag, bit5 moving, bit6 unauthorizedMove
`riskBand`	u8	1	0 low / 1 med / 2 high

Delta tail: after the rows, a u32 removedCount then removedCount × u32 ids that left the viewport/cell. Events channel (separate, JSON, low-volume): {type, id, sev, label, code, ts} for swap / fault / thermal / security / KaaG / money-risk changes.

Redis live-state keys: veh:{id} (hash: lat,lng,heading,speed,soc,status,ts,cell) · cell:{h3} (set of ids) · pub/sub channel cell:{h3} (binary delta) · events channel radar:events.

Appendix B — Glossary

Track A / Track B — the display (look) vs accuracy (truth) layers; the core decoupling.
Map-matching / snapping — correcting noisy GPS to the nearest plausible road edge (Valhalla Meili).
H3 cell — Uber’s hexagonal geospatial index; the unit of viewport subscription + clustering.
FrameCodec — the binary snapshot/delta wire format (App. A).
Dead reckoning — advancing a vehicle’s drawn position between server ticks using heading/speed (or matched polyline).
BFF — Backend-for-Frontend; the single gateway the browser reads from.
Cagg — Timescale continuous aggregate (materialized rollup).
VcuActivityStatus — the GREEN→BLACK staleness heartbeat from last-seen.
SRRR — Station/Rider Revenue Run-Rate (rechaj-backend revenue metric).
KaaG — Keke-as-a-Generator (a keke’s inverter powering a home/shop).

Appendix C — Decision log (ADR index)

To be filled during Stage 0; each is a short docs/adr/NNN-*.md.

ADR-001 — Basemap: Carto Dark Matter vs bespoke MapTiler dark.
ADR-002 — Gateway language: Go vs Node for WS fan-out.
ADR-003 — H3 resolution: 7/8/9 against Lagos density.
ADR-004 — Wire format: typed-array columns vs FlatBuffers.
ADR-005 — BFF API: GraphQL vs batched REST for profile composition.
ADR-006 — Repo: new radar-web vs a module of an existing frontend.
ADR-007 — Online vs batch matching split and the confidence threshold θ.