Telemetry Pipeline

Everything the dashboard shows — gauges, plots, settings labels, staleness indicators — arrives over a single one-way push stream from the regulator to the browser. This page explains how that stream is structured, why it looks the way it does, and the one rule you must never break when changing it.

The big picture

The regulator pushes data to the browser using server-sent events (SSE, consumed in the browser with the EventSource API). The dashboard never polls for telemetry: it opens one connection to the /events endpoint and the firmware sends frames as data becomes available. Each frame is a named event on that one connection — there are four data channels plus a text console channel, and the browser registers a separate listener for each name.

Each data frame is a flat, comma-separated string of numbers (positional CSV). The meaning of each number is determined entirely by its position in the line. There are no keys in the payload — instead, both sides carry a matching index map:

• On the firmware side, a C-style list of named positions (an enum per channel in 3_functions.ino, e.g. Csv1Index) defines which value goes in which slot.
• On the browser side, an ordered list of field names (a JavaScript array per channel in web_src/script.js, e.g. CSV1_FIELDS) gives each slot its name back.

The browser splits the line on commas and zips the values together with the name array, producing a normal { name: value } object for the rest of the UI to consume:

firmware:  "<count>,1234,567,..."          // values packed by enum position
browser:   values = frame.split(',')
           data   = zip(CSV1_FIELDS, values)   // names restored by index

Why positional CSV instead of JSON?

The fast channel fires roughly ten times per second, and the slow channels carry several hundred values per frame. Sending JSON would mean repeating every field name in every frame — for the larger channels that's thousands of bytes of keys per frame, plus the cost of building and parsing structured text on a microcontroller. Positional CSV is a fraction of the bytes and a fraction of the formatting cost: the firmware builds each frame with a single formatted-print call (snprintf), and the browser parses it with one split(','). The price is that both sides must agree on field order — which is why the integrity rules below exist.

The four channels

Channel (SSE event name)	Cadence	What belongs here
Live data (CSVData)	Every web-gauges interval — user-adjustable, ~10 Hz by default	Fast-changing numbers the UI shows live: control-loop values, voltages, currents, RPM, anything plotted in real time. A few dozen fields.
Status and diagnostics (CSVData2)	Every 5 seconds (faster during a system-identification test)	Slow-changing state: temperatures, counters, per-session worst values, function-timing statistics, runtime flags. Several hundred fields.
Settings echo (CSVData3)	When a setting changes (a dirty-flag rising edge), plus a 60-second heartbeat	Every user-configurable setting, echoed back from the device so the UI displays confirmed device-side values rather than what the user thinks they submitted. A few hundred fields.
Staleness timestamps (TimestampData)	Every 3 seconds	Last-update times (millis() watermarks) per sensor source, so the UI can grey out inputs that have stopped arriving. A few dozen fields.

Live data (CSVData) drives all real-time dashboard displays. Its period is a user setting (webgaugesinterval), 100 ms by default. If a value changes faster than every five seconds and the user needs to watch it move, it belongs here — and nowhere else, because this is the only channel that keeps up.

Status and diagnostics (CSVData2) carries everything that's interesting but not fast: accumulated statistics, error counters, worst-case timings, hardware health. During an automated plant-characterization run (system identification), it temporarily speeds up so the UI's progress display tracks the firmware state closely.

Settings echo (CSVData3) exists so the UI never has to guess. When the user submits a setting, the request handler flags the settings state as changed (settingsDirty), and the echo channel fires on the next dispatch tick with the device's actual current values. A slow heartbeat (every 60 seconds) covers clients that connect mid-session. Putting a setting in the diagnostics channel instead is a recurring error — it would re-send constantly for no reason and defeat the on-change behavior.

Staleness timestamps (TimestampData) is a watchdog feed, not a data feed. Each field is "when did this sensor last report", letting the UI distinguish a frozen value from a live one. Only add a field here if the input has its own concept of going stale (an external data source like a NMEA field, for example).

Priority gating: one channel per pass

The dispatcher (SendWifiData() in 3_functions.ino) runs every loop pass but sends at most one channel per pass, in fixed priority order:

1. live data (CSVData)        — if its interval elapsed
2. console text messages      — throttled, a few per second max
3. diagnostics (CSVData2)     — only if nothing above sent
4. settings echo (CSVData3)   — only if nothing above sent
5. timestamps (TimestampData) — only if nothing above sent

A sentSomething flag short-circuits the lower priorities once anything has gone out, and a small cooldown spaces consecutive sends apart. The practical consequence for contributors: adding a field never adds per-tick CPU cost. It only makes that one channel's payload slightly longer when its turn comes around. The slow channels naturally fit into the gaps between live-data ticks.

Integrity by construction

Every payload begins with a declared field count — the first number on the line is how many values should follow. On the firmware side this is the sentinel value at the bottom of each channel's enum (CSV1_FIELD_COUNT, CSV2_FIELD_COUNT, CSV3_FIELD_COUNT, TS_FIELD_COUNT), printed as the first item in the format string. The browser checks two things before accepting a frame:

1. The actual number of values matches the declared count.
2. The declared count matches the length of its own field-name array.

On any mismatch the entire frame is rejected — no partial parse, no best-effort. The stream simply goes dark for that channel, with a rate-limited warning in the browser console (schema mismatch / length mismatch).

The failure mode that motivates all this: the formatted-print call (snprintf) that builds each payload has a long format string and a long argument list. If you add an argument but forget its conversion specifier (%d and friends) — or vice versa — snprintf does not crash. It silently drops the trailing fields, the browser sees declared = N, actual = N-1, and rejects every frame. The dashboard looks like a connection problem when it's actually a one-character formatting bug.

The iron rule: three things change together, always

Every time a field is added to or removed from any channel, you must update all three of these in the same change, and verify they agree:

1. The firmware position list — the channel's enum, whose *_FIELD_COUNT sentinel is the declared count.
2. The format string — the conversion specifiers in that channel's snprintf call must number exactly the declared count plus one (the leading count field itself).
3. The browser name list — the matching CSV1_FIELDS / CSV2_FIELDS / CSV3_FIELDS / TS_FIELDS array in web_src/script.js, whose length must equal the declared count.

A mismatch anywhere kills the whole channel, not just your field. This bug has recurred repeatedly in this project's history; count the specifiers, don't eyeball them — and count all specifier types, not just %d.

Scaling convention: integers on the wire

Frames contain only integers. Fractional values are multiplied up before sending and divided back at the display layer. The firmware helper for this is SafeInt(value, scale), which also converts not-a-number and infinity into a sentinel (-1) so a bad sensor reading can never corrupt the CSV line.

Stored / sent as	Displayed as	Conversion at display layer
value × 100 (two decimal places preserved)	float	divide by 100
milliseconds	seconds	divide by 1000
milliseconds	minutes	divide by 60000
milliseconds	hours	divide by 3600000
raw integer	raw integer	none

The rule of thumb: store and transmit in the natural internal unit, convert only where a human reads it. The scale factor used in SafeInt(...) on the firmware side must match the divisor used wherever the browser displays that field.

The client side

The browser opens one EventSource against /events and attaches a listener per channel name (CSVData, CSVData2, CSVData3, TimestampData, plus console for text messages). Each listener follows the same shape:

on frame:
  split on commas, parse numbers
  validate declared count vs actual count vs field-array length  (reject on mismatch)
  build { name: value } via the channel's field array
  route values to gauges / plots / labels

One thing to know before touching this code: each listener callback is its own scope. A helper or local variable defined inside the live-data handler is not visible inside the diagnostics handler. Anything genuinely shared lives on window or at module scope — don't assume a name you saw in one dispatcher exists in another.

The settings-echo display system

Echoed settings don't route through the normal gauge logic. A registration table (the echoUpdates array inside updateAllEchosOptimized() in script.js) maps each echoed field to a label element and a per-setting display transform:

{ key: 'BulkVoltage', id: 'BulkVoltage_echo', transform: v => (v / 100).toFixed(2) }

The element naming convention is the setting name plus an _echo suffix, and the transform function owns all unit conversion for that label (scaled-integer division, milliseconds to seconds, on/off text, and so on). When you add a setting, you add one entry here — the echo system handles change detection and updates only labels whose values actually changed.

Adding a field? Classify first

The most common mistake when extending telemetry is picking the wrong channel. Before plumbing anything, decide what kind of value you have:

• Fast live telemetry — changes quicker than 5 seconds and the user watches it move → live data channel. See Adding Telemetry.
• Slow diagnostic or runtime state — a 5-second cadence is fine, no user input involved → diagnostics channel. See Adding Telemetry.
• User-configurable setting — has a form input, persists across reboot, needs a confirmed echo → settings-echo channel plus persistence. See Adding a Setting.

Then follow the matching guide step by step — including the three-way count verification from the iron rule above.