Skip to content

Memory and Persistence

The regulator has two very different kinds of RAM and three very different kinds of permanent storage, and almost every architectural choice in the firmware comes back to using the right one for the job. This page is the map: what lives where, and why.

Two kinds of RAM

The ESP32-S3 module carries a large external RAM chip (8 MB of PSRAM) alongside a much smaller pool of internal RAM (a few hundred kilobytes, much of it consumed by the WiFi stack and task stacks).

The project rule: every large buffer goes to external RAM, allocated with ps_malloc(). Look at the top of setup() in Xregulator.ino — the sensor history rings, telemetry payload buffers, console message queue, IMU sample buffers, and tuning logs are all allocated there, explicitly, from PSRAM. Even the network upload path follows the rule: when the firmware queues a cloud upload, the payload buffer itself lives in PSRAM and only a small pointer-carrying request goes on the task queue.

Internal RAM is reserved for the things that genuinely need it: task stacks, the radio stack, and — most importantly — secure-connection handshakes.

Why internal RAM matters: the TLS trap

Opening a secure connection (a TLS handshake, used for all HTTPS) requires a large contiguous block of internal RAM. Total free memory is not the number that matters — the largest single allocatable block is. As internal RAM fragments over a long session, that largest block can shrink below what the TLS library needs, and at that point HTTPS requests start failing with generic connection errors that look exactly like network problems but are actually memory problems.

The diagnostic is ESP.getMaxAllocHeap() — the largest contiguous internal block currently available. The firmware checks it itself before opening outbound secure connections (see the heap guard at the top of doCloudPOST() in 3_functions.ino), and it should be the first thing you check before attributing an upload failure to the network.

The persistence map

Permanent storage is split across the ESP32's key-value flash store (NVS) and an on-flash filesystem (LittleFS). What goes where is determined by churn and by what must survive a filesystem wipe.

1. User settings → key-value storage (NVS), as strings

Every user-configurable setting — voltage targets, control gains, feature toggles — lives in a dedicated key-value namespace, stored as a string. The access layer is three small helpers in 2_functions.ino: settingExists(), settingRead(), and settingWrite() (plus settingRemove()).

Two conventions to know:

  • Short keys. NVS limits key names to a small number of characters, so long setting names map to short keys through a block of NK_* macros in 2_functions.ino (for example, a long setting name gets a fifteen-character-or-less key). Always go through the macro, never a bare string.
  • Compare before write. settingWrite() reads the current stored value first and returns early if it is unchanged. Re-submitting a settings form with the same values costs zero flash wear.

Settings are loaded once at boot by InitSystemSettings() in 4_functions.ino (missing keys are created with their defaults) and written by the web server's settings handler whenever the user submits a change.

The vessel description — boat dimensions and type, engine, battery bank, alternator, and the regulator's physical mounting geometry — lives in this same store. It is written as one batched record (a single flash commit for all nineteen fields) when the Vessel Info form is saved, and served back to the dashboard as a JSON view built on demand from the live values, not from a stored file.

Historical note: older firmware stored each setting as its own small text file on the filesystem. That scheme is gone. Firmware through 0.0.4 carried a one-time boot sweep (importLegacySettingsFromLittleFS()) that migrated any leftover legacy files into key-value storage and deleted them; every fielded device has long since migrated, so that sweep — and its table of legacy filenames — was removed in 0.0.5. A device still running the old text-file scheme must therefore be upgraded through 0.0.4 first, or it will fall back to compile-time defaults.

2. Self-updating values → a separate key-value path with change detection

Values the firmware writes on its own — lifetime energy totals, engine hours, extrema, power-cycle counts, IMU event counters — use a different namespace and a different mechanism: loadNVSData() restores them at boot, and saveNVSDataFull() persists them (both in 5_functions.ino).

Because these values change constantly, two protections keep flash wear and loop timing under control:

  • Shadow-variable change detection. Every persisted value has a prev_* shadow; the save routine writes only the keys whose live value differs from the shadow, and commits nothing at all if nothing changed. initNVSCache() seeds the shadows at boot so the first save after a reboot doesn't falsely rewrite everything.
  • Timing discipline. A key-value commit can stall the CPU for a long moment during a flash sector erase, so the full save deliberately runs only at safe moments — primarily when the alternator field has just switched off and during the shutdown sequence — never in the middle of active voltage control.

3. The filesystem (LittleFS) → larger blobs only

The filesystem is for things that are genuinely file-shaped:

  • History rings and logs — RAM-resident history buffers are periodically or at-shutdown dumped to backup files and restored at the next boot (see dumpLongTermRing() / restoreLongTermRing() and restoreSensorRingFromLittleFS() in 2_functions.ino), plus the tuning and step-test logs.
  • The web dashboard assets — the compressed (gzip) HTML/JS/CSS bundle is served straight off flash. The web assets live in their own filesystem areas, separate from the user-data area, with a factory copy kept as a fallback alongside the updatable production copy.

There are no configuration documents here. The vessel description was the last one, and it moved to key-value storage — see the note below.

Historical note: the vessel description used to be a JSON file on the filesystem (/vessel_info.json), separate from every other setting. Two things were wrong with that. A filesystem wipe — which the mount recovers from automatically — took the boat's identity, mounting geometry and battery chemistry with it, while the settings that depend on those values survived in key-value storage. And because the config export is built from the settings store, most of the vessel description was invisible to config sharing and to the fleet snapshot. The record now lives in key-value storage with everything else. A one-time boot migration folds any surviving file in and deletes it, so no re-entry is needed on upgrade.

4. WiFi credentials → key-value storage too

Network credentials (station SSID and password, the access-point credentials, the first-boot-complete flag) are ordinary entries in the settings namespace — see the NK_* keys used by setupWiFi() in 3_functions.ino. They are not files, and they migrate through the same one-time legacy import as everything else.

Why the split exists

The user-data filesystem is high-churn — rings, logs, and backups are written constantly — and it is mounted with format-on-fail (see ensureLittleFS() in 5_functions.ino): if the filesystem is ever corrupted, it is silently reformatted so the device always comes up. That tradeoff is acceptable for replaceable history data, but not for user settings. Moving settings, WiFi credentials and the vessel description into key-value storage means a filesystem wipe costs you some history plots, never your configuration.

Flash layout, qualitatively

Without getting into the actual table (which is a build artifact and changes), the flash is divided into:

  • Two firmware areas — a factory image and an updatable production image. The factory image doubles as the recovery installer: over-the-air updates reboot into it, and it downloads, verifies, and writes the new production image. See OTA Updates.
  • A key-value store (NVS) — settings, lifetime counters, learned tables, and update flags.
  • Filesystem areas — the web dashboard assets (factory and production copies) and the user-data filesystem for JSON, rings, and logs described above.

If you need the live layout on a real device, the firmware prints it at boot — see printPartitionInfo() called from setup().