Skip to content

Memory and Persistence

The regulator has two very different kinds of RAM and three very different kinds of permanent storage, and almost every architectural choice in the firmware comes back to using the right one for the job. This page is the map: what lives where, and why.

Two kinds of RAM

The ESP32-S3 module carries a large external RAM chip (8 MB of PSRAM) alongside a much smaller pool of internal RAM (a few hundred kilobytes, much of it consumed by the WiFi stack and task stacks).

The project rule: every large buffer goes to external RAM, allocated with ps_malloc(). Look at the top of setup() in Xregulator.ino — the sensor history rings, telemetry payload buffers, console message queue, IMU sample buffers, and tuning logs are all allocated there, explicitly, from PSRAM. Even the network upload path follows the rule: when the firmware queues a cloud upload, the payload buffer itself lives in PSRAM and only a small pointer-carrying request goes on the task queue.

Internal RAM is reserved for the things that genuinely need it: task stacks, the radio stack, and — most importantly — secure-connection handshakes.

Why internal RAM matters: the TLS trap

Opening a secure connection (a TLS handshake, used for all HTTPS) requires a large contiguous block of internal RAM. Total free memory is not the number that matters — the largest single allocatable block is. As internal RAM fragments over a long session, that largest block can shrink below what the TLS library needs, and at that point HTTPS requests start failing with generic connection errors that look exactly like network problems but are actually memory problems.

The diagnostic is ESP.getMaxAllocHeap() — the largest contiguous internal block currently available. The firmware checks it itself before opening outbound secure connections (see the heap guard at the top of doCloudPOST() in 3_functions.ino), and it should be the first thing you check before attributing an upload failure to the network.

The persistence map

Permanent storage is split across the ESP32's key-value flash store (NVS) and an on-flash filesystem (LittleFS). What goes where is determined by churn and by what must survive a filesystem wipe.

1. User settings → key-value storage (NVS), as strings

Every user-configurable setting — voltage targets, control gains, feature toggles — lives in a dedicated key-value namespace, stored as a string. The access layer is three small helpers in 2_functions.ino: settingExists(), settingRead(), and settingWrite() (plus settingRemove()).

Two conventions to know:

  • Short keys. NVS limits key names to a small number of characters, so long setting names map to short keys through a block of NK_* macros in 2_functions.ino (for example, a long setting name gets a fifteen-character-or-less key). Always go through the macro, never a bare string.
  • Compare before write. settingWrite() reads the current stored value first and returns early if it is unchanged. Re-submitting a settings form with the same values costs zero flash wear.

Settings are loaded once at boot by InitSystemSettings() in 4_functions.ino (missing keys are created with their defaults) and written by the web server's settings handler whenever the user submits a change.

Historical note: older firmware stored each setting as its own small text file on the filesystem. That scheme is gone. A one-time sweep at boot (importLegacySettingsFromLittleFS() in 2_functions.ino) migrates any leftover legacy files into key-value storage and deletes them, so upgraded devices carry their settings forward automatically.

2. Self-updating values → a separate key-value path with change detection

Values the firmware writes on its own — lifetime energy totals, engine hours, extrema, power-cycle counts, IMU event counters — use a different namespace and a different mechanism: loadNVSData() restores them at boot, and saveNVSDataFull() persists them (both in 5_functions.ino).

Because these values change constantly, two protections keep flash wear and loop timing under control:

  • Shadow-variable change detection. Every persisted value has a prev_* shadow; the save routine writes only the keys whose live value differs from the shadow, and commits nothing at all if nothing changed. initNVSCache() seeds the shadows at boot so the first save after a reboot doesn't falsely rewrite everything.
  • Timing discipline. A key-value commit can stall the CPU for a long moment during a flash sector erase, so the full save deliberately runs only at safe moments — primarily when the alternator field has just switched off and during the shutdown sequence — never in the middle of active voltage control.

3. The filesystem (LittleFS) → larger blobs only

The filesystem is for things that are genuinely file-shaped:

  • JSON documents — for example the vessel description (/vessel_info.json, read in InitSystemSettings()).
  • History rings and logs — RAM-resident history buffers are periodically or at-shutdown dumped to backup files and restored at the next boot (see dumpLongTermRing() / restoreLongTermRing() and restoreSensorRingFromLittleFS() in 2_functions.ino), plus the tuning and step-test logs.
  • The web dashboard assets — the compressed (gzip) HTML/JS/CSS bundle is served straight off flash. The web assets live in their own filesystem areas, separate from the user-data area, with a factory copy kept as a fallback alongside the updatable production copy.

4. WiFi credentials and the interface password → key-value storage too

Network credentials (station SSID and password, the access-point credentials, the first-boot-complete flag) and the dashboard password are ordinary entries in the settings namespace — see the NK_* keys used by setupWiFi() in 3_functions.ino and loadPasswordHash() in 5_functions.ino. They are not files, and they migrate through the same one-time legacy import as everything else.

Why the split exists

The user-data filesystem is high-churn — rings, logs, and backups are written constantly — and it is mounted with format-on-fail (see ensureLittleFS() in 5_functions.ino): if the filesystem is ever corrupted, it is silently reformatted so the device always comes up. That tradeoff is acceptable for replaceable history data, but not for user settings. Moving settings (and WiFi credentials, and the password) into key-value storage means a filesystem wipe costs you some history plots, never your configuration.

Flash layout, qualitatively

Without getting into the actual table (which is a build artifact and changes), the flash is divided into:

  • Two firmware areas — a factory image and an updatable production image. The factory image doubles as the recovery installer: over-the-air updates reboot into it, and it downloads, verifies, and writes the new production image. See OTA Updates.
  • A key-value store (NVS) — settings, lifetime counters, learned tables, and update flags.
  • Filesystem areas — the web dashboard assets (factory and production copies) and the user-data filesystem for JSON, rings, and logs described above.

If you need the live layout on a real device, the firmware prints it at boot — see printPartitionInfo() called from setup().