Building the Firmware from Scratch with the Arduino IDE¶
This page is for developers who want to modify the firmware and compile it from source. To simply reinstall or recover a regulator with stock firmware — no development tools involved — use Firmware Reset instead.
Procedure for compiling and flashing the XREG-010 firmware from source with the Arduino IDE. The build is a standard Arduino sketch; the items that require attention are the board settings and the library set, documented below.
Prerequisites¶
- Arduino IDE 2.x with the Espressif board package installed (Boards Manager: "esp32 by Espressif Systems").
- USB-C cable for the first flash.
- Target hardware: ESP32-S3 module with octal-interface external RAM (OPI PSRAM) and 16 MB flash (ESP32-S3-WROOM-1-N16R8). The firmware allocates all large buffers in external RAM (
ps_malloc()throughout), so a board without OPI PSRAM will compile but fail at runtime.
Getting the Source¶
Clone the public mirror:
git clone https://github.com/XEngineeringLLC/XREG-010.git Xregulator
The trailing Xregulator is required, not cosmetic. The Arduino IDE demands that the sketch folder name match the main sketch file, which is Xregulator.ino. Cloning without it gives you a folder named XREG-010, and the IDE will refuse to open the sketch. If you have already cloned, just rename the folder to Xregulator.
Xregulator.ino holds the globals, setup(), and loop(). The numbered companion files (2_functions.ino through 8_functions.ino) hold supporting functions and compile as tabs of the same sketch.
Board Settings¶
Arduino IDE Tools menu settings, as of this writing:
| Setting | Value |
|---|---|
| Board | ESP32S3 Dev Module |
| USB CDC On Boot | Enabled |
| PSRAM | OPI PSRAM |
| Flash Size | 16MB |
| CPU Frequency | 240 MHz |
| Partition Scheme | Custom |
Settings not listed remain at their defaults.
USB CDC On Boot routes Serial output to the USB-C connector. It defaults to Disabled; left that way, the upload succeeds but the serial monitor shows nothing.
The PSRAM setting is mandatory. With it wrong, or on a board without OPI PSRAM, external-RAM allocations fail at runtime and produce symptoms unrelated to memory.
The custom partition scheme is defined by a partitions.csv file in the sketch folder. See Distribution status below.
Required Libraries¶
Forked dependencies
Seven libraries are forks or carry local modifications. Installing the same-named library from the Arduino Library Manager installs the upstream version, which either fails to compile against this code or misbehaves at runtime. This is the most common build failure for this project. Check the table below before installing.
Modified or fork-dependent libraries¶
| Library | Function | Source | Status | What was changed |
|---|---|---|---|---|
| ESPAsyncWebServer | Serves the web dashboard | Community fork at github.com/ESP32Async/ESPAsyncWebServer (formerly under the mathieucarbou account); tested with 3.11.0 |
Fork required — the abandoned upstream is incompatible with current ESP32 cores | Nothing — used exactly as published by the ESP32Async fork |
| AsyncTCP | Network layer under the web server | Same community fork: github.com/ESP32Async/AsyncTCP; tested with 3.4.10 |
Fork required, plus one local edit | In AsyncTCP.h, the default network-task core assignment (CONFIG_ASYNC_TCP_RUNNING_CORE) is changed from -1 (any core) to 0, pinning the network task to core 0. Without the pin, the network task can preempt the alternator control loop on core 1 mid-sensor-read, producing 30–62 ms control stalls. Command-line builds can pass -DCONFIG_ASYNC_TCP_RUNNING_CORE=0 as a compiler flag instead; Arduino IDE builds do not support per-sketch flags and require the header edit. Re-apply after any library update. |
| PID_v1_xeng | Field-control PID loop | Project fork of Brett Beauregard's Arduino PID library v1.2.2 — see Distribution status | Fork required — the firmware calls methods that do not exist in the stock library | Actuator-aware tracking anti-windup: TrackAppliedOutput() reports the post-governor applied output back to the PID, ResetIntegratorTo() provides bumpless mode transfers, SetTrackingGain() / EnableTracking() configure the tracking loop, and accessors expose the individual P/I/D terms and pre-clamp output. Header renamed PID_v1.h → PID_v1_xeng.h. Stock methods are unchanged. |
| BMP388_DEV | Barometric pressure and board temperature sensor | Patched copy of Martin Lindupp's BMP388_DEV — see Distribution status | Patched — do not substitute the upstream release | Hardened I2C error handling. The stock library does not check bus transaction results; the patched I2C layer (Device.cpp) checks every transaction, verifies the expected byte count arrived, drains partial data from the bus on failure, and returns status. The sensor driver (BMP388_DEV.cpp) propagates those failures, so a faulty bus read returns an error instead of invalid pressure/temperature values. |
| VeDirectFrameHandler_xeng | Victron VE.Direct serial protocol | Project fork of Chris Terwilliger's VeDirectFrameHandler (github.com/cterwilliger/VeDirectFrameHandler), itself derived from Victron's reference implementation — see Distribution status |
Fork required — the firmware reads a member the stock library does not have | A frameCounter that increments once per checksum-valid frame, so the application can tell a genuinely new frame from a re-read of the parser's stale public table; plus corrupted-stream bounds fixes (a frameIndex clamp in textRxEvent, truncate-and-terminate on overlong field names, and zero-initialised name/value buffers). Header renamed VeDirectFrameHandler.h → VeDirectFrameHandler_xeng.h. Stock parsing behaviour is otherwise unchanged. |
| NMEA2000_esp32_xeng | ESP32-S3 CAN (TWAI) transport under the NMEA2000 library | Project fork of Svante Karlsson's NMEA2000_twai — see Distribution status. Note: the older, similarly named upstream NMEA2000_esp32 library targets the original ESP32 and does not work on the S3 |
Fork required — stock transmit blocks, and the raw-receive tap does not exist upstream | Transmit is unconditionally non-blocking: stock maps the library's wait_sent flag (set on every fast-packet continuation frame) to a 100 ms wait when the TWAI queue is full, which is exactly what happens with no bus attached — long enough to stall the control loop. The fork returns immediately and lets refused frames fall through to the core library's send-frame retry ring. It also adds a static raw-receive hook (SetRawRxHook), fired for every received frame in ParseMessages() caller context, so the sketch can decode RV-C and proprietary frames the core dispatch cannot deliver. Header renamed NMEA2000_esp32.h → NMEA2000_esp32_xeng.h. The NMEA2000 library itself is used unmodified. |
| LSM6DSOX_xeng | Six-axis motion sensor (IMU) | Project fork of STMicroelectronics' STM32duino LSM6DSOX 2.3.4 (github.com/stm32duino/LSM6DSOX) — see Distribution status |
Fork required — the sketch includes the renamed header | Hardened I2C error handling, the same class of fix as BMP388_DEV above. The stock library's I2C read and write discard the transfer result and report success unconditionally: endTransmission() is ignored and the number of bytes requestFrom() actually delivered is never checked. Every status test in the caller is therefore meaningless over I2C, and a failed read leaves the caller's buffer holding its previous contents rather than being cleared — so a sensor that stops answering keeps yielding its last batch as if it were new data. The fork returns the real result from both directions, bounds the receive loop to the requested length, and refuses a request larger than the 255-byte count the Wire API can express. The SPI path, the class name and the API are unchanged. Header renamed LSM6DSOXSensor.h → LSM6DSOXSensor_xeng.h. |
Standard libraries¶
Install these as published; no modifications.
| Library | Function | Source |
|---|---|---|
| NMEA2000 | NMEA 2000 (marine CAN) message library | Timo Lappalainen, github.com/ttlappalainen/NMEA2000 |
| INA228 | Battery shunt monitor (voltage/current) | Rob Tillaart, github.com/RobTillaart/INA228 |
| ADS1115_lite | Auxiliary analog inputs | Terry Myers, github.com/terryjmyers/ADS1115-Lite |
| OneWire + DallasTemperature | DS18B20 temperature probes | Arduino Library Manager |
| ArduinoJson | JSON parsing and serialization | Benoit Blanchon, Arduino Library Manager |
| TinyGPSPlus | NMEA 0183 GPS sentence parsing | Mikal Hart, Arduino Library Manager. Included but unused — the shipped NMEA 0183 receiver is a hand-rolled parser in 5_functions.ino; the include must still resolve for the sketch to compile |
The remaining includes — WiFi, WiFiClientSecure/mbedTLS, LittleFS, ESPmDNS, DNSServer, NVS, esp_ota_ops — ship with the Espressif board package and require no separate installation.
The Web Dashboard Bundle¶
The browser dashboard is part of the firmware project. Source files live in web_src/ (index.html, script.js, styles.css, and the uPlot charting files). They are not flashed as-is: each file is gzip-compressed and the compressed copies are written to the device filesystem (LittleFS). The web server serves the .gz versions transparently — a request for script.js is answered with script.js.gz and a gzip Content-Encoding header. Page references are unchanged.
Consequence: an edit to a file in web_src/ has no effect on the device until the compressed copies are regenerated and reflashed. Regeneration is a gzip pass over each web_src/ file into the staging folder the filesystem image is built from; there is no bundler or build system. Do not edit the compressed output folder directly — it is overwritten on every rebuild.
Writing the web bundle to the device is a separate step from the firmware upload: the compressed folder is packed into a LittleFS image (mklittlefs) and written with esptool.py to one of the two web-file partitions (factory_fs or prod_fs). That step never re-flashes firmware.
First Flash¶
With the board settings and libraries in place, the first flash is the standard Arduino flow: connect over USB-C, select the serial port, Upload. The serial monitor, at 230400 baud, shows the boot sequence; it ends with a PSRAM after setup: line reporting free and total external RAM. Verify on first boot that the total is non-zero.
USB uploads on a previously updated device
Production devices receive firmware through an over-the-air update system (see Over-the-Air Updates). After a device has taken one over-the-air update, the bootloader prefers the slot that update was written into: a USB upload can complete successfully, the device reboots, and the previous over-the-air firmware is still what runs. If a USB-flashed change does not appear to execute, this is the probable cause. To recover, run the Firmware Reset procedure — it reinstalls firmware over USB regardless of which slot the bootloader prefers. If problems persist, email joe@xengineering.net.
Forcing programming mode¶
The ESP32 can typically be programmed over USB without pulling GPIO0 low — this fallback is rarely needed.
To force programming mode: short Cable 3 pin 8 (GPIO0, brown wire) to GND before powering up, then upload firmware via USB. To reboot into normal operation: release the short and ground Cable 4 pin 13 (RESET, blue/white wire) momentarily, or power-cycle the board. Cable numbering and wire colors are in Data Cables & Pinout.
Forked Libraries and Partition Table — Distribution Status¶
The patched library copies (PID_v1_xeng, VeDirectFrameHandler_xeng, NMEA2000_esp32_xeng, LSM6DSOX_xeng, BMP388_DEV) and the custom partition table (partitions.csv) are not yet packaged for public distribution. Packaging is planned. Until it lands, email joe@xengineering.net for early access to these files.