Awesome
ESP32-FlashZ
an arduino library that provides zlib compressed OTA update feature for esp32
Unlike esp8266, esp32's bootloader does not (yet) support compressed image updates. But esp32 has a miniz lib decompressor in ROM, so it can be used to decompress zlib packed stream on the fly during OTA update and write decompressed data to SPI flash. This could be done both for firmware and filesystem image.
I wrote this lib just for fun to get the idea of miniz functions hidden in esp32 ROM. FlashZ
lib is integrated into EmbUI framework as OTA update handler.
The code inspired by esptool that does the same with it's serial flasher stub.
Features
- low code overhead, deco algo is already present in ROM
- both firmware and filesystem compressed images upload supported
- compatible with ESP32 WebServer, autodetect compressed/non-compressed images
- compatible with ESPAsyncWebServer), autodetect compressed/non-compressed images
- stream decompression, i.e. via http client (fetch and flash compressed image from remote URL)
- PlatformIO integration via post_flash.py script that automates on-the-fly compression and OTA upload for your project
- on-the-fly compress/decompress upload from browser example via pako js lib (tnx to @playmiel for contribution)
esp32-flashz
is used in BLEOTA library to upload updates to esp32 via BlueTooth
Requirements
- 32k of heap memory during decompression for dict data
- derives from Arduino's UpdaterClass to perform flash operation on inflated data
- firmware images must be compressed in zlib stream (not a gzip format file!). There are many ways to do this, i.e.
- use pigz tool with
-z
flag - use python's zlib module (check post_flash.py script for Platformio)
- use perl's Compress::Zlib module
- use pako js lib
- use pigz tool with
Check examples to get more idea on how to intergate this lib into platformio projects
Compatibility
Platform | Firmware | Filesystem |
---|---|---|
ESP32 | :heavy_check_mark: | :heavy_check_mark: |
ESP32-S2 | :heavy_check_mark: | :heavy_check_mark: |
ESP32-S3 | :heavy_check_mark: | :heavy_check_mark: |
ESP32-c3 | :heavy_check_mark: | :heavy_check_mark: |
Comparison
some basic tests
OTA update | Origin | zlib |
---|---|---|
esp32, fw, ~1MiB | 9.8 s | 10.1 s |
esp32, fs, ~1.5MiB, 90% sparse | 9 s | 6 s |
esp32-c3, fw, ~1MiB | 9.2 s | 9.1 s |
esp32-c3, fs, ~1.5MiB, 90% sparse | 4.1 s | 2.5 s |
esp32-s2, fw, ~1MiB | 7.5 s | 7.5 s |
esp32-s2, fs, ~1.5MiB, 90% sparse | 4.7 s | 1.8 s |
Using FlashZ lib
FlashZ Lib consists of a low level FlashZ
singleton that derives from a built-in Arduino's UpdateClass class and provides additional methods to handle zlib compressed data. It tries to maintain same API as Update
.
FlashZ::beginz
suposed to be called instead of UpdateClass::begin
call to intialize update procedure. It allocates zlib Inflator structures and dictionary memory. For uncompressed data FlashZ::begin
could be used as usual with UpdaterClass
.
FlashZ::writez
is called to inflate and flash zlib compressed block of data. bool final
flag is used to signal last piece of data input.
FlashZ::writezStream
can read a standart Stream
class objects, decompress and write decompressed stream to flash.
NOTE: total stream length must be known in advance, so that Inflator insures a proper end of stream is reached on decompression.
FlashZ::abortz
or FlashZ::endz
must be called to end the update and release dynamically allocated Inflator memory.
To stich FlashZ
with networking and OTA updates here is a FlashZhttp
class. This is not a complete OTA updater solution but more of a reference implementation example. Any real-life projects could easily implement something similar with more features, bells and whistles.
FlashZhttp
class integrates WebServer or AsyncWebServer file upload feature with FlashZ
low level methods. Also it can initiate streamed download via http client from a remote URL (only plain http).
FlashZhttp
methods includes some heuristic in attempt to autodetect file image format and type, so that it can handle both compressed and uncompressed images transparently. But for compressed file it can't autodetect between firmware and FS image, so it need some metadata to differetiate. This is implemented via additional POST data fields.
Build-time options
By default AsyncWebServer
support is not build into lib, do not want to intorduce dependency for external lib.
To get AsyncWebServer
support, FlashZ
lib must be build with FZ_WITH_ASYNCSRV
flag. This could be done via PlatformIO build_flags. AsyncWebServer
and ESP32 WebServer
support options are mutually exclusive due to some definitions clashing.
See EXAMPLES projects for further details.
By default FlashZhttp
class within the lib includes support for HTTP Client to be able to download and flash an image from a remote URL. It uses Arduino's HTTPClient.h
lib for that, which by default includes SSL support, even if client code does not meant to be using it. SSL support makes a huge impact on resulting firmware image size, it grows in about 100k. If you do not need SSL support in HTTP Client for the sake of reducing resulting image size you can define HTTPCLIENT_NOSECURE
to disable SSL and allow linker exclude SSL-related code from the resulting firmware image.
This flag is available only since Arduino Core v3.0.2. Added in PR #9893.
For previous versions of Arduino core you can define FZ_NOHTTPCLIENT
build flag to completely disable HTTP Client support in this lib and reduce firmware size.
Also you should always specify NO_GLOBAL_UPDATE
build flag for your project to prevent Arduino's UpdateClass creating it's instance by default. FlashZ uses it's own instance of a derived class and default one just wastes your memory (about 180 bytes). See arduino-esp32/pull#8500
On-the-fly compression of uploaded images via pako js lib
An example of implementing firmware updating web page with embedded js code that compresses raw uploaded images on-the-fly could be found in asyncserver-flashz-pakojs.
Integration with PlatformIO
It is pretty easy to integrate PlatformIO with HTTP OTA update via post build scripting. Python's zlib module could be used to compress firmware image after building and http-client module to upload a compressed image to ESP32 board Over-the-Air. See a reference implementation in post_flashz.py example. It relies on FlashZhttp
class methods to process POST form data but could be adjusted easily. Additional platformio.ini
variables are used to set remote address of a board. Uploading compressed firmware/FS is done automagicaly via simple pio run -t upload
. (MCU must be connected to network and reachable).
Using CLI tools for updates
I'm a linux user and prefer to use cli tools for automating tasks rather than web browser. So here are some oneliners for ESP32-FlashZ updating
- compress and upload firmware image to ESP
pigz -9kzc .pio/build/esp32c3/firmware.bin | curl -v http://$ESPHOST/update -F file=@-
- compress and upload File System image to ESP
pigz -9kzc .pio/build/esp32-s2/littlefs.bin | curl -v http://$ESPHOST/update -F "img=fs" -F file=@-
- trigger esp32's firmware self-update from a remote host
curl http://$ESPHOST/update -F "img=fw" -F "url=http://$REMOTE/download/firmware.bin.zz"
- trigger esp32's FS self-update from a remote host
curl http://$ESPHOST/update -F "img=fs" -F "url=http://$REMOTE/download/littlefs.bin.zz"
License
Since I get the idea from a esptool code, this lib inherits esptool's GNU General Public License v2.0