How can I implement over-the-air (OTA) firmware updates for a fleet of embedded devices?

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust, secure, and fault-tolerant architecture. If an update fails mid-way, you don't want a fleet of expensive, unrecoverable "brick" devices in the field.

Here is a clear, structured roadmap to building a reliable OTA pipeline:

1. Hardware & Memory Architecture (Dual-Bank Flash)
   • Dual-Bank (A/B) Partitioning : Divide your non-volatile storage (Flash) into two separate application partitions: Bank A (Active/Running) and Bank B (Inactive/Staging). The bootloader boots from Bank A while the new firmware downloads safely into Bank B.
   • The Bootloader : Keep your primary bootloader small, secure, and rarely changed. It is responsible for verifying the integrity of the newly flashed image, swapping the active pointer, and handling rollbacks if the new firmware crashes.
   • Fallback/Rollback Mechanism : If the new firmware fails its health check or fails to report a "success" status to the bootloader within a watchdog timeout window, the bootloader automatically reverts to the previous working image (Bank A).
2. Security & Cryptography
   • Cryptographic Signing : Never push unsigned binaries. Sign your firmware images on your build server using a private key (e.g., using Ed25519 or RSA-3072). The embedded device verifies the signature with an embedded public key before applying the update.
   • Transport Security : Use strict TLS (1.3) for downloading images over Wi-Fi, Cellular, or Ethernet to prevent man-in-the-middle tampering or snooping.
   • Encrypted Images : If your firmware contains proprietary algorithms or sensitive keys, encrypt the binary payload itself and decrypt it on-the-fly or inside a secure element/microcontroller flash controller.
3. Backend & Fleet Management Infrastructure
   • Device Registry & Metadata : Maintain a database of your fleet tracking each device's hardware revision, current firmware version, and cryptographic identifier (like a UUID or hardware-bound certificate).
   • Delta Updates : For constrained bandwidth (like Cellular/NB-IoT), generate binary diffs (using tools like bsdiff) so devices download only the changed bytes rather than a full monolithic image, saving data and time.
   • Phased Rollouts (Canary Deployments) : Avoid updating the entire fleet at once. Push updates to a 1% canary group, monitor telemetry for errors or crash loops, and scale up to 10%, 50%, and 100% progressively.
4. Open-Source & Managed Ecosystem Tools
   • MCUboot : A secure bootloader for 32-bit microcontrollers that natively supports dual-image swappable slots and signature verification.
   • AWS IoT Jobs / Azure Device Update : Cloud-managed frameworks that handle device grouping, rollout scheduling, and security for connected fleets.
   • Mender or Balena : End-to-end open-source and commercial OTA update software specifically engineered for Linux-based or RTOS embedded edge devices.

To help tailor this design, let me know:

• What microcontroller or processor family (e.g., ESP32, STM32, ARM Cortex-A Linux) are you using?
• What is your connectivity medium (Wi-Fi, Cellular, LoRaWAN, Ethernet)?

A robust OTA system for an embedded fleet should be designed as secure boot + signed firmware + fail-safe installation + controlled fleet rollout, rather than simply “download a binary and flash it.”

Recommended architecture

                 ┌─────────────────────┐
                 │ Firmware CI/CD       │
                 │ build + test + sign  │
                 └──────────┬──────────┘
                            │
                     signed image
                            │
                 ┌──────────▼──────────┐
                 │ Firmware repository │
                 │ object storage/CDN  │
                 └──────────┬──────────┘
                            │ HTTPS
                            │
       ┌────────────────────▼────────────────────┐
       │              OTA service                │
       │ device inventory / jobs / rollout       │
       └────────────────────┬────────────────────┘
                            │ MQTT/HTTPS
              ┌─────────────┴─────────────┐
              │                           │
        ┌─────▼─────┐               ┌─────▼─────┐
        │ Device A  │      ...      │ Device N  │
        │ OTA agent │               │ OTA agent │
        │ bootloader│               │ bootloader│
        └───────────┘               └───────────┘

1. Put a secure bootloader underneath the application

The bootloader should verify that an image is:

• cryptographically authentic
• intact
• compatible with the hardware
• sufficiently new
• authorized to run

For MCU-class devices, MCUboot is one established option. It supports signed images and upgrade/rollback mechanisms, including a test boot followed by confirmation of the new firmware.mcuboot+1

A typical flash layout is:

+-----------------------+
| Bootloader            |  <-- immutable/protected
+-----------------------+
| Application slot A    |  <-- currently running
+-----------------------+
| Application slot B    |  <-- downloaded update
+-----------------------+
| Persistent data       |
+-----------------------+

This A/B scheme is extremely valuable. Don't overwrite the only known-good application image while performing an OTA update.

2. Sign firmware in CI/CD

Your build pipeline should produce something like:

firmware.bin
firmware.manifest
firmware.sig

The manifest might contain:

{
  "product": "sensor-v3",
  "version": "2.7.4",
  "hardware": ["rev-c", "rev-d"],
  "size": 483920,
  "sha256": "...",
  "security_counter": 42
}

Then sign the image/metadata with a release signing key.

The device should have only the public verification key (or a chain of trust); the private signing key should never exist on the device.

Also implement anti-rollback. Otherwise an attacker who obtains an old but validly signed vulnerable firmware image could potentially downgrade a device. MCUboot, for example, supports version/security-counter-based downgrade prevention.GitHub

3. Have the device ask whether an update exists

Don't necessarily push a several-megabyte binary directly over your command channel.

A device can periodically report:

{
  "device_id": "abc123",
  "firmware": "2.7.3",
  "hardware": "rev-c"
}

The server responds:

{
  "update": true,
  "version": "2.7.4",
  "url": "...",
  "sha256": "...",
  "size": 483920
}

The device then downloads the image, preferably over HTTPS.

For constrained devices, MQTT-based block transfer can also make sense. AWS's OTA implementation, for example, supports HTTP and MQTT delivery and separates notification, downloading, cryptographic verification, and installation.AWS Documentation+1

4. Download into the inactive slot

The device should:

1. Check that enough storage is available.
2. Download into slot B.
3. Verify the complete image hash.
4. Verify the firmware signature.
5. Verify hardware/product compatibility.
6. Verify version/security counter.
7. Mark slot B as a pending/test image.
8. Reboot.

Critically, don't erase slot A until the new firmware has successfully booted.

5. Make the first boot a trial

The bootloader/application state machine should look approximately like:

             update available
                    │
                    ▼
             Download image
                    │
                    ▼
              Verify image
                    │
                    ▼
             Boot new image
                    │
              ┌─────┴─────┐
              │           │
          self-test OK   crash/watchdog
              │           │
              ▼           ▼
          CONFIRM       ROLLBACK
              │
              ▼
          normal boot

The application should confirm the new firmware only after passing meaningful health checks, such as:

• initialization succeeds
• flash/filesystem is healthy
• communications work
• critical peripherals respond
• configuration can be loaded
• watchdog operates normally

MCUboot explicitly supports this test/confirm/revert pattern.GitHub

6. Treat fleet deployment separately from firmware delivery

This is where many OTA implementations become dangerous.

Don't immediately deploy:

10,000 devices → firmware 2.7.4

Instead use staged rollout:

1–5 internal devices
       ↓
10 devices
       ↓
1% of fleet
       ↓
10%
       ↓
25%
       ↓
50%
       ↓
100%

Automatically pause the rollout if metrics deteriorate.

Track things such as:

download_success_rate
install_success_rate
boot_success_rate
rollback_rate
watchdog_resets
device_disconnects
battery_level
firmware_version

Cloud fleet-management systems such as AWS IoT Device Management provide rollout controls, failure thresholds, device groups, and monitoring for this purpose.Amazon Web Services, Inc.+1

7. Make updates resumable

Embedded devices will lose connectivity.

Don't assume:

download 8 MB
             ↓
connection stays alive
             ↓
success

Instead download in chunks:

chunk 0 ✓
chunk 1 ✓
chunk 2 ✓
chunk 3 ✓
connection lost
       ↓
reconnect
       ↓
resume at chunk 4

You can store a small download-progress record in persistent storage.

For cellular devices especially, also consider:

• bandwidth limits
• battery state
• data costs
• update windows
• maximum retry count

8. Separate firmware identity from device identity

Every device should have a unique cryptographic identity, for example:

device certificate/private key
             ↓
       authenticated
             ↓
        OTA service

The OTA service should know:

device ID
hardware revision
current firmware
region
deployment group
last successful update
last failed update

That lets you say:

Deploy firmware 2.7.4 only to sensor-v3, hardware revisions C/D, excluding devices currently reporting low battery. This is much safer than broadcasting a firmware URL to every device.

9. Design for power loss

Assume the device loses power at every possible point:

during download
during flash write
after flash write
during reboot
during first boot
during confirmation

Your state machine should still result in either:

known-good firmware

or

new firmware + ability to revert

Never require the network to recover a device after a failed update. The recovery mechanism must be local to the device.

10. Keep application data separate from firmware

Don't casually erase:

configuration
calibration
device identity
keys/certificates
user data

when installing firmware.

Use separate flash partitions or a carefully designed migration mechanism:

Bootloader
Firmware A
Firmware B
Device credentials
Configuration
Application data
OTA metadata

If firmware 2.7 changes the configuration schema, make migrations explicit and preferably reversible.

A practical technology stack

For a typical MCU product, one reasonable stack is:

Layer	Example
Bootloader	MCUboot
RTOS	Zephyr / FreeRTOS
Transport	HTTPS, optionally MQTT
Firmware storage	S3/object storage + CDN
Fleet management	AWS IoT Jobs or equivalent

MCUboot is OS-independent and supports several embedded operating systems and MCU platforms.mcuboot

If you're already using AWS IoT, AWS's OTA Update Manager can create OTA updates targeting device groups, deliver via MQTT/HTTP, and integrate signing and IoT Jobs.AWS Documentation

The key security rule

The most important principle is:

Never trust the update server alone. The device itself must cryptographically decide whether an image is allowed to execute. Even if your cloud account, CDN, network, or update metadata is compromised, an attacker should not be able to make the device execute an unsigned or unauthorized firmware image.

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52/nRF53), RTOS/Linux, connectivity (Wi-Fi/cellular/LoRaWAN), and approximate fleet size, I can lay out a concrete flash layout, bootloader state machine, signing scheme, cloud architecture, and implementation skeleton for that platform.

A production OTA system is best designed as a secure update pipeline + a fail-safe boot/update mechanism + fleet rollout controls. The exact implementation depends heavily on whether your devices run embedded Linux or a microcontroller RTOS.

1. Use this architecture

                    ┌─────────────────────┐
                    │   CI/CD build       │
                    │ compile + test       │
                    │ sign firmware       │
                    └──────────┬──────────┘
                               │
                         signed artifact
                               │
                    ┌──────────▼──────────┐
                    │ Artifact storage    │
                    │ HTTPS / object      │
                    │ storage/CDN         │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ OTA control plane   │
                    │                     │
                    │ targeting           │
                    │ rollout percentage  │
                    │ health monitoring   │
                    │ rollback            │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ Device OTA agent    │
                    │                     │
                    │ download           │
                    │ verify signature    │
                    │ install             │
                    │ reboot              │
                    │ health-check        │
                    └──────────┬──────────┘
                               │
                     ┌─────────▼─────────┐
                     │ Secure bootloader │
                     │ A/B slots         │
                     │ rollback          │
                     └───────────────────┘

A service such as aws.amazon.com provides the fleet-management/job portion, while alternatives such as Mender provide both server and device-side OTA infrastructure. AWS IoT Jobs, for example, can send jobs instructing devices to download/install firmware and report their status.AWS Documentation+1

2. Make the firmware cryptographically signed

Don't rely on HTTPS alone.

Your release pipeline should produce something like:

firmware.bin
manifest.json
signature

The manifest might contain:

{
  "product": "sensor-v2",
  "hardware_revision": "B",
  "version": "3.7.2",
  "size": 524288,
  "sha256": "...",
  "min_bootloader": "2.1.0",
  "security_counter": 42
}

The device should:

1. Download the artifact over TLS.
2. Verify its hash.
3. Verify the manufacturer's digital signature using a public key embedded/protected on the device.
4. Check hardware compatibility.
5. Check that the version/security counter is acceptable.
6. Only then install it.

For example, MCUboot supports signed images and downgrade prevention using version/security-counter mechanisms.GitHub

Keep the signing private key off the device and preferably out of ordinary developer workstations. Your CI/CD release process should be the only thing authorized to produce production-signed firmware.

3. Use A/B firmware partitions

This is probably the most important reliability feature.

For a Linux device:

Flash
┌─────────────────────┐
│ Bootloader          │
├─────────────────────┤
│ Root filesystem A   │ ← currently running
├─────────────────────┤
│ Root filesystem B   │ ← install update here
├─────────────────────┤
│ Persistent data     │
└─────────────────────┘

The device downloads the new image into the inactive partition. It doesn't overwrite the currently working firmware.

Then:

A running
   ↓
download firmware
   ↓
verify signature/hash
   ↓
install into B
   ↓
boot B
   ↓
health check
   ↓
mark B good

If power disappears during the download or installation, A remains intact.

If B boots but crashes repeatedly, the bootloader should automatically return to A.

This is the model used by systems such as Mender and RAUC; Mender explicitly uses redundant A/B partitions for fail-safe OS updates, while RAUC supports atomic redundant-system updates and cryptographic bundle verification.Mender Docs+1

For an MCU, a bootloader such as MCUboot can provide the equivalent two-slot mechanism. Mender's Zephyr integration, for example, combines a device-side client with MCUboot's two-slot update mechanism and rollback behavior.Mender Docs

4. Separate "download" from "activate"

Your OTA agent should behave roughly like this:

if (!update_available())
    return;

download_to_inactive_slot();

if (!verify_hash())
    fail();

if (!verify_signature())
    fail();

if (!compatible_with_hardware())
    fail();

if (!version_is_allowed())
    fail();

mark_slot_pending();
reboot();

After reboot:

if (self_test_passed() &&
    peripherals_working() &&
    application_started()) {

    mark_slot_good();

} else {
    reboot_into_previous_slot();
}

Don't consider an update successful merely because the binary was written successfully.

Successful OTA means the new firmware boots and passes meaningful health checks.

5. Build fleet rollout controls

Never go directly from:

version 3.7.1 → 100% of devices

Instead:

Internal test devices
       ↓
1%
       ↓
5%
       ↓
25%
       ↓
50%
       ↓
100%

At each stage monitor things such as:

• boot failures
• rollback rate
• crash/watchdog rate
• connectivity
• battery consumption
• application-specific health metrics
• update completion rate
• unexpected device resets

Then automatically stop the rollout if failure rates exceed your threshold.

AWS IoT Device Management, for example, supports fleet grouping, rollout velocity and failure thresholds for OTA deployments.Amazon Web Services, Inc.+1

6. Give every device useful metadata

Your OTA service should know at least:

device_id
hardware_model
hardware_revision
current_firmware
bootloader_version
region
environment
last_seen
battery_level
update_status

This lets you target:

hardware_model == "sensor-v2"
AND hardware_revision == "B"
AND current_firmware < "3.7.2"
AND environment == "production"

rather than accidentally sending a firmware image intended for one hardware revision to the entire fleet.

AWS's continuous OTA mechanism, for example, supports targeting devices using metadata attributes.AWS Documentation

7. Design for bad networks

The update process should tolerate:

• Wi-Fi/cellular disconnections
• device power loss
• partial downloads
• server failures
• intermittent connectivity
• devices being offline for weeks

Useful techniques include:

• resumable downloads
• chunked downloads
• checksums
• download-to-inactive-slot
• retry with exponential backoff
• randomized polling/jitter
• CDN/object storage for firmware
• update bandwidth limits
• persistent update state

Don't have 50,000 devices all wake up at exactly midnight and request a 20 MB image.

8. Protect against rollback attacks

There's an important distinction between:

"Rollback automatically if the new firmware crashes." and:

"Allow anyone to install an old, vulnerable firmware." You want the first but generally not the second.

Use a monotonically increasing security counter or equivalent anti-downgrade mechanism. MCUboot, for example, supports security counters for preventing installation of images below the trusted version.GitHub

A typical policy is:

current security counter = 42

image counter 41 → reject
image counter 42 → policy-dependent
image counter 43 → accept

9. Don't forget the bootloader

A common OTA mistake is designing a beautiful application update system while making the bootloader impossible to recover.

Treat these separately:

Immutable / recovery bootloader
             ↓
      OTA-capable bootloader
             ↓
       application A/B

Ideally you have a protected recovery mechanism—such as a factory image, ROM bootloader, hardware recovery interface, or redundant bootloader strategy—so a failed bootloader update doesn't permanently brick the device.

For embedded Linux, frameworks such as RAUC and Mender can handle substantial portions of this update architecture.GitHub+1

10. Choose the implementation based on your device

Embedded Linux

A practical stack could be:

Yocto/Buildroot
      +
U-Boot
      +
RAUC or Mender
      +
HTTPS/object storage
      +
OTA management server

Mender is particularly oriented toward fleet management and supports managed deployments, while RAUC is a lightweight update client/bundle system.Mender Docs+1

MCU / RTOS

A typical architecture is:

FreeRTOS / Zephyr
       +
MCUboot
       +
A/B flash slots
       +
HTTPS/MQTT
       +
OTA service

For MCUboot-based systems, the bootloader performs image validation and manages the slot transition/rollback.Mender Docs

Cloud-managed fleet

If you're already heavily invested in AWS, a reasonable architecture is:

CI/CD
  ↓
Code signing
  ↓
S3
  ↓
AWS IoT Jobs
  ↓
Device OTA agent
  ↓
A/B bootloader

AWS's OTA agent model explicitly expects the device to download the firmware, verify its digital signature, install it, and report the job result.AWS Documentation

11. A good minimum production design

If I were designing a new fleet today, I'd make these requirements non-negotiable:

The key principle is: an OTA system shouldn't merely make firmware downloadable; it should make an interrupted, malicious, incompatible, or broken update fail safely.

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52, Raspberry Pi), OS/RTOS, connectivity (Wi-Fi/cellular/etc.), approximate fleet size, and whether you have a cloud backend, I can sketch a concrete architecture and implementation—including partition layout, bootloader flow, manifest format, and device-side OTA code.

Then automatically stop the rollout if failure rates exceed your threshold.

AWS IoT Device Management, for example, supports fleet grouping, rollout velocity and failure thresholds for OTA deployments.

6. Give every device useful metadata

Your OTA service should know at least:

Implementing over-the-air (OTA) firmware updates for an embedded fleet requires a robust dual-bank memory architecture, cryptographic signing, and a reliable device management backend to prevent bricking devices in the field.mender.io+2

Core Implementation Architecture

• Dual-Bank (A/B) Partitioning : Allocate your non-volatile flash memory into two separate application slots (Bank A and Bank B). The bootloader executes the active image while the OTA client downloads the new firmware binary into the inactive bank, executing an atomic switch upon success.youtube.com+1
• A Fallback/Rollback Mechanism : Program your primary bootloader to verify the integrity and health of a newly booted image. If the application crashes repeatedly or fails a health check post-update, the bootloader automatically reverts to the previous working partition.memfault.com+1
• Cryptographic Verification : Sign every firmware image using private-public key cryptography (such as Ed25519 or ECDSA) during your CI/CD build process. The embedded device must use a hardcoded public key—often stored in a hardware Root of Trust or secure element—to verify the signature before writing or booting the image.memfault.com
• Transport and Update Client : Implement a lightweight client protocol (such as MQTT, HTTPS, or specialized device management agents) for the device to poll or receive payloads from a secure cloud platform.witekio.com+2
• Fleet Management Server : Deploy a backend distribution system (such as AWS IoT Device Management, Mender , or Memfault ) to manage staged rollouts, target specific hardware groups, and monitor deployment health metrics.memfault.com+1

Step-by-Step Deployment Workflow

1. Build and Sign : Compile your application binary and generate a cryptographic signature hash (e.g., SHA-256) of the image.
2. Staged Upload : Push the validated artifact package to your secure server repository or OTA orchestration platform.
3. Notification and Download : The device client queries the server (via scheduled polling or broker alert) and downloads the update into the inactive flash bank over an encrypted TLS connection.
4. Integrity Check : The device validates the file checksum and verifies the digital signature against the stored public key.
5. Atomic Swap and Reboot : The bootloader flags the new bank as active, reboots the system, and runs initial smoke tests before finalizing the slot commitment.youtube.com+4

If you'd like to narrow this down, tell me:

• What is your target hardware (e.g., bare-metal microcontroller vs. embedded Linux)?
• Which connectivity protocol are you using (Wi-Fi, cellular, BLE)?

I can provide a tailored architecture recommendation or configuration example.

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust, secure, and fault-tolerant architecture. A failure mid-update can "brick" a device, making physical recovery expensive or impossible.

Here is a structured guide on how to design and implement an end-to-end OTA update system for embedded devices.

1. Core Architectural Patterns

Choose a partitioning scheme that prevents the device from becoming inoperable during an interrupted update.

• Dual-Bank (A/B) Partitioning (Recommended): The flash memory is divided into two identical application partitions (Bank A and Bank B) plus a small bootloader partition. The active bank runs the current firmware, while the update is downloaded into the inactive bank. Once verified, the bootloader switches the active flag. If the new firmware crashes or fails validation, the bootloader safely rolls back to the previous working bank.
• Single-Bank Partitioning: The new image is written directly over the running firmware or stored in an external flash/EEPROM chip before being applied. This is cost-effective for memory-constrained devices but carries a higher risk of bricking if power is lost during the flash-writing process.
• A Secure Bootloader: This is the root of trust. The bootloader must be lightweight, immutable (or locked against unauthorized modification), and responsible for verifying the integrity and authenticity of the firmware image before jumping to it.

2. The Implementation Workflow

1. Image Generation & Signing:
   • Compile your firmware binary and generate a cryptographic hash (e.g., SHA-256).
   • Sign the hash using a private key via an asymmetric cryptography algorithm like ECDSA (secp256r1) or RSA-3072 . The device stores only the public key to verify the signature.
   • Package the binary, version metadata, and signature into a structured update container (using formats like JSON/CBOR headers combined with binary payloads).
2. Transport & Distribution:
   • Use lightweight, reliable protocols suited for IoT, such as MQTT or HTTPS , to securely transfer the firmware chunk-by-chunk or as a single stream.
   • Implement resume capabilities or chunk-level acknowledgment if operating over flaky cellular or Wi-Fi connections.
3. Secure Download & Verification:
   • Store the incoming binary in the inactive memory bank or external storage.
   • Verify the image by checking the cryptographic signature against the embedded public key before altering any boot flags.
4. Activation & Rollback:
   • Set the bootloader configuration to boot from the new image on the next reset.
   • On first boot, the new firmware must run a self-test (verifying peripherals, cloud connectivity, etc.) and send a "commit" signal back to the bootloader. If the commit signal isn't received within a specific timeout (e.g., three boot cycles), the bootloader rolls back to the stable golden image.

3. Essential Security Best Practices

• Confidentiality: If your firmware contains proprietary algorithms or keys, encrypt the payload on the server using symmetric encryption (like AES-256) and decrypt it on the device during or prior to flashing.
• Anti-Rollback Protection: Store the minimum allowed firmware version number in secure, one-time programmable (OTP) non-volatile memory or protected flash. This prevents an attacker from rolling a device back to an older version with known security vulnerabilities.
• Unique Device Credentials: Ensure each device has unique cryptographic identifiers and authentication tokens to prevent unauthorized or rogue devices from pulling firmware updates meant for other hardware revisions.

4. Existing Tools and Frameworks

Building an entire OTA infrastructure from scratch is a massive engineering undertaking. Consider leveraging open-source or managed solutions:

• Open-Source Frameworks:
  • MCUboot : A secure bootloader for 32-bit microcontrollers that supports dual-image swappable slots and integrates heavily with Zephyr and Mynewt.
  • Mender : An open-source OTA software updater designed specifically for embedded Linux and microcontroller-based IoT devices.
  • UpdateHub : A robust enterprise-grade solution for managing firmware updates on embedded Linux systems.
• Cloud Platforms:
  • : Offers built-in secure, scalable, and monitored OTA rollout features for fleets.

A robust OTA system is best designed as two separate problems:

1. Device-side update safety — can the device authenticate, install, boot, and recover from a firmware image safely?
2. Fleet-side orchestration — can you roll that image out gradually, monitor it, and stop/rollback if something goes wrong?

A good architecture looks like this:

        CI/CD
          │
   build + test firmware
          │
     sign firmware
          │
          ▼
   ┌───────────────┐
   │ Artifact store│
   │  version/hash │
   └───────┬───────┘
           │
      OTA service
   ┌───────▼────────┐
   │ deployment/job │
   │   controller   │
   └───────┬────────┘
           │ TLS
    ┌──────┼───────────┐
    ▼      ▼           ▼
 device  device      device
   │       │            │
   ▼       ▼            ▼
verify → install → reboot
           │
      health check
           │
      confirm/rollback

1. Use a bootloader that supports authenticated updates

For MCU-class devices, MCUboot is a strong starting point. It provides a secure-boot/update framework and supports signed images, test boots, rollback, and downgrade protection.mcuboot+1

Your flash layout would typically resemble:

+-------------------------+
| Bootloader              |  <- immutable / protected
+-------------------------+
| Application slot A      |  <- currently running
+-------------------------+
| Application slot B      |  <- downloaded update
+-------------------------+
| Persistent config/data  |
+-------------------------+

The important part is never overwrite your only known-good firmware image.

With A/B slots, the device can:

1. Download the new firmware into the inactive slot.
2. Verify its hash and signature.
3. Mark it as a candidate.
4. Reboot.
5. Boot the new image in "test" mode.
6. Run a health/self-test.
7. Mark the image as confirmed if healthy.
8. Automatically revert to the old image if the new firmware fails.

MCUboot explicitly supports this test/confirm/revert model.GitHub

2. Cryptographically sign every firmware image

Don't rely on TLS alone.

TLS protects the download channel, but your device should independently determine:

"Was this firmware produced by an authorized release process?" Your release pipeline should therefore do something like:

source
  ↓
compile
  ↓
unit/integration tests
  ↓
firmware.bin
  ↓
hash
  ↓
sign with production private key
  ↓
firmware-v1.8.3.signed.bin

The private signing key should never be placed on the device. The bootloader contains the corresponding public verification key.

MCUboot's signing model follows this approach, with the public key embedded in the bootloader and the private key kept in the signing environment.GitHub+1

For production, I'd also consider:

• Hardware-backed key storage/HSM for the signing key.
• Separate development and production signing keys.
• Key rotation/revocation strategy.
• Firmware version/security counter.
• Protection against firmware downgrade.

MCUboot supports security counters specifically for preventing installation of older, vulnerable firmware.GitHub

3. Make the device an OTA state machine

Don't implement OTA as one giant function.

A state machine is considerably easier to recover from interrupted downloads and power failures:

IDLE
 │
 │ update available
 ▼
DOWNLOADING
 │
 │ download complete
 ▼
VERIFYING
 │
 ├── invalid ──> FAILED
 │
 ▼
READY_TO_BOOT
 │
 ▼
REBOOTING
 │
 ▼
TRIAL_BOOT
 │
 ├── health check fails ──> ROLLBACK
 │
 ▼
CONFIRMED

Persist the important state in nonvolatile storage so that a power loss at any point doesn't brick the device.

For example:

struct ota_state {
    uint32_t update_version;
    uint32_t download_offset;
    uint8_t  image_hash[32];
    enum {
        OTA_IDLE,
        OTA_DOWNLOADING,
        OTA_PENDING,
        OTA_TESTING,
        OTA_CONFIRMED
    } state;
};

The exact implementation depends heavily on your MCU/RTOS.

4. Make downloads resumable

Assume the network will fail.

Your OTA agent should support:

• Chunked downloads.
• Resume after disconnect.
• Resume after reboot.
• Hash verification.
• Timeouts.
• Retries with exponential backoff.
• Storage-space checks before starting.
• Optional bandwidth/radio constraints.

For example:

Firmware: 1,500,000 bytes

chunk 0    ✓
chunk 1    ✓
chunk 2    ✓
chunk 3    ✗ network lost

          reboot

chunk 3    → resume
chunk 4    ✓
...
chunk 146 → ✓

whole-image hash → verify

Don't consider the update downloaded merely because the HTTP transfer completed. Verify the complete artifact before allowing the bootloader to select it.

5. Give every device a unique identity

Each device should have a unique cryptographic identity, rather than a shared fleet password.

A common design is:

Device
  │
  ├── unique device ID
  ├── private key
  └── device certificate
             │
             ▼
        MQTT / HTTPS
             │
             ▼
        OTA backend

The backend can then determine:

• Which firmware a device currently has.
• Which hardware revision it is.
• Which update channel it belongs to.
• Whether it is eligible for a particular update.
• Whether it successfully completed the update.

For large IoT deployments, an OTA job service can provide this orchestration. For example, AWS IoT Jobs lets a backend issue jobs to individual devices and track their progress, while the device downloads and validates the firmware.Amazon Web Services, Inc.+1

You can build an equivalent system yourself if you don't want a cloud-specific dependency.

6. Don't deploy to the entire fleet at once

This is probably the most important fleet-management rule.

Instead of:

100,000 devices
       ↓
    v2.4.0

use staged rollout:

10 devices
   ↓
100 devices
   ↓
1,000 devices
   ↓
10,000 devices
   ↓
25%
   ↓
50%
   ↓
100%

After every stage, monitor things like:

OTA success rate
boot failure rate
rollback rate
crash rate
battery consumption
network reconnect rate
application health
device-specific error codes

Then automatically pause if thresholds are exceeded.

A modern fleet-management implementation can even treat deployment as a sequence of waves and roll back affected devices when health metrics deteriorate.Amazon Web Services, Inc.

7. Separate firmware version from hardware compatibility

Your update manifest should contain more than:

{
  "version": "2.4.0"
}

I'd use something closer to:

{
  "product": "sensor-x",
  "hardware": {
    "min_revision": 3,
    "max_revision": 7
  },
  "version": "2.4.0",
  "security_counter": 18,
  "size": 1847296,
  "sha256": "...",
  "url": "...",
  "release_channel": "stable"
}

The device should reject an image that doesn't apply to its hardware.

This prevents a surprisingly nasty class of OTA failures: validly signed firmware that is simply incompatible with the device.

8. Build OTA into your CI/CD pipeline

A practical pipeline could be:

Git commit
   ↓
Build
   ↓
Unit tests
   ↓
Hardware-in-loop tests
   ↓
Static/security analysis
   ↓
Create firmware image
   ↓
Sign image
   ↓
Publish artifact
   ↓
Deploy to lab devices
   ↓
Automatic soak test
   ↓
Deploy canary fleet
   ↓
Health evaluation
   ↓
Progressive rollout

Critically, signing should happen after the firmware has passed your release gates.

You don't want an accidentally signed artifact to become an authorized production image.

9. Design for recovery from day one

Ask what happens if each of these occurs:

If you can't answer these before implementation, the OTA design isn't finished.

A concrete stack

If you're using a typical embedded MCU + RTOS, I'd consider:

Device

• Zephyr/FreeRTOS/etc.
• MCUboot
• A/B firmware slots
• TLS
• Device certificate/private key
• Persistent OTA state
• Watchdog
• Health/self-test
• Signed firmware

Backend

• HTTPS or MQTT
• Object storage for firmware binaries
• Device registry
• OTA job service
• Deployment groups
• Telemetry/metrics
• Rollout controller

Release infrastructure

• CI/CD
• Reproducible builds
• Automated hardware testing
• Production signing service/HSM
• Artifact/version database

For example, AWS IoT Jobs provides a ready-made fleet orchestration layer: jobs tell devices what update to perform, devices report progress/status, and firmware can be hosted in S3 with signing and TLS protecting the delivery path.AWS Documentation

The key principle

I'd design the system around this invariant:

At every point during an OTA update, the device must have a path back to known-good firmware. That leads naturally to:

signed image → inactive slot → verify → trial boot → health check → confirm → rollback if necessary → staged fleet rollout.

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52/nRF53), RTOS, connectivity (Wi-Fi/LTE/BLE), and approximate fleet size, I can give you a concrete OTA architecture and flash layout for that platform.

{
  "version": "2.4.0"
}

I'd use something closer to:

{
  "product": "sensor-x",
  "hardware": {
    "min_revision": 3,
    "max_revision": 7
  },
  "version": "2.4.0",
  "security_counter": 18,
  "size": 1847296,
  "sha256": "...",
  "url": "...",
  "release_channel": "stable"
}

The device should reject an image that doesn't apply to its hardware.

This prevents a surprisingly nasty class of OTA failures: validly signed firmware that is simply incompatible with the device.

8. Build OTA into your CI/CD pipeline

A practical pipeline could be:

Git commit
   ↓
Build
   ↓
Unit tests
   ↓
Hardware-in-loop tests
   ↓
Static/security analysis
   ↓
Create firmware image
   ↓
Sign image
   ↓
Publish artifact
   ↓
Deploy to lab devices
   ↓
Automatic soak test
   ↓
Deploy canary fleet
   ↓
Health evaluation
   ↓
Progressive rollout

Critically, signing should happen after the firmware has passed your release gates.

You don't want an accidentally signed artifact to become an authorized production image.

9. Design for recovery from day one

Ask what happens if each of these occurs:

If you can't answer these before implementation, the OTA design isn't finished.

A concrete stack

If you're using a typical embedded MCU + RTOS, I'd consider:

Device

• Zephyr/FreeRTOS/etc.
• MCUboot
• A/B firmware slots
• TLS
• Device certificate/private key
• Persistent OTA state
• Watchdog
• Health/self-test
• Signed firmware

Backend

• HTTPS or MQTT
• Object storage for firmware binaries
• Device registry
• OTA job service
• Deployment groups
• Telemetry/metrics
• Rollout controller

Release infrastructure

• CI/CD
• Reproducible builds
• Automated hardware testing
• Production signing service/HSM
• Artifact/version database

For example, AWS IoT Jobs provides a ready-made fleet orchestration layer: jobs tell devices what update to perform, devices report progress/status, and firmware can be hosted in S3 with signing and TLS protecting the delivery path.

The key principle

A robust OTA system for embedded devices is best designed as a secure update pipeline + a resilient bootloader + fleet rollout controls. The exact implementation depends on whether you're using an MCU, embedded Linux, ESP32, etc., but the architecture is broadly the same.

1. Use this architecture

                    ┌─────────────────────┐
                    │ Firmware CI/CD      │
                    │                     │
                    │ build → test → sign │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │ Firmware repository │
                    │ / object storage    │
                    └──────────┬──────────┘
                               │
                         HTTPS download
                               │
                               ▼
┌─────────────┐        ┌─────────────────────┐
│ Device      │◄──────►│ Fleet/OTA service   │
│ registry    │ MQTT   │                     │
└─────────────┘        │ rollout + telemetry │
                       └─────────────────────┘
                               │
                               ▼
                       ┌──────────────┐
                       │ Device OTA   │
                       │ agent        │
                       └──────┬───────┘
                              │
                    verify signature/hash
                              │
                              ▼
                    ┌─────────────────┐
                    │ A/B partitions  │
                    │                 │
                    │ A = running     │
                    │ B = candidate   │
                    └────────┬────────┘
                             │ reboot
                             ▼
                    ┌─────────────────┐
                    │ Secure boot     │
                    │ + health check  │
                    └────────┬────────┘
                             │
                   success → commit
                   failure → rollback

The important principle is: never overwrite the only known-good firmware image.

2. Build an A/B firmware layout

For a typical MCU, reserve flash roughly like:

+--------------------+
| Bootloader         |
+--------------------+
| Metadata           |
+--------------------+
| Application A      |  ← currently running
+--------------------+
| Application B      |  ← OTA download target
+--------------------+
| Scratch / NVM      |
+--------------------+

The OTA agent downloads the new firmware into the inactive slot.

It then:

1. Checks the image size.
2. Checks its cryptographic hash.
3. Verifies the manufacturer's digital signature.
4. Marks the new slot as a candidate.
5. Reboots.
6. Bootloader starts the candidate.
7. Firmware performs health checks.
8. Firmware marks itself confirmed.
9. Otherwise, the bootloader rolls back to the previous version.

This is substantially safer than downloading directly over the currently executing image.

A bootloader such as MCUboot can provide much of the image-selection, upgrade, and rollback machinery rather than requiring you to invent it yourself.mcuboot

3. Cryptographically sign every firmware image

Don't rely on TLS alone.

TLS protects the download channel, but your device should independently establish:

"This firmware was produced by someone authorized to update this device." A typical release artifact might contain:

{
  "product": "sensor-v2",
  "version": "3.7.12",
  "hardware": "revB",
  "size": 524288,
  "sha256": "...",
  "signature": "...",
  "min_bootloader": "2.1.0"
}

Your signing infrastructure should keep the private signing key off the build machines that compile ordinary firmware. Ideally, production signing requires a protected signing service/HSM and explicit authorization.

The device contains the corresponding trusted public key, or a certificate chain rooted in a key stored in protected hardware.

AWS's OTA documentation similarly recommends digitally signing firmware so devices can detect modification, and its OTA system supports signature verification on the device.AWS Documentation+1

4. Separate control plane from firmware transfer

Don't send a multi-megabyte firmware binary through your device-management messaging protocol unless there's a good reason.

Instead:

Control plane:

MQTT:
    "Firmware 3.7.12 is available"

Data plane:

HTTPS:
    GET /firmware/sensor-v2/3.7.12/image.bin

The OTA message can contain:

{
  "job_id": "ota-2026-09-00142",
  "version": "3.7.12",
  "url": "https://...",
  "sha256": "...",
  "signature": "...",
  "size": 524288
}

The device downloads the file in chunks, verifies it, and installs it.

AWS IoT's OTA architecture follows essentially this separation: IoT Jobs handles update notification/control while firmware can be downloaded over HTTP or MQTT.AWS Documentation+1

5. Make the OTA agent resumable

Assume that connections will fail.

Instead of:

download 0 → 100%

implement:

download:
    0-64 KB      ✓
    64-128 KB    ✓
    128-192 KB   ✓
    connection lost

reconnect

resume:
    192 KB → ...

Store:

• firmware version
• expected size
• bytes received
• hash/state information
• update/job ID
• target partition

Use integrity checks on downloaded chunks and a final cryptographic verification before activation.

For cellular devices in particular, resumability and download throttling can make a huge difference.

6. Add fleet-level rollout controls

Never update 100,000 devices simultaneously.

Use staged deployment:

Firmware 3.7.12

        10 devices
           │
           ▼
       0.1% fleet
           │
           ▼
          1%
           │
           ▼
          5%
           │
           ▼
         25%
           │
           ▼
         100%

After each stage, monitor:

• update success rate
• reboot failures
• crash/reset rate
• connectivity
• battery consumption
• application-specific health metrics
• rollback rate
• device availability

Automatically stop the rollout if those metrics exceed thresholds.

Fleet systems such as AWS IoT Jobs provide rollout, timeout, abort, and per-device job status capabilities for this purpose.AWS Documentation+1

7. Give every device a state machine

For example:

IDLE
 │
 │ update available
 ▼
DOWNLOADING
 │
 │ download complete
 ▼
VERIFYING
 │
 ├── invalid ──► FAILED
 │
 ▼
READY
 │
 │ reboot
 ▼
TESTING
 │
 ├── health check failed ──► ROLLBACK
 │
 ▼
CONFIRMED

Persist the important state in nonvolatile storage so a power failure doesn't leave the update mechanism confused.

For example:

typedef enum {
    FW_CONFIRMED,
    FW_PENDING,
    FW_TESTING,
    FW_ROLLBACK
} firmware_state_t;

8. Design explicitly for power loss

Imagine this happens:

Device downloads 98%
        ↓
power disappears
        ↓
device reboots

The device should simply boot the existing firmware and resume or discard the incomplete update.

Likewise:

new firmware installed
        ↓
device reboots
        ↓
power disappears during first boot

The bootloader should still have enough persistent information to determine that the new image wasn't confirmed and revert to the previous known-good image.

This is one of the biggest reasons to use A/B slots and a capable bootloader.

9. Protect against downgrade attacks

Suppose:

Current:  firmware 10
Attacker: signed firmware 3

A valid signature isn't necessarily enough.

Include an anti-rollback counter/version:

minimum_allowed_version = 10

firmware 9  → reject
firmware 10 → accept
firmware 11 → accept

Where possible, store the minimum version in secure/nonvolatile hardware that attackers cannot simply rewrite.

10. Bind firmware to hardware/product variants

Don't let firmware intended for:

Device A / HW rev 1

accidentally install on:

Device A / HW rev 3

Include identifiers such as:

product_id
hardware_revision
bootloader_compatibility
minimum_version
firmware_version

and validate them before activation.

11. Build the backend around immutable releases

I would structure the server side something like:

/releases/
    sensor-v2/
        3.7.10/
            manifest.json
            firmware.bin
        3.7.11/
            manifest.json
            firmware.bin
        3.7.12/
            manifest.json
            firmware.bin

Never modify an already-published firmware artifact.

A release should be immutable and identified by something like:

release_id = sensor-v2-3.7.12

Your fleet database can then track:

device_id
hardware_revision
current_version
target_version
update_state
last_seen
last_update_result
rollback_count

12. Secure the device identity separately

Every device should have a unique identity/credential.

For example:

Device
  │
  ├── device ID
  ├── private key
  └── certificate

Ideally, the private key lives in a secure element or hardware-backed keystore rather than ordinary flash.

The device's cloud credentials should authorize only what that device needs to do. In particular, don't give every device permission to publish arbitrary fleet-management commands.

13. Your CI/CD pipeline should look like this

git push
   │
   ▼
compile
   │
   ▼
unit tests
   │
   ▼
static analysis
   │
   ▼
hardware-in-the-loop tests
   │
   ▼
generate firmware
   │
   ▼
generate manifest
   │
   ▼
sign firmware
   │
   ▼
publish immutable artifact
   │
   ▼
deploy to internal devices
   │
   ▼
canary fleet
   │
   ▼
staged rollout
   │
   ▼
100% deployment

The signing step should happen after all modifications to the firmware are complete.

14. Don't overlook observability

For every OTA job, you want to answer:

"What happened to device 18472?" with something like:

Device:       18472
Current:      3.7.11
Target:       3.7.12

Download:     SUCCESS
Verification: SUCCESS
Install:      SUCCESS
Reboot:       SUCCESS
Health check: SUCCESS
Confirmed:    YES

Duration:     94 sec

And for a failed device:

Device:       18472
Target:       3.7.12
State:        ROLLED_BACK

Reason:
    application health check failed

Previous:
    3.7.11

Rollback:
    SUCCESS

This is much more useful than simply logging "OTA failed."

A practical technology stack

If you're starting from scratch, a reasonable stack is:

For example, AWS IoT Jobs is specifically designed to send remote operations—including firmware updates—to individual devices or groups and track their execution.AWS Documentation+1

AWS also provides an embedded OTA library that handles notification, download, cryptographic verification, self-testing, and update status reporting, which can save substantial implementation effort on supported platforms.AWS Open Source

The minimum secure design

If you want the shortest version, I would not ship an OTA system without these seven properties:

1. Signed firmware
2. Secure boot / authenticated boot
3. A/B or equivalent atomic update mechanism
4. Automatic rollback
5. Anti-downgrade protection
6. Resumable downloads
7. Staged fleet rollout with automatic abort

Those turn OTA from "download a new .bin file and reboot" into a system that can survive bad firmware, unreliable networks, power loss, compromised networks, and large-scale deployments.

7. Give every device a state machine

For example:

IDLE
 │
 │ update available
 ▼
DOWNLOADING
 │
 │ download complete
 ▼
VERIFYING
 │
 ├── invalid ──► FAILED
 │
 ▼
READY
 │
 │ reboot
 ▼
TESTING
 │
 ├── health check failed ──► ROLLBACK
 │
 ▼
CONFIRMED

Persist the important state in nonvolatile storage so a power failure doesn't leave the update mechanism confused.

For example:

typedef enum {
    FW_CONFIRMED,
    FW_PENDING,
    FW_TESTING,
    FW_ROLLBACK
} firmware_state_t;

8. Design explicitly for power loss

Imagine this happens:

Device downloads 98%
        ↓
power disappears
        ↓
device reboots

The device should simply boot the existing firmware and resume or discard the incomplete update.

Likewise:

new firmware installed
        ↓
device reboots
        ↓
power disappears during first boot

The bootloader should still have enough persistent information to determine that the new image wasn't confirmed and revert to the previous known-good image.

This is one of the biggest reasons to use A/B slots and a capable bootloader.

9. Protect against downgrade attacks

Suppose:

Current:  firmware 10
Attacker: signed firmware 3

A valid signature isn't necessarily enough.

Include an anti-rollback counter/version:

minimum_allowed_version = 10

firmware 9  → reject
firmware 10 → accept
firmware 11 → accept

Where possible, store the minimum version in secure/nonvolatile hardware that attackers cannot simply rewrite.

10. Bind firmware to hardware/product variants

Don't let firmware intended for:

Device A / HW rev 1

accidentally install on:

Device A / HW rev 3

Include identifiers such as:

product_id
hardware_revision
bootloader_compatibility
minimum_version
firmware_version

and validate them before activation.

11. Build the backend around immutable releases

I would structure the server side something like:

/releases/
    sensor-v2/
        3.7.10/
            manifest.json
            firmware.bin
        3.7.11/
            manifest.json
            firmware.bin
        3.7.12/
            manifest.json
            firmware.bin

Never modify an already-published firmware artifact.

A release should be immutable and identified by something like:

release_id = sensor-v2-3.7.12

Your fleet database can then track:

device_id
hardware_revision
current_version
target_version
update_state
last_seen
last_update_result
rollback_count

12. Secure the device identity separately

Every device should have a unique identity/credential.

For example:

Device
  │
  ├── device ID
  ├── private key
  └── certificate

Ideally, the private key lives in a secure element or hardware-backed keystore rather than ordinary flash.

The device's cloud credentials should authorize only what that device needs to do. In particular, don't give every device permission to publish arbitrary fleet-management commands.

13. Your CI/CD pipeline should look like this

git push
   │
   ▼
compile
   │
   ▼
unit tests
   │
   ▼
static analysis
   │
   ▼
hardware-in-the-loop tests
   │
   ▼
generate firmware
   │
   ▼
generate manifest
   │
   ▼
sign firmware
   │
   ▼
publish immutable artifact
   │
   ▼
deploy to internal devices
   │
   ▼
canary fleet
   │
   ▼
staged rollout
   │
   ▼
100% deployment

The signing step should happen after all modifications to the firmware are complete.

14. Don't overlook observability

For every OTA job, you want to answer:

"What happened to device 18472?" with something like:

Device:       18472
Current:      3.7.11
Target:       3.7.12

Download:     SUCCESS
Verification: SUCCESS
Install:      SUCCESS
Reboot:       SUCCESS
Health check: SUCCESS
Confirmed:    YES

Duration:     94 sec

And for a failed device:

Device:       18472
Target:       3.7.12
State:        ROLLED_BACK

Reason:
    application health check failed

Previous:
    3.7.11

Rollback:
    SUCCESS

This is much more useful than simply logging "OTA failed."

A practical technology stack

If you're starting from scratch, a reasonable stack is:

For example, AWS IoT Jobs is specifically designed to send remote operations—including firmware updates—to individual devices or groups and track their execution.

AWS also provides an embedded OTA library that handles notification, download, cryptographic verification, self-testing, and update status reporting, which can save substantial implementation effort on supported platforms.

The minimum secure design

If you want the shortest version, I would not ship an OTA system without these seven properties:

implement: