How can I implement over-the-air (OTA) firmware updates for a fleet of embedded devices?

Implementing OTA (over-the-air) firmware updates for a fleet of embedded devices is primarily a reliability, security, and fleet-management problem, not just a “download a new binary” problem. A production OTA system usually has four major pieces:

1. Device-side update agent
2. Firmware distribution backend
3. Secure update package format
4. Recovery and rollout controls

A common architecture looks like this:

             CI/CD Pipeline
                  |
                  v
        Build + Test Firmware
                  |
                  v
        Sign Firmware Artifact
                  |
                  v
        OTA Update Server
                  |
        -------------------
        |        |        |
      Device  Device   Device
        |        |        |
        v        v        v
   OTA Agent + Bootloader
   Verify → Download → Install → Reboot → Confirm

A reference architecture for IoT firmware updates is also described in Internet Engineering Task Force RFC 9019, which discusses secure update components such as manifests, firmware verification, and device-side update processing.www.rfc-editor.org

1. Design the firmware update flow

A robust update sequence is typically:

1. Device checks for an update:

   • Periodically polls the server, or
   • Receives a notification through MQTT, HTTPS, etc.

2. Server responds with:

   • New firmware version
   • Compatibility requirements
   • Download URL
   • Cryptographic metadata

3. Device:

   • Downloads firmware
   • Verifies signature
   • Writes firmware to inactive storage
   • Reboots
   • Runs self-test
   • Marks update as successful

4. If anything fails:

   • Boot previous firmware
   • Report failure
   • Retry or wait for another deployment

2. Use A/B firmware partitions (recommended)

For field devices, avoid overwriting the currently running firmware.

Example flash layout:

+----------------+
| Bootloader     |
+----------------+
| Firmware A     | <-- currently running
+----------------+
| Firmware B     | <-- new update written here
+----------------+
| Configuration  |
+----------------+
| Logs / Data    |
+----------------+

Update process:

1. Download new image into inactive slot.
2. Verify checksum and signature.
3. Mark slot B as pending.
4. Reboot.
5. Bootloader starts slot B.
6. Device confirms successful operation.
7. Slot B becomes permanent.

If the device loses power during installation, it can return to slot A.

A/B updates with rollback are commonly used in OTA systems such as Mender, where the inactive partition is updated first and committed only after a successful boot.docs.mender.io

3. Secure the update process

Minimum security requirements:

Firmware signing

Do not trust downloaded binaries based only on HTTPS.

Typical flow:

Developer
   |
   | sign firmware with private key
   v
Firmware package
   |
   | download
   v
Device verifies with embedded public key

Use:

• ECDSA
• Ed25519
• RSA (less common for constrained devices)

The private signing key should never be stored on devices.

Secure boot

Your bootloader should verify:

• Firmware signature
• Firmware version
• Hardware compatibility
• Integrity hash

The bootloader should reject:

• Modified firmware
• Unsigned firmware
• Downgrade attempts (if required)

Transport security

Use:

• TLS
• Certificate validation
• Device authentication

Avoid:

• Plain HTTP
• Shared global passwords
• Anonymous update endpoints

4. Build a firmware update backend

A fleet backend normally stores:

Device inventory

Example:

{
  "device_id": "sensor-48291",
  "hardware": "rev3",
  "firmware": "2.1.4",
  "location": "plant-a"
}

Firmware releases

Example:

{
  "version": "2.2.0",
  "hardware": "rev3",
  "sha256": "...",
  "signature": "...",
  "rollout": "staged"
}

Deployment controls

Useful features:

• Device groups
• Rollback
• Maintenance windows
• Failure reporting
• Staged rollout
• Audit logs

5. Add staged rollouts

Do not update 100,000 devices immediately.

A safer deployment:

Release 2.2.0

  1%   -> internal test devices
 10%   -> early production devices
 50%   -> larger fleet
100%   -> full deployment

Monitor:

• Boot failures
• Crash rates
• Battery drain
• Connectivity problems
• User-reported issues

6. Handle unreliable networks

Embedded devices often have:

• Cellular connections
• Intermittent Wi-Fi
• Low bandwidth
• Limited storage

Design for:

Resumable downloads

Store:

downloaded: 43%
remaining: 57%

and continue later.

Delta updates

Instead of:

Firmware 2.1.0
      |
      | 50 MB
      v
Firmware 2.2.0

send:

Patch:
+ changed blocks only

Delta updates can reduce bandwidth usage significantly when supported.docs.mender.io

7. Separate application updates from bootloader updates

A dangerous mistake is making the bootloader easy to replace.

A safer hierarchy:

Immutable Boot ROM
        |
        v
Secure Bootloader
        |
        v
OTA Application Firmware
        |
        v
User Data

The bootloader should be updated rarely and with extra safeguards.

8. Choose an OTA framework or build your own

Common approaches:

Embedded Linux devices

Options include:

• Mender — client/server OTA architecture with fleet deployment management and A/B update support.docs.mender.io
• RAUC
• SWUpdate

Microcontrollers

Typical stack:

MCUboot
   +
RTOS OTA agent
   +
Cloud update service

For example, Mender documents MCU OTA integration using Zephyr and MCUboot with A/B-style firmware handling.docs.mender.io

9. Example implementation stack

For a fleet of STM32/ESP32-class devices:

Device

• MCUboot bootloader
• Dual firmware slots
• Secure element for keys
• HTTPS/MQTT client
• OTA manager task

Backend

• Device registry
• Firmware artifact storage
• Deployment scheduler
• Metrics dashboard

CI/CD

• Build firmware
• Run hardware tests
• Generate manifest
• Sign artifact
• Publish release

10. Test failure scenarios

Before deploying, test:

A practical production checklist:

• ✅ Signed firmware images
• ✅ Secure boot verification
• ✅ A/B partitions or equivalent rollback mechanism
• ✅ Device authentication
• ✅ Version and downgrade protection
• ✅ Staged deployments
• ✅ Failure telemetry
• ✅ Recovery path for bricked devices
• ✅ Automated build/sign/release pipeline

The exact implementation depends heavily on whether your fleet uses bare-metal MCUs (STM32, ESP32, nRF52, etc.), RTOS devices (Zephyr/FreeRTOS), or embedded Linux (Yocto, Buildroot, Debian).

A production OTA system is best thought of as a secure deployment pipeline plus a fail-safe boot/update mechanism, not simply “download a new .bin file.”

A solid architecture looks like this:

Developer
   │
   ▼
CI/CD ──► Build firmware ──► Sign artifact
                              │
                              ▼
                       OTA backend / CDN
                              │
                    deployment + targeting
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
        Device A                         Device B
       OTA client                       OTA client
           │                               │
       TLS/auth                        TLS/auth
           │                               │
           ▼                               ▼
      Bootloader                       Bootloader
       A/B slots                        A/B slots
       ┌──────┐                         ┌──────┐
       │ FW A │                         │ FW A │
       ├──────┤                         ├──────┤
       │ FW B │                         │ FW B │
       └──────┘                         └──────┘

1. Make the device update mechanism fail-safe

For an MCU, a typical layout is:

+-------------------------+
| Bootloader              |
| - verifies signatures   |
| - selects firmware      |
+-------------------------+
| Firmware slot A         |
+-------------------------+
| Firmware slot B         |
+-------------------------+
| Persistent configuration|
+-------------------------+

The device normally executes A. The OTA client downloads the new image into B, verifies it, marks B as the next boot target, and reboots.

On the first boot, the new firmware must prove that it successfully initialized. Only then does the bootloader permanently commit B. Otherwise, the bootloader returns to A.

This A/B approach protects against failures such as power loss during installation or a firmware image that boots incorrectly. It's a common pattern in robust OTA systems.Mender Docs+1

For embedded Linux, the same concept is commonly implemented with two root filesystem partitions. Mender, for example, writes the new image to the inactive partition and rolls back if the new system doesn't successfully boot and commit.Mender+1

2. Cryptographically sign every firmware image

Don't rely on TLS alone.

You want two separate protections:

• TLS protects the connection while downloading.
• Firmware signatures establish that the binary itself was produced by you and hasn't been modified.

Conceptually:

Build server
    │
    ├── firmware.bin
    │
    └── Sign with private release key
             │
             ▼
       firmware + signature
             │
             ▼
          Device
             │
       verify signature
             │
       ┌─────┴─────┐
     valid        invalid
       │             │
       ▼             ▼
    install         reject

Keep the signing private key out of ordinary developer machines. Ideally, signing happens through a controlled CI/CD process backed by an HSM or equivalent key-management system.

The device should contain only the public verification key or certificate chain. Hardware-backed key storage can provide another layer of protection.Mender+1

Also include:

• Firmware version
• Hardware/model identifier
• Image size
• Cryptographic hash
• Signature
• Minimum bootloader version
• Optional minimum hardware revision
• Release metadata

3. Prevent firmware downgrade attacks

Suppose an attacker obtains a perfectly valid firmware image from six months ago that contains a vulnerability.

A valid signature doesn't make that image safe.

Have the bootloader enforce a monotonic version/security counter:

Installed security version: 42

Incoming image:
    security version = 41

→ reject

A legitimate emergency rollback should therefore be explicitly designed into your security model rather than allowing arbitrary older signed images.

4. Give every device a strong identity

Each device should have its own identity rather than sharing a fleet-wide password.

For example:

Device 001
  device ID: abc123
  credential: unique certificate/private key

Device 002
  device ID: def456
  credential: unique certificate/private key

X.509 device certificates are one common approach for IoT fleets. AWS's guidance similarly recommends unique device identities and credentials rather than shared fleet credentials.AWS Documentation

Ideally, private device keys are generated/stored in a secure element, TPM, or other hardware-backed storage when your hardware supports it.

5. Separate the OTA client from the bootloader

A useful division is:

OTA client

• Contacts server
• Authenticates device
• Determines whether an update exists
• Downloads it
• Resumes interrupted downloads
• Verifies metadata/hash
• Places image into inactive slot
• Tells bootloader to attempt update
• Reports success/failure

Bootloader

• Validates firmware signature
• Checks compatibility/version policy
• Selects boot slot
• Tracks boot attempts
• Rolls back failed images

That separation means a compromised or buggy application doesn't get unrestricted authority to replace the bootloader's security decisions.

6. Design for unreliable networks

Assume the connection will disappear halfway through the download.

Your update protocol should support:

• HTTP range/resumable downloads or equivalent
• Chunked downloads
• Hash/checksum validation
• Retries with exponential backoff
• Power-loss recovery
• Persistent download state
• Sufficient free flash/storage
• Verification before activation

Don't make an interrupted download equivalent to a failed firmware installation.

For cellular devices in particular, also consider bandwidth and data costs. Full-image updates are simple and robust, while delta updates can substantially reduce bandwidth at the cost of additional complexity.Mender

7. Don't update the whole fleet simultaneously

Your backend should support staged deployments.

For example:

Firmware 2.7.0

  └── Internal test devices       20
          │
          ▼
      Canary fleet               100
          │
          ▼
      1% of production
          │
          ▼
      10% of production
          │
          ▼
      25%
          │
          ▼
      100%

At each stage, monitor:

• Download failures
• Installation failures
• Boot failures
• Rollback rate
• Device connectivity
• Application health
• Crash/reset rates
• Battery consumption
• Relevant application-specific metrics

Automatically pause the rollout if failure rates exceed your thresholds.

Fleet deployment systems such as AWS IoT Jobs explicitly support controlled deployment/rollback patterns, while OTA guidance recommends gradual rollouts rather than updating every device simultaneously.AWS Documentation

8. Use device groups and compatibility constraints

Don't send one generic firmware to every device.

Your update manifest might look conceptually like:

{
  "version": "2.7.0",
  "hardware": "sensor-rev-C",
  "min_bootloader": "1.4.0",
  "image": "firmware-2.7.0.bin",
  "sha256": "...",
  "signature": "..."
}

The backend can then target:

model = sensor
hardware_revision = C
region = US
current_version < 2.7.0

This becomes particularly important when you have multiple MCU types, board revisions, bootloader versions, or product generations.

9. Keep application data separate from firmware

Don't overwrite configuration and user/device state when replacing firmware.

For example:

Flash
├── Bootloader
├── Firmware A
├── Firmware B
├── Device identity
└── Persistent data
     ├── configuration
     ├── calibration
     └── application state

Your firmware update should be able to replace either firmware slot without destroying persistent data.

For Linux systems, this separation is also important because replacing an entire root filesystem can otherwise remove files that applications expect to persist.Mender Docs

10. Pick the implementation according to your hardware

MCU / RTOS

If you're using something like Zephyr, a strong starting point is MCUboot plus an OTA management layer. MCUboot provides image verification and A/B-style slot management; Mender, for example, integrates its MCU client with Zephyr and MCUboot for atomic OTA updates and rollback.Mender Docs

Embedded Linux

For Linux devices, an image-based A/B system is usually attractive.

Options include:

• Mender
• RAUC
• SWUpdate
• OSTree-based systems
• A custom U-Boot + A/B implementation

Mender, for example, provides a device client and server architecture with A/B root filesystem updates and rollback.Mender+1

Cloud backend

You can either build the control plane yourself or use an OTA fleet-management service.

A managed architecture might be:

CI/CD
  │
  ├── Build
  ├── Test
  ├── Sign
  └── Publish
        │
        ▼
   OTA service
        │
   ┌────┴────┐
   │         │
 device    device

AWS IoT Jobs is one example of a service providing fleet targeting and OTA job management.AWS Documentation+1

Mender is another example, with a server/client architecture and support for both embedded Linux and, through its MCU integration, constrained microcontrollers.Mender Docs+1

A practical implementation plan

I'd build it in this order:

1. Bootloader

• Signature verification
• A/B slots
• Version protection
• Boot-attempt counter
• Automatic rollback

2. Local updater

• Download firmware
• Resume interrupted downloads
• Validate hash/signature
• Write inactive slot
• Request reboot
• Report result

3. Device identity

• Unique device credentials
• TLS
• Secure credential storage

4. Artifact pipeline

• Reproducible firmware build
• Automated tests
• Firmware signing
• Artifact repository

5. OTA backend

• Device inventory
• Firmware versions
• Device groups
• Deployment jobs
• Status reporting

6. Staged rollout

• Test → canary → percentage rollout → full fleet
• Automatic pause/rollback criteria

7. Observability

• Update state per device
• Failure reason
• Current/previous firmware
• Boot failures
• Rollbacks
• Connectivity

The key design principle

Never make “downloaded successfully” mean “updated successfully.”

A robust lifecycle is:

AVAILABLE
   ↓
DOWNLOADING
   ↓
DOWNLOADED
   ↓
SIGNATURE VERIFIED
   ↓
INSTALLED IN INACTIVE SLOT
   ↓
PENDING REBOOT
   ↓
BOOTING NEW IMAGE
   ↓
HEALTH CHECK
   ├── FAIL → ROLLBACK
   │
   └── PASS → COMMIT

That state machine, combined with cryptographic verification, A/B storage, rollback, unique device identity, and staged fleet deployment, gives you the foundation of a production-grade OTA system.Mender+2

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52, i.MX), RTOS/Linux choice, connectivity (Wi-Fi/cellular), and approximate fleet size, I can lay out a concrete OTA architecture and flash partition map for it.

Implementing over-the-air (OTA) firmware updates for a fleet of embedded devices requires a robust, secure, and fault-tolerant architecture. If an update fails mid-way, you don't want a fleet of expensive, unrecoverable "brick" devices in the field.

Here is a clear, structured roadmap to building a reliable OTA pipeline:

1. Hardware & Memory Architecture (Dual-Bank Flash)
   • Dual-Bank (A/B) Partitioning : Divide your non-volatile storage (Flash) into two separate application partitions: Bank A (Active/Running) and Bank B (Inactive/Staging). The bootloader boots from Bank A while the new firmware downloads safely into Bank B.
   • The Bootloader : Keep your primary bootloader small, secure, and rarely changed. It is responsible for verifying the integrity of the newly flashed image, swapping the active pointer, and handling rollbacks if the new firmware crashes.
   • Fallback/Rollback Mechanism : If the new firmware fails its health check or fails to report a "success" status to the bootloader within a watchdog timeout window, the bootloader automatically reverts to the previous working image (Bank A).
2. Security & Cryptography
   • Cryptographic Signing : Never push unsigned binaries. Sign your firmware images on your build server using a private key (e.g., using Ed25519 or RSA-3072). The embedded device verifies the signature with an embedded public key before applying the update.
   • Transport Security : Use strict TLS (1.3) for downloading images over Wi-Fi, Cellular, or Ethernet to prevent man-in-the-middle tampering or snooping.
   • Encrypted Images : If your firmware contains proprietary algorithms or sensitive keys, encrypt the binary payload itself and decrypt it on-the-fly or inside a secure element/microcontroller flash controller.
3. Backend & Fleet Management Infrastructure
   • Device Registry & Metadata : Maintain a database of your fleet tracking each device's hardware revision, current firmware version, and cryptographic identifier (like a UUID or hardware-bound certificate).
   • Delta Updates : For constrained bandwidth (like Cellular/NB-IoT), generate binary diffs (using tools like bsdiff) so devices download only the changed bytes rather than a full monolithic image, saving data and time.
   • Phased Rollouts (Canary Deployments) : Avoid updating the entire fleet at once. Push updates to a 1% canary group, monitor telemetry for errors or crash loops, and scale up to 10%, 50%, and 100% progressively.
4. Open-Source & Managed Ecosystem Tools
   • MCUboot : A secure bootloader for 32-bit microcontrollers that natively supports dual-image swappable slots and signature verification.
   • AWS IoT Jobs / Azure Device Update : Cloud-managed frameworks that handle device grouping, rollout scheduling, and security for connected fleets.
   • Mender or Balena : End-to-end open-source and commercial OTA update software specifically engineered for Linux-based or RTOS embedded edge devices.

To help tailor this design, let me know:

• What microcontroller or processor family (e.g., ESP32, STM32, ARM Cortex-A Linux) are you using?
• What is your connectivity medium (Wi-Fi, Cellular, LoRaWAN, Ethernet)?

A robust OTA system for an embedded fleet should be designed as secure boot + signed firmware + fail-safe installation + controlled fleet rollout, rather than simply “download a binary and flash it.”

Recommended architecture

                 ┌─────────────────────┐
                 │ Firmware CI/CD       │
                 │ build + test + sign  │
                 └──────────┬──────────┘
                            │
                     signed image
                            │
                 ┌──────────▼──────────┐
                 │ Firmware repository │
                 │ object storage/CDN  │
                 └──────────┬──────────┘
                            │ HTTPS
                            │
       ┌────────────────────▼────────────────────┐
       │              OTA service                │
       │ device inventory / jobs / rollout       │
       └────────────────────┬────────────────────┘
                            │ MQTT/HTTPS
              ┌─────────────┴─────────────┐
              │                           │
        ┌─────▼─────┐               ┌─────▼─────┐
        │ Device A  │      ...      │ Device N  │
        │ OTA agent │               │ OTA agent │
        │ bootloader│               │ bootloader│
        └───────────┘               └───────────┘

1. Put a secure bootloader underneath the application

The bootloader should verify that an image is:

• cryptographically authentic
• intact
• compatible with the hardware
• sufficiently new
• authorized to run

For MCU-class devices, MCUboot is one established option. It supports signed images and upgrade/rollback mechanisms, including a test boot followed by confirmation of the new firmware.mcuboot+1

A typical flash layout is:

+-----------------------+
| Bootloader            |  <-- immutable/protected
+-----------------------+
| Application slot A    |  <-- currently running
+-----------------------+
| Application slot B    |  <-- downloaded update
+-----------------------+
| Persistent data       |
+-----------------------+

This A/B scheme is extremely valuable. Don't overwrite the only known-good application image while performing an OTA update.

2. Sign firmware in CI/CD

Your build pipeline should produce something like:

firmware.bin
firmware.manifest
firmware.sig

The manifest might contain:

{
  "product": "sensor-v3",
  "version": "2.7.4",
  "hardware": ["rev-c", "rev-d"],
  "size": 483920,
  "sha256": "...",
  "security_counter": 42
}

Then sign the image/metadata with a release signing key.

The device should have only the public verification key (or a chain of trust); the private signing key should never exist on the device.

Also implement anti-rollback. Otherwise an attacker who obtains an old but validly signed vulnerable firmware image could potentially downgrade a device. MCUboot, for example, supports version/security-counter-based downgrade prevention.GitHub

3. Have the device ask whether an update exists

Don't necessarily push a several-megabyte binary directly over your command channel.

A device can periodically report:

{
  "device_id": "abc123",
  "firmware": "2.7.3",
  "hardware": "rev-c"
}

The server responds:

{
  "update": true,
  "version": "2.7.4",
  "url": "...",
  "sha256": "...",
  "size": 483920
}

The device then downloads the image, preferably over HTTPS.

For constrained devices, MQTT-based block transfer can also make sense. AWS's OTA implementation, for example, supports HTTP and MQTT delivery and separates notification, downloading, cryptographic verification, and installation.AWS Documentation+1

4. Download into the inactive slot

The device should:

1. Check that enough storage is available.
2. Download into slot B.
3. Verify the complete image hash.
4. Verify the firmware signature.
5. Verify hardware/product compatibility.
6. Verify version/security counter.
7. Mark slot B as a pending/test image.
8. Reboot.

Critically, don't erase slot A until the new firmware has successfully booted.

5. Make the first boot a trial

The bootloader/application state machine should look approximately like:

             update available
                    │
                    ▼
             Download image
                    │
                    ▼
              Verify image
                    │
                    ▼
             Boot new image
                    │
              ┌─────┴─────┐
              │           │
          self-test OK   crash/watchdog
              │           │
              ▼           ▼
          CONFIRM       ROLLBACK
              │
              ▼
          normal boot

The application should confirm the new firmware only after passing meaningful health checks, such as:

• initialization succeeds
• flash/filesystem is healthy
• communications work
• critical peripherals respond
• configuration can be loaded
• watchdog operates normally

MCUboot explicitly supports this test/confirm/revert pattern.GitHub

6. Treat fleet deployment separately from firmware delivery

This is where many OTA implementations become dangerous.

Don't immediately deploy:

10,000 devices → firmware 2.7.4

Instead use staged rollout:

1–5 internal devices
       ↓
10 devices
       ↓
1% of fleet
       ↓
10%
       ↓
25%
       ↓
50%
       ↓
100%

Automatically pause the rollout if metrics deteriorate.

Track things such as:

download_success_rate
install_success_rate
boot_success_rate
rollback_rate
watchdog_resets
device_disconnects
battery_level
firmware_version

Cloud fleet-management systems such as AWS IoT Device Management provide rollout controls, failure thresholds, device groups, and monitoring for this purpose.Amazon Web Services, Inc.+1

7. Make updates resumable

Embedded devices will lose connectivity.

Don't assume:

download 8 MB
             ↓
connection stays alive
             ↓
success

Instead download in chunks:

chunk 0 ✓
chunk 1 ✓
chunk 2 ✓
chunk 3 ✓
connection lost
       ↓
reconnect
       ↓
resume at chunk 4

You can store a small download-progress record in persistent storage.

For cellular devices especially, also consider:

• bandwidth limits
• battery state
• data costs
• update windows
• maximum retry count

8. Separate firmware identity from device identity

Every device should have a unique cryptographic identity, for example:

device certificate/private key
             ↓
       authenticated
             ↓
        OTA service

The OTA service should know:

device ID
hardware revision
current firmware
region
deployment group
last successful update
last failed update

That lets you say:

Deploy firmware 2.7.4 only to sensor-v3, hardware revisions C/D, excluding devices currently reporting low battery. This is much safer than broadcasting a firmware URL to every device.

9. Design for power loss

Assume the device loses power at every possible point:

during download
during flash write
after flash write
during reboot
during first boot
during confirmation

Your state machine should still result in either:

known-good firmware

or

new firmware + ability to revert

Never require the network to recover a device after a failed update. The recovery mechanism must be local to the device.

10. Keep application data separate from firmware

Don't casually erase:

configuration
calibration
device identity
keys/certificates
user data

when installing firmware.

Use separate flash partitions or a carefully designed migration mechanism:

Bootloader
Firmware A
Firmware B
Device credentials
Configuration
Application data
OTA metadata

If firmware 2.7 changes the configuration schema, make migrations explicit and preferably reversible.

A practical technology stack

For a typical MCU product, one reasonable stack is:

Layer	Example
Bootloader	MCUboot
RTOS	Zephyr / FreeRTOS
Transport	HTTPS, optionally MQTT
Firmware storage	S3/object storage + CDN
Fleet management	AWS IoT Jobs or equivalent

MCUboot is OS-independent and supports several embedded operating systems and MCU platforms.mcuboot

If you're already using AWS IoT, AWS's OTA Update Manager can create OTA updates targeting device groups, deliver via MQTT/HTTP, and integrate signing and IoT Jobs.AWS Documentation

The key security rule

The most important principle is:

Never trust the update server alone. The device itself must cryptographically decide whether an image is allowed to execute. Even if your cloud account, CDN, network, or update metadata is compromised, an attacker should not be able to make the device execute an unsigned or unauthorized firmware image.

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52/nRF53), RTOS/Linux, connectivity (Wi-Fi/cellular/LoRaWAN), and approximate fleet size, I can lay out a concrete flash layout, bootloader state machine, signing scheme, cloud architecture, and implementation skeleton for that platform.

A production OTA system is best designed as a secure update pipeline + a fail-safe boot/update mechanism + fleet rollout controls. The exact implementation depends heavily on whether your devices run embedded Linux or a microcontroller RTOS.

1. Use this architecture

                    ┌─────────────────────┐
                    │   CI/CD build       │
                    │ compile + test       │
                    │ sign firmware       │
                    └──────────┬──────────┘
                               │
                         signed artifact
                               │
                    ┌──────────▼──────────┐
                    │ Artifact storage    │
                    │ HTTPS / object      │
                    │ storage/CDN         │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ OTA control plane   │
                    │                     │
                    │ targeting           │
                    │ rollout percentage  │
                    │ health monitoring   │
                    │ rollback            │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │ Device OTA agent    │
                    │                     │
                    │ download           │
                    │ verify signature    │
                    │ install             │
                    │ reboot              │
                    │ health-check        │
                    └──────────┬──────────┘
                               │
                     ┌─────────▼─────────┐
                     │ Secure bootloader │
                     │ A/B slots         │
                     │ rollback          │
                     └───────────────────┘

A service such as aws.amazon.com provides the fleet-management/job portion, while alternatives such as Mender provide both server and device-side OTA infrastructure. AWS IoT Jobs, for example, can send jobs instructing devices to download/install firmware and report their status.AWS Documentation+1

2. Make the firmware cryptographically signed

Don't rely on HTTPS alone.

Your release pipeline should produce something like:

firmware.bin
manifest.json
signature

The manifest might contain:

{
  "product": "sensor-v2",
  "hardware_revision": "B",
  "version": "3.7.2",
  "size": 524288,
  "sha256": "...",
  "min_bootloader": "2.1.0",
  "security_counter": 42
}

The device should:

1. Download the artifact over TLS.
2. Verify its hash.
3. Verify the manufacturer's digital signature using a public key embedded/protected on the device.
4. Check hardware compatibility.
5. Check that the version/security counter is acceptable.
6. Only then install it.

For example, MCUboot supports signed images and downgrade prevention using version/security-counter mechanisms.GitHub

Keep the signing private key off the device and preferably out of ordinary developer workstations. Your CI/CD release process should be the only thing authorized to produce production-signed firmware.

3. Use A/B firmware partitions

This is probably the most important reliability feature.

For a Linux device:

Flash
┌─────────────────────┐
│ Bootloader          │
├─────────────────────┤
│ Root filesystem A   │ ← currently running
├─────────────────────┤
│ Root filesystem B   │ ← install update here
├─────────────────────┤
│ Persistent data     │
└─────────────────────┘

The device downloads the new image into the inactive partition. It doesn't overwrite the currently working firmware.

Then:

A running
   ↓
download firmware
   ↓
verify signature/hash
   ↓
install into B
   ↓
boot B
   ↓
health check
   ↓
mark B good

If power disappears during the download or installation, A remains intact.

If B boots but crashes repeatedly, the bootloader should automatically return to A.

This is the model used by systems such as Mender and RAUC; Mender explicitly uses redundant A/B partitions for fail-safe OS updates, while RAUC supports atomic redundant-system updates and cryptographic bundle verification.Mender Docs+1

For an MCU, a bootloader such as MCUboot can provide the equivalent two-slot mechanism. Mender's Zephyr integration, for example, combines a device-side client with MCUboot's two-slot update mechanism and rollback behavior.Mender Docs

4. Separate "download" from "activate"

Your OTA agent should behave roughly like this:

if (!update_available())
    return;

download_to_inactive_slot();

if (!verify_hash())
    fail();

if (!verify_signature())
    fail();

if (!compatible_with_hardware())
    fail();

if (!version_is_allowed())
    fail();

mark_slot_pending();
reboot();

After reboot:

if (self_test_passed() &&
    peripherals_working() &&
    application_started()) {

    mark_slot_good();

} else {
    reboot_into_previous_slot();
}

Don't consider an update successful merely because the binary was written successfully.

Successful OTA means the new firmware boots and passes meaningful health checks.

5. Build fleet rollout controls

Never go directly from:

version 3.7.1 → 100% of devices

Instead:

Internal test devices
       ↓
1%
       ↓
5%
       ↓
25%
       ↓
50%
       ↓
100%

At each stage monitor things such as:

• boot failures
• rollback rate
• crash/watchdog rate
• connectivity
• battery consumption
• application-specific health metrics
• update completion rate
• unexpected device resets

Then automatically stop the rollout if failure rates exceed your threshold.

AWS IoT Device Management, for example, supports fleet grouping, rollout velocity and failure thresholds for OTA deployments.Amazon Web Services, Inc.+1

6. Give every device useful metadata

Your OTA service should know at least:

device_id
hardware_model
hardware_revision
current_firmware
bootloader_version
region
environment
last_seen
battery_level
update_status

This lets you target:

hardware_model == "sensor-v2"
AND hardware_revision == "B"
AND current_firmware < "3.7.2"
AND environment == "production"

rather than accidentally sending a firmware image intended for one hardware revision to the entire fleet.

AWS's continuous OTA mechanism, for example, supports targeting devices using metadata attributes.AWS Documentation

7. Design for bad networks

The update process should tolerate:

• Wi-Fi/cellular disconnections
• device power loss
• partial downloads
• server failures
• intermittent connectivity
• devices being offline for weeks

Useful techniques include:

• resumable downloads
• chunked downloads
• checksums
• download-to-inactive-slot
• retry with exponential backoff
• randomized polling/jitter
• CDN/object storage for firmware
• update bandwidth limits
• persistent update state

Don't have 50,000 devices all wake up at exactly midnight and request a 20 MB image.

8. Protect against rollback attacks

There's an important distinction between:

"Rollback automatically if the new firmware crashes." and:

"Allow anyone to install an old, vulnerable firmware." You want the first but generally not the second.

Use a monotonically increasing security counter or equivalent anti-downgrade mechanism. MCUboot, for example, supports security counters for preventing installation of images below the trusted version.GitHub

A typical policy is:

current security counter = 42

image counter 41 → reject
image counter 42 → policy-dependent
image counter 43 → accept

9. Don't forget the bootloader

A common OTA mistake is designing a beautiful application update system while making the bootloader impossible to recover.

Treat these separately:

Immutable / recovery bootloader
             ↓
      OTA-capable bootloader
             ↓
       application A/B

Ideally you have a protected recovery mechanism—such as a factory image, ROM bootloader, hardware recovery interface, or redundant bootloader strategy—so a failed bootloader update doesn't permanently brick the device.

For embedded Linux, frameworks such as RAUC and Mender can handle substantial portions of this update architecture.GitHub+1

10. Choose the implementation based on your device

Embedded Linux

A practical stack could be:

Yocto/Buildroot
      +
U-Boot
      +
RAUC or Mender
      +
HTTPS/object storage
      +
OTA management server

Mender is particularly oriented toward fleet management and supports managed deployments, while RAUC is a lightweight update client/bundle system.Mender Docs+1

MCU / RTOS

A typical architecture is:

FreeRTOS / Zephyr
       +
MCUboot
       +
A/B flash slots
       +
HTTPS/MQTT
       +
OTA service

For MCUboot-based systems, the bootloader performs image validation and manages the slot transition/rollback.Mender Docs

Cloud-managed fleet

If you're already heavily invested in AWS, a reasonable architecture is:

CI/CD
  ↓
Code signing
  ↓
S3
  ↓
AWS IoT Jobs
  ↓
Device OTA agent
  ↓
A/B bootloader

AWS's OTA agent model explicitly expects the device to download the firmware, verify its digital signature, install it, and report the job result.AWS Documentation

11. A good minimum production design

If I were designing a new fleet today, I'd make these requirements non-negotiable:

The key principle is: an OTA system shouldn't merely make firmware downloadable; it should make an interrupted, malicious, incompatible, or broken update fail safely.

If you tell me your MCU/SoC (e.g. ESP32, STM32, nRF52, Raspberry Pi), OS/RTOS, connectivity (Wi-Fi/cellular/etc.), approximate fleet size, and whether you have a cloud backend, I can sketch a concrete architecture and implementation—including partition layout, bootloader flow, manifest format, and device-side OTA code.

Implementing over-the-air (OTA) firmware updates for an embedded fleet requires a robust dual-bank memory architecture, cryptographic signing, and a reliable device management backend to prevent bricking devices in the field.[mender.io](https://mender.io/resources/reports-and-guides/ota-updates-best-practices)[[2](https://memfault.com/blog/ota-for-iot)](https://memfault.com/blog/ota-for-iot/)[[3](https://witekio.com/embedded-software/embedded-linux/ota-update)](https://witekio.com/embedded-software/embedded-linux/ota-update/)

Core Implementation Architecture

• Dual-Bank (A/B) Partitioning : Allocate your non-volatile flash memory into two separate application slots (Bank A and Bank B). The bootloader executes the active image while the OTA client downloads the new firmware binary into the inactive bank, executing an atomic switch upon success.[mender.io](https://www.youtube.com/watch?v=SipMi1VcYTU&t=379)[[2](https://memfault.com/blog/ota-for-iot)](https://mender.io/resources/reports-and-guides/ota-updates-best-practices)
• A Fallback/Rollback Mechanism : Program your primary bootloader to verify the integrity and health of a newly booted image. If the application crashes repeatedly or fails a health check post-update, the bootloader automatically reverts to the previous working partition.[mender.io](https://memfault.com/blog/ota-for-iot/)[[2](https://memfault.com/blog/ota-for-iot)](https://witekio.com/embedded-software/embedded-linux/ota-update/)
• Cryptographic Verification : Sign every firmware image using private-public key cryptography (such as Ed25519 or ECDSA) during your CI/CD build process. The embedded device must use a hardcoded public key—often stored in a hardware Root of Trust or secure element—to verify the signature before writing or booting the image.[mender.io](https://memfault.com/blog/ota-for-iot/)
• Transport and Update Client : Implement a lightweight client protocol (such as MQTT, HTTPS, or specialized device management agents) for the device to poll or receive payloads from a secure cloud platform.[mender.io](https://witekio.com/embedded-software/embedded-linux/ota-update/)[[2](https://memfault.com/blog/ota-for-iot)](https://mender.io/resources/reports-and-guides/ota-updates-best-practices)[[3](https://witekio.com/embedded-software/embedded-linux/ota-update)](https://www.youtube.com/watch?v=C0zaW6SajTk&t=366)
• Fleet Management Server : Deploy a backend distribution system (such as AWS IoT Device Management, Mender , or Memfault ) to manage staged rollouts, target specific hardware groups, and monitor deployment health metrics.[mender.io](https://memfault.com/blog/ota-for-iot/)[[2](https://memfault.com/blog/ota-for-iot)](https://witekio.com/embedded-software/embedded-linux/ota-update/)

Step-by-Step Deployment Workflow

1. Build and Sign : Compile your application binary and generate a cryptographic signature hash (e.g., SHA-256) of the image.
2. Staged Upload : Push the validated artifact package to your secure server repository or OTA orchestration platform.
3. Notification and Download : The device client queries the server (via scheduled polling or broker alert) and downloads the update into the inactive flash bank over an encrypted TLS connection.
4. Integrity Check : The device validates the file checksum and verifies the digital signature against the stored public key.
5. Atomic Swap and Reboot : The bootloader flags the new bank as active, reboots the system, and runs initial smoke tests before finalizing the slot commitment.[mender.io](https://www.youtube.com/watch?v=C0zaW6SajTk&t=366)[[2](https://memfault.com/blog/ota-for-iot)](https://memfault.com/blog/ota-for-iot/)[[3](https://witekio.com/embedded-software/embedded-linux/ota-update)](]()](]()]()

If you'd like to narrow this down, tell me:

• What is your target hardware (e.g., bare-metal microcontroller vs. embedded Linux)?
• Which connectivity protocol are you using (Wi-Fi, cellular, BLE)?

I can provide a tailored architecture recommendation or configuration example.