Recently, I had the pleasure of chatting with Paul Asadoorian, Principal Security Researcher at Eclypsium and the host of the legendary Paul’s Security Weekly podcast. Our conversation dove into the often-murky waters of embedded systems and the Internet of Things (IoT), sparked by a specific vulnerability discussion on Paul’s show concerning reference code for the popular ESP32 microcontroller.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

During the chat it became clear is that “embedded” isn’t a single, simple category. Paul helped break this down into a couple of categories that made sense to me.

Beneath the OS: Even standard PCs, servers, and laptops contain embedded systems below the main operating system. This includes the UEFI (Unified Extensible Firmware Interface) BIOS, which initializes hardware, as well as other components like TPM (Trusted Platform Module) chips and BMCs (Baseboard Management Controllers) – essentially separate computers running inside your computer, often on Linux.

Enterprise Appliances: These are the network and security devices common in corporate environments – firewalls, routers, VPN concentrators from vendors like Fortinet and Palo Alto, which have frequently been targeted by attackers. Paul firmly categorizes these appliances as embedded systems. Also included here are printers, building access controls, security cameras, and other devices more common in businesses than homes.

“True” IoT: Paul has a strict definition for IoT (Internet of Things). These devices typically run firmware, communicate with a cloud backend, and are managed via a mobile app. This architecture introduces multiple potential vulnerability points – the device firmware, the cloud service, and the app itself.

Microcontroller-Based Devices: This category includes popular chips like the ESP32 from Espressif. These systems often run code “closer to the metal,” but frequently rely on underlying real-time operating systems (RTOS), such as FreeRTOS. Development can range from lower-level control using vendor-specific toolkits like Espressif’s IDF to higher-level abstractions like Arduino’s libraries. These chips power everything from hobbyist projects to commercial products like smart home devices.

OT/ICS: Operational Technology and Industrial Control Systems (like PLCs) represent another class, often characterized by extremely long lifecycles of 20+ years and unique software stacks, though open source is still prevalent alongside commercial options like VxWorks.

A critical thread running through almost all these categories is open source is the foundation fo everything.

Reference code is production code

The conversation circled back to the ESP32 vulnerability that prompted the discussion. The issue wasn’t in Espressif’s core libraries, but in example code they provided to demonstrate Bluetooth functionality. Developers, naturally, often copy and paste directly from these examples into their own projects.

Espressif initially argued against assigning a CVE identifier to the vulnerable example code, stating it wasn’t a versioned software package but merely an example. But that creates the problem where if there’s no CVE for the source of the vulnerability (the example code in this instacne), how do you track it down the supply chain? Once copied and potentially modified in countless projects, finding the vulnerable snippet becomes incredibly difficult. Identifying this copied code requires techniques like source code fingerprinting, but even that can fail if developers rename variables or functions. The rise of AI code generation, potentially trained on or using such examples, adds yet another layer of complexity to detection (since the AI will probably also include the code, but you don’t know it’s copied)

The ESP32 story is a good example of the challenge of software supply chain security in the embedded and IoT space. The reliance on open source, combined with practices like copying reference code, it’s a tough problem.