ESP8266’s bricked: 2
Many months ago, I wrote on some experience I had glitching the ESP8266. Yesterday, I picked up where I left off, and attempted to explore various options for this type of attack, as well as refine my approach to glitching (particularly with crowbar circuit glitching, due to laziness).
I will write about my experience in this post.
As a brief word of forewarning, I’ll note that I wasn’t able to achieve a successful limited-instruction glitch (successfully modifying a hash calculation without impacting the rest of the operation). This isn’t a post about a known-successful method.
A special thanks!
Usually, I will provide my acknowledgements and thanks to the author of CTF challenges and whatnot towards the end of a post, but as this post will continue to grow, I will provide it here:
- Thanks to Silvio and the BSides team of 2017 for providing the initial hardware which got me started.
- Thanks to everyone who has, or will, donate their badge to science. I would normally source them myself, but in an attempt to minimize errors caused by variation in hardware, I am looking for donations of BSides badges from 2017. I understand two different variations are in circulation, I don’t believe it makes a difference.
- Thanks to everyone who has answered my questions and helped me learn this, without giving away answers. Something something teaching men to fish.
On with the show.
On first attacking the ESP8266, my first task was to remove the metal shielding, and inspect the circuit layout. This looked like this:
Note what appears to be a flash memory chip below the CPU: this is, in my opinion, also worth consideration, but is out of scope for our current work. I then began to review the datasheet of the ESP8266EX processor:
We can immediately see some interesting pins (marked here with stars). Section 2 of the datasheet helpfully summarizes their functions:
- CHIP_EN: Chip Enable. High: On, chip works properly. Low: Off, small current consumed.
- EXT_RSTB: External reset signal (Low voltage level: active)
- VDDPST: Digital/IO Power Supply (1.8V ~ 3.6V)
- XTAL_OUT: Connect to crystal oscillator output, can be used to provide BT
Until now, I had been using a simple counting loop style Arduino sketch as the “target” for my glitching work. The ESP8266 runs at a speed of many times an ATMega328P, so I modified my target accordingly: I used a script which SHA1’ed the character “A”, then SHA1’ed the result of that etc, for 7500 times, printing the result at 2500 and 5000 times. This would allow me to see glitches affecting both maths operations and memory management operations. You can find the new script here. Unfortunately, many of the screenshots were taken with old versions of the code.
Building the Glitch Platform
This is perhaps the only successful part of this experiment – this works to great effect on an ATMega processor (glitching the entire power supply, or via a crowbar circuit).
The next step of this was to build the glitch platform. This used an Arty FPGA board to control the glitch generation, and a BSS138PW mosfet to create a crowbar circuit, connecting the target rail to a local ground. It looked something like this:
The thing on the right is a Schmartboard: it’s just a fancy breakout board with these “solder rail” things on it – it sure as hell beats soldering a tiny fucking 0.65mm pitch transistor without one.
For the software, I improved upon my previous work, ending up with an FPGA design which allowed both “sustained” glitches and “101010101” glitches. It’s probably easier to describe in picture form:
My final code included a few toggle-able modes of operation, which would hopefully help me automatically find glitches (with manual review of the output). You can find the code here.
As I would be soldering to little tiny pins, I found that normal soldering wouldn’t work – instead, the following method seemed to work OK:
- Firstly, coat the target pin with some solder. It doesn’t matter if you “miss” and some gets on neighbouring pins, as long as you don’t form a bridge.
- Then, use wire cutters to sharpen the point of your wire.
- Coat the wire with solder. Use a very small amount – don’t leave a dangling “tip” of solder, or this will melt and form a blob.
- Press the wire against the target pin, and hold the iron against it’s side for short bursts of time. Holding it with tweezers didn’t give me enough control, so this was mostly to avoid burning myself too badly.
In diagram form:
This was of particular use in soldering to the pins between the flash chip and the microcontroller.
CHIP_EN and EXT_RSTB
My first targets were CHIP_EN and EXT_RSTB. These could easily be glitched without any further modification, by soldering directly to the pin, and pulling them down momentarily.
CHIP_EN produced an interesting behaviour when glitched – no matter when the glitch was, it would “desynchronise” the UART-over-USB output, and only correct itself upon a device reset (via the reset pin). In logfiles, it looks like this:
Note that while the output made no sense visually, if we throw it into a logic analyzer, we can clearly see the repeated data:
Glitching EXT_RSTB didn’t seem to produce any interesting results (though this is likely due to me sucking, as opposed to the glitch not working).
VDDPST (and the Voltage Regulator)
I attempted to glitch the VDDPST rail independently. On further discussion (with people who actually know how2play electronics), I discovered the purpose of two VDDPST pins was to resist momentary glitches caused by an undersupply of current on one of the pins – this did it’s job admirably, and glitching either one of the pins caused no observable glitches.
My next step would have been to glitch them both together (using two mosfets), but unfortunately, I bricked the target device before I got a chance to – read on for this particular tale.
Along the same lines, the device also makes use of an AMS1117 voltage regulator. I attempted to build a crowbar circuit to short out the output to ground, which in hindsight was probably not a good idea considering it was directly connected via (and drawing power from) USB. This also produced no noticeable effects, and I don’t know why (and didn’t have a chance to investigate further).
Glitching XTAL_OUT bore the most interesting results of the lot. Both forms of glitching (grounding for an extended period of time, or 010101 glitching) produced interesting results, like this:
While we can now see output corruption, the SHA1 calculation in this case was still intact (and when repeated with the 7500-iteration SHA, this still held true).
In some regards, one could consider transient faults in output, corresponding to glitches on a clock crystal line, a success – though I’m not sure why the SHA1 calculation holds up in this case. What do you think?
My next attempt was to simply remove the crystal oscillator and replace it with my own 26MHz signal, also driven off the FPGA. Not wanting to turn on a noisy hot air gun late at night, I attempted to cut the trace on the PCB, and solder my own clock signal wire directly to the MCU:
Unfortunately, I cut too close to the actual pin and ended up mangling the pin irrecoverably – I don’t have a microscope or fine enough soldering kit (solder, soldering iron tip – though I do have some 0.15mm diameter wire that “gives off toxic fumes when soldered” – thanks China!) to unfuck this manually. At this point, this particular ESP moduel was salvaged for parts.
Bringing Method to Madness
Unfortunately, the screenshots will not be 100% accurate in this section, as I had failed to screenshot some things. I will be more careful in future, and use the best screenshot applicable.
At this point, I proceeded to unwrap a second board, and resumed the experiments above. In an attempt to bring method to the madness, I used an oscilloscope to measure the effect of each variance in glitching.
To start with, we can resume with what is the logically simplest test case – to use a crowbar circuit to temporarily ground one of the VDDPST lines, after having removed (what I think is) the decoupling capacitor within the ESP module itself.
With the probe set to 10x, we can see a small dip in voltage on the VDDPST rail, followed by a spike – I assume this is caused by some manner of decoupling capacitor discharging. Note that this doesn’t necessarily need to be an actual capacitor – this can be anything with capacitance.
At this point, I was a little concerned that it was my setup – that something might limit how much current can pass through the mosfet. I figured I would use a piece of reference hardware – I got out my chipwhisperer board, and used it to produce a single shot glitch:
After breathing a sigh of relief that it wasn’t “just me”, I identified and removed more decoupling capacitors, and to be safe, powered the board from the VCC / GND pins on the ESP module itself:
This didn’t seem to produce *too* much of an effect. There appears to be less “ringing” after the glitch itself, I’m not sure what the implications of this are – I am confident this will become clear in time as I learn more about electronics.
As a next step, I did some reading about power consumption of the board, which led me to this link: at this point, I thought I’d follow the link, to attempt to isolate the microcontroller as much as possible. My first step was to (horrendously) desolder the voltage regulator:
And attempt another glitching exercise:
This is a poor screenshot, glitching at this point regularly produced dips to approximately 2V. To me, this indicates that removing the voltage regulator, even though nothing is supplied on the 5V input pin, has a significant and reliable impact on the success of glitching against the board.
Following the article above, I attempted to remove the USB-to-Serial chip, but was sidetracked by what appeared to be two decoupling capacitors next to the USB port. Unfortunately, desoldering these produced an unhealthy-looking pattern on the VDDPST power trace (with or without glitches introduced), and the UART transmissions from the board finishing.
At this point, I figured I would remove the ESP module and breadboard it, in an attempt to revive the chip and glitch the power supply in a more isolated manner – there’s no glue used, so this comes off easily with a heat gun and an electronics opening spatula thingy from under the WiFi antenna:
Unfortunately, the module refused to respond to my attempts to revive it.
Unfortunately, the bricking of the target device means I am momentarily unable to progress from here. My next test cases are as follows:
- Attempt to breadboard the module from the start, powering it completely independent of the NodeMCU board.
- Attempt to isolate the power for the microcontroller and the Flash, so when I ground a power rail, it’s impacting only the microcontroller (shouldn’t have an impact, but maybe).