Notes on (Unsuccessfully) Glitching the ESP8266EX

ESP8266’s bricked: 2

Many months ago, I wrote on some experience I had glitching the ESP8266. Yesterday, I picked up where I left off, and attempted to explore various options for this type of attack, as well as refine my approach to glitching (particularly with crowbar circuit glitching, due to laziness).

I will write about my experience in this post.

As a brief word of forewarning, I’ll note that I wasn’t able to achieve a successful limited-instruction glitch (successfully modifying a hash calculation without impacting the rest of the operation). This isn’t a post about a known-successful method.

A special thanks!

Usually, I will provide my acknowledgements and thanks to the author of CTF challenges and whatnot towards the end of a post, but as this post will continue to grow, I will provide it here:

  • Thanks to Silvio and the BSides team of 2017 for providing the initial hardware which got me started.
  • Thanks to everyone who has, or will, donate their badge to science. I would normally source them myself, but in an attempt to minimize errors caused by variation in hardware, I am looking for donations of BSides badges from 2017. I understand two different variations are in circulation, I don’t believe it makes a difference.
  • Thanks to everyone who has answered my questions and helped me learn this, without giving away answers. Something something teaching men to fish.

On with the show.

Attack Planning

On first attacking the ESP8266, my first task was to remove the metal shielding, and inspect the circuit layout. This looked like this:

Note what appears to be a flash memory chip below the CPU: this is, in my opinion, also worth consideration, but is out of scope for our current work. I then began to review the datasheet of the ESP8266EX processor:

We can immediately see some interesting pins (marked here with stars). Section 2 of the datasheet helpfully summarizes their functions:

  • CHIP_EN: Chip Enable. High: On, chip works properly. Low: Off, small current consumed.
  • EXT_RSTB: External reset signal (Low voltage level: active)
  • VDDPST: Digital/IO Power Supply (1.8V ~ 3.6V)
  • XTAL_OUT: Connect to crystal oscillator output, can be used to provide BT
    clock input

Until now, I had been using a simple counting loop style Arduino sketch as the “target” for my glitching work. The ESP8266 runs at a speed of many times an ATMega328P, so I modified my target accordingly: I used a script which SHA1’ed the character “A”, then SHA1’ed the result of that etc, for 7500 times, printing the result at 2500 and 5000 times. This would allow me to see glitches affecting both maths operations and memory management operations. You can find the new script here. Unfortunately, many of the screenshots were taken with old versions of the code.

Building the Glitch Platform

This is perhaps the only successful part of this experiment – this works to great effect on an ATMega processor (glitching the entire power supply, or via a crowbar circuit).

The next step of this was to build the glitch platform. This used an Arty FPGA board to control the glitch generation, and a BSS138PW mosfet to create a crowbar circuit, connecting the target rail to a local ground. It looked something like this:

The thing on the right is a Schmartboard: it’s just a fancy breakout board with these “solder rail” things on it – it sure as hell beats soldering a tiny fucking 0.65mm pitch transistor without one.

For the software, I improved upon my previous work, ending up with an FPGA design which allowed both “sustained” glitches and “101010101” glitches. It’s probably easier to describe in picture form:

My final code included a few toggle-able modes of operation, which would hopefully help me automatically find glitches (with manual review of the output). You can find the code here.

Soldering

As I would be soldering to little tiny pins, I found that normal soldering wouldn’t work – instead, the following method seemed to work OK:

  • Firstly, coat the target pin with some solder. It doesn’t matter if you “miss” and some gets on neighbouring pins, as long as you don’t form a bridge.
  • Then, use wire cutters to sharpen the point of your wire.
  • Coat the wire with solder. Use a very small amount – don’t leave a dangling “tip” of solder, or this will melt and form a blob.
  • Press the wire against the target pin, and hold the iron against it’s side for short bursts of time. Holding it with tweezers didn’t give me enough control, so this was mostly to avoid burning myself too badly.

In diagram form:

This was of particular use in soldering to the pins between the flash chip and the microcontroller.

CHIP_EN and EXT_RSTB

My first targets were CHIP_EN and EXT_RSTB. These could easily be glitched without any further modification, by soldering directly to the pin, and pulling them down momentarily.

CHIP_EN produced an interesting behaviour when glitched – no matter when the glitch was, it would “desynchronise” the UART-over-USB output, and only correct itself upon a device reset (via the reset pin). In logfiles, it looks like this:

Note that while the output made no sense visually, if we throw it into a logic analyzer, we can clearly see the repeated data:

Glitching EXT_RSTB didn’t seem to produce any interesting results (though this is likely due to me sucking, as opposed to the glitch not working).

VDDPST (and the Voltage Regulator)

I attempted to glitch the VDDPST rail independently. On further discussion (with people who actually know how2play electronics), I discovered the purpose of two VDDPST pins was to resist momentary glitches caused by an undersupply of current on one of the pins – this did it’s job admirably, and glitching either one of the pins caused no observable glitches.

My next step would have been to glitch them both together (using two mosfets), but unfortunately, I bricked the target device before I got a chance to – read on for this particular tale.

Along the same lines, the device also makes use of an AMS1117 voltage regulator. I attempted to build a crowbar circuit to short out the output to ground, which in hindsight was probably not a good idea considering it was directly connected via (and drawing power from) USB. This also produced no noticeable effects, and I don’t know why (and didn’t have a chance to investigate further).

XTAL_OUT

Glitching XTAL_OUT bore the most interesting results of the lot. Both forms of glitching (grounding for an extended period of time, or 010101 glitching) produced interesting results, like this:

While we can now see output corruption, the SHA1 calculation in this case was still intact (and when repeated with the 7500-iteration SHA, this still held true).

In some regards, one could consider transient faults in output, corresponding to glitches on a clock crystal line, a success – though I’m not sure why the SHA1 calculation holds up in this case. What do you think?

My next attempt was to simply remove the crystal oscillator and replace it with my own 26MHz signal, also driven off the FPGA. Not wanting to turn on a noisy hot air gun late at night, I attempted to cut the trace on the PCB, and solder my own clock signal wire directly to the MCU:

Unfortunately, I cut too close to the actual pin and ended up mangling the pin irrecoverably – I don’t have a microscope or fine enough soldering kit (solder, soldering iron tip – though I do have some 0.15mm diameter wire that “gives off toxic fumes when soldered” – thanks China!) to unfuck this manually. At this point, this particular ESP moduel was salvaged for parts.

Bringing Method to Madness

Unfortunately, the screenshots will not be 100% accurate in this section, as I had failed to screenshot some things. I will be more careful in future, and use the best screenshot applicable.

At this point, I proceeded to unwrap a second board, and resumed the experiments above. In an attempt to bring method to the madness, I used an oscilloscope to measure the effect of each variance in glitching.

To start with, we can resume with what is the logically simplest test case – to use a crowbar circuit to temporarily ground one of the VDDPST lines, after having removed (what I think is) the decoupling capacitor within the ESP module itself.

With the probe set to 10x, we can see a small dip in voltage on the VDDPST rail, followed by a spike – I assume this is caused by some manner of decoupling capacitor discharging. Note that this doesn’t necessarily need to be an actual capacitor – this can be anything with capacitance.

At this point, I was a little concerned that it was my setup – that something might limit how much current can pass through the mosfet. I figured I would use a piece of reference hardware – I got out my chipwhisperer board, and used it to produce a single shot glitch:

After breathing a sigh of relief that it wasn’t “just me”, I identified and removed more decoupling capacitors, and to be safe, powered the board from the VCC / GND pins on the ESP module itself:

This didn’t seem to produce *too* much of an effect.  There appears to be less “ringing” after the glitch itself, I’m not sure what the implications of this are – I am confident this will become clear in time as I learn more about electronics.

As a next step, I did some reading about power consumption of the board, which led me to this link: at this point, I thought I’d follow the link, to attempt to isolate the microcontroller as much as possible. My first step was to (horrendously) desolder the voltage regulator:

And attempt another glitching exercise:

This is a poor screenshot, glitching at this point regularly produced dips to approximately 2V. To me, this indicates that removing the voltage regulator, even though nothing is supplied on the 5V input pin, has a significant and reliable impact on the success of glitching against the board.

Following the article above, I attempted to remove the USB-to-Serial chip, but was sidetracked by what appeared to be two decoupling capacitors next to the USB port. Unfortunately, desoldering these produced an unhealthy-looking pattern on the VDDPST power trace (with or without glitches introduced), and the UART transmissions from the board finishing.

At this point, I figured I would remove the ESP module and breadboard it, in an attempt to revive the chip and glitch the power supply in a more isolated manner – there’s no glue used, so this comes off easily with a heat gun and an electronics opening spatula thingy from under the WiFi antenna:

Unfortunately, the module refused to respond to my attempts to revive it.

Next Steps

Unfortunately, the bricking of the target device means I am momentarily unable to progress from here. My next test cases are as follows:

  • Attempt to breadboard the module from the start, powering it completely independent of the NodeMCU board.
  • Attempt to isolate the power for the microcontroller and the Flash, so when I ground a power rail, it’s impacting only the microcontroller (shouldn’t have an impact, but maybe).
Posted in Bards, Computers, Jesting | Leave a comment

A Reflection on 2017

As we prepare to wind down for the year, it is appropriate that we take some time to reflect on our successes and failures over 2017.

Thoughts On Community

Over this past year, much has occurred. As the Platypus Initiative, we have had a second (and yet-again, trend-setting) conference. We have explored new avenues of collaboration with our friends from Electrofringe, and we have kicked off Australia’s first domestic hardware capture-the-flag. With each event that we do, more volunteers approach us, with content which they can help present to everyone. In this way, we are slowly but surely building a self-sustaining pool of content.

As a community, we have swelled to almost 200 members yet again – it warms my frozen heart that we usher in the new year with another Great Purge (our third? I lose count). This brings back memories of the IRC’s of old, where dozens of people would lurk in a channel – no-one would ask questions for fear of looking stupid, and there would be little discussion of actual technology. It is my intent that this does not become the case for the Platypus Initiative.

remember this?

For all our progress – we must not forget our roots. I must think back to what we were doing a year and a bit ago – we were just a group of nerds who wanted to do cyber stuff, so we got together and did cyber stuff, and treated one another as equals. It is my hope we do this once more.

Thoughts On Technology

Over the past year, I have spent some time learning a few hardware security parlor tricks. While conceptually simple, this is a tremendous eye-opener – both that there is tremendous profit in wiring up some GPIO buttons and a USB HID gadget, and that most “hardware hacking” that gets talked about is mostly the same few simple concepts.

The practical change is noticeable – I can now understand attacks which were once the equivalent of sorcery (think Heart of Darkness / HID, USB descriptor glitching, that kind of jazz). The thing about advanced technology being indistinguishable from magic works in reverse too – with just a little bit of basic understanding, the pieces of the magic show come together, and the whole thing becomes understandable, and with practice, very repeatable.

In an effort to focus my time, I have taken to using Trello to build a rudimentary task list. For me, the motivating effect of this cannot be understated, as long as I:

  • Keep the list up to date
  • Write items up to a consistent standard.

It is a sobering reminder that there is much to learn and do, and looking to 2018, I intend to spend more time analyzing different targets: shifting away from CTF-style content to attacking more real hardware (but still attempting as many CTF’s as I can, and trying to score as high as I can).

The Long Road Ahead

In 2017, to me, our greatest success is in the amount of activity we were able to provoke – from people spending time with us at our hands-on workshops, to people having a great time learning at hw101 with our friends from Sectalks, to the number of people eagerly taking part in our hardware capture-the-flag. Truth be told, it doesn’t matter what someone learns in a closed-classroom / traditional content format: learning for themselves that it’s OK to take risks learning is worth more than every workshop under the sun.

It’s the old adage of teaching someone to fish, versus giving them a fish, except with more cyber.

Looking to 2018, it is my intent to provoke yet more activity. Prep work is already underway for events hopefully to kick off in late January. I hope to establish stronger working relationships with the rest of the Australian “infosec scene”, to the extent that it helps everyone in bringing more varied content to the communal table. A special thanks goes to Sectalks and Pedram here, for putting up with all my shit this year.

For those of you reading this on New Year’s Eve, I wish you all a most ethical holiday. May you take this day and the next to rest and recover, to take stock of your achievements over the past year, and of the year which lies ahead.

Happy New Year’s, see you in Insomnihack Teaser CTF 🙂

 

Posted in Bards, Computers | Leave a comment

Writeup – Morph (34C3)

This week, I participated in the 34C3 CTF. Unfortunately, I was only able to solve a single challenge in the time available (and not a high-value one) – as always, I will present the writeup here.

Morph

This challenge was presented as a Linux binary, which you can download here.

On first opening the binary in IDA, the beginning of the main function immediately gives away the heart of the challenge:

It is obvious that there is self-modifying code, currently located at “src”. Travelling further down into the disassembly, we see the beginning of the payload:

Further down still, we can see the “next stage” decryption code:

There’s no immediate clue to what our decryption key is – but given that we know where the “original” self-modifying code is, we can modify the decryption stub to include a breakpoint before retn, allowing the code to decrypt itself in memory (wherever it is), copy out the decrypted memory, and stick it back into IDA. A little gdb later, and we have some limited success:

At this point, the astute reader will notice that the “parts” of the payload actually follow a very static pattern, with the first instructions being “push esi” for both the first and second part. Therefore, the easiest solution would be to brute force the decryption keys for the remainder of the challenge.

A bit of Python later, and the flag is ours. You’ll also need the “1.bin” file extracted from the challenge itself.

I would like to thank the 34C3 CTF organisers for putting together this event – well done. See you all in SANS Holiday Hack, followed by the Insomnihack Teaser CTF early next year 🙂

Posted in Bards, Computers, Jesting | Leave a comment

Reversing the R216H (Brief Note)

After working out how to shell the E5573 device, I briefly turned my attention to the Huawei R216H. Despite the considerable difference in packaging, the device looked somewhat familiar. On opening it, I was met with a similar scene, with JTAG and serial test points in precisely the same place:

For reference, the basic JTAG pin out is (from 0 at the top):

  • TCK 1
  • TMS 3
  • TDI 4
  • TDO 5
  • Reset 7

Pin 2 is grounded, and the voltage reference is ~1.5V from the boot pin.

The only visible difference I could see was the addition of a small green LED up the top. According to the packaging, this indicates if you have a new SMS or not (because clearly, SMS is what people use this for).

Initially, I had some trouble getting the wires to stick, but I found that curving the ends of the magnet wire inwards and “dipping” it into a blob of solder seemed to work a bit better than regular soldering – and was much more forgiving of poorly knife-stripped wire ends:

Exactly the same trick works as the E5573 (patching authconsole in memory), for exactly the same expected result:

My next task was to gain persistent shell, so I could continue investigating the device. On investigating the various bin directories, I figured my approach would be to FTP the authconsole binary to a second workstation, patch the executable and FTP it back. After a little bit of network fuckery (and wrestling against ftpput – you need to specify all the arguments), imagine my surprise when I couldn’t find the very bytes which I patched in memory to get the shell.

Instead, I found the following (refer to my last post on the E5573 for the original source chunk):

To me, this indicated that the binary “on disk” was affected by our in-memory patch, but was somehow overridden at the next reboot, based on behavioral analysis. Still, this presented a simpler problem than FTP trickery. I remounted /app as a rw partition, and renamed the file a bit.

To my astonishment, this committed the change to disk – and the device is now correctly perma-rooted, accepting any password, without overwriting NVRam or downgrading firmware.

Posted in Bards, Computers, Jesting | Tagged , , | Leave a comment

Reversing the e5573 – Completing the Compromise

Recently, I picked up reverse engineering of the E5573 device. The overall goal was to compromise the “authconsole” binary, and gain access to a shell on the device, without use of the “reset pin” trick outlined by forth32 (and therefore, preserving NVRAM).

We can accomplish this via the JTAG interface, using a J-Link to affect compromise. From previous extraction of vendor firmware, we know that the binary controlling authentication is “authconsole”. We first locate this in memory: from practice, I’ve found that this is generally located within 0x200000 from 0xc3000000. Ensure that you do not dump too much memory at once, or your device’s watchdog timer will complain.

You can quickly check for the presence of this binary by searching for “Welcom to enter” or any of the binary blocks below. Once identified, we can patch two key code paths within the authconsole binary. Firstly, the password check itself (replacing 20 B9 with 05 46 is fine):

As well as a “return code” check for the application:

Here, we replace the offending instruction with 00 20 (mov r0,#0), therefore passing the next check, and avoiding the “eUAP login return” path.

Once this is done, you should get the “eUAP>” prompt, which acts as a shell. Simply use “sh” to turn this into an actual shell, on which you will be the root user.

At this point, I have derived a second method to access the e5573 device, without resetting the non-volatile RAM or overwriting the firmware completely. In hindsight, I had been extremely close to this solution before, but was unable to identify the authconsole binary in memory, due to subtle changes in code between the version running on my device, and the version running on the target.

Success!

Posted in Bards, Computers, Jesting | 6 Comments