The VMU was a Dreamcast add-on that primarily stored game data in its flash memory, but it also had a small LCD display, a D-pad, and inter-VMU communications functions. It also had room for a standalone game which could interact with the main Dreamcast games in limited ways. [Dmitry] wanted to see what else he could do with it. Basically everything.
We can’t do this hack justice in a short write-up, but the outline is that he starts out with the datasheet for the VMU’s CPU, and goes looking for interesting instructions. Then he started reverse engineering the ROM that comes with the SDK, which was only trivially obfuscated. Along the way, he wrote his own IDA plugin for the chip. Discovery of two ROP gadgets allowed him to dump the ROM to flash, where it could be easily read out. Those of you in the VMU community will appreciate the first-ever ROM dump.
On to doing something useful with the device! [Dmitry]’s definition of useful is to have it emulate a modern CPU so that it’s a lot easier to program for. Of course, nobody writes an emulator for modern hardware directly on obsolete hardware — you emulate the obsolete hardware on your laptop to get a debug environment first. So [Dmitry] ported the emulator for the VMU’s CPU that he found in MAME from C++ to C (for reasons that we understand) and customized it for the VMU’s hardware.
Within the emulated VMU, [Dmitry] then wrote the ARM Cortex emulator that it would soon run. But what ARM Cortex to emulate? The Cortex-M0 would have been good enough, but it lacked some instructions that [Dmitry] liked, so he ended up writing an emulator of the not-available-in-silicon Cortex-M23, which had the features he wanted. Load up the Cortex emulator in the VMU, and you can write games for it in C. [Dmitry] provides two demos, naturally: a Mandlebrot set grapher, and Flappy Bird.
Amazed? Yeah, we were as well. But then this is the same guy emulated an ARM chip on the AVR architecture, just to run Linux on an ATMega1284p.