The rep movsb instruction is a super common way to move around memory in x86. The destination, direction and amount are all set in this call, but the processor does stuff under the hood.
In x86, the instruction decoding is very relaxed. Sometimes, compilers use redundant prefixes to pad a single instruction to get a nice alignment boundary. There are several prefixes that can be used, such as rex, vex and evex. On i386, there are only 4 registers which were encoded in the instruction. When this was doubled to 8 registers, there was no where to go.
So, the rex instruction adds an additional byte to the beginning of the instruction to encode this information. If this is found before an instruction like movsb, then it's silently ignored. Well, in most cases. The fast short repeat move instruction; the feature is all about moving small (less than 128 bytes) strings quickly
To test for architecture level issues, the author of this post uses Oracle Serialization. This generates two programs but transforms it to include micro architecture changes like fencing instructions. If the state of the program after serializing it is different, then something weird has happened.
While fuzzing using this technique, they noticed that adding redundant rex.r prefix instructions to an FSRM optimized operation caused unpredictable results. For instance, branches to random locations, branches being ignored and many other weird things. Somehow, this had corrupted the state.
Within a few days, they found out that triggering this on multiple cores led to exceptions and halts. Within an unprivileged guest VM, this could be used to crash the computer! So, what's going on?
The CPU has two main components: frontend and backend. The frontend fetches, decoding and generates the ops for the backend to execute. The backend then executes these instructions. The authors of the post think that there is a miscalculation in the movsb instruction size, which leads to extra backend entries to be processed.
Is this exploitable? Probably! However, there is no insight into what's being processed under the hood. So, the information above is just a guess from the author. Awesome post once again!