The author of this post decided to take a trip down memory lane by reviewing a vulnerability they found in VirtualBox 5 years ago. The post is heavy into the methodology, which I always appreciate!
While doing recon and looking at previous research, they came to the conclusion that the VBVA (Virtual Box Video Acceleration) was good to look at. Why? One thing is that video subsystems are known to be hard to implement securely not just in VBox but other virtualization software as well. This is because they are many handling of pointers, offsets and things, which can lead to memory corruption.
To look at the source code, they decided to look for sources. Any input that was directed controlled by users within the virtual machine was considered. They were curious about how integers overflows were being prevented. One way of doing this is by saving the result of two smaller integers into a single integer, such as uint16_t into uint32_t. Although this is fine in most cases, there is an additional multiplication by 4, that can lead to an overflow occurring.
What can we do with this? The height * width * 4 value is used for a sanity check to ensure we're not writing outside the bounds of the VRAM buffer. However, since this check will overflow, we can cause memory corruption in future writes. They noticed that the function crMClrFillMem() for filling in a rectangle img can be used to write outside of the buffer! The OOB write has a controlled value and a controlled offset, which is an amazing primitive. This also grants an OOB via the same bug.
Overall, a good post on the discovery of a super powerful vulnerability within VBox. To me, I don't like the preventative strategy of overflows used, as it seems prone to errors with extra addition being done.