ZKSync was launching the Aave V3 pool on their chain. While activating this they noticed a major bug. The bug only happened after a complex flow of supplying and borrowing assets. Since things looked weird, they paused the protocol to investigate.
The contract functioned perfectly on other chains so what was going wrong here? Their initial hypothesis was an issue with the bitmap math which had user data for 256 bits of boolean pairs. Since this bitmap was crucial to the protocol, being used for a token being collateral/borrowed, it was real bad.
From dynamic testing, they noticed that in situations with multiple assets enabled as collateral and borrowed, disabling the isBorrowing flag would zero out the wrong isCollateral flags. All flags to the right were being disabled for some reason.
They have a great example of the bug. For instance, if you have three assets that are being used as collateral and borrowed, the values would be 000...111111. If one of the debts was paid back, it should be 000...101111. In reality, this was becoming 000...100000. It believes that the user hadn't taken out a position, even though this simply isn't true.
From reading the compiler output, it became clear what was going on. The compiler had made an optimization that was wrong! Both parties that had reviewed it, Certora and Matter Labs, came to the same conclusion at the same time.
ZKSync takes the EVM bytecode and translates it to ZKSync bytecode using an intermediate representation in LLVM. The LLVM compiler had a
bug in it when handling 256 bit number optimizations.
The code xor (shl 1, x), -1 was optimized to rotl ~1, x. When the ~1 was converted, it would turn into 2^64-1 to be zero extended instead of sign extended. The code should have been roti 2^256 - 1, x but was incorrectly turned into rotl 2^64 - 1, x.
I appreciate the detailed write up from the zkSync dev team. Additionally, their phased rollout of Aave to check for bugs really paid off in this case. Even though the bug wasn't the teams fault, it's still their customers funds at risk. Great write up and incident response by the ZKEra team!