Juno is a proof of stake based blockchain built on Cosmos and CosmWasm. While going through an upgrade of the eco-system, a bunch of issues were occurring... forks, lack of consensus and many other things. What was going on?
In 2021 the Terra blockchain was dealing with the same issue. What was the problem? Different versions of libc were outputting different gas calculations. The solution? Statically link the same libc version into the binary so that everyone can use it. This didn't work though. Hmmm.
At this point, they had attempted to role out the static version of the binary but this didn't work. They stopped the chain, told everybody to stop and decided to triage and analyze what was really going on. Sometimes, taking a step back is the best way to move forward.
The author of this post decided to review the pending transactions of their own validator's memqueue. While doing this, they noticed a weird transaction that was a call to a CosmWasm contract with some indirection. When they started reviewing the state, they noticed that the chain halted while changing the name. Why did this occur?
Every single node that the operator queried had a different name field. In particular, it had split into 125 different networks! This was at the fault of CosmWasm and led to them not reaching consensus. This vulnerability is known as indeterminatism; this is when the operations being performed do not have a deterministic path, leading to rapid forking.
This is not the end of this though. More issues were still occurring. While trying to get a node up and going, they kept going down with a very similar looking transaction. It was another known CosmWasm
non-deterministic bug. The bug a non-deterministic ABCI query being made from a CosmWasm smart contract.
Overall, an interesting lesson on non-deterministic behavior on blockchain drastically affected the software. Thanks for the good write up!