During the authors internship at Trail of Bits, they setup the tool
wycheproof on a JavaScript library called elliptic. The idea behind this package is great: let's take a collection of known attacks against cryptographic protocols and run them against the library. This plugin drastically improves the security confidence in a library imo. They claim this test suite would be good for CI/CD.
To do this, they had to setup a harness around elliptic for wycheproof. Once this was done, they ran the tool over the library. They had several findings that they started to triage as either false positives or real findings. When trying to integrate, there's also a question of did I set this up correctly or are these issues my fault?
The first issue they call out is around EdDSA signature malleability. Ellipic curves have two valid y points. In reality, only one of these should be allowed, according to specifications. This is technically valid math but bad for many cryptography purposes. This could lead to consensus failures or replay protection bypasses. Personally, I don't think that cryptographic libraries should enforce the malleability because it is desirable in some cases but I'm not a cryptography expert so what do I know. This was the vulnerability that was fixed while the next one was not.
The second bug is that hashes with leading zeros can cause a signature to become invalid. This appears to be a string parsing bug where the conversion new BN(msg, 16) removes the leading zeros. When it's used later, some offset math is wrong as a result. This bug was never fixed.
The usage of the cryptography testing library is interesting. However, I'm not sure that these are security "vulnerabilities". I agree that they differ from the specification. In the context of blockchain where two libraries need to have perfect parity, these are both bugs for sure. In the context of causing damage via signature validation, the first one has merit in specific situations while the second does not.
Funny point at the end: Wycheproof test developer Daniel Bleichenbacher independently discovered and disclosed issue #321, which is related to this discovery. This is a really famous cryptography person who discovered some attacks on RSA back in the day. It's cool he's still in the game!