This is an interesting article about how people discover flaws within protocols in Cryptography. However, this article can be used for other types of bug finding.
To find bugs, meaningful engagement is required. What does this actually mean? Put in the effort!
The first claim is to write a proof. While writing a proof for the paper, some prevalent issues may appear that compromise the security of the system. This worked for an attacker on the Zcash cryptocurrency system.
What if there already is a proof? Then understand the proof and write one for yourself. This worked for an analysis of RSA-OAEP to find a bug and again for forgery an authenticated encryption scheme called OCB2.
What is next? Formal verification. This is defined as "In the context of hardware and software systems, formal verification is the act of proving or disproving the correctness of intended algorithms underlying a system with respect to a certain formal specification or property, using formal methods of mathematics." - Wikipedia
This technique is a programmatic way to verify the soundness of an algorithms implementation. But, now what?
Sometimes, the integration between two points is the issue. This can be seen with the KRACK exploit on WPA WiFi networks. This is essentially just a trivial replay attack that was not considered previously.
Finally, implementing the protocol can allow people to discover their own bugs. For instance, a vulnerability in
Zcash was discovered prior to releasing from exactly this. As a side note, this is an interesting article so it should be read!
Finding bugs is difficult. It requires a deep understanding of the system being worked with. Above are some ideas for attacking cryptography systems.