Resources

People often ask me "How did you learn how to hack?" The answer: by reading. This page is a collection of the blog posts and other articles that I have accumulated over the years of my journey. Enjoy!

Jane Street is a quantitative trading firm that takes code quality seriously. One of the significant ways to improve code quality is through tests, as they act as documentation, a reminder of mistakes, and boost confidence during a refactor. Because of this, they wrote a framework called Aria to test their complex systems.

They have a list of tests types that they use internally:

• Unit tests. Modules and data structures without any side effects.
• Integration Tests. Simulated networking layer that allows for fine-grained interactions between services.
• Quickcheck. Random orderings of events that can feed a simulation.
• Version Skew. New client library changes work with existing servers and vice versa.
• Fuzz tests. Random data and see what happens.
• Lab tests. Performance regressions that run nightly are similar to production.
• Choas Testing. Change the environment with things like service restarts to see how the service reacts to it.

All of these have value, but the integration testing is the most crucial bit. Expressive tests, fsat, and deterministic allow for better coverage. The Antitheseis tool runs in a virtual machine with a completely deterministic hypervisor. This allows for faults to be created at weird points, that can potentially find bugs as a result.

The configuration of this tool allows for simulated production in test to find crazy edge cases. This is a double-edged sword though: a larger input space takes more time to run. So, the tool includes a powerful exploration engine for finding edge cases.

They have an example vulnerability that they found via this testing framework. It only happened after a specific server was restarted, before a ring buffer was filled and if the client sends a request for data prior to a snapshot. Because of this case, the client read corrupted data. But why? When the client was written, the server didn't have a snapshot feature so this issue wasn't even possible. Antithesis also gave them debugging tools and reproduction steps to make it possible to reproduce.

A good post on the benefits of testing. A bit too focused on the specific testing framework they used by the end, but the product demo was cool nonetheless.