Resources

People often ask me "How did you learn how to hack?" The answer: by reading. This page is a collection of the blog posts and other articles that I have accumulated over the years of my journey. Enjoy!

The author of this post found several vulnerabilities purely using LLMs. In this post, they outline some takeaways from the hundreds of prompts they have used. To start with, they debunk an intuition: more data isn't necessarily better. LLMs have a recency bias and suffer from context rot. Providing too much context, overly-scaffolded chaining of prompts, and bloated SKILLS.md files hurts the process.

They start from the beginning: why can't you just say "LLM find me bugs". The first issue is that LLMs don't have a notion of impact. So they will end up with a nonsensical threat model. Without clear trust boundaries, it does a horrible job. Second, general prompts get general answers. The model will pattern-match common bug classes and not go very deep. Threat modeling is the ultimate form of context compression.

The process for creating a good threat model drew on prior CVEs: list several previously discovered bugs and how they were triggered. From this threat model, they created plausible bug classes to review the codebase for. On ParseServer, the LLM identified an interesting trust boundary with multiple keys at different privilege levels. After surveying the code, they identified that several handlers gate access on isMaster() without checking isReadOnly(). With a narrower prompt focusing on this, they found 3 bugs and a fourth on in JWT aud claim lacking verification.

In the HonoJS library, they reviewed the JWT verification flow. They gave it previous CVE's and general JWT implementation vulnerabilities. From there, they asked the LLM to identify high-risk areas, which it identified as algorithm confusion and default fallback behavior. With some direction, they scanned for the specific flaws, and the LLM surfaced two concerning patterns. Finally, they asked in what exact conditions an LLM could trigger these vulnerabilities. This popped out two classic algorithm confusion bugs.

On ElysiaJS the LLM flagged secrets rotation as being sketchy. After further digging, the LLM found that a Boolean initialization error caused signature validation checks to never fail during secrets rotation. On harden-runner, a GitHub Actions tool for preventing data exfiltration, it found several gaps in the syscall filtering that allowed data to be exfiltrated. On a similar product called BullFrog, they identified three distinct bypasses via similar methods.

Finally, in Better-Hub, an alernative GitHub frontend, they found six XSS variants, all stemming from unsanitized Markdown. One of the rabbit holes was on caching, which led to two authorization bypasses for private repository content leaks. Instead of deriving the threat model from other data, they described it themselves. The LLM helped find high-risk areas with the prompt "What happens when user-controlled content is rendered unsafely in a context that has access to stored credentials?"

None of these code reviews had a giant checklist, huge scaffolding or anything else. They started with a simple threat model that identified sections of codebases that had security-critical operations. From there, they let the LLM focus heavily on that section of code and try to violate various security invariants. The claim here is twofold: use threat models and pick small yet sensitive slices of the codebase to find bugs.

They have a few general tips for prompting. First, assert that vulnerabilities exist; this is important because LLMs have been trained to be agreeable. Similar to the first one, asserting search pressure by saying a vulnerability has already been found makes it much more likely to find issues. Again, telling it to assume that developers make mistakes, rather than rationalizing the code, seems useful for finding bugs as well. Second, ask for the exploit, not an assessment; this forces the model to produce a concrete, testable input to trigger the bug.

LLMs are very good at comparative analysis. It can leverage its training data against known patterns to find bugs. Another tip is not settling on the first answer, you can ask for more with different criteria. A simple "what else" can be useful. The final tip they give is constraining how the application can be attacked. For instance, stating that it's a remote unauthenticated attacker. This eliminates many false positives and helps the model stay focused.

This was a fantastic post that is definitely going against the grain. In this case, the author was in the loop the whole time, instead of having an agent do everything. Less context and more focus seem to be the general theme of this post.