Resources

People often ask me "How did you learn how to hack?" The answer: by reading. This page is a collection of the blog posts and other articles that I have accumulated over the years of my journey. Enjoy!

We hacked Google’s A.I Gemini and leaked its source code (at least some part)- 1628

Roni "Lupin" CartaPosted 11 Months Ago
  • At a live hacking event, the author of this post Roni decided to look at a new release of Gemini. In this update, the interface had Run in Sandbox as an option for Python execution. This was a sandbox used for executing Python in the context of Gemini.
  • The sandbox is powered by Google's GVisor, which is a user-space kernel. It does some magic with intercepting system calls and does a great job at restrictions access. It has a 100K bug bounty so this was likely impossible to escape. So, what's in the sandbox itself? Remember, this is a custom Python binary that is being ran.
  • They rewrote Front End to run their own Python code in the sandbox, to give them more flexibility. To see what they could do, they downloaded the Python interpreter itself. The custom Python binary had the os package in it, making it easy to see the file system. They were unable to pop a shell so this was the best they could do.
  • Exploring the file system led to some interesting files. In particular, the /usr/bin/entry/entry_point file. It was 579Mb in size though. Trying to base64 print it over the network just led to a network hang. Eventually, they read parts of the file in 10Mb chunks to get the whole file out.
  • Using binwalk, even though it's just a binary file, lead to some files being revealed. Upon reviewing these files, it appeared that some source code had been leaked. This code was not meant to be public, as pointed out by some comments in the interpreter program itself.
  • The sandbox had access to some Google specific network services but failed on other outbound connections. From reviewing some papers and reading internal source code, they realized that the process for opening up network information was done via Gemini itself. Gemini self-prompts to unlock necessary access. For instance, writing a prompt to access the Google Flights API will spawn a sandbox with the proper privileges - this wasn't noticed earlier because of how the sandbox was being started in their session.
  • By using a secondary prompt injection, access to a more privileged sandbox was given that did have this type of access. The hope was that they could get access to two extra file descriptors that they were not always given there. They were hopeful this would lead to a P0 type of issue to do things on Google infrastructure. The Google team did some assessments and believed that nothing extra on the box could be accessed even with a secondary prompt injection.
  • Within the sandbox, they found several protobuf files that were meant to be "internal-only" that exposed some sensitive information. Although I can't see the data they saw, I felt like the leakage of data was over-hyped. Maybe I'm missing something here though.
  • Overall, the post was okay. The sandbox escape discussion was fun! However, instead of memes and commands that they can't show the result of for "legal reasons", diagrams of the flow would have been appreciated. I also like posts that are more into the weeds with less story telling but that's a style choice I suppose.