OpenText Directory Services (OTDS) is a Java web application that provides authentication and user management for OpenText applications. They had seen this in various client projects and decided to take a closer look. Upon reviewing the code, nothing obvious came up besides a potential deserialization sink.
The function cookieToMap had the exact traits of a deserialization-to-RCE bug: using readObject() on user-controlled data. Except, there was a catch: the cookie must be signed. So, this seems like a dead end... the entire post is about their method for exploiting this.
The signed data is generated using an HMAC-SHA1 hash with a controlled message and IV. The message and IV were user-controlled and concatenated together to create the HMAC. The cookie had the following format:
- 2 bytes for the length of the signature
- Signature data
- Length of IV (2 bytes)
- IV.
- Length of message (2 bytes)
- Message. A serialized Java object.
The concatenation had no separation between the IV and the message. So, AA,BB is the same as A,ABB. If a part of the message could contain a deserialization payload, then this could be turned into RCE! So, what's the catch? There's a lot of controllable data in the cookie, but it must be valid UTF-8 strings. Additionally, the HMAC was using the compressed message rather than the actual message. This sounds like a CTF!
The compression library was zlib. For each implementation of the deflate algorithm, there's a table that must be provided. There is nothing that mandates the exact bytes and table to be used. So, you can just use a table from only 0x01 to 0x7F; there's even a tool called ascii-zip that already does this! So, they generated a payload with ysoserial and compressed it with ascii-zip. Now, this just had to land in the middle of the message.
They ended up needing to create their own encoder due to constraints with zlib, thinking the data was uncompressible. They ended up encoding everything by hand due to the complexity of the requirements. After encoding everything by hand, they got a DNS request to work!
They were able to use some of the data as a signing oracle, which was super interesting to me. Overall, a great, technical post on hacking software under real constraints.