Pentesting Web Services with Proprietary Formatted Input

Introduction

From time to time I come across a web service that expects its input in some proprietary format, usually JSON distorted in one way or another. A vulnerability scanner knows nothing about that stuff and can't properly fuzz it. (At the time of this writing Acunetix and Burp Pro support JSON only in HTTP responses.) In this case one has to resort to pure manual testing or partially automatic test with a fuzzer. Both approaches have their limitations, and I decided to finally find a way to run an automated scanner against proprietary web services.

In this blog post I will talk about how this problem can be approached. It includes the description of a practical setup, which I used twice to pentest proprietary web services with Burp (one expecting JSON-formatted input in the body, another in a POST parameter). It can be adapted for any other input format.

The described setup has a lot of limitations (see note 3 in the end of this document) and is somewhat cumbersome. Unfortunately, at the moment it seems to be the only way to approach testing the wheels being reinvented by developers over and over again.

The approach

Consider an ideal situation, with Burp supporting JSON-formatted input:

  1. The client connects to Burp and sends JSON-formatted request
  2. Burp receives the JSON-formatted request in the body
  3. Burp fuzzes the request
  4. Burp sends the request to the web service
  5. The web service receives the request and returns the response
  6. Burp receives the response and possibly forwards it back to the client

Our problem is step 3. Burp cannot properly fuzz the request it can't understand. To let it do so, we need to translate the original request into something Burp can understand and fuzz, and then translate it back.

Consider this request:

POST /method HTTP/1.0
Host: example.com
Content-Type: application/json; charset=utf-8

{"a": 1; b: "xyz"}

It needs to be reformatted to expose the inner structure of JSON-formatted data. This will be done with a custom PHP script which parses the request and resubmits it to Burp in the following form:

GET /tpl2http.php?P1_NAME=a&P1_VALUE=1&P2_NAME=b&P2_VALUE=xyz HTTP/1.0
Host: localhost

Given the latter request, Burp can fuzz. For example, it can try SQL injection test:

GET /tpl2http.php?P1_NAME=a&P1_VALUE=1&P2_NAME=b&P2_VALUE=xyz' or 'a'='a HTTP/1.0
Host: localhost

This request can be translated back into the following:

POST /method HTTP/1.0
Host: example.com
Content-Type: application/json; charset=utf-8

{"a": 1; b: "xyz' or 'a'='a"}

Altered request flow

To achieve the above, the request flow has to be changed, for example as following.

  1. The client resolves the host name (from URL) trying to get the IP address of the WS (web service), but instead arrives to our intercepting proxy. I use Burp Pro set to listen on port TCP/8080 with redirection to localhost:80.
  2. The client sends HTTPS POST with JSON-encoded body. For example, consider the following HTTP request coming from the client.

    POST /method HTTP/1.1
    Host: example.com
    Content-Type: application/json; charset=utf-8
    Content-Length: 110

    {"id":2, "jsonrpc":"2.0", "method":"getPurchaseHistory", "params":{"params":{"startIndex":0, "maxResults":5}}}

  3. Burp receives this HTTP POST and forwards to Apache listening on localhost:80 and set to redirect everything to a custom PHP script called http2tpl.php. The following addition Apache vhost does the redirection:

    RewriteEngine on
    RewriteCond %{HTTP_HOST} !=localhost
    RewriteCond /var/www/%{REQUEST_FILENAME} !-f
    RewriteRule ^(.*) /http2tpl.php/$1 [L]

  4. http2tpl.php script transforms HTTP POST request into plain HTTP GET with parameters, which can be parsed and fuzzed by scanners and sends this request back to Burp (localhost:8080, hardcoded in the script).
  5. Now Burp received a request with all the parameters exposed. It can fuzz them if asked to do so. Next Burp, being a proxy, sends the request back to Apache on localhost:8080, but now it goes to another PHP script.
  6. When tpl2http.php script is invoked, it re-assembles the original request, possibly with some parameter values altered by scanner.
  7. After the request is assembled, tpl2http.php script forwards it to Burp again, but to another listener set to forward the requests through to the target server. The port the second listener runs on is hardcoded into this script (I use TCP/8081). This resulting request will looks very much like the original one (it is slightly different from the original because PHP JSON decode/encode functions drop redundant white spaces). It will be proxied by Burp towards the target server.

    POST /*** HTTP/1.1
    Host: ***
    Content-Type: application/json; charset=utf-8
    Content-Length: 106

    {"id":2,"jsonrpc":"2.0","method":"get***History","params":{"params":{"startIndex":0,"maxResults":5}}}

    Here is another sample, note Burp's attempt to test for SQL injection:

    POST /*** HTTP/1.1
    Host: ***
    Content-Type: application/json; charset=utf-8
    Connection: Keep-Alive
    Content-Length: 125

    {"id":2,"jsonrpc":"2.0","method":"get***History","params":{"params":{"startIndex11952520' or 1=2--":0,"maxResults":5}}}

Summary

Here is the summary of the test setup for HTTP services:

  1. The first Burp listener (:8080) set to redirect requests to local Apache (:80)
  2. The client set to use this listener for HTTP requests.
  3. The second Burp listener (:8081) is set to invisible mode
  4. Local Apache (:80) set to redirect all requests to http2tpl.php
  5. Variables proxyhost/proxyport in http2tpl.php are set to the first Burp listener localhost:8080 (default)
  6. Variables proxyhost/proxyport in tpl2http.php are set to the second Burp listener localhost:8081 (default)
  7. http2tpl.php and tpl2http.php are deployed in Apache's webroot.

If HTTPS needs to be supported:

  1. The third Burp listener (:8443) set to redirect requests to local Apache (:443)
  2. The client set to use this listener for HTTPS requests.
  3. The fourth Burp listener (:8444) is set to invisible mode
  4. Local Apache (:443) set to redirect all requests to http2tpl.php
  5. Variables $proxyhost/$proxyport in http2tpl.php are set to the first Burp listener localhost:8080 (default)
  6. Variables $proxyhostssl/$proxyportssl in tpl2http.php are set to the fourth Burp listener localhost:8444
  7. http2tpl.php and tpl2http.php are deployed in Apache's webroot.

Notes:

  1. The same approach can be used to handle other input formats by changing the implementation of http2tpl.php (see comments inside). Script tpl2http.php is not JSON-specific, it merely gets HTTP request line, headers, body template, and set of parameter substitutions it needs to make and assembles the request back.
  2. The sample requests above are simplified, real GET /tpl2http.php requests carry serialized HTTP request line, headers, and body template. These values are sent in DST parameter, its integrity protected with HMAC in DST_SIGN parameter (to make sure Burp can't mess with them).
  3. The current implementation is extremely ugly, and probably does not let Burp identify all vulnerabilities it could have identified. It breaks binary content, for example images. This is because it uses fgets() function of PHP, which drops \x0d characters. It does not correctly relay HTTP headers. Still, it is better than nothing and has already helped me to find some input validation flaws.

Here is a zip file containing http2tlp/tpl2http.php scripts and Apache vhost configs.

Comments

Just came across another attempt to pentest a web service using non-standard input encoding format
http://blog.gdssecurity.com/labs/2010/5/6/fuzzing-gwt-rpc-requests.html . Here it is about GWT encoding, used by applications based on Google Web Toolkit.

Ammonite (ammonite.ryscc.com) supports JSON and XML request fuzzing.