Because I always love to push the envelope, and love learning how different services work in the background, I find myself running into undocumented APIs fairly regularly. Through writing many random bots, I've come up with a pretty nice workflow for handling, and documenting these APIs.
This article is going to cover how I typically go about the endeavor of documenting unknown APIs.
Whisper is at its core an app that allows people to anonymously post messages (called whispers). It also allows people to reply to other users, like posts, and change their profile name at any time.
I use this app sometimes, looked up their API, and found no information on it. I figured that this would be the perfect target for learning how their backend system works.
Burp Suite (Burp for short) is a MiTM proxy. This is useful while trying to find APIs, and figure out how the application normally uses it.
Burp allows us to set up a proxy server, and log or edit anything that comes through the proxy. This includes HTTP headers, HTTP data, WebSocket Data, and much more.
To set this up, I fire up Burp, go to the
Proxy tab, and choose the
Options tab. This tab allows us to configure our proxy as needed.
Because I'm going to be aiming to work on the app "Whisper", which is a mobile app, I need to set my proxy to listen for connections from the network, not just on the loopback interface (
To set this up, click on
Add in the
Proxy Listeners section, choose a port (I'm choosing
8080), and set the
Bind to address option to
Specific address, with my internal IP.
Now that Burp is listening for proxy connections, I'm going to configure my Android phone to use Burp's proxy. Although I could write about this process here, Burp Suite's help site has a good article about doing just this here: https://support.portswigger.net/customer/portal/articles/1841101-configuring-an-android-device-to-work-with-burp
Next we need to install the root CA that Burp set up automatically so that we can intercept HTTPS traffic. We can do this by downloading our CA certificate by opening our computer's web browser, and going to
http://127.0.0.1:8080, and clicking on
CA Certificate in the top right.
Once we put this certificate on our device, we can go to
Encryption & credentials ->
Install from storage and select the certificate file.
Some phones do not like the default file extension of the certificate. If you run into problems with the certificate not being selectable, rename it to end in
After tapping on
OK, Burp will be able to intercept HTTPS traffic on your phone as well as HTTP.
Configuring Better Interception
Going back to Burp Suite, switch to the
Intercept tab, and click on
Intercept is on, which will turn it off. We're turning off the interception so that we are able to quickly find out what domains the APIs are located on, and focus only on the domains we find.
Because our phone likely makes tons of HTTP requests that we don't want to waste time analyzing, as they don't pertain to Whisper, we go to
Options, and check the
And URL is in target scope in the
Intercept Client Requests.
Now we're going to go to the
Target tab, which will allow us to see which domains our phone is connecting to. This may already be populating a list, which we can ignore at this point. Once we're on this tab, we can open Whisper, and look at what gets added to the list here.
Upon trying this, I'm running into an issue which Burp is showing me back on the
Dashboard tab. Here's what Burp is complaining about:
This likely means that the app is using a TLS feature called "Certificate Pinning", in which the app developers have attempted to protect against what we're doing by only allowing the app to communicate with their certificate. This just won't do in our situation. Fortunately, there's a way around this with
frida, which you can find information about here: https://frida.re/
I'm not going to cover using frida to get around certificate pinning, as there's already an excellent article written about it here: https://medium.com/@ved_wayal/hail-frida-the-universal-ssl-pinning-bypass-for-android-e9e1d733d29
Once we have frida up and running, we'll be able to switch over to the
Target tab of Burp Suite and see all the domains the app is connecting to.
Looking at the domains, I'm seeing
ads.mopub.com, and many other advertising websites. I'm only interested in finding out how whisper works, not the ad websites. Because of this, I'm going to narrow down my scope, as then the proxy will only show us details about communication to their web server, not the analytic and advertising domains.
I'm seeing a bunch of URLs with the domain
[subdomain].whisper.sh. These are likely the servers that Whisper uses for their backend. Right clicking on each of these, and selecting
Add to Scope will allow us to narrow down our logging of communications.
Now that Burp is intercepting communications, we can go to the
Proxy tab, and go to the
HTTP history section. This section shows us a history of all requests it's intercepted that are within our scope (which we defined earlier).
Now we're going to do a set of steps for everything we do in whisper (the reasoning is that we want to document the API, so we need to learn how all of it works).
Right click on the list of requests in the
HTTP history tab, and select
Clear History. Click on
Yes in the confirmation box. This allows us to work on each request in a clean state.
On our device, we do something (I'm going to open a whisper post). Notice that when you open a post, Burp shows some stuff in the
HTTP history suddenly. What Burp is showing is the details of every request Whisper made to their web server.
You can see that there are two requests (one has the title
400 Bad Request, which I am going to ignore because the request failed). Clicking on the other request pulls up detailed information about what was sent to the server, with a tab on the bottom section to view the reply.
Now that we have the request information, we take notes about parts of it that may determine what the server gives us information about.
In this request, I'm seeing that it is a request to
GET without the question mark or anything after it). What's being sent to this API endpoint is a
uid (highlighted in blue, and the value in red), along with something called
sme (no clue what it does),
wid (which I'd guess is the whisper's ID#), and the
locale (language we're using).
Switching over to the
Response tab, we can see that the server replies to us with some JSON.
The JSON is:
We can see that the server gives us a
scroll_id (no clue), a
total (number of replies), and an array of replies.
Now we take all this information (including what we did, where the request went to, what was returned), and take notes on this using whatever way we prefer.
Personally, I use a tool called
insomnia, which is specifically designed for documenting and testing APIs. I really like insomnia because we can right click on a request, and hit
Copy as curl command, and we can paste it into a new request in insomnia. Insomnia then configures a request in the exact same way our device sent it, and lets us disable headers our device sent to the server. This allows us to narrow down that the API requires to be sent.
At this point, we have one request documented. Now we repeat the same steps (do something -> look at request and response -> document) for every action we can do in the app.
Once we've done documentation
Once we've documented every action we can take in the app, we can start writing a bot to perform these on our behalf. I'm likely going to write a bot for this in the future, but since that topic is a fairly complicated topic, I'm going to leave that for another day.
As an example of what documentation can look like when done, I'm releasing my documentation on the Whisper API. As a note, my documentation is incomplete, as I haven't worked out some fairly important aspects (such as how new accounts are set up). This isn't a huge problem, as I can set up a device on my phone, and make a bot log into that account on their API fairly easily.
My documentation can be viewed here: https://whisperdocs.randomcpu.com
As always, I'd love to hear your feedback on this post, and am totally welcome to any help on figuring out how whisper generates the
work query parameters are generated (as these are necessary to register new accounts).