Hacker News Re-Imagined

Show HN: Mitmproxy2swagger – Automagically reverse-engineer REST APIs

  • 691 points
  • 16 days ago

  • @alufers
  • Created a post

Show HN: Mitmproxy2swagger – Automagically reverse-engineer REST APIs


@jwong_ 15 days

Replying to @alufers 🎙

Really neat! Gives me an idea on using something like this to generate e.g., CURL commands to mimic SSO flows.

Even just documenting an SSO flow as a diagram would be quite neat.

Reply


@a-dub 15 days

Replying to @alufers 🎙

lol!

step 2: features for training a language model on the request and response variables in the mitm stream and a shim for standing up a fully ml data driven zero code mock backend.

Reply


@POPOSYS 15 days

Replying to @alufers 🎙

Can we have this as a browser dev tool please? F12 -> Tab REST -> Create spec from API

Reply


@h1fra 15 days

Replying to @alufers 🎙

very nice !

Reply


@ducktective 15 days

Replying to @alufers 🎙

Is it possible to do this on wireshark/tcpdump pcap dumps? Like for finding out hostnames, endpoints and request packets of HTTPS requests that an android app is making?

Reply


@julianlam 15 days

Replying to @alufers 🎙

This is great work!

This would come in very handy for codebases where an OpenAPI v3 spec would be welcome, but is too onerous to create by hand. Run this for a bit, have it spit out a nearly complete spec, and tweak it a bit to output the final product.

In fact, it is precisely what we did to generate the OpenAPI docs for NodeBB [1]. We had an undocumented API that we turned into an OpenAPI v3 file.

[1] https://docs.nodebb.org/api/read

Reply


@mutant 15 days

Replying to @alufers 🎙

This is absolutely phenomenal!

Reply


@mro_name 15 days

Replying to @alufers 🎙

awesome take

Reply


@nattaylor 14 days

Replying to @alufers 🎙

I gave this a try today. It was silky smooth! Is it possible to tell Swagger to omit OPTIONS methods?

Reply


@dnnssl2 15 days

Replying to @alufers 🎙

Starred. Does this work with non-emulated iOS or Android http calls in which you may need to disable app level security?

Reply


@Sytten 15 days

Replying to @alufers 🎙

Super nice! We might integrate something similar in Caido proxy.

Reply


@aleksiy123 15 days

Replying to @alufers 🎙

Really awesome, I tried my hand at writing something similar and was surprised at how well it actually ended up working.

I feel liken the next step is automatically generating load tests and/or fuzzing tests. Felt like that could be a real product.

Reply


@Labo333 15 days

Replying to @alufers 🎙

Very nice!

On the same note, I wrote a program to generate Python code (requests) from a HAR capture: https://github.com/louisabraham/har2requests

I think using HAR captures is simpler for the end user than spawning mitmproxy as they don't require any installation and are extracted from the network tab of the browser devtools. Is there a reason why you didn't use them?

EDIT: I realized that mitmproxy can also get traffic from other devices like phones. Very cool project, I will think about modifying mine to support mitmproxy captures!

Reply


@klyr 15 days

Replying to @alufers 🎙

Hi, I would also like to add another tool I'm contributing to at work (cisco) called APIClarity [1]. It aims at reconstructing swagger specifications of REST microservices running in K8S, but can also be run locally.

This is a challenging task and we don't support OpenAPI v3 specs yet (we are working on it).

Feel free to have a look, and get ideas from it :)

We'll also be presenting it at next Kubecon 2022.

[1]: https://github.com/openclarity/apiclarity

Reply


@SemanticStrengh 15 days

Replying to @alufers 🎙

Can this be used to generate a REST documentation for your own frontend just by interacting with it? This should be augmented via a crawler, that click everyclickable element recursively.

Reply


@useful 15 days

Replying to @alufers 🎙

bravo, I've wanted something like this

Reply


@Divyeshkharade 15 days

Replying to @alufers 🎙

This looks amazing. Will it also capture data types like enumerators by someway detecting patters?

Reply


@difu_disciple 14 days

Replying to @alufers 🎙

This is fantastic. Thank you

Reply


@alufers 16 days

Replying to @alufers 🎙

Wanted to show off my little project which helps whith reverse engneering APIs used by various apps. It takes HTTP traffic capturewd by mitmproxy and generates an OpenAPI specification for a given REST API.

I have used it already on two apps and the results are good enough to write an alternative client or quickly automate some stuff.

Reply


@captn3m0 15 days

Replying to @alufers 🎙

Almost exactly a fit against my idea[1] to generate OpenAPI from HAR files. Going to read through to see if I can add HAR support.

[1]: https://github.com/captn3m0/ideas#openapi-specification-gene...

Reply


@instagary 15 days

Replying to @alufers 🎙

How did you bypass cert pinning in the video for the Airbnb app?

Reply


@BWStearns 15 days

Replying to @alufers 🎙

This is fantastic!

Reply


@lsferreira42 13 days

Replying to @alufers 🎙

Congrats, this is really awsome and i have a use for it right now, it will be really useful for debuging old and undocumented api's

Reply


@efitz 15 days

Replying to @alufers 🎙

This is awesome; I’m going to try it as soon as I get back to my desk. I’ve been working on trying to glue together tools to translate Charles proxy output to OpenAPI (swagger). I think it would be a great tool to have in a web app reverse engineering toolbox.

Reply


@eligro91 15 days

Replying to @alufers 🎙

Really amazing.

We're having hundreds of undocumented endpoints created over the years, and running this tool on our backends will create instantly good documentation

Thanks for that! Will give feedbacks if any issues

Reply


@Cilvic 15 days

Replying to @alufers 🎙

The question is maybe a bit off-topic a d vague. That's because I struggle to express it with the right terms:

I'm looking for a generic tool to build and then serve:

Accept Incoming request (API contract A) Send outgoing request (API contract B) potentially with parameters from the incoming request Receiving incoming response (API contract B) Do some translations/string manipulation Send outgoing response (API contract A)

Reply


@dsfiguer 15 days

Replying to @alufers 🎙

Oh I love this so much! This would help me with scraping certain sites.

Reply


@chrisweekly 15 days

Replying to @alufers 🎙

Awesome idea! Thank you for creating and sharing!

Reply


@andrewstuart2 15 days

Replying to @alufers 🎙

I've always wanted to build something similar to this, by reading HAR files captured right out of the devtools. Have you given any thought to that as an alternative input?

Reply


@dudus 15 days

Replying to @alufers 🎙

This is a great idea. Kudos.

Reply


@jeroenhd 15 days

Replying to @alufers 🎙

Very interesting! Would this also be able to determine what kind of auth (header tokens, cookies, etc) the APIs require or is that something you still need to detect manually?

Reply


@oneweekwonder 15 days

Replying to @alufers 🎙

little bit off-topic, but do anybody know of something similar for soap/wsdl? I'm aware of soapui mock service.

Reply


@andrewstuart 15 days

Replying to @alufers 🎙

Be interesting to run a fuzzer on the API whilst doing this.

Reply


@upupandup 15 days

Replying to @alufers 🎙

this is absolutely insane!!! I understand capturing the REST api network part, is it then examining the request body, headers being sent back and forth to figure out the API?

Reply


@renewiltord 15 days

Replying to @alufers 🎙

This is great. Good example too since Airbnb could use with some improvement to the user chrome: include cleaning fees, etc

Reply


@thefilmore 15 days

Replying to @alufers 🎙

This is one of the most clever projects I've seen in a while. Nice work.

Reply


@nickysielicki 15 days

Replying to @alufers 🎙

This is really incredible. With a rooted android phone and these tools, plus a couple others [1,2,3], you can get a skeleton to implement a backend for any app you want.

[1]: https://github.com/koxudaxi/fastapi-code-generator

[2]: https://github.com/ioxiocom/openapi-to-fastapi

[3]: https://infosecwriteups.com/hail-frida-the-universal-ssl-pin...

Reply


@evnix 14 days

Replying to @alufers 🎙

I did something similar a year ago at the company which I work, I basically wrote a middleware that intercepts all the requests(express JS) and writes to a OpenAPI YAML file. It diffs previous requests to see which parts of the request path could be variables. The system isn't perfect but you are 95% there which is better than having no documentation or to hand write documentation or keep that spec file updated with changes that people introduce in the code. (got promoted to tech lead after this :-) )

Reply


About Us

site design / logo © 2022 Box Piper