Debugging an HTTP Client Library

I’ve been trying to use Benoit Chesneau’s hackney library to hit the Wistia API from an Elixir program. Between the fact that hackney is still under rapid development, and the fact that I’m not great at reading Erlang code, it’s been slow going. I found myself needing a way to see exactly what the requests I was generating with the library looked like.

For various reasons I wanted to point the client at a local server rather than hit the actual Wistia API server while working the kinks out. What I really wanted was a local server that would accept any HTTP request, reply with a 200 OK, and dump detailed information about the request it had received.

I asked for suggestions on Twitter, and I got a bunch of replies. I thought I’d summarize some of them here for future reference and for anyone else who is looking to solve the same problem.

About seven thousand people said “netcat“. This was the solution I had been avoiding; I knew it was a possibility, but I was hoping there would be something more purpose-built out there.

For the record, the following command line will suffice to dump any HTTP request to STDOUT and reply with a 200 OK.

printf "HTTP/1.1 200 OK\r\n\r\n" | netcat -l -v 4040

However, I found that I had to manually kill and restart netcat after each request.

Josh Cheek suggested a Ruby one-liner using Rack and the PP standard library:

ruby -rrack -rpp -e 'p Rack::Server.start(app: -> e {[200,{}, [e.pretty_inspect]]}, Port: 9292)'

This didn’t give me the raw information about the way the message body was encoded that I was looking for, but it could probably be modified to do so pretty easily.

RequestBin is a hosted service for inspecting requests, but their code is open source so it could be run locally.

Similarly, httpbin can be run locally.

Chris Hunt offered this one-line thin configuration. It works quite well, dumping similar raw request information to the netcat version, but without having to be restarted after each request.

Alex Conner offered a similar solution using WEBrick.

A few people recommended Charles Proxy, not as an endpoint, but as a way to monitor and examine HTTP traffic between my client and another server. It looks nice, but I was unable to get it to work with SSL connections.

A similar, free software tool is mitmproxy. I haven’t had time to try it yet. UPDATE: Yes I have, and it’s pretty spiffy.

Brian Goff suggested the packet sniffing tool ngrep, which apparently has some HTTP-specific functions.

Joel Oliveira brought up ngrok, which I think solves a slightly different problem(?), but which is interesting all the same.

This entry was posted in Howto and tagged , . Bookmark the permalink.
  • http://www.manuelmeurer.com/ Manuel Meurer

    Soo… which one did you choose? :)

  • Bryan Barnard

    I’m using RequestBin running locally for just this purpose and I’ve found it to work well. Curious which solution you chose here.

  • http://xanview.com Roman Gaufman

    Why is restarting netcat a big deal? – Just put it in a loop like this: while true; do netcat; done

    • http://avdi.org Avdi Grimm

      Yeah, I tried that. The trouble was it would never actually terminate; it would get into a state where it was “done”, but wouldn’t exit. This was probably because the library I was debugging wasn’t closing connections (like I said, I was debugging it…) The upshot was I had to manually kill it every time.