Lost in a World of Data

Lately I’ve been using the Erlang “Hackney” library to interact with HTTP APIs from Elixir. The principle entry point to that library is a function called request. At its simplest, it looks like this:

request(:get, "http://example.com")

But calls to request can get a lot more complex. Here’s a rough approximation of a call I used recently.

:hackney.request(:post,
                 "http://example.org",
                 [],
                 {:multipart, [{"api_password",
                                api_password, 
                                {"form-data", []}, []},
                               {"project_id",
                                project_id, 
                                {"form-data", []}, []},
                               {:file, 
                                filename, 
                                {"form-data", [{"name", "file"}]}, 
                                [{"Content-Transfer-Encoding", "binary"}]}]},
                  {:proxy, proxy_host, proxy_port})

Of course, I didn’t actually call it like that all at once; I assigned smaller parts to intermediate variables and used those variables to build up the overall structure piece by piece. But I still had to understand the overall shape of the data that was expected, and know where to slot in each individual part of it.

Note in particular that even where I didn’t care about the value a particular piece of the data, and was fine with the defaults, I still needed to supply a blank list ([]) to hold its place. Looking back at that call I’m not sure what some of those instances of [] mean; I just know they need to be there.

Now, I want to make it clear that I don’t think Hackney has a bad API at all. I think Benoit is doing terrific work on it. As far as I can tell its design is consistent with FP and Erlang best practices. This is a good library.

On top of that, Benoit has done a great job documenting this API. It’s probably fair to say that I probably never would have succeeded in using it without his documentation.

But there is no denying that the function call above is a monstrosity. It took me a long time, building it piece by piece, looking at my code, then at the docs, then at the code again, over and over, to get it right. And when I got it wrong, the failures were less-than-obvious pattern-matching failures deep down inside the hackney source code, wherever the offending bit of data was actually being used.

Because as in any well-decomposed functional program, each part of that mass of data passed to request is handled by its own dedicated function. And those functions delegate to other functions, and so on. See that section that starts with {:multipart, …}? Everything inside there is handled by a family of functions named stream_multipart. These functions process the many variations on multipart/form-data that Hackney can handle. One of them handles the form {:file, …}. It, in turn delegates to other functions. Which eventually delegate to functions in a completely different project, called hackney_lib.

At each level, functions break apart the data and operate on the fragment that they care about and understand. Individually, these functions are small and easy to understand. Individually, these functions know when the data is in the wrong format, and can complain about it.

From the top level, those functions are all implementation details. And intentionally so; I’m not supposed to know about how the Hackney library is decomposed. As a client programmer, I am cheerfully oblivious to how the sausage is made.

Except I’m not, really. Because despite not knowing about all those little functions, I’m responsible for coming up with arguments that can satisfy them: tuples with the precise right number of elements; empty placeholder arrays; atoms with the right name, and so forth.

Of course, Benoit can document all these little data “shapes”. Elixir (and Erlang) even have a special language for documenting them, called typespecs. But if I don’t know which functions are called for which part of the request arguments, this doesn’t do me a lot of good. And Elixir doesn’t have any way of automatically seeing how the request function delegates its data to subsidiary functions, and taking all those little typespecs and assembling them into a great composite typespec for request.

Another option would be to write a great big mongo-typespec for the request function, either duplicating the information on the individual smaller functions or simply skipping the smaller typespecs entirely. But this means keeping that typespec, which is far away from the individual functions that define it, in sync with all those little functions scattered around. What’s worse, that great big typespec would likely turn out to be nearly unreadable, and not terribly useful as usage documentation.

Still another possibility would be to use Records for each of these little bits of an HTTP request. For instance, there might be a ProxyOptions record, with :host and :port fields. This might make it easier to understand how to build up a complex HTTP request.

But this goes against one of the principle philosophies of programming in dynamic functional languages like Elixir or Clojure, which is that it is better to use “plain old” data structures like lists, tuples, and maps over specialized ones whenever possible.

Instead, Benoit has reasonably opted to document all the different usage possibilities, in English and markup, on the main request function. As a result, he has to manually keep this documentation up-to-date every time he makes a change to any of dozens of different functions, in two different libraries, which will ultimately process the request options.

Consider now what a typical Object-Oriented API for the same task might look like. Here’s some pseudocode, in no particular language.

request = HTTP::Request.new();
request.url = "http://example.org";
request.method = "post";
request.body.multipart = true;
request.body.add_multipart_field("api_password", api_password);
request.body.add_multipart_field("project_id", project_id);
file = request.body.add_multipart_file(filename);
file.content_disposition = "form-data";
file.content_disposition_params["name"] = "file";
file.add_header("Content-Transfer-Encoding", "binary");
request.proxy_options.host = proxy_host;
request.proxy_options.port = proxy_port;
request.send();

I don’t want to get too bogged down into whether this is an optimal OO API, or on details like why I would need to manually set the file’s content-disposition to “form-data”. Let’s instead talk about some of the ways this differs from the Elixir code.

First, obviously, it’s more verbose.

Second: let us assume that all we know about this API is that we need to start with an HTTP::Request object. We have no other documentation. Assuming we have either a REPL and some introspection capabilities; or an IDE with code completion, we can immediately discover what methods our new request responds to. We can see, for instance, that it has a url=(url) setter method. It’s easy to surmise that we need to set this to determine what host the request will be submitted to. In Ruby, this might look something like this:

irb> request.methods
=> [:body, :url, ...]

Likewise, once we discover the body attribute, we can try it out and discover that it returns an HTTP::Request::Body, which we can then probe for methods.

irb> request.body
=> #<HTTP::Request::Body>
irb> request.body.methods
=> [:add_multipart_field, :multipart, ...]

This kind of discoverability is particularly powerful when we come across existing code we need to modify. Let’s say we need to add proxy authentication to our code. If we see that part of the request is {:proxy, {proxy_host, proxy_port}}, there is no way to “ask” that data structure how to add a login and password. Nor is there a way to find this information out from the request function, without painstakingly tracing down through the code until we find where the {:proxy, …} data structure is actually used.

On the other hand, in our OO version, once we see:

request.proxy_options.host = proxy_host;
request.proxy_options.port = proxy_port;

It is trivial to discover that request.proxy_options is an HTTP::ProxyOptions object, and then to introspect on that class and see that it has proxy_user and proxy_password attributes.

Third: remember those empty “placeholder” arrays in the call to :hackney.request? We have nothing of the sort in our OO version. Configuration that isn’t needed simply isn’t specified, and doesn’t clutter up the code.

Fourth: most of the lines of code in the OO version could be re-ordered without changing the outcome. Having method names for every part of the request configuration means that the meaning of any given line is explicit, rather than implicit based on position.

Fifth, and finally: every part of this code is unambiguous. We don’t have to refer back to a pages-long description of request arguments to remember what each piece of it is about. Nor do we need to break it down into a series of local variables just to remind ourselves of the significance of each bit of data.

You’re probably expecting me to say something like “…and this is why OO is better!” at this point. But that’s not what I’m trying to get across. For one thing, this is only really a critique of dynamic functional languages like Clojure and Elixir/Erlang; programs in statically-typed languages like Haskell tend to spend a lot more time on defining (hopefully self-documenting) types for things. And given data of a particular type, it’s easier to ask the system “what can operate on this type”?

Also, it is clear that functional programming in general does have a number of compelling advantages. And a lot of good arguments are made, by people like Rich Hickey and others, for the value of functions that act on simple data structures rather than opaque and specialized objects. I’m not going to rehash all the arguments here; I just want to emphasize that they are real and shouldn’t be casually dismissed.

But it seems to me that there is an approach-ability gap here between the dynamic functional approach and the OO approach shown above. There may also be a comprehensibility gap as well. The Elixir code may be semantically equivalent to the OO code. It might break down into a similar number of nicely orthogonal pieces internally, albeit organized very differently. But it’s difficult to make a case that it has the same level of explore-ability as the OO version, or that the finished product is as easy to comprehend and modify without lengthy consultation of documentation.

It seems to me that programmers in languages like Erlang, Elixir and Clojure are going to have to find strategies to contend with that gap in order to build programs that are accessible, either to developers who are new to the project, or to people who are new to programming in general. I don’t know if this will take the form of additions to the languages, better tools, or best practices for building libraries. I’m curious what people with more dynamic functional programming experience have to say about this, and what strategies they have for making functional APIs welcoming and approachable.