Menu Sidebar

Avdi Grimm

Hacker; code documentarian.

Active Record Soup

Once upon a time there was a small but prosperous village. This village had a reputation for being tech-centric. It was populated mainly by enterprise consultants, software architects, and agile coaches. The denizens of the village had grown wealthy and contented from their lucrative careers. And while none of them were bad people, they had a reputation for being a bit cold to outsiders.

One day a pair of itinerant programmers named Susan and Bob wandered into the village, and set up a makeshift camp in an abandoned hackerspace.  No sooner than they had unpacked their laptops, then they began taping up posters all over town: “Web Applications: lightweight, fast, and cheap! No big frameworks!”

This quickly got the attention of the townspeople. They gathered at the hackerspace, bemused and a bit disdainful. “There are just two of you, and you claim you don’t use any big frameworks. How do you propose to build useful applications?” “Come and see!” the programmers replied. “All we need is the Active Record pattern!” The crowd scoffed. “How can you build real applications without entities, and beans, and containers, and repositories?” they jeered.

But the two programmers calmly set to work defining simple Active Record classes that mapped directly to database tables.

After a little while, one of the programmers turned to the other and said “you know Bob, the one thing that goes great with Active Record is a good Domain Model to handle validations.” As soon as she said this one of the onlookers shouldered his way forward and said “I’ve written dozens of domain models, let me pair with you on that”. The two programmers happily let him sit in. He quickly defined dozens of validations designed to ensure the application data stayed consistent. As an experienced domain modeler, he threw in some Contextual Validation  as well, because domain objects often have different rules in different contexts.

Soon after, as Susan and Bob were fleshing out this domain model, Bob remarked: “I keep running into a situation where I have partial models that need to be filled in if corresponding data exists in the database. I feel like our Active Record needs some logic dedicated to mapping columns to domain model attributes”. No sooner were the words out of his mouth than another audience member piped up and said “you need Data Mapper!” Bob gestured towards a free seat next to him, and the new volunteer sat down and began writing code. After conferring with Bob a bit she also threw in some Lazy Load to avoid excessive database traffic.

Meanwhile, Susan was writing more and more specialized Active Record finder methods, each with hardcoded finder logic. One particular observer was becoming audibly frustrated by her approach the longer she kept at it. Finally, he walked up and said “you’re doing it all wrong, let me show you something”. Susan graciously turned the keyboard over to him, and before long the codebase could boast Query Objects build with a convenient DSL for building up abstract queries. While he was at it, he tweaked the Active Record objects to behave more like Repositories, taking abstract queries and turning them into iterators over data sets.

And so the project continued. As the day waned, one after another village member jumped in to add their own expertise to the simple, Active Record-based application. One added Association Table Mapping so that models could link to each other. Another added Embedded Value for aggregating multiple database fields into a single nested object. A particularly abstract thinker added Metadata Mapping to reduce some of the tedium in writing new extensions to the Active Record classes. Someone else stepped in to help with a Unit of Work, allowing whole trees of objects to be written to the database if any fields had been modified (or “dirtied”) by domain logic.

They worked and worked. Finally, exhausted, Susan and Bob made one last commit, watched the tests turn green, and stepped back from their work tables. The townspeople looked around at each other, and realized that they had been at this task until late into the night. But then they experienced a much bigger revelation: not only had they helped write an app in a day; but Susan and Bob had been right: they’d built the whole application with just Active Record, and nothing else.

That night they ordered pizza, and the whole village sat on beanbags in the hackerspace and discussed ideas for new applications without all the bloated layers they had grown accustomed to. Nobody noticed when Bob and Susan quietly slipped out, and set out to bring their message of simple, lightweight design to a new town.

Storyteller’s notes:

First off, let me tell you the inspiration for this post. I’ve been working on an app of my own, and I’ve been writing my own classes for mapping database rows to domain objects, rather than using any off-the shelf frameworks or libraries. It’s all been very keep-it-simple, only-write-what-I-need, keep-concerns separate-in-small-objects.

And yet today, as I was struggling with trying to understand the interactions in the database mapping layer that I myself wrote, I had a shocking realization: I had somehow managed to conflate Data Mapper and Repository in the same class without even realizing it. Mind you, I’d practically been writing this code with a keyboard in one hand and a copy of PoEAA in the other. And yet somehow how I’d still managed to bash together two distinct concerns (mapping rows to attributes vs. fetching and iterating over rows). It was an embarrassing and slightly humbling moment.

Critiques of ActiveRecord-the-Ruby-library sometimes fall into the error of suggesting that there is something fundamentally wrong with the pattern. Nothing could be further from the truth; for many applications, especially heavily CRUD-oriented ones, Active Record is a perfect fit.

However, it is equally erroneous to equate use of ActiveRecord-the-library with the simple implementation of ActiveRecord-the-pattern. As it stands today, ActiveRecord without any extensions comprises all of the patterns I linked to above, and probably many more besides. All of them are baked together in a way that can make it difficult to know where one ends and the next begins. Most idiomatic uses of ActiveRecord in Rails applications bear little resemblance to the examples of Active Record in PoEAA.

Which, again, is not inherently a bad thing. But newcomers to Rails can perhaps be forgiven when they find the ostensible “simplifying assumption” of Active Record to be less of a simplification than they might have hoped. It also makes it easy to talk at cross purposes when discussing whether and how to use Active Record in a Rails application, since Active Record-the-pattern comprises maybe 10% of ActiveRecord-the-library’s current functionality. Are we talking about a few basic finders and some objects that map directly to database rows? Or are we talking about dependent associations, lazy loading, and arbitrarily complex Arel queries?Space Shuttle cockpit

I’m trying to work my away around to a moral for this story, but I’m having some trouble finding one. I guess one lesson is that, as with my example above of my handcrafted artisinal locally-grown fair-trade data layer, it’s all too easy to conflate concerns without even realizing it. So we should cut framework designers some slack.

But I wouldn’t even have had the terms to recognize that conflation if I hadn’t read PoEAA. Now that I’ve recognized my mistake I’m pretty excited, because I can see the way to a design that isn’t going to hurt my head nearly as much. (It’ll also be easier to test, but don’t tell anyone that). So I guess that’s the other lesson: read good books!

Which I suppose is a bit of a letdown after all of this build-up. Sorry about that. Want some stone soup?

Jim Weirich on exceptions

Back in 2011 I was doing research for the talk that became Exceptional Ruby, and Jim Weirich was nice enough to let me pick his brain on the topic. I was reminded of this email today, and thought I’d share it. There’s a lot of accumulated wisdom about dealing with failure in Ruby (or any language) packed into this short email.

Here’s my basic philosophy (and other random thoughts) on exceptions.

When you call a method, you have certain expectations about what the method will accomplish. Formally, these expectations are called post-conditions. A method should throw an exception whenever it fails to meet its postconditions.

To effectively use this strategy, it implies you must have a small understanding of Design by Contract and the meaning of pre- and post-conditions. I think that’s a good thing to know anyways.

Here’s some concrete examples. The Rails model save method:!
-- post-condition: The model object is saved.

If the model is not saved for some reason, then an exception must be raised because the post-condition is not met.
-- post-condition: (the model is saved && result == true) ||
                   (the model is not saved && result == false)

If save doesn’t actually save, then the returned result will be false, but the post-condition is still met, hence no exception.

I find it interesting that the save! method has a vastly simpler post-condition.

On the topic of rescuing exceptions, I think an application should have strategic points where exceptions are rescued. There is little need for rescue/rethrows for the most part. The only time you would want to rescue and rethrow is when you have a job half-way done and you want to undo something so avoid a partially complete state. Your strategic rescue points should be chosen carefully so that the program can continue with other work even if the current operation failed. Transaction processing programs should just move on to the next transaction. A Rails app should recover and be ready to handle the next http request.

Most exception handlers should be generic. Since exceptions indicate a failure of some type, then the handler needs only make a decision on what to do in case of failure. Detailed recovery operations for very specific exceptions are generally discouraged unless the handler is very close (call graph wise) to the point of the exception.

Exceptions should not be used for flow control, use throw/catch for that. This reserves exceptions for true failure conditions.

(An aside, because I use exceptions to indicate failures, I almost always use the “fail” keyword rather than the “raise” keyword in Ruby. Fail and raise are synonyms so there is no difference except that “fail” more clearly communcates that the method has failed. The only time I use “raise” is when I am catching an exception and re-raising it, because here I’m *not* failing, but explicitly and purposefully raising an exception. This is a stylistic issue I follow, but I doubt many other people do).

There you have it, a rather rambling memory dump on my thoughts on exceptions.

TL;DR Amazon Sucks

You’re probably here because you tried to send one of my ebooks to your Kindle, it didn’t work, you asked me why not, and I sent this to you in response. I wrote a long rant on this topic, but it’s not really fair to make you wade through that just because you want to read my book on the beach. Here’s the short version.

  • Amazon Kindle uses a proprietary, undocumented variant of the open, industry-standard EPUB3 format. They call it KF8.
  • Amazon provides a free-as-in-beer tool called Kindlegen to enable authors to generate well-formed KF8 files without knowing the format specification.
  • According to their terms of use, I am not allowed to directly sell you the high-quality KF8 files generated by that tool. I can only use it to generate copies for personal review and submission to the Kindle store.
  • Instead, I use a free-as-in-speech tool called Calibre to generate the KF8 (.mobi) file I bundle in my direct ebook sales. Unfortunately, Calibre-generated KF8 files are lower in quality than those generated by Kindlegen, largely because Calibre doesn’t really support EPUB3 freatures.
  • Amazon’s mail-to-kindle service has begun rejecting the files generated by Calibre. Which is probably why you are here, reading this.
  • In other words, Amazon has been very careful to make it inconvenient to buy Kindle-format books outside of the Kindle store. I know, shocking, right?

Again, if you want the full story it’s here. Practically speaking, here are your options:

  1. Install Kindlegen on your computer. Take the EPUB file I shipped you, and use Kindlegen to convert it to KF8 format. Then send the resulting file to your Kindle. I have taken pains to ensure that my EPUB3 files translate very well to KF8 using Kindlegen. So if you want the best reading experience on your Kindle, and you’re willing to go through a few extra steps to get it, this is the way to go.
  2. Alternately, you can take the Calibre-generated file I shipped you, and transfer it via USB rather than email-to-kindle.


“Why don’t you use [EBOOK CREATION TOOL X]?” 

Trust me, I’m familiar with virtually every document conversion tool and ebook toolchain project on the planet. They all suffer from the same issues. Whatever else goes into the toolchain, if they produce KF8 they use either Kindlegen or Calibre to do it.  And thus they are subject to the same licensing and quality issues explained above.

“But the ebooks I receive from [PUBLISHER X] don’t have this problem!”

Established publishing houses have their own agreements with Amazon. I’m not an established publishing house.

A case study on civility and constructive criticism

The other day Gregory Brown posted a remarkable story on Parley. He told how he had given Robert “Uncle Bob” Martin some heated Twitter criticism about one of his recent articles. But then, instead of allowing the discussion to spiral into un-constructive sniping, he engaged Robert privately. Together, they found some common ground and mutual respect, and created a re-worked version of the article that sparked it all.

I found his story profoundly inspiring and instructive, and encouraged him to publish it publicly. Instead, he gave me permission to re-print his words here. The following are Greg’s words, with a little editing by me.

The other day I had criticized Uncle Bob for his style of argumentation, which has at times gotten fairly fast-and-loose in a way I feel is unproductive.

Initially I thought that “this was just his style” and that he’d be uninterested in a dialogue on the topic, but I was wrong about that. After some private discussion and going through the exercise of revising one of his articles to show what I thought could be changed, it seems like we were able to get through to one another meaningfully and both learn something from it.

It doesn’t matter if what I said here was true or not, because I was acting like an asshole. I had gotten so worked up over the last week or so with how many of our community leaders were acting and was trying to hold back, but I failed at that and lashed out in a harsh way.

I also told Bob at first that I didn’t want to talk to him, because I didn’t respect him. This was based on the very flawed assumption that a person’s writing is equivalent to the person themselves. I.e. I was attacking the person, but what I really should have been doing was addressing issues in the person’s work.

But coming to my senses, I realized I needed to practice what I preached. To show respect for the person even if I didn’t like the way they presented ideas, and also to bring things down the ground by discussing issues in the context of a real example. The very first email I sent to Bob in our exchange was an apology for saying I didn’t respect him, and a set of guidelines that I believe lead to good discourse, which I had tweeted but were probably lost in the stream:

  • The hallmark of a good discussion is that even if you disagree and leave with your position unchanged, you have learned a lot anyway
  • To have a good discussion with someone you disagree with, you need to acknowledge common ground and understand why differences exist.
  • To have a good discussion with someone you disagree with, you need to give many, many examples to establish context.
  • To have a good discussion with someone you disagree with, you have to be sure that you are defining terms the same way.
  • To have a good conversation with someone you disagree with, you need to respect that person, even if you don’t respect their views.

I also sent him a summary of his article in my own words, trying to read beyond the rhetoric and guess at what he intended to express, rather than the way he chose to express it. In doing this, I was able to make sure that I wasn’t misunderstanding his intentions, and that was hugely valuable.

When we were talking on twitter, he came across as really defensive, because he was responding just to the individual tweets I was sending his way, and not the broader picture. Switching to email gave us each the time and space we needed to talk a whole lot more openly and honestly with each other, and without an audience of people who’s primary contribution were one line insults to both of us, taken completely out of context.

It cannot be understated how much the difference in environment can make for a discussion. The same people, with pretty much the same views, can either learn a lot from each other or nothing at all based primarily on where the communications medium pushes you to go. Going to email and agreeing to publish only after each of us had a chance to share our thoughts and acknowledge one another’s perspective totally changed the game.

You don’t need to avoid fucking up to get results like this, you just need to be able to recognize when you do, apologize for your bad behavior, then ask “what good can come of this?”. Often times, the answer to that last question is much more significant than we’d think.

We’ve published the result of our discussion and the editing exercise on a gist, along with some commentary. Feedback is welcome either here or on that gist. I think this is a lesson we can all learn something from, so I’d be very interested to hear what you think!

Avdi again: I’d like to thank Greg for giving me permission to publish this account, and moreover for his courage in doing the hard work of civil discourse.

Speaking for myself, I find it very difficult to establish these kinds of dialogues, especially after things have heated up on Twitter. Many people have commented on the polarizing nature of Twitter discussions. As a consequence of the brevity of the format, nuance and qualification gets tossed by the wayside.

However, a factor that I think gets less recognition is the role that Twitter’s public nature plays. Because I know I have an audience, I often feel a strong temptation to score as many rhetorical “points” as possible with each Tweet. Going to email gives a conversation space to breathe, both in terms of message length and time. But beyond that, it also removes the pressure to “perform”, and makes it easier to admit uncertainty and other forms of rhetorical “weakness”.

In this story, Greg has given me an example to hold myself to the next time I find myself lobbing rhetorical hand grenades. If I say “let’s take it to email”, this is why.

One last note: Greg didn’t ask me to plug any of his stuff when I reprinted this. But you should know that he publishes thoughtful, in-depth articles and courses on Ruby and software design at

Learn advanced Rake in 7 episodes

Rake is ubiquitous in Ruby-land, but its power is often under-appreciated and under-used. Over the course of the past two weeks I’ve been posting a series of RubyTapas videos I did on Rake. They go into advanced features of Rake that can help you elegantly automate common tasks. Here, for convenience, is a list of all of the episodes in one place. I’ve included links to the original posts, which contain the episode script and source code for those who want to follow along at home.

Part 1: Files and Rules

Part 2: File Lists

Part 3: Rules

Part 4: Pathmap

Part 5: File Operations

Part 6: Clean and Clobber

Part 7: MultiTask

This is not a complete guide to all of the features in Rake; but I hope you find something here that helps you work more efficiently.

If you are grateful for the wonderful tool that is Rake, consider making a donation to the Weirich Fund. If you like the style of these videos, check out for more.

Oh, and if you’ve used Rake for anything particularly interesting, I want to hear from you.

Happy hacking!

Newer Posts
Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.