This episode garnered a lot of discussion after I initially sent it out to RubyTapas subscribers. Ryan Bates posted an alternative refactoring, as did Evan Light.

Periodically I start a new project using Rails, with RSpec, Capybara, and Selenium for acceptance testing, and a short way into it I find myself banging my head against bizarre inconsistencies with the test database. I’ll set up some records in the test DB, only to have the Selenium-driven browser-based tests act like those records never existed. Eventually, I’ll realize what I did wrong and curse my feeble brain for not remembering the last time I solved the same problem.

The problem is always the same: the tests are being wrapped in database transactions, so any code running outside the actual test process (like, say, a server process servicing a Selenium-driven browser request) does not see the database fixture I’ve so carefully assembled.

I just walked a pairing client through the same fix, so in the interests of remembering the steps, and hopefully preventing some other folks from tearing their hair out, here are the steps needed.

First of all, and this is very important, go into spec/spec_helper.rb and change this line:

config.use_transactional_fixtures = true

To:

config.use_transactional_fixtures = false

This will disable rspec-rails’ implicit wrapping of tests in a database transaction. Without disabling this, none of the following configuration will matter.

Now configure database_cleaner. I usually create a separate file called spec/support/database_cleaner.rb for this. Inside, I put something like this:

RSpec.configure do |config|

  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

end

Let’s take that step by step.

config.before(:suite) do
  DatabaseCleaner.clean_with(:truncation)
end

This says that before the entire test suite runs, clear the test database out completely. This gets rid of any garbage left over from interrupted or poorly-written tests—a common source of surprising test behavior.

config.before(:each) do
  DatabaseCleaner.strategy = :transaction
end

This part sets the default database cleaning strategy to be transactions. Transactions are very fast, and for all the tests where they do work—that is, any test where the entire test runs in the RSpec process—they are preferable.

config.before(:each, :js => true) do
  DatabaseCleaner.strategy = :truncation
end

This line only runs before examples which have been flagged :js => true. By default, these are the only tests for which Capybara fires up a test server process and drives an actual browser window via the Selenium backend. For these types of tests, transactions won’t work, so this code overrides the setting and chooses the “truncation” strategy instead.

config.before(:each) do
  DatabaseCleaner.start
end

config.after(:each) do
  DatabaseCleaner.clean
end

These lines hook up database_cleaner around the beginning and end of each test, telling it to execute whatever cleanup strategy we selected beforehand.

And that’s it!

Note that this is all for RSpec, and does not cover Cucumber configuration.

Hopefully this will help someone else out there avoid the frustrations I’ve run into!

EDIT: A few people have asked me why I don’t just force all threads to share the same ActiveRecord connection, as demonstrated in this Gist. A few reasons:

• Using database_cleaner implies that I want ORM neutrality. database_cleaner supports ActiveRecord, DataMapper, MongoMapper, and others. The solution linked above only works for ActiveRecord.
• It’s a monkey-patched kludge which will only work so long as AR refrains from changing its connection-sharing internals. And frankly I’m not sure I trust it to work across all Ruby VM and database combinations (UPDATE: And indeed, I’ve now seen two different people say there are race conditions with the current Postgres adapter). I’d be more inclined to use it if ActiveRecord had a published configuration option which was known to work in all contexts.
• As I stressed above, I’m careful to set things up so that only the tests that need them fall back to truncation. Since :js => true tests generally don’t form the bulk of my suite (and since they are unavoidably slow anyway, due to the overhead of driving a browser), I’m not overly concerned about the added overhead. Perhaps if all of my acceptance tests drove a live browser I’d be more worried about it.
• In cases where database truncation is taking up a significant amount of test time, you can usually speed things up with some judicious control of which subset of tables get truncated for a given test. This is something database_cleaner makes pretty easy. Maybe that would make a good topic for a followup post.
• UPDATE: Oh yeah, and as @donaldball points out, sharing a transaction between test and test server means acceptance tests don’t run quite the same as they would in production. Specifically, they’ll never trigger after_commit hooks.

Sock.build_fuzzy # => #<Sock type: "fuzzy" id: nil drawer_id: nil>
sock_drawer.socks.build_fuzzy # => #<Sock type: "fuzzy" id: nil drawer_id: 123>

At first we might think to put it in an a collection extension module:

class Sock < ActiveRecord::Base
  module SockCollectionHelpers
    def build_fuzzy
      build(type: "fuzzy")
    end
  end
  # ...
end

Now we have to remember to add it to every Sock association:

class Drawer < ActiveRecord::Base
  has_many :socks, extend: Sock::SockCollectionHelpers
  # ...
end

And we also need to include it in Sock:

class Sock < ActiveRecord::Base
  # ...
  extend SockCollectionHelpers
end

…but this doesn’t work:

sock_drawer.socks.build_fuzzy # => #<Sock type: "fuzzy" id: nil drawer_id: 123>
Sock.build_fuzzy # => Raises no method exception for Sock.build()

And anyway, we want this to be available globally without explicitly extending associations.

Next, we try adding a class method to the Sock class. Since ActiveRecord association proxies delegate missing methods to the assoication class, it seems like this should work.

class Sock < ActiveRecord::Base
  def self.build_fuzzy
    build(type: "fuzzy")
  end
  # ...
end

This is a complete failure. While the association does forward the .build_fuzzy call to Sock, once in the call it is operating in the context of the class object, which as we saw before has no #build method.

sock_drawer.socks.build_fuzzy # Raises no method exception for Sock.build()
Sock.build_fuzzy # => Raises no method exception for Sock.build()

However, a slight change makes everything work as hoped:

class Sock < ActiveRecord::Base
  def self.build_fuzzy
    scoped.build(type: "fuzzy")
  end
end

Now when we call #build_fuzzy on the class it builds a fuzzy sock unassociated with any drawer, and when we call it on an association it builds a fuzzy sock with the appropriate drawer ID set:

Sock.build_fuzzy # => #<Sock type: "fuzzy" id: nil drawer_id: nil>
sock_drawer.socks.build_fuzzy # => #<Sock type: "fuzzy" id: nil drawer_id: 123>

All this thanks to the #scoped method, which is aware of the current scope.

Thank you to Dan Kubb for figuring this out for me.

UPDATE: Fixed def build_fuzzy to be def self.build_fuzzy in the last two examples.

require 'active_record'

class Greeting < ActiveRecord::Base
  establish_connection(adapter:  'mysql', 
                       database: 'scratch', 
                       password: ENV['MYSQL_PASSWORD'])

  connection.create_table(:greetings) do |t|
    t.string :text, limit: 10
  end
end

g = Greeting.create(text: "Greetings and salutations, esteemed world!")
g.reload
puts g.text # => "Greetings "

This is potentially surprising to users. To improve usability, we can set a validation:

require 'active_record'

class Greeting < ActiveRecord::Base
  validates :text, length: { maximum: 10 }
end

g = Greeting.create(text: "Greetings and salutations, esteemed world!")
g.valid? # => false
g.errors.full_messages # => ["Text is too long (maximum is 10 characters)"]

Now we have a new problem: duplication of the length cap. The number “10″ now appears in both the database migrations, and the model validations.

We could extract it out into a constant, although referencing models can be problematic in migrations. But that still doesn’t account for the (many) cases where we don’t specify an explicit field limit in the migration, and instead rely on the built-in ActiveRecord defaults for field limits.

Here’s a solution that uses database reflection to validate a field is within its DB column size limit:

validates :text,
          :length => {
              :maximum => columns_hash['text'].limit
          },

ActiveRecord provides the columns_hash to get at column metadata gathered from the database backend. In the code above we query it for the limit attribute, and use that as the max field length. By pulling the limit from the DB, we avoid duplication of knowledge.

I often get brought in to help preexisting Ruby/Rails projects evolve and mature in a sustainable way. As a result, the vast majority of Ruby projects I’ve worked on have been well-established by the time I arrived. In fact, offhand I can only think of one commercial greenfield Ruby project I’ve participated in. All the rest have been “legacy” from my perspective, in the sense that there was a sizable codebase in production before I showed up. (I’m not counting personal and internal projects.)

I’ve realized that in this my experience may be somewhat unusual among Ruby and Rails developers. With its fast churn and startup-heavy community, a lot of Rubyists are working on projects that they started recently.  My work is more in the codebases where the original programmers have moved on.

Rails’ dirty secret

In the days before I got paid to write Ruby, I worked on some legacy codebases that had histories spanning multiple decades and 100s of KLOCs. That’s a lot of opportunity for bad code to accumulate; and in some cases, the accumulations were impressive.

But here’s the dirty little secret of Rails development: the messiest, nastiest big-ball-of-mud code I have seen in my entire career has been in Ruby on Rails projects. I’ve seen Rails projects that accumulated enough technical debt and waste in two years to make 10 year-old C/C++ programs look clean and elegant by comparison. And it wasn’t just one project. I’ve seen it over and over.

In a way I think this is a testament to the power of the platform. If you’re getting a 500 error in a Rails app, you can keep adding kludge after kludge and hitting “reload” until it works. No need to ever write a test or refactor. In languages and frameworks with a slower turnaround time, this kind of tweak-it-till-it-works workflow is simply impractical. Ruby on Rails has an impressively low barrier to fiddling.

Unfortunately, as a result a lot of projects I come to on have hit what I think of as the productivity crash. At some point the cumulative effect of all those little shortcuts catches up with the development team, and changes that would once have taken a day start taking two weeks as all the dependencies and unintended consequences are sorted out.

As a somewhat ranty aside: this is also the point where, often, original members of the team start moving on to bigger and better things. Meanwhile the crew that inherited the codebase is left to field questions from management about why they can’t seem to push out changes nearly as fast as the old team. The new team is confronted with the problem of getting the codebase under better test coverage and a little more modularized before they can ramp velocity back up; thus perpetuating the notion among the business types that testing and refactoring just slows things down. And/or that the original team were some kind of wizards.

Okay, rant over.

But Rails is different!

Rails developers are sometimes accused of being arrogant and judgmental. I’m not sure how true this is; I don’t see it all that much, but maybe I’m too close to the community and/or arrogant and judgemental myself to be a fair observer.

What I do see is a kind of “Rails exceptionalism”.  Remember back in the first dot-com boom, when some economists were saying that no, this time it was different, the Internet had changed the game this time the markets would just keep going up and up? The phenomenon I see is similar in spirit. it’s a belief, perhaps not fully conscious, that Ruby on Rails development is somehow different, and not subject to the forces affecting other software projects.

Here are a few examples, just to give you an idea of what I’m talking about:

• “Design Patterns are a Java thing. In Ruby you just write code.”
• “The warnings Ruby produces are dumb; just disable them.”
• “Sure they aren’t technically Unit Tests, but isolating objects turned out to be kind of hard and besides nobody else is doing it.”
• “Monkeypatching is frowned on in other languages, but in Ruby it’s fine. The downsides almost never materialize.”
• “Stuff like the Law of Demeter isn’t really as important in Ruby code”
• “Dividing methods into private and public is for control freaks, you don’t need it in Ruby”
• “That’s only a code smell when it’s in Java code”
• “That’s only a problem in large projects” (implying that this project will never become large).

I also see a fair amount of project or subsystem-level exceptionalism: “I know they say classes shouldn’t be this big, but for this class it just makes sense for all of that stuff to be in one place”.

Welcome to Lilliput

The truth is, Ruby on Rails projects are exceptional in a way: they are really small. In James Gray’s terrific keynote at Lone Star Ruby Conf this past week, he mentioned “huge projects” of 40+ KLOC. That gave me a smile, because the first two Rails projects I was ever paid to work on were 50KLOC and 70KLOC, respectively. And while that may seem like a lot of code, that’s small by industry standards.

There are a few reasons for this. Ruby is a more expressive language than, say, Java, so to some degree Rails projects will always be smaller than equivalent projects in higher-ceremony languages.

It’s also possible that Rails programmers have embraced the wisdom of breaking systems many, small, intercommunicating apps. I’d like to believe this, but experience suggests this strategy has seen only spotty uptake.

No, I think the biggest reason for the diminutive nature of Rails apps is also the most obvious: they are all pretty young. It’s a young framework, and there’s a lot of churn in this community. A Rails app that lasts three years is ancient.

I think it’s safe to say that this situation won’t last. We’re going to see larger and larger codebases. And here’s a not-very-daring prediction: a lot of projects are going to hit the very same architectural roadblocks that Lisp, Smalltalk, Pascal, C++, and Java projects hit before them.

You are not a special snowflake

It’s funny reading programming literature from the 80s. Dynamic, object-oriented systems navigating the transition from “small” to “medium-sized”. Sound familiar?

Every revolutionary believes his revolution is special, and won’t devolve into the partisan bickering and venal bureaucracy that the last revolution led to. And it’s easy to believe at first. Everyone’s excited and eager to help; the problems are relatively small; and the marketing drones haven’t latched onto the movement yet.

The truth is, the problem you are solving probably isn’t as special as you think it is. And those byzantine patterns you thought were a relic of a bygone age were invented by people using languages surprisingly similar to Ruby.

Don’t panic

Relax. I’m not here to tell you that the last few years were all just a lovely dream, and you’re really still strapped to a chair in the Ministry of UML.

Ruby is still a wonderful language, and the terrific thing about it is that it adapts to large-system design patterns remarkably easily, and with very little ceremony. Dependency Injection? It’s a one-liner. Object delegation and composition? Piece of cake. Contrary to misconceptions, Ruby doesn’t obviate solid design patterns and SOLID principles; what it does is make them very easy to express. In fact, the ease of expressing robust architectural styles was what attracted some of us early-adopters to the language in the first place.

Just please, do me a favor: before you tell me that Ruby and Rails doesn’t need any of this discipline, have a chat with the guy or gal who is still maintaining the first Rails app you worked on.

I was working on a form recently where the steps to show it went something like this:

1. Go to the home page
2. Log in
3. Click “New Thingit” (domain terms have been changed to protect the innocent)
4. Click “Edit” on the Thingit
5. Click “Settings”
6. Click “Enable Facebook integration”
7. Click “Save”
8. Click “New Wodget”
9. Fill in the “New Wodget” form
10. Click “Save”
11. Click “Edit” on the saved wodget

All this just to get to the “Edit” view of a Wodget. Once you got this far the iterations were a little tighter, but because the forms were presented via AJAX, you couldn’t just hit F5 to refresh the form after making a view change. You still had to take a few steps:

1. Click “Cancel”
2. Click “Edit”

All this rigmarole got in the way of quickly seeing the results of view tweaks.

The way I finally decided to address the issue was to create a “view sandbox” inside the app where I could easily fiddle with arbitrary views.

I created a routing section for the sandbox which would only be enabled in the “development” environment:

if Rails.env.development? 
  match 'sandbox/:action' => 'sandbox' 
end

Then I created a very basic SandboxController, with an action dedicated to the view I wanted to play with:

class SandboxController < ApplicationController
  helper 'thingit'
  helper 'wodget'
  
  layout 'application'

  def edit_wodget
    @thingit = Thingit.find_or_create_by_name("SANDBOX_THINGIT") do |thingit|
      thingit.wodgets.create(...)
    end
    @wodget = @thingit.wodgets.first
  end
end

The controller action created just enough example model data to keep the view happy.

Finally, I created a views/sandbox/edit_wodget.html.haml view corresponding to the action, which simply included the Wodget form partial.

With this in place, I was able to simply navigate to http://localhost:3000/sandbox/edit_wodget to experiment with changes to the view, hitting reload when I wanted to see the effect.

Maybe there’s a better way to do this, and if there is I hope you’ll clue me in. I present this in case anyone else has run into a situation where the change->preview->change roundtrip is just too darn slow.

decent_exposure helps you program to an interface, rather than an implementation in your Rails controllers.

Sharing state via instance variables in controllers promotes close coupling with views. decent_exposure gives you a declarative manner of exposing an interface to the state that controllers contain, thereby decreasing coupling and improving your testability and overall design.

via voxdolo/decent_exposure – GitHub.

It’s almost as if the program were composed out of some kind of discrete “objects”, which only communicated with each other via well-defined “interfaces”, and “encapsulated” their implementation details. It’s like a whole new paradigm of programming!

(Just in case anyone misses the joke, this is not an indictment of decent_exposure. It’s a commentary on just how broken the Rails Controller/View interface is and has always been. I find it ironic that one of the most OO languages in modern use, the notion of a public method interface has to be hacked into Rails controllers with a special syntax.)

<% @products.each_with_index do |product, i| %>
  <li class="product_<%= i %>"><%= product.name %></li>
<% end %>

They are handy for testing, among other things.

Of course, if the content of the block gets too big I’ll factor it out into a partial. The idiomatic Rails way of doing this is to factor the block contents out into a partial and then render it with the :collection option:

< %= render :partial => "product", :collection => @products %>

Which is very clean, except now where do we get our row counter? As it turns out, Rails provides one for us:

  <li class="product_<%= product_counter %>"><%= product.name %></li>

Note the use of product_counter. Rails takes the name of the partial and adds _counter to it when providing the built-in counter.

Just be aware that unlike the counter provided by each_with_index, this counter is 1-based. I have seen some references to this counter being 1-based, but at least in the version of Rails I am using (2.3.9) the counter is 0-based.

Culprit #1: Rails

Rails’ ActiveSupport added a handy little method called #chars to the String class. In and of itself this doesn’t seem like such a bad thing, and a lot of other handy methods in ActiveSupport are built on top of #chars. However, as we’ll see, taking advantage of Ruby’s open classes to extend core types has a way of drawing the unwanted attention from the Law of Unintended Consequences.

Culprit #2: Ruby

It’s not set in stone anywhere, but there’s a fairly well accepted convention in open source projects that versions are divided into a major version, a minor version, and a tiny or patch version. New major versions indicate API-breaking changes. A new minor version may introduce new features, but existing code should continue to work as-is. And a new tiny version indicates that the API remains fixed; the only difference is that bugs have been fixed and security holes patched.

Ruby 1.8.7 is a minor release masquerading as a tiny release. Among the features backported into 1.8.7 from Ruby 1.9 is a new #chars attribute. Unfortunately, it is incompatible with the Rails 2.0 implementation of #chars. This, incidentally, is a prime example of one of the subtler ways that patching the core classes can bite you. Even if you are adding new methods rather than re-writing existing ones, the chances are good that someone else will have the same idea only with a slightly different implementation and semantics. Bang, incompatibility.

Culprit #3: MacPorts

We have an app which has not yet been ported to Rails 2.1. This, in itself, would not have been a problem; we can keep running it under Ruby 1.8.6 with Rails 2.0, no problem. However, I have a nasty habit of trying to keep my software up to date. So I run sudo port upgrade outdated periodically, and watch all the errors from unmaintained ports go scrolling across my terminal for 24 hours or so.

The last time I did this, one of the ports that did manage to build was Ruby. Version 1.8.7. The next time I ran our app, it of course promptly crashed.

This is the point at which I discovered something I hadn’t realized about MacPorts: it has no downgrade path. Coming from the world of Debian, Ubuntu, and apt-get, I just expected any package management system to handle the case where the user specifies an older version to be installed.

In fact, there’s a way to do it in MacPorts, but it’s painful.

Fail.

So there I was with a broken app, no time in the iteration to upgrade it to Rails 2.1, and no easy way to get back to Ruby 1.8.6. Lame.

Rescue

After bitching and moaning on Twitter for awhile, I decided Bob helps those who help themselves, so I took a look at the crash backtrace I was getting. I traced it back to a line in vendor/rails/activesupport/lib/activesupport/core_ext/string/access.rb:

        def first(limit = 1)
          chars[0..(limit - 1)].to_s
        end

In the Rails 2.0 version of String#chars, #chars returns an Array or Array-like object which can be subscripted with #[]. The Ruby 1.8.7 version, by contrast, returns an Enumerable::Enumerator.

“That’s easy enough” thought I, and, fully expecting that patching this one issue would just reveal another incompatibility, and another, and another…, I changed the code to:

        def first(limit = 1)
          chars.to_a[0..(limit - 1)].to_s
        end

Lo and behold, the app worked perfectly.

Of course, YMMV. But as a quick kludge this one was surprisingly painless.

Lessons Learned

Here’s what I took away from this experience:

1. Be wary of adding methods to core classes. What could possibly go wrong? More than you think.
2. Patch releases should be true patch releases. It’s tempting to include a neat new feature as a bonus —
   again, what could possibly go wrong? Resist this urge.
3. Macs are shiny, but for industrial-strength development support, nothing beats a Debian-based system with APT.
4. Every now and then taking a clawhammer to vendor code is the shortest (short-term) way from point A to point B. Personally I prefer to either keep this kind of change local or, if necessary, version it with something like Piston, rather than maintaining it as a monkey-patch.

[ad#PostInline]