Menu Sidebar
Menu

Avdi Grimm

Hacker; code documentarian.

New book! Birthday sale!

TL;DR: New book project, The Rake Field Manual. Today only, buy early access to it at half-price ($12.50) with coupon code HAPPY0X22. Or get any of my other books and videos at half price with code BDAY0X22. Or get your first three months of RubyTapas for the price of one by signing up using this link.

Hi folks! It’s that time of year again. The time of year when I celebrate another revolution around the sun by announcing new projects and putting everything on sale!

Oh, and drinking lots of beer and playing Badminton all afternoon, but sadly our yard isn’t big enough to invite you all over for that part.

The Rake Field Manual

First off! It’s been way too long since I started a new project. I’ve mentioned it once or twice on Twitter, but today I’m officially announcing my new book project, The Rake Field Manual.

Some of you may have seen the video/blog post series I released about advanced Rake. In the past year as I have transitioned my digital bookbinding toolchain over from GNU Make to Rake, I’ve learned a lot about the power and flexibility of the Rake build tool. Of course I had used Rake for many years in conjunction with various Ruby projects. But as it turns out, I had barely scratched the surface of what it is capable of.

So in this book I’m going to share what I’ve learned, and no doubt learn even more along the way. It will be laid out in a “cookbook” style, each short chapter addressing a real-world automation problem with a practical Rake solution. However, I don’t want the book to only be applicable to people who are already familiar with Rake. So I will be structuring the recipes in such a way that if you read it from the beginning, you’ll get a solid Rake tutorial starting with basic principles.

Here’s are some example chapters:

I also believe that Rake is applicable far beyond automating just Ruby project builds. In order to give the book the widest potential audience, I’ll be constraining most examples to use a small subset of Ruby 1.9 features. In addition, in the finished product I’ll include an appendix containing a brief introduction to just enough Ruby to understand the examples in the book.

I’m not announcing a timeline for this book; it’ll be done when it’s done. A lot of that will hinge on the feedback I get from readers. And that’s where you come in.

As a self-publishing author, I don’t get publisher advances to cover the cost of writing a book. Instead, in the past I’ve pre-sold my books while they were in an early state, both to get reader feedback and to fund the writing process.

I’m doing that again with The Rake Field Manual, but this time I’ll be collaborating with early readers more closely than ever before. Early buyers of this book will receive immediate access to the working Github repo where I’m writing it. There, you’ll be able to see the daily changes as I make them, report errata, and file issues requesting coverage of topics that are important to you. More than any other book I’ve written, the direction of this one will be guided by the input of my readers.

Oh yeah, one other thing: I’ve decided to donate 10% of the proceeds from this book to the Weirich Fund, in honor of the man who gave us Rake (and so much besides).

Every year I throw a big sale on my birthday, and this year is no exception. Today only, you can purchase early access to The Rake Field Manual as I write it at half price, only $12.50. Just use coupon code HAPPY0X22 when you check out.

So if you want to support my work, give me a sweet birthday present, support a good cause, help guide the evolution of my new book, and get a great deal, buy it now!

Buy The Rake Field Manual

Sale on other products

What about the rest of my books and videos, like Confident Ruby, Exceptional Ruby, and The Making of Cowsays.com? They are all on sale too! For one day, take 50% off of anything in my store using the coupon code: BDAY0X22 (note that this is a different code).

One more thing: if you’re not already a RubyTapas subscriber, today only you can get your first three months for the price of one by signing up using this link: http://rubytapas.com/bday2014

Thanks for all the business over the years! Happy hacking!

Typist Wanted

I love most aspects of making RubyTapas. I love the research, and the writing. I enjoy doing voice-over. I even like editing the video, at least some of the time. And of course I love the feedback that the videos generate.

What I don’t really enjoy is the actual screen-recording part. I am a terrible typist, and once I’ve scripted an episode and come up with all the sample code I find recording those samples for the screen to be a tedious process.

So, as part of my overall strategy of streamlining my process as much as possible, I’m thinking of delegating the screen recording part of my production pipeline. Read on if you think you might be interested in the job. Here are some notes on the job:

  • Probably ~1 hour a week of work, based on how long it takes me.
  • Modest compensation. I’m not prepared to pay consultant-level rates, since this isn’t a consulting job; but I’m not expecting work for free. Also (it should go without saying) all the RubyTapas you can eat.
  • You should know Ruby, but you don’t need to be an expert. You should at least know how to diagnose the problem if something in a code example doesn’t work the way it was supposed to.
  • I don’t care what editor you use, but you should be able to set it up with the same theme, font, etc. that I use on RubyTapas, as well as set up xmpfilter integration in order to show inline results. There may be some other editor setup nitpicks as well.
  • You’d be responsible for figuring out how to record your screen, although if you use Linux I have some tools you could try. Importantly, you’d be responsible for coming up with a 16×9 recording without any window decorations. I don’t want to have to crop borders in post.
  • You don’t have to be a perfect typist. When I record for RubyTapas I just keep trying until I get a section right, and then edit out all the bad tries in post. That said, the less I have to edit out the better.
  • No, you wouldn’t have to synchronize with voice-over. I record VO after the screen-recording, and edit the screen recording to match the pacing of the VO.
  • No, I can’t replace this job with a program. While I occasionally automate typing large code samples, a lot of RubyTapas screen recordings involve following the script and moving around the code adding, changing, and evaluating things just as if I were live-coding during a presentation. Scripts aren’t really good for that kind of thing.
  • You’d need to have the spare time and inclination to record two ‘casts a week, every week.
  • Perks, other than compensation: you get to see RubyTapas scripts early, and you get to yell at me if an example doesn’t make sense.

Interested? Drop me an email. (Warning: I’m mostly heads-down on a new project right now, so I can’t guarantee a timely reply.)

Rake Part 2: File Lists

If you're a Ruby programmer, you've almost certainly used Rake, the build utility created by the late Jim Weirich. But you might not realize just how powerful and flexible a tool it can be. I certainly didn't, until I decided to use it as the basis for Quarto, my e-book production toolchain.

This post is part of a series on Rake, starting with the basics and then moving on to advanced usage. It originated as a series of RubyTapas videos published to subscribers in August-September 2013. Each post begins with a video, followed by the script for those who prefer reading to viewing.

My hope in publishing these episodes for free is that more people will come to know and love the full power of this ubiquitous but under-appreciated tool. If you are grateful for Rake, please consider donating to the Weirich Fund in Jim's memory.


In the last episode, we wrote this Rakefile. It automates building three Markdown files into HTML files.

task :default => :html
task :html => %W[ch1.html ch2.html ch3.html]

rule ".html" => ".md" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

We really don’t want to have to edit this file every time we add a new file to process though. Instead, we’d like to have the Rakefile automatically find files to be built.

To give us something to experiment with, I’ve set up a sample project directory. It contains four Markdown chapter files and one appendix file in a subdirectory, all of which should be built into HTML files. It also has some other stuff which we don’t want to build. There’s a ~ch1.md file which is some kind of temporary file left behind by an editor. And there’s a scratch directory, the contents of which should be ignored.

$ tree
.
├── ~ch1.md
├── ch1.md
├── ch2.md
├── ch3.md
├── ch4.markdown
├── scratch
│   └── test.md
├── subdir
│   └── appendix.md
└── temp.md

This project is under Git revision control. If we tell Git to list the files it knows about, we see a subset of the files from before. Notably missing is a file called temp.md, which has not been registered with Git and probably never will. It too should be left out of the list of files to build.

$ git ls-files
ch1.md
ch2.md
ch3.md
ch4.markdown
scratch/test.md
subdir/appendix.md
~ch1.md

In order to automatically discover just the files which should be built, we turn to Rake file lists. Let’s explore what file lists are, and what they are capable of.

To create a file list, we use the subscript operator on the Rake::FileList class, passing in a list of strings representing files.

require 'rake'
files = Rake::FileList["ch1.md", "ch2.md", "ch3.md"]
files # => ["ch1.md", "ch2.md", "ch3.md"]

So far this isn’t very exciting. But we’re just getting started. Instead of listing files individually, with a FileList we can instead pass in a shell glob pattern. Let’s give it the pattern *.md

require 'rake'
Dir.chdir "project"
files = Rake::FileList["*.md"]
files # => ["ch1.md", "temp.md", "ch3.md", "ch2.md", "~ch1.md"]

Now we start to see the power of a FileList. But this isn’t quite the list of files we want. It contains some files we don’t care about, and it’s missing some files we do want.

We’ll address the missing files first. We add a *.markdown pattern to find files which use the long-form extension.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["*.md", "*.markdown"]
files # => ["ch1.md", "temp.md", "ch3.md", "ch2.md", "~ch1.md", "ch4.markdown"]

But we’re still missing the appendix file. To fix this, we change the glob patterns to match any level in the project directory tree.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> scratch/test.md
# >> ~ch1.md
# >> subdir/appendix.md
# >> ch4.markdown

Now we’ve found all four chapters and the appendix, but we’ve picked up a lot of junk along the way. Let’s start winnowing down the list of files. For this, we’ll use exclusion patterns.

We start by ignoring files that begin with a ~ character.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> scratch/test.md
# >> subdir/appendix.md
# >> ch4.markdown

Next we’ll ignore files in the scratch directory. Just to demonstrate that it’s possible, we’ll use a regular expression for this exclusion instead of a shell glob.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
files.exclude(/^scratch\//)
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

We’ve still got the file temp.md hanging around. As we saw before, this file isn’t registered with Git. We’d like to make an exclusion rule that says to ignore any non-Git-controlled file. To do this, we pass a block to .exclude. Inside, we put an incantation which will determine if Git is aware of the file.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
files.exclude(/^scratch\//)
files.exclude do |f|
  `git ls-files #{f}`.empty?
end
puts files 

# >> ch1.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

This filters out the temp file, and finally we are left with the list of just the files we care about.

Next we update the code to make the FileList definition a little more self-contained. We change from the subscript shorthand to FileList.new, and pass a block to the constructor. The FileList will yield itself to this block, which means we can set up all of our exclusions inside the block.

require 'rake'
Dir.chdir "project"
files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end
puts files 

# >> ch1.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

We need to make one more change to our list of files before we can return to our Rakefile. In the Rakefile what we needed was a list of the files to be built, not the source files that correspond to them. To convert our list of input files to a list of output files, we use the #ext method. We give it a .html file extension, and it returns a new list of files with all of the original Markdown extensions replaced with .html.

require 'rake'
Dir.chdir "project"
files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end
puts files.ext(".html")

# >> ch1.html
# >> ch3.html
# >> ch2.html
# >> subdir/appendix.html
# >> ch4.html

Now we’re ready to come back to our Rakefile. We replace our hardcoded list of target files with the FileList we just built.

Since we are now supporting Markdown files with either a .md or .markdown extension, we have to make one more change to tell Rake it can build an HTML file for either one. For now, we’ll do this by simply duplicating the rule. In the future we’ll look at a way to avoid this duplication.

source_files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end

task :default => :html
task :html => source_files.ext(".html")

rule ".html" => ".md" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

rule ".html" => ".markdown" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

When we run rake, we can see that it builds all the right HTML files:

$ rake
pandoc -o ch1.html ch1.md
pandoc -o ch2.html ch2.md
pandoc -o ch3.html ch3.md
pandoc -o subdir/appendix.html subdir/appendix.md
pandoc -o ch4.html ch4.markdown

I think that’s enough Rake for today. Happy hacking!

(This post is also available in Japanese at Nilquebe Blog.)


I hope you've enjoyed this episode/article on Rake. If you've learned something today, please consider "paying it forward" by donating to the Weirich Fund, to help carry on Jim's legacy of educating programmers. If you want to see more videos like this one, check out RubyTapas. If you want to learn more about Rake, check out my book-in-progress The Rake Field Manual.

P.S. Have you used Rake in a particularly interesting way? I want to hear from you.

Active Record Soup

Once upon a time there was a small but prosperous village. This village had a reputation for being tech-centric. It was populated mainly by enterprise consultants, software architects, and agile coaches. The denizens of the village had grown wealthy and contented from their lucrative careers. And while none of them were bad people, they had a reputation for being a bit cold to outsiders.

One day a pair of itinerant programmers named Susan and Bob wandered into the village, and set up a makeshift camp in an abandoned hackerspace.  No sooner than they had unpacked their laptops, then they began taping up posters all over town: “Web Applications: lightweight, fast, and cheap! No big frameworks!”

This quickly got the attention of the townspeople. They gathered at the hackerspace, bemused and a bit disdainful. “There are just two of you, and you claim you don’t use any big frameworks. How do you propose to build useful applications?” “Come and see!” the programmers replied. “All we need is the Active Record pattern!” The crowd scoffed. “How can you build real applications without entities, and beans, and containers, and repositories?” they jeered.

But the two programmers calmly set to work defining simple Active Record classes that mapped directly to database tables.

After a little while, one of the programmers turned to the other and said “you know Bob, the one thing that goes great with Active Record is a good Domain Model to handle validations.” As soon as she said this one of the onlookers shouldered his way forward and said “I’ve written dozens of domain models, let me pair with you on that”. The two programmers happily let him sit in. He quickly defined dozens of validations designed to ensure the application data stayed consistent. As an experienced domain modeler, he threw in some Contextual Validation  as well, because domain objects often have different rules in different contexts.

Soon after, as Susan and Bob were fleshing out this domain model, Bob remarked: “I keep running into a situation where I have partial models that need to be filled in if corresponding data exists in the database. I feel like our Active Record needs some logic dedicated to mapping columns to domain model attributes”. No sooner were the words out of his mouth than another audience member piped up and said “you need Data Mapper!” Bob gestured towards a free seat next to him, and the new volunteer sat down and began writing code. After conferring with Bob a bit she also threw in some Lazy Load to avoid excessive database traffic.

Meanwhile, Susan was writing more and more specialized Active Record finder methods, each with hardcoded finder logic. One particular observer was becoming audibly frustrated by her approach the longer she kept at it. Finally, he walked up and said “you’re doing it all wrong, let me show you something”. Susan graciously turned the keyboard over to him, and before long the codebase could boast Query Objects build with a convenient DSL for building up abstract queries. While he was at it, he tweaked the Active Record objects to behave more like Repositories, taking abstract queries and turning them into iterators over data sets.

And so the project continued. As the day waned, one after another village member jumped in to add their own expertise to the simple, Active Record-based application. One added Association Table Mapping so that models could link to each other. Another added Embedded Value for aggregating multiple database fields into a single nested object. A particularly abstract thinker added Metadata Mapping to reduce some of the tedium in writing new extensions to the Active Record classes. Someone else stepped in to help with a Unit of Work, allowing whole trees of objects to be written to the database if any fields had been modified (or “dirtied”) by domain logic.

They worked and worked. Finally, exhausted, Susan and Bob made one last commit, watched the tests turn green, and stepped back from their work tables. The townspeople looked around at each other, and realized that they had been at this task until late into the night. But then they experienced a much bigger revelation: not only had they helped write an app in a day; but Susan and Bob had been right: they’d built the whole application with just Active Record, and nothing else.

That night they ordered pizza, and the whole village sat on beanbags in the hackerspace and discussed ideas for new applications without all the bloated layers they had grown accustomed to. Nobody noticed when Bob and Susan quietly slipped out, and set out to bring their message of simple, lightweight design to a new town.

Storyteller’s notes:

First off, let me tell you the inspiration for this post. I’ve been working on an app of my own, and I’ve been writing my own classes for mapping database rows to domain objects, rather than using any off-the shelf frameworks or libraries. It’s all been very keep-it-simple, only-write-what-I-need, keep-concerns separate-in-small-objects.

And yet today, as I was struggling with trying to understand the interactions in the database mapping layer that I myself wrote, I had a shocking realization: I had somehow managed to conflate Data Mapper and Repository in the same class without even realizing it. Mind you, I’d practically been writing this code with a keyboard in one hand and a copy of PoEAA in the other. And yet somehow how I’d still managed to bash together two distinct concerns (mapping rows to attributes vs. fetching and iterating over rows). It was an embarrassing and slightly humbling moment.

Critiques of ActiveRecord-the-Ruby-library sometimes fall into the error of suggesting that there is something fundamentally wrong with the pattern. Nothing could be further from the truth; for many applications, especially heavily CRUD-oriented ones, Active Record is a perfect fit.

However, it is equally erroneous to equate use of ActiveRecord-the-library with the simple implementation of ActiveRecord-the-pattern. As it stands today, ActiveRecord without any extensions comprises all of the patterns I linked to above, and probably many more besides. All of them are baked together in a way that can make it difficult to know where one ends and the next begins. Most idiomatic uses of ActiveRecord in Rails applications bear little resemblance to the examples of Active Record in PoEAA.

Which, again, is not inherently a bad thing. But newcomers to Rails can perhaps be forgiven when they find the ostensible “simplifying assumption” of Active Record to be less of a simplification than they might have hoped. It also makes it easy to talk at cross purposes when discussing whether and how to use Active Record in a Rails application, since Active Record-the-pattern comprises maybe 10% of ActiveRecord-the-library’s current functionality. Are we talking about a few basic finders and some objects that map directly to database rows? Or are we talking about dependent associations, lazy loading, and arbitrarily complex Arel queries?Space Shuttle cockpit

I’m trying to work my away around to a moral for this story, but I’m having some trouble finding one. I guess one lesson is that, as with my example above of my handcrafted artisinal locally-grown fair-trade data layer, it’s all too easy to conflate concerns without even realizing it. So we should cut framework designers some slack.

But I wouldn’t even have had the terms to recognize that conflation if I hadn’t read PoEAA. Now that I’ve recognized my mistake I’m pretty excited, because I can see the way to a design that isn’t going to hurt my head nearly as much. (It’ll also be easier to test, but don’t tell anyone that). So I guess that’s the other lesson: read good books!

Which I suppose is a bit of a letdown after all of this build-up. Sorry about that. Want some stone soup?

Jim Weirich on exceptions

Back in 2011 I was doing research for the talk that became Exceptional Ruby, and Jim Weirich was nice enough to let me pick his brain on the topic. I was reminded of this email today, and thought I’d share it. There’s a lot of accumulated wisdom about dealing with failure in Ruby (or any language) packed into this short email.

Here’s my basic philosophy (and other random thoughts) on exceptions.

When you call a method, you have certain expectations about what the method will accomplish. Formally, these expectations are called post-conditions. A method should throw an exception whenever it fails to meet its postconditions.

To effectively use this strategy, it implies you must have a small understanding of Design by Contract and the meaning of pre- and post-conditions. I think that’s a good thing to know anyways.

Here’s some concrete examples. The Rails model save method:

model.save!
-- post-condition: The model object is saved.

If the model is not saved for some reason, then an exception must be raised because the post-condition is not met.

model.save
-- post-condition: (the model is saved && result == true) ||
                   (the model is not saved && result == false)

If save doesn’t actually save, then the returned result will be false, but the post-condition is still met, hence no exception.

I find it interesting that the save! method has a vastly simpler post-condition.

On the topic of rescuing exceptions, I think an application should have strategic points where exceptions are rescued. There is little need for rescue/rethrows for the most part. The only time you would want to rescue and rethrow is when you have a job half-way done and you want to undo something so avoid a partially complete state. Your strategic rescue points should be chosen carefully so that the program can continue with other work even if the current operation failed. Transaction processing programs should just move on to the next transaction. A Rails app should recover and be ready to handle the next http request.

Most exception handlers should be generic. Since exceptions indicate a failure of some type, then the handler needs only make a decision on what to do in case of failure. Detailed recovery operations for very specific exceptions are generally discouraged unless the handler is very close (call graph wise) to the point of the exception.

Exceptions should not be used for flow control, use throw/catch for that. This reserves exceptions for true failure conditions.

(An aside, because I use exceptions to indicate failures, I almost always use the “fail” keyword rather than the “raise” keyword in Ruby. Fail and raise are synonyms so there is no difference except that “fail” more clearly communcates that the method has failed. The only time I use “raise” is when I am catching an exception and re-raising it, because here I’m *not* failing, but explicitly and purposefully raising an exception. This is a stylistic issue I follow, but I doubt many other people do).

There you have it, a rather rambling memory dump on my thoughts on exceptions.

Newer Posts
Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of Cowsays.com

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.

Archives

Categories