Menu Sidebar
Menu

Avdi Grimm

Hacker; code documentarian.

Where do ideas come from?

Someone wrote in asking:

you always have ideas… How do you generate ideas to build something?

Which is a great opportunity to pontificate. (I swear, I did not make this question up!)

Let’s get this out of the way: I am the wrong person to ask. I am not an effective “idea guy”. I’m white, male, I was born in the USA, I’m 34 years old, and I’ve been writing software since I was in my teens. If I were any good at ideas, I’d be a millionaire by now.

So you probably shouldn’t listen to a word I say. But I’m going to say it anyway, because there’s nothing quite so much fun as making up answers to the big questions.

Read More

Zero to Smoke Test with Sinatra

The other day I put an app in production with (gasp) no automated tests. I’m careful to say no automated tests, because of course I had been testing it manually throughout the ~12 hours it took me to write the initial version of the app. In lieu of tests, I made sure to log copious amounts of information, which I then collected from my deployed app into Papertrail.

For a single day’s work this was acceptable. But I knew that once I had a version in production, pushing out even the smallest, seemingly benign change would be an extremely dicey proposition without some automated regression tests in place.

Read More

New book! Birthday sale!

TL;DR: New book project, The Rake Field Manual. Today only, buy early access to it at half-price ($12.50) with coupon code HAPPY0X22. Or get any of my other books and videos at half price with code BDAY0X22. Or get your first three months of RubyTapas for the price of one by signing up using this link.

Hi folks! It’s that time of year again. The time of year when I celebrate another revolution around the sun by announcing new projects and putting everything on sale!

Oh, and drinking lots of beer and playing Badminton all afternoon, but sadly our yard isn’t big enough to invite you all over for that part.

The Rake Field Manual

First off! It’s been way too long since I started a new project. I’ve mentioned it once or twice on Twitter, but today I’m officially announcing my new book project, The Rake Field Manual.

Some of you may have seen the video/blog post series I released about advanced Rake. In the past year as I have transitioned my digital bookbinding toolchain over from GNU Make to Rake, I’ve learned a lot about the power and flexibility of the Rake build tool. Of course I had used Rake for many years in conjunction with various Ruby projects. But as it turns out, I had barely scratched the surface of what it is capable of.

So in this book I’m going to share what I’ve learned, and no doubt learn even more along the way. It will be laid out in a “cookbook” style, each short chapter addressing a real-world automation problem with a practical Rake solution. However, I don’t want the book to only be applicable to people who are already familiar with Rake. So I will be structuring the recipes in such a way that if you read it from the beginning, you’ll get a solid Rake tutorial starting with basic principles.

Here’s are some example chapters:

I also believe that Rake is applicable far beyond automating just Ruby project builds. In order to give the book the widest potential audience, I’ll be constraining most examples to use a small subset of Ruby 1.9 features. In addition, in the finished product I’ll include an appendix containing a brief introduction to just enough Ruby to understand the examples in the book.

I’m not announcing a timeline for this book; it’ll be done when it’s done. A lot of that will hinge on the feedback I get from readers. And that’s where you come in.

As a self-publishing author, I don’t get publisher advances to cover the cost of writing a book. Instead, in the past I’ve pre-sold my books while they were in an early state, both to get reader feedback and to fund the writing process.

I’m doing that again with The Rake Field Manual, but this time I’ll be collaborating with early readers more closely than ever before. Early buyers of this book will receive immediate access to the working Github repo where I’m writing it. There, you’ll be able to see the daily changes as I make them, report errata, and file issues requesting coverage of topics that are important to you. More than any other book I’ve written, the direction of this one will be guided by the input of my readers.

Oh yeah, one other thing: I’ve decided to donate 10% of the proceeds from this book to the Weirich Fund, in honor of the man who gave us Rake (and so much besides).

Every year I throw a big sale on my birthday, and this year is no exception. Today only, you can purchase early access to The Rake Field Manual as I write it at half price, only $12.50. Just use coupon code HAPPY0X22 when you check out.

So if you want to support my work, give me a sweet birthday present, support a good cause, help guide the evolution of my new book, and get a great deal, buy it now!

Buy The Rake Field Manual

Sale on other products

What about the rest of my books and videos, like Confident Ruby, Exceptional Ruby, and The Making of Cowsays.com? They are all on sale too! For one day, take 50% off of anything in my store using the coupon code: BDAY0X22 (note that this is a different code).

One more thing: if you’re not already a RubyTapas subscriber, today only you can get your first three months for the price of one by signing up using this link: http://rubytapas.com/bday2014

Thanks for all the business over the years! Happy hacking!

Typist Wanted

I love most aspects of making RubyTapas. I love the research, and the writing. I enjoy doing voice-over. I even like editing the video, at least some of the time. And of course I love the feedback that the videos generate.

What I don’t really enjoy is the actual screen-recording part. I am a terrible typist, and once I’ve scripted an episode and come up with all the sample code I find recording those samples for the screen to be a tedious process.

So, as part of my overall strategy of streamlining my process as much as possible, I’m thinking of delegating the screen recording part of my production pipeline. Read on if you think you might be interested in the job. Here are some notes on the job:

  • Probably ~1 hour a week of work, based on how long it takes me.
  • Modest compensation. I’m not prepared to pay consultant-level rates, since this isn’t a consulting job; but I’m not expecting work for free. Also (it should go without saying) all the RubyTapas you can eat.
  • You should know Ruby, but you don’t need to be an expert. You should at least know how to diagnose the problem if something in a code example doesn’t work the way it was supposed to.
  • I don’t care what editor you use, but you should be able to set it up with the same theme, font, etc. that I use on RubyTapas, as well as set up xmpfilter integration in order to show inline results. There may be some other editor setup nitpicks as well.
  • You’d be responsible for figuring out how to record your screen, although if you use Linux I have some tools you could try. Importantly, you’d be responsible for coming up with a 16×9 recording without any window decorations. I don’t want to have to crop borders in post.
  • You don’t have to be a perfect typist. When I record for RubyTapas I just keep trying until I get a section right, and then edit out all the bad tries in post. That said, the less I have to edit out the better.
  • No, you wouldn’t have to synchronize with voice-over. I record VO after the screen-recording, and edit the screen recording to match the pacing of the VO.
  • No, I can’t replace this job with a program. While I occasionally automate typing large code samples, a lot of RubyTapas screen recordings involve following the script and moving around the code adding, changing, and evaluating things just as if I were live-coding during a presentation. Scripts aren’t really good for that kind of thing.
  • You’d need to have the spare time and inclination to record two ‘casts a week, every week.
  • Perks, other than compensation: you get to see RubyTapas scripts early, and you get to yell at me if an example doesn’t make sense.

Interested? Drop me an email. (Warning: I’m mostly heads-down on a new project right now, so I can’t guarantee a timely reply.)

Rake Part 2: File Lists

If you're a Ruby programmer, you've almost certainly used Rake, the build utility created by the late Jim Weirich. But you might not realize just how powerful and flexible a tool it can be. I certainly didn't, until I decided to use it as the basis for Quarto, my e-book production toolchain.

This post is part of a series on Rake, starting with the basics and then moving on to advanced usage. It originated as a series of RubyTapas videos published to subscribers in August-September 2013. Each post begins with a video, followed by the script for those who prefer reading to viewing.

My hope in publishing these episodes for free is that more people will come to know and love the full power of this ubiquitous but under-appreciated tool. If you are grateful for Rake, please consider donating to the Weirich Fund in Jim's memory.


In the last episode, we wrote this Rakefile. It automates building three Markdown files into HTML files.

task :default => :html
task :html => %W[ch1.html ch2.html ch3.html]

rule ".html" => ".md" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

We really don’t want to have to edit this file every time we add a new file to process though. Instead, we’d like to have the Rakefile automatically find files to be built.

To give us something to experiment with, I’ve set up a sample project directory. It contains four Markdown chapter files and one appendix file in a subdirectory, all of which should be built into HTML files. It also has some other stuff which we don’t want to build. There’s a ~ch1.md file which is some kind of temporary file left behind by an editor. And there’s a scratch directory, the contents of which should be ignored.

$ tree
.
├── ~ch1.md
├── ch1.md
├── ch2.md
├── ch3.md
├── ch4.markdown
├── scratch
│   └── test.md
├── subdir
│   └── appendix.md
└── temp.md

This project is under Git revision control. If we tell Git to list the files it knows about, we see a subset of the files from before. Notably missing is a file called temp.md, which has not been registered with Git and probably never will. It too should be left out of the list of files to build.

$ git ls-files
ch1.md
ch2.md
ch3.md
ch4.markdown
scratch/test.md
subdir/appendix.md
~ch1.md

In order to automatically discover just the files which should be built, we turn to Rake file lists. Let’s explore what file lists are, and what they are capable of.

To create a file list, we use the subscript operator on the Rake::FileList class, passing in a list of strings representing files.

require 'rake'
files = Rake::FileList["ch1.md", "ch2.md", "ch3.md"]
files # => ["ch1.md", "ch2.md", "ch3.md"]

So far this isn’t very exciting. But we’re just getting started. Instead of listing files individually, with a FileList we can instead pass in a shell glob pattern. Let’s give it the pattern *.md

require 'rake'
Dir.chdir "project"
files = Rake::FileList["*.md"]
files # => ["ch1.md", "temp.md", "ch3.md", "ch2.md", "~ch1.md"]

Now we start to see the power of a FileList. But this isn’t quite the list of files we want. It contains some files we don’t care about, and it’s missing some files we do want.

We’ll address the missing files first. We add a *.markdown pattern to find files which use the long-form extension.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["*.md", "*.markdown"]
files # => ["ch1.md", "temp.md", "ch3.md", "ch2.md", "~ch1.md", "ch4.markdown"]

But we’re still missing the appendix file. To fix this, we change the glob patterns to match any level in the project directory tree.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> scratch/test.md
# >> ~ch1.md
# >> subdir/appendix.md
# >> ch4.markdown

Now we’ve found all four chapters and the appendix, but we’ve picked up a lot of junk along the way. Let’s start winnowing down the list of files. For this, we’ll use exclusion patterns.

We start by ignoring files that begin with a ~ character.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> scratch/test.md
# >> subdir/appendix.md
# >> ch4.markdown

Next we’ll ignore files in the scratch directory. Just to demonstrate that it’s possible, we’ll use a regular expression for this exclusion instead of a shell glob.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
files.exclude(/^scratch\//)
puts files 

# >> ch1.md
# >> temp.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

We’ve still got the file temp.md hanging around. As we saw before, this file isn’t registered with Git. We’d like to make an exclusion rule that says to ignore any non-Git-controlled file. To do this, we pass a block to .exclude. Inside, we put an incantation which will determine if Git is aware of the file.

require 'rake'
Dir.chdir "project"
files = Rake::FileList["**/*.md", "**/*.markdown"]
files.exclude("~*")
files.exclude(/^scratch\//)
files.exclude do |f|
  `git ls-files #{f}`.empty?
end
puts files 

# >> ch1.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

This filters out the temp file, and finally we are left with the list of just the files we care about.

Next we update the code to make the FileList definition a little more self-contained. We change from the subscript shorthand to FileList.new, and pass a block to the constructor. The FileList will yield itself to this block, which means we can set up all of our exclusions inside the block.

require 'rake'
Dir.chdir "project"
files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end
puts files 

# >> ch1.md
# >> ch3.md
# >> ch2.md
# >> subdir/appendix.md
# >> ch4.markdown

We need to make one more change to our list of files before we can return to our Rakefile. In the Rakefile what we needed was a list of the files to be built, not the source files that correspond to them. To convert our list of input files to a list of output files, we use the #ext method. We give it a .html file extension, and it returns a new list of files with all of the original Markdown extensions replaced with .html.

require 'rake'
Dir.chdir "project"
files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end
puts files.ext(".html")

# >> ch1.html
# >> ch3.html
# >> ch2.html
# >> subdir/appendix.html
# >> ch4.html

Now we’re ready to come back to our Rakefile. We replace our hardcoded list of target files with the FileList we just built.

Since we are now supporting Markdown files with either a .md or .markdown extension, we have to make one more change to tell Rake it can build an HTML file for either one. For now, we’ll do this by simply duplicating the rule. In the future we’ll look at a way to avoid this duplication.

source_files = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end

task :default => :html
task :html => source_files.ext(".html")

rule ".html" => ".md" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

rule ".html" => ".markdown" do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

When we run rake, we can see that it builds all the right HTML files:

$ rake
pandoc -o ch1.html ch1.md
pandoc -o ch2.html ch2.md
pandoc -o ch3.html ch3.md
pandoc -o subdir/appendix.html subdir/appendix.md
pandoc -o ch4.html ch4.markdown

I think that’s enough Rake for today. Happy hacking!

(This post is also available in Japanese at Nilquebe Blog.)


I hope you've enjoyed this episode/article on Rake. If you've learned something today, please consider "paying it forward" by donating to the Weirich Fund, to help carry on Jim's legacy of educating programmers. If you want to see more videos like this one, check out RubyTapas. If you want to learn more about Rake, check out my book-in-progress The Rake Field Manual.

P.S. Have you used Rake in a particularly interesting way? I want to hear from you.

Newer Posts
Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of Cowsays.com

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.

Archives

Categories