Menu Sidebar
Menu

Avdi Grimm

Hacker; code documentarian.

TL;DR Amazon Sucks

You’re probably here because you tried to send one of my ebooks to your Kindle, it didn’t work, you asked me why not, and I sent this to you in response. I wrote a long rant on this topic, but it’s not really fair to make you wade through that just because you want to read my book on the beach. Here’s the short version.

  • Amazon Kindle uses a proprietary, undocumented variant of the open, industry-standard EPUB3 format. They call it KF8.
  • Amazon provides a free-as-in-beer tool called Kindlegen to enable authors to generate well-formed KF8 files without knowing the format specification.
  • According to their terms of use, I am not allowed to directly sell you the high-quality KF8 files generated by that tool. I can only use it to generate copies for personal review and submission to the Kindle store.
  • Instead, I use a free-as-in-speech tool called Calibre to generate the KF8 (.mobi) file I bundle in my direct ebook sales. Unfortunately, Calibre-generated KF8 files are lower in quality than those generated by Kindlegen, largely because Calibre doesn’t really support EPUB3 freatures.
  • Amazon’s mail-to-kindle service has begun rejecting the files generated by Calibre. Which is probably why you are here, reading this.
  • In other words, Amazon has been very careful to make it inconvenient to buy Kindle-format books outside of the Kindle store. I know, shocking, right?

Again, if you want the full story it’s here. Practically speaking, here are your options:

  1. Install Kindlegen on your computer. Take the EPUB file I shipped you, and use Kindlegen to convert it to KF8 format. Then send the resulting file to your Kindle. I have taken pains to ensure that my EPUB3 files translate very well to KF8 using Kindlegen. So if you want the best reading experience on your Kindle, and you’re willing to go through a few extra steps to get it, this is the way to go.
  2. Alternately, you can take the Calibre-generated file I shipped you, and transfer it via USB rather than email-to-kindle.

FAQ:

“Why don’t you use [EBOOK CREATION TOOL X]?” 

Trust me, I’m familiar with virtually every document conversion tool and ebook toolchain project on the planet. They all suffer from the same issues. Whatever else goes into the toolchain, if they produce KF8 they use either Kindlegen or Calibre to do it.  And thus they are subject to the same licensing and quality issues explained above.

“But the ebooks I receive from [PUBLISHER X] don’t have this problem!”

Established publishing houses have their own agreements with Amazon. I’m not an established publishing house.

A case study on civility and constructive criticism

The other day Gregory Brown posted a remarkable story on Parley. He told how he had given Robert “Uncle Bob” Martin some heated Twitter criticism about one of his recent articles. But then, instead of allowing the discussion to spiral into un-constructive sniping, he engaged Robert privately. Together, they found some common ground and mutual respect, and created a re-worked version of the article that sparked it all.

I found his story profoundly inspiring and instructive, and encouraged him to publish it publicly. Instead, he gave me permission to re-print his words here. The following are Greg’s words, with a little editing by me.

The other day I had criticized Uncle Bob for his style of argumentation, which has at times gotten fairly fast-and-loose in a way I feel is unproductive.

Initially I thought that “this was just his style” and that he’d be uninterested in a dialogue on the topic, but I was wrong about that. After some private discussion and going through the exercise of revising one of his articles to show what I thought could be changed, it seems like we were able to get through to one another meaningfully and both learn something from it.

It doesn’t matter if what I said here was true or not, because I was acting like an asshole. I had gotten so worked up over the last week or so with how many of our community leaders were acting and was trying to hold back, but I failed at that and lashed out in a harsh way.

I also told Bob at first that I didn’t want to talk to him, because I didn’t respect him. This was based on the very flawed assumption that a person’s writing is equivalent to the person themselves. I.e. I was attacking the person, but what I really should have been doing was addressing issues in the person’s work.

But coming to my senses, I realized I needed to practice what I preached. To show respect for the person even if I didn’t like the way they presented ideas, and also to bring things down the ground by discussing issues in the context of a real example. The very first email I sent to Bob in our exchange was an apology for saying I didn’t respect him, and a set of guidelines that I believe lead to good discourse, which I had tweeted but were probably lost in the stream:

  • The hallmark of a good discussion is that even if you disagree and leave with your position unchanged, you have learned a lot anyway
  • To have a good discussion with someone you disagree with, you need to acknowledge common ground and understand why differences exist.
  • To have a good discussion with someone you disagree with, you need to give many, many examples to establish context.
  • To have a good discussion with someone you disagree with, you have to be sure that you are defining terms the same way.
  • To have a good conversation with someone you disagree with, you need to respect that person, even if you don’t respect their views.

I also sent him a summary of his article in my own words, trying to read beyond the rhetoric and guess at what he intended to express, rather than the way he chose to express it. In doing this, I was able to make sure that I wasn’t misunderstanding his intentions, and that was hugely valuable.

When we were talking on twitter, he came across as really defensive, because he was responding just to the individual tweets I was sending his way, and not the broader picture. Switching to email gave us each the time and space we needed to talk a whole lot more openly and honestly with each other, and without an audience of people who’s primary contribution were one line insults to both of us, taken completely out of context.

It cannot be understated how much the difference in environment can make for a discussion. The same people, with pretty much the same views, can either learn a lot from each other or nothing at all based primarily on where the communications medium pushes you to go. Going to email and agreeing to publish only after each of us had a chance to share our thoughts and acknowledge one another’s perspective totally changed the game.

You don’t need to avoid fucking up to get results like this, you just need to be able to recognize when you do, apologize for your bad behavior, then ask “what good can come of this?”. Often times, the answer to that last question is much more significant than we’d think.

We’ve published the result of our discussion and the editing exercise on a gist, along with some commentary. Feedback is welcome either here or on that gist. I think this is a lesson we can all learn something from, so I’d be very interested to hear what you think!

Avdi again: I’d like to thank Greg for giving me permission to publish this account, and moreover for his courage in doing the hard work of civil discourse.

Speaking for myself, I find it very difficult to establish these kinds of dialogues, especially after things have heated up on Twitter. Many people have commented on the polarizing nature of Twitter discussions. As a consequence of the brevity of the format, nuance and qualification gets tossed by the wayside.

However, a factor that I think gets less recognition is the role that Twitter’s public nature plays. Because I know I have an audience, I often feel a strong temptation to score as many rhetorical “points” as possible with each Tweet. Going to email gives a conversation space to breathe, both in terms of message length and time. But beyond that, it also removes the pressure to “perform”, and makes it easier to admit uncertainty and other forms of rhetorical “weakness”.

In this story, Greg has given me an example to hold myself to the next time I find myself lobbing rhetorical hand grenades. If I say “let’s take it to email”, this is why.

One last note: Greg didn’t ask me to plug any of his stuff when I reprinted this. But you should know that he publishes thoughtful, in-depth articles and courses on Ruby and software design at practicingruby.com.

Learn advanced Rake in 7 episodes

Rake is ubiquitous in Ruby-land, but its power is often under-appreciated and under-used. Over the course of the past two weeks I’ve been posting a series of RubyTapas videos I did on Rake. They go into advanced features of Rake that can help you elegantly automate common tasks. Here, for convenience, is a list of all of the episodes in one place. I’ve included links to the original posts, which contain the episode script and source code for those who want to follow along at home.

Part 1: Files and Rules

Part 2: File Lists

Part 3: Rules

Part 4: Pathmap

Part 5: File Operations

Part 6: Clean and Clobber

Part 7: MultiTask

This is not a complete guide to all of the features in Rake; but I hope you find something here that helps you work more efficiently.

If you are grateful for the wonderful tool that is Rake, consider making a donation to the Weirich Fund. If you like the style of these videos, check out RubyTapas.com for more.

Oh, and if you’ve used Rake for anything particularly interesting, I want to hear from you.

Happy hacking!

Rake Part 7: MultiTask

If you're a Ruby programmer, you've almost certainly used Rake, the build utility created by the late Jim Weirich. But you might not realize just how powerful and flexible a tool it can be. I certainly didn't, until I decided to use it as the basis for Quarto, my e-book production toolchain.

This post is part of a series on Rake, starting with the basics and then moving on to advanced usage. It originated as a series of RubyTapas videos published to subscribers in August-September 2013. Each post begins with a video, followed by the script for those who prefer reading to viewing.

My hope in publishing these episodes for free is that more people will come to know and love the full power of this ubiquitous but under-appreciated tool. If you are grateful for Rake, please consider donating to the Weirich Fund in Jim's memory.


This miniseries on Rake is winding its way to a close. I hope that over the course of the last several videos you’ve come to appreciate the power of Rake as I have. But perhaps you’re still skeptical about the benefit of using Rake over plain-old Ruby or shell scripts. If so, I think today might just change that impression. I want to show you an amazingly powerful capability that Rake gives us more or less for free.

Let’s say we’re putting an ebook together. We have a directory full of several hundred code listings which we’ve stripped out of the text in preparation to be turned into syntax-highlighted HTML using the pygmentize utility.

Here’s a little Rakefile to take care of this task. It defines a list of listings, a list of “highlights”, which are the HTML end products, and a task to produce all of them called highlight. Finally, it defines a rule to produce a .html file from a listing file by running pygmentize on it. We’ve also defined the default task to depend on the highlight task.

require "rake/clean"

task :default => :highlight

LISTINGS   = FileList["listings/*"]
HIGHLIGHTS = LISTINGS.ext(".html")
CLEAN.include(HIGHLIGHTS)

task :highlight => HIGHLIGHTS

rule ".html" => ->(f){ FileList[f.ext(".*")].first } do |t|
  sh "pygmentize -o #{t.name} #{t.source}"
end

Highlighting source code with pygmentize takes time. When we have a lot of source files, it takes a lot of time. If we run rake under the time command, it tells us that the process takes about 48 seconds.

$ time rake
...
pygmentize -o listings/fd673484d50a66ea67fcd20e0c55f038a729e4d7.html listings/fd673484d50a66ea67fcd20e0c55f038a729e4d7.rb
pygmentize -o listings/ff6e24090e794c4db847b10ca993c872ca804101.html listings/ff6e24090e794c4db847b10ca993c872ca804101.rb

real    0m47.961s
user    0m41.912s
sys     0m4.852s

Currently these highlighted files are being built one at a time. But this is 2013, and I have a computer with two physical cores and, through hyperthreading, four virtual cores. Why can’t we build more than one file at a time?

As it turns out, we can. And all we have to do is change one line of the rakefile, from task to multitask.

multitask :highlight => HIGHLIGHTS

This tells Rake that it can process the prerequisites of the :highlight task in parallel. Note that we make this change to the task which depends on the task we want to be parallelized; not to the parallelizable task itself

We run rake again. We see some rather messy output, as Rake fires up a few hundred parallel Rake subprocesses simultaneously and they all talk to the same STDOUT.

A little over 25 seconds later, the build is done. With this one change, we’ve cut the processing time nearly in half!

$ time rake
...

real    0m25.701s
user    1m13.492s
sys     0m8.272s

If we want to fine tune how many tasks are run in parallel, we can use the -j option to Rake to tell it the maximum number of processes to run at once. I’ll specify 4, one for each virtual core.

Interestingly, this actually takes a little bit longer. I’m not sure why.

$ time rake -j 4

real    0m26.752s
user    1m10.300s
sys     0m7.208s

Earlier I said that all it takes is a one-line change to the code to parallize execution, but that was a bit of a fib. In truth, we can tell Rake to run tasks in parallel with no changes to the code whatsoever. Let’s change the multitask back to a task. Then we’ll run rake with the -m option. This tells Rake to treat ever task as if it is a multitask.

Again, we see distorted output. And when the dust settles, we once again see a total time of a little over 25 seconds.

$ time rake -m

real    0m25.606s
user    1m13.060s
sys     0m7.856s

Rake’s parallelization is smart, too: if other tasks were dependent on the :highlight task, it would still wait until all the pygmentize processes finished before moving on to the next phase.

So what do we gain from automating our builds with Rake? Not just an easy way to declare complex dependencies and rules for accomplishing tasks. Not just a set of convenience methods for file operations. Not just a handy command-line front-end. In addition to all that, we get parallelization of repetitive tasks for free. And that’s what I call happy hacking!

(This post is also available in Japanese from Nilquebe Blog.)


I hope you've enjoyed this episode/article on Rake. If you've learned something today, please consider "paying it forward" by donating to the Weirich Fund, to help carry on Jim's legacy of educating programmers. If you want to see more videos like this one, check out RubyTapas. If you want to learn more about Rake, check out my book-in-progress The Rake Field Manual.

P.S. Have you used Rake in a particularly interesting way? I want to hear from you.

Rake Part 6: Clean and Clobber

If you're a Ruby programmer, you've almost certainly used Rake, the build utility created by the late Jim Weirich. But you might not realize just how powerful and flexible a tool it can be. I certainly didn't, until I decided to use it as the basis for Quarto, my e-book production toolchain.

This post is part of a series on Rake, starting with the basics and then moving on to advanced usage. It originated as a series of RubyTapas videos published to subscribers in August-September 2013. Each post begins with a video, followed by the script for those who prefer reading to viewing.

My hope in publishing these episodes for free is that more people will come to know and love the full power of this ubiquitous but under-appreciated tool. If you are grateful for Rake, please consider donating to the Weirich Fund in Jim's memory.


In the last episode we defined a Rake task to clean up the products of our build script. It did this through the simple expedient of recursively removing the “outputs” directory.

task :clean do
  rm_rf "outputs"
end

Sometimes cleanup isn’t that simple. Today, we’re going to start with a somewhat modified version of the Rakefile we’ve been developing.

As before, we’re converting Markdown files to HTML. Unlike in the immediately preceding episode, the HTML files will be generated next to their source files—there are no separate source and output directories.

In addition to the rule for building Markdown files to HTML, we’ve added some new rules. We now have a rule for concatenating all the HTML fragment files into a single book.html. Then there’s a rule to convert the book.html into an EPUB format ebook using the ebook-convert command from the Calibre ebook package. Finally, there’s a rule to take the EPUB file and convert it into a Kindle-compatible .mobi file using Amazon’s kindlegen.

As a last tweak, we’ve updated the :default rule to depend on these .epub and .mobi targets.

SOURCE_FILES = Rake::FileList.new("**/*.md", "**/*.markdown") do |fl|
  fl.exclude("~*")
  fl.exclude(/^scratch\//)
  fl.exclude do |f|
    `git ls-files #{f}`.empty?
  end
end

task :default => ["book.epub", "book.mobi"]
task :html => SOURCE_FILES.ext(".html")

rule ".html" => ->(f){source_for_html(f)} do |t|
  sh "pandoc -o #{t.name} #{t.source}"
end

file "book.html" => SOURCE_FILES.ext(".html") do |t|
  chapters   = FileList["**/ch*.html"]
  backmatter = FileList["backmatter/*.html"]
  sh "cat #{chapters} #{backmatter} > #{t.name}"
end

file "book.epub" => "book.html" do |t|
  sh "ebook-convert book.html #{t.name}"
end

file "book.mobi" => "book.epub" do |t|
  sh "kindlegen book.epub -o #{t.name}"
end

def source_for_html(html_file)
  SOURCE_FILES.detect{|f| f.ext('') == html_file.ext('')}
end

This build script produces two different categories of file:

  • It generates intermediate files. All of the HTML files fall into this category. Another name for these files is temporary files, since they aren’t needed once the full build process has finished.
  • It also generates deliverable ebook files. These files are the end goal of the whole process.

When it comes to automatically cleaning up our project directory, we’d like to treat these two categories of files differently. At times we may want to clean up just the intermediate files, leaving the ebook products intact. At other times we may want to blow away every generated file and start with a clean slate.

We could write our own tasks to handle these two types of cleanup. Or, we could make use of Rake’s optional rake/clean library.

To use rake/clean, we first require it. Once we do this, a new global constant called CLEAN is available to us. This constant is a FileList, which is initially empty.

require 'rake/clean'

CLEAN                           # => []
CLEAN.class                     # => Rake::FileList

We can use the CLEAN list to tell Rake which files are intermediate files. First, let’s add the list of HTML files generated from Markdown files

CLEAN.include(SOURCE_FILES.ext(".html"))

Then we’ll add the concatenated book.html to the list.

file "book.html" => SOURCE_FILES.ext(".html") do |t|
  chapters   = FileList["**/ch*.html"]
  backmatter = FileList["backmatter/*.html"]
  sh "cat #{chapters} #{backmatter} > #{t.name}"
end
CLEAN.include("book.html")

Next, we’ll add files to another list called CLOBBER. This list tells Rake which files are considered final products. To the CLOBBER list, we add the .epub and .mobi ebook files.

file "book.epub" => "book.html" do |t|
  sh "ebook-convert book.html #{t.name}"
end
CLOBBER << "book.epub"

file "book.mobi" => "book.epub" do |t|
  sh "kindlegen book.epub -o #{t.name}"
end
CLOBBER << "book.mobi"

We could have added these files to the CLEAN and CLOBBER lists all in one place in the Rakefile. But by making each addition next to the rules for building that file, we make it more likely that if we ever remove or change those rules, we’ll remember update the associated entry in the CLEAN or CLOBBER list.

When we go to the command line and tell Rake to list the available tasks with the rake -T command, we can see that there are two tasks available that we didn’t define: clean, and clobber.

We run rake without any arguments first, to build our ebook files. When we list the files in the project, we can see various .html intermediate files as well as the final product .epub and .mobi files.

$ rake
$ tree
.
├── backmatter
│   ├── appendix.html
│   └── appendix.md
├── book.epub
├── book.html
├── book.mobi
├── ch1.html
├── ~ch1.md
├── ch1.md
├── ch2.html
├── ch2.md
├── ch3.html
├── ch3.md
├── ch4.html
├── ch4.markdown
├── Rakefile
├── scratch
│   └── test.md
└── temp.md

If we then run rake clean, we don’t see any output. But when we list the project contents again, we can see that all the .html files have disappeared:

$ rake clean
avdi@hazel:~/Dropbox/rubytapas/134-rake-clean/project$ tree
.
├── backmatter
│   └── appendix.md
├── book.epub
├── book.mobi
├── ~ch1.md
├── ch1.md
├── ch2.md
├── ch3.md
├── ch4.markdown
├── Rakefile
├── scratch
│   └── test.md
└── temp.md

If we then run rake clobber, we see a bunch of warnings about files that could not be found. That’s because clobber first executes the clean task, which we already ran. That task is trying to remove a bunch of files which are already gone. Don’t worry though; these warnings are harmless and can be safely ignored.

When we look at the project contents after running clobber, we can see that the ebook files have vanished along with the intermediate files.

And that’s all there is to it—with rake/clean, we should never again need to write our own project cleanup tasks to remove build files. We can just add the appropriate files or file patterns to the CLEAN and CLOBBER lists, and let Rake do the rest. Happy hacking!

(This post is also available in Japanese from Nilquebe Blog.)


I hope you've enjoyed this episode/article on Rake. If you've learned something today, please consider "paying it forward" by donating to the Weirich Fund, to help carry on Jim's legacy of educating programmers. If you want to see more videos like this one, check out RubyTapas. If you want to learn more about Rake, check out my book-in-progress The Rake Field Manual.

P.S. Have you used Rake in a particularly interesting way? I want to hear from you.

Newer Posts
Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of Cowsays.com

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.

Archives

Categories