Menu Sidebar
Menu

The real 10x developer

Ask me for a 10x developer, and I’ll show you a hacker who can take a list of 10 tickets, and reject 9 of them because they don’t appreciably advance the project goals.

Like some kind of “project manager”, or something.

The Inescapable Pragmatism of Procedures

Dr. Ben Maughan writes:

At the moment I am rewriting some LaTeX notes into org mode to use in lecture slides. This involves several repetitive tasks, like converting a section heading like this

into this

Whenever I come across a problem like this, my first inclination is always to write a regular expression replacement for it.

A regular expression solution would likely be concise. It would be elegant. It would neatly state the abstract transform that needs to be performed, rather than getting bogged down in details of transformation. A clean, beautiful, stateless function from input to output.

And by definition, a regular expression replacement solution would have a well-defined model of valid input data. Only lines that match the pattern would be touched.

Like I said, a regular expression is always my first thought. But then I’ll work on the regex for a while, and start to get frustrated. There’s always some aspect that’s just a little bit tricky to get right. Maybe I’ll get the transform to work right on one line, but then fail on the next, because of a slight difference I hadn’t taken into account.

Minutes will tick by, and eventually I’ll decide I’m wasting time, throw it away, and just do the editing manually.

Or, on a good day, when I’ve had just the right amount of coffee, I will instead remember that macros exist. Macros are the subject of Maughan’s article.

The trick to making a good macro is to make it as general as possible, like searching to move to a character instead of just moving the cursor. In this case I did the following:

  1. Start with the cursor somewhere on the line containing the subsection and hit C-x C-( to start the macro recording
  2. C-a to go to the start of the line
  3. C-SPC to set the mark
  4. C-s { to search forward to the “{” character
  5. RET to exit the search
  6. C-d to delete the region
  7. Type “** ” to add my org style heading
  8. C-e to move to the end of the line
  9. BACKSPACE to get rid of the last “}”
  10. C-x ) to end the recording

Now I can replay my macro with C-x e but I know I’ll need this again many times in the future so I use M-x name-last-kbd-macro and enter a name for the macro (e.g. bjm/sec-to-star ).

If I ask Emacs to show me an editable version of Maughan’s macro, I see this:

This is the antithesis of a pattern-matching, functional-style solution. This is imperative code. It’s a procedure.

Let’s list some of the negatives of the procedural style:

  • It reveals nothing about the high-level transformation being performed. You can’t look at that procedure definition and get any sense of what it’s for.
  • It’s almost certainly longer than a pattern-replacement solution.
  • It implies state: the “point” and “mark” variables that Emacs uses to track cursor and selection position, as well as the mutable data of the buffer itself.
  • It has no clear statement of the acceptable inputs. It might start working on a line and then break halfway through.

Now let’s talk about some of the strengths of the procedural approach:

  • It is extraordinarily easy to arrive at using hands-on trial and error.
  • The hands-on manipulation becomes the definition, rather than forcing the writer to first identify the transforms, then mentally convert them into a transformation language.
  • It has a fair amount of robustness built-in: by using actions like “go to the next open bracket”, it’s likely to work on a variety of inputs without any specific effort on the part of the programmer.
  • It can get part of the work done and then fail and ask for help, instead of rejecting input that fails to match pattern.
  • It lends itself to a compelling human-oriented visualization: a cursor, moving around text and adding and deleting characters. In other words, it can tell its own story.
  • You can edit it without thinking too hard. You don’t have to hold a whole pattern in your head. You can just advance through the story until you get to the point where something different needs to happen, and add, delete, or edit lines at that point.
  • As the transforms become more elaborate, a regex-transformational approach will eventually hit a wall where regex is no longer a sufficiently powerful model, and the whole thing has to be rewritten. There’s no such inflection point with procedural code.

Time after time, the pattern-matching, functional, transformational approach is the first that appeals to me. And time after time it becomes a frustrating time-sink process of formalising the problem. And time after time, I then turn to the macro approach and just get shit done.

The procedural solution strikes me as being at the “novice” level on the Dreyfus Model of Skill Acquisition. We tell the computer: do this sequence of steps. If something goes wrong, call me.

By contrast, more “formal” solutions strike me as an attempt to jump straight to the “competent” or even “proficient” level: here is an abstract model of the problem. Get it done.

One problem with this, at least looking at it from an anthropomorphic point of view, is that this isn’t how knowledge transfer normally works. People work up to the point of advanced beginner, then competent, then proficient by doing the steps, and gradually intuiting the relations between them, understanding which parts are constant and which parts vary, and then gaining a holistic model of the problem.

Of course, we make it work with computers. We do all the hard steps of modeling the problem, of gaining that level-three comprehension, and then freeze-dry that model and give it to the computer.

But this imposes an artificially high “first step”: witness me trying, and failing, to get a regex solution working in a short period of time. Before reverting to the “dumb” solution of writing a procedural macro through trial and error.

And I worry about the scalability of this approach, as we have to do the hard work of modeling the problem for every last little piece of an application. And then re-modeling when our understanding turns out to be flawed.

This is one reason I’m not convinced that fleeing the procedural paradigm as fast as possible is the best approach for programming languages. I fear that by assuming that a problem must always be modeled before being addressed, we’re setting ourselves up for the exhausting assumption that we have to be the ones doing the modeling.

(And I think there might be a tiny bit of elitism there, as well: so long as someone has to model the problem before telling the computer how to solve it, we’ll always have jobs.)

This is also why I worry a little about a movement toward static systems. The interactive process described above works because Emacs is a dynamic lisp machine. A machine which can both do a thing and reflect on and record the fact that it is doing a thing, and then explain what it did, and then take a modified version of that explanation and do that instead.

I’ve recently realized that I’m one of those nutjobs who wants to democratize programming. And I think in order for that to happen, we need computing systems which are dynamic, but which moreover are comfortable sitting down at level 1 of the Dreyfus model. Systems that can watch us do things, and then repeat back what we told them. Systems that can go through the rote steps, and ask for help when something doesn’t go as expected.

Systems that have a gradual, even gradient from manual to automated.

And then, gradually, systems that can start to come up with their own models of the problem based on those procedures. And revise those models when the procedures change. Maybe they can come up with internal, efficient, elegant transformational solutions which accomplish the same task. But always with the procedure to fall back on when the model falls apart. And the users to fall back on when the procedure falls apart.

Now, there are some false dichotomies that come up when we talk about procedural/functional, formal/informal. For instance: there’s no reason that stateful, destructive procedures can’t be built on top of persistent immutable data structures. The bugbear of statefulness needn’t haunt every discussion of imperative coding.

But anyway, getting back the point at hand: there is an inescapable pragmatism to imperative, procedural code that mutates data (at least locally). There is a powerful convenience to it. And I think that convenience is a signal of a deeper dichotomy between how we show things to other people, vs. how we [think we should] explain things to computers. And for that reason, I’m nervous about discarding the procedural model.

P.S: I’m going to flagrantly exploit the popularity of this post to say: if you like software thinky-thoughts such as this one, you might also enjoy my newsletter!

Contempt as a sign of organizational incompetence

A few months back I wrote on my personal journal about how incompetently-written firmware in a VTech child’s camera led to my 5 year old daughter losing cherished memories. I also recorded their dismissive response to a flaw that would be considered recall-worthy in any camera made for adults.

Sadly, there is a deeply ingrained seam of reflexive apologetics for negligent software among hackers, as several of my peers tried to tell me it was my own fault.

Here’s the thing about negligence, though: it’s rarely found in isolation. This week it came to light that VTech had been hacked. Turns out, it was far worse than first thought: the attackers were able to access not only home addresses and passwords, bu 190 gigabytes of children’s photos.

A couple weeks ago, hackers successfully broke into the servers of connected toy maker Vtech and stole the personal information of nearly 5 million parents and over 200,000 kids. What we didn’t know until now: The hackers stole pictures of kids, too.This is very bad. The hacker’s identity is still unknown, but he’s been updating Motherboard with details about the hack. When the story broke a couple days ago, the site reported that the hacker broke into Vtech’s servers and stole the names, emails, passwords, download histories, and home addresses of 4,833,678 parents who bought the company’s devices. The massive batch of data also contained the first names, genders, and birthdays of over 200,000 children.

Source: The Horrifying Vtech Hack Let Someone Download Thousands of Photos of Children

Just in case there was any doubt as to whether this was a case of negligence:

For example, there is no SSL anywhere. All communications are over unencrypted connections including when passwords, parent’s details and sensitive information about kids is transmitted. These days, we’re well beyond the point of arguing this is ok – it’s not. Those passwords will match many of the parent’s other accounts and they deserve to be properly protected in transit.

Obviously, VTech should never be trusted again. In an ideal world they would face criminal prosecution, be dropped from store shelves, and/or be driven into bankruptcy by civil suits.

But this experience also serves to reinforce a larger lesson: outward contempt for users is always a sign of deeper organizational flaws. And the more data we entrust to corporations, the more their flaws become our problems.

I think this also suggests that an old rule for evaluating people is just as true for organizations. That rule being: how you treat children reveals a lot about your values and overall integrity.

Are you smart enough to be a programmer?

I often talk to non-programmers who think I must have to be extraordinarily clever to be a software developer, and that they could never do it. In my experience, people often underestimate their own ability. I thought I’d write down some heuristics to help you determine if you have the brains it takes to be a programmer.

Read More

Hell’s code kitchen

Linus Torvalds is infamous for periodically tearing into kernel hackers who submit patches he doesn’t like.

Dave Eisenberg took it upon himself to rewrite one of Linus’ rants. Without the invective it’s a fraction of its original length. It still gets the same information across. Arguably, it conveys more information, by putting the essential message front and center instead of burying it in a lot of speculations about the recipient’s intelligence.

37202605

There’s no question that writing a solid rant can be cathartic. But writing doesn’t imply sending. Dale Carnegie tells a story about Abraham Lincoln. Lincoln wrote an infuriated letter to General Meade in 1863, castigating him for not pressing the advantage after the battle of Gettysburg. The letter was discovered in Lincoln’s effects, written but un-sent.

In the classic How to Win Friends and Influence People, Carnegie observes that Lincoln, like most of the renowned leaders he studied, made a conscious choice to avoid giving criticism. Instead, he focused on praising people for what they did well.

Some view Linus’ angry protectiveness of the kernel code as an essential element of his success. But this view ignores the many large, successful open-source projects with leaders who don’t regularly insult and belittle contributors. When was the last time you saw a Hacker News item about an Apache contributor being treated like a dog who piddled on the rug?

And as FreeBSD hacker Randi Harper has repeatedly pointed out, while it’s easy to find stories of people who have exited Open Source development due to a toxic environment, that doesn’t really represent the true extent of the damage. It’s impossible to know how many potential contributors have been silently driven off after observing how other developers are treated.

There’s a line of thinking that says this is just the acid test of whether someone “belongs”. In other words, “if you can’t take the heat, get out of the kitchen”. But when exactly did we decide that Open Source development was supposed to be like an episode of a Gordon Ramsay show? I don’t remember voting on that.

For that matter, even Mr. Ramsay himself is reportedly far more supportive off-camera than most people would imagine. Like many successful people, he has a reputation for hiring talented people and then getting out of their way. His TV fame may be built on abrasiveness, but that’s not the personality trait his restaurant empire rests on.

Being an abusive jackass isn’t an essential piece of the leadership puzzle. In fact, there is research to suggest that selecting for “alpha asshole” behavior is actively damaging organizations.

All of this reminds me of a conversation I used to hear in various forms as I was growing up as a home-schooled kid:

Concerned Adult: But how will the kids learn to deal with normal, real-world challenges like bullying?

Homeschooling Parent: …you consider bullying “normal”?

As an adult, I feel like my homeschooled background equipped me to opt-out of, or route around, bullying behavior wherever I encountered it. I assumed I was worthy of a certain level of basic human respect, and if I didn’t find it, I moved on. This ability has been a kind of super-power in my career.

(Note: this “super-power” was also predicated on the pre-existing privilege of having a lot of options. I’ll probably write more about this another time.)

If you’re just getting started in this community, know this: abuse in open source, or in software generally, is not a given. It’s not a rite of passage. It’s something we can choose to accept, participate in, and enable. Or not.

Newer Posts
Older Posts

Virtuous Code

"The three virtues of a programmer: laziness, impatience, and hubris" — Larry Wall

Newsletter

News, notes, and commentary from Avdi, weekly-ish.

Books and Screencasts

RubyTapas Screencasts

RubyTapas Screencasts

Small plates of gourmet Ruby code.

Confident Ruby

Confident Ruby cover

32 Patterns for joyful coding.

The Making of Cowsays.com

Confident Ruby cover

Watch me build an app in Sinatra and Rails

Objects on Rails

Objects on Rails

A developer notebook on applying classic Object-Oriented principles to Ruby on Rails projects.

Exceptional Ruby

Exceptional Ruby

The definitive guide to exceptions and failure handling in Ruby.

Archives

Categories