Jison

Jison is a parser generator for JavaScript that has separate lex and bnf definitions. It’s used by Handlebars.js.

Treetop

Treetop is a parser generator for Ruby. It’s installed with Ruby On Rails, through the mail gem which is installed by ActionMailer.

Citrus

Citrus is another promising parsing gem for Ruby. It seems to be very easy to get started with, and I like many of the design decisions.

Temple

Temple is a templating-specific library that helps with a lot of the AST transformation. It doesn’t seem to have a CFG syntax so it seems that using treetop or citrus would make sense for complex grammars. Otherwise, strscan could be used.

temple-mustache

This is an implementation of a mustache renderer in . It uses strscan to generate the initial parse tree, and Temple to generate the ruby code. It supports mustache sections.

Slim

Slim is a Haml-like templating library for Ruby that’s used in production by many. It uses Temple, with a line-based parser, which works well because it uses significant indentation for nesting.

import unittest

class Frame(object):
    def __init__(self, tenth=False):
        self.rolls = []
        self.tenth = tenth

    def full(self):
        if len(self.rolls) >= self.max_rolls():
            return True
        if self.tenth:
            has_special = all([roll in ('X', '/') for roll in self.rolls])
            return len(self.rolls) == 2 and not has_special
        else:
            return self.strike()

    def roll(self, score):
        if self.full():
            raise RuntimeError('attempted to record a roll on a full frame')
        self.rolls.append(score)

    def pins(self):
        if len(self.rolls) == 0:
            return 0
        if self.strike() or self.spare():
            return 10
        else:
            return sum([int(roll) for roll in self.rolls])

    def first_roll_pins(self):
        if len(self.rolls) == 0:
            return 0
        elif self.rolls[0] == 'X':
            return 10
        else:
            return int(self.rolls[0])

    def score(self, subsequent_frames):
        if self.tenth:
            return self.tenth_frame_score()

        score = self.pins()
        if self.strike():
            score += sum([frame.pins() for frame in subsequent_frames])
        elif self.spare():
            if len(subsequent_frames) > 0:
                score += subsequent_frames[0].first_roll_pins()
        return score

    def tenth_frame_score(self):
        return min(Game(''.join(self.rolls)).score(), 40)

    def strike(self):
        return len(self.rolls) > 0 and self.rolls[0] == 'X'

    def spare(self):
        return len(self.rolls) > 0 and self.rolls[-1] == '/'

    def max_rolls(self):
        return 3 if self.tenth else 2

class Game(object):
    def __init__(self, roll_scores=''):
        self.frames = []
        for score in roll_scores:
            self.roll(score)

    def roll(self, score):
        if len(self.frames) == 0 or self.frames[-1].full():
            tenth = len(self.frames) == 9
            self.frames.append(Frame(tenth))
        self.frames[-1].roll(score)

    def score(self):
        return sum(self.frame_scores())

    def frame_scores(self):
        frame_scores = []
        for frame_index in xrange(len(self.frames)):
            frame = self.frames[frame_index]
            subsequent_frames = []
            if frame.strike() or frame.spare():
                subsequent_frames = self.frames[frame_index+1:]
                added_frames = 1
                if frame.strike() and len(subsequent_frames) > 0:
                    added_frames = 2 if subsequent_frames[0].strike() else 1
                subsequent_frames = subsequent_frames[:added_frames]
            frame_scores.append(frame.score(subsequent_frames))
        return frame_scores

class GameTest(unittest.TestCase):
    def test_initial_strike(self):
        self.assertEqual(Game('X').score(), 10)

    def test_two_strikes(self):
        self.assertEqual(Game('XX').score(), 20+10)

    def test_three_strikes(self):
        self.assertEqual(Game('XXX').score(), 30+20+10)

    def test_strike_spare_strike(self):
        self.assertEqual(Game('X9/X').score(), 20+20+10)

    def test_strike_strike_spare(self):
        self.assertEqual(Game('XX9/').score(), 30+20+10)
        self.assertEqual(Game('XX9/71').score(), 30+20+17+8)

    def test_perfect_game(self):
        game = Game('X'*12)
        self.assertEqual(len(game.frames), 10)
        self.assertEqual(game.score(), 300)

    def test_made_up_game(self):
        game = Game('X907/818/X70070/72')
        self.assertEqual(len(game.frames), 10)
        self.assertEqual(game.score(), 132)

if __name__ == '__main__':
    unittest.main()

I think that the tenth frame calculation is incorrect. My limited understanding of bowling slowed me down a fair bit. I got an object system that I’m fairly happy with, though!

First I looked at the Gemfile.lock. Had I known the format I would have found out my answer more quickly. I didn’t, though, and so when I found my first result for treetop, I stopped. It showed treetop below specs in the hierarchy.

GEM
  remote: http://rubygems.org/
  specs:
    # ...snip...
    thor (0.14.6)
    tilt (1.3.3)
    treetop (1.4.10)
      polyglot
      polyglot (>= 0.3.1)
    tzinfo (0.3.30)
    uglifier (1.0.3)

The next thing I did was run find . -iname '*.treetop' in ~/.rbenv. It found the following results:

(mbp) ~/.rbenv/versions/1.9.2-p290 $ find . -iname '*.treetop'
./lib/ruby/gems/1.9.1/gems/erector-0.8.3/lib/erector/erect/rhtml.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/address_lists.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/content_disposition.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/content_location.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/content_transfer_encoding.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/content_type.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/date_time.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/envelope_from.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/message_ids.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/mime_version.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/phrase_lists.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/received.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/rfc2045.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/rfc2822.treetop
./lib/ruby/gems/1.9.1/gems/mail-2.3.0/lib/mail/parsers/rfc2822_obsolete.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/examples/lambda_calculus/arithmetic.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/examples/lambda_calculus/lambda_calculus.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/lib/treetop/compiler/metagrammar.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/compiler/test_grammar.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/compiler/test_grammar_do.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/a.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/b.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/c.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/d.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/f.treetop
./lib/ruby/gems/1.9.1/gems/treetop-1.4.10/spec/composition/subfolder/e_includes_c.treetop
(mbp) ~/.rbenv/versions/1.9.2-p290 $

Aha, so there are numerous treetop files in actionmailer! I have my answer. Seems like a good use of a parser, plus those may be worth using as examples.

Then I took another look at a Gemfile.lock from a rails project, and saw that it was plainly listed there. I just didn’t see it and didn’t keep looking after I found one.

GEM
  remote: http://rubygems.org/
  specs:
    XMLCanonicalizer (1.0.1)
      log4r (>= 1.0.4)
    actionmailer (3.1.1)
      actionpack (= 3.1.1)
      mail (~> 2.3.0)
  # ...snip...

I noticed something: Gemfile.lock doesn’t show an arbitrarily nested hierarchy; instead it shows a list of gems and their dependencies, where the list of gems includes all gems. Then, separately at the end of the file, it shows the top-level gems from the Gemfile.

To see a deeply nested, a graph could be constructed from the Gemfile.lock, using the two nesting levels under specs as an adjacency list.

Heroku has five megabytes for database space, which often sounds like it ought to be enough when it isn’t. Want auditing and comments? Nah, that’ll take up too much space. Its single dyno free plan serves one request at a time. The next steps up are twenty dollars a month and five cents an hour for databases and dynos, respectively. These aren’t that expensive for a major project, but for several side projects it quickly adds up.

I realized this and switched back to running a VPS, this time on Zerigo, which I pay for annually. There is no limit to the number of apps I have. Concurrent requests are supported. They can use the same databases. Database backups are free and uncomplicated. I’m also happy to be outside of the cloud oligopoly that seems to be forming.

Besides that, it’s fun! I get to try niche language platforms. Node.js was building steam long before Heroku supported it, and it still doesn’t support websockets. It’s not hard to find, with a little thinking, other interesting platforms to try. How about Racket or Factor? Or setting up your own Lucene server, or a web server that uses the git command line tool? Those can’t (easily) be run with Heroku.

I find anecdotally that most developers don’t have their own websites or non-trivial side projects. I only have the first, but I can sense that my personal website is helping me prepare to launch non-trivial side projects. I’ve done very little work to set up this server, yet despite tweeting about it and having visitors and occasional commenters, it stays up. That gives me the confidence I need to launch something bigger.

My plea to other developers (and aspiring developers) out there is to draw parallels between programming and other creative works and find out how much you could responsibly be spending for hosting side projects, and then realize that there’s no reason you shouldn’t have at least a VPS.

In vim, there are separate modes for searching and commands. This generally works well as they have separate completion buffers and since I do them a lot, it keeps my command history from cluttering up my search history and vice versa. The area where I’ve wished that they were unified is when I want to convert from a search to a substitution expression. Normally in vim you do search expressions in search mode and command expressions in command mode. For example, if I wanted to search for trailing whitespace I might type:

/\s\s*

If I want to replace it with an empty string through the whole file I might type:

:%s/\s\s*//g

I can’t remove the / at the front of the command line with a backspace and replace it with a colon, so until I learned the tip I’m about to reveal I had to retype it. No longer.

The StackOverflow answer I found for the query, Convert vim / search to search and replace without retyping regular expression, suggests this remarkably simple technique:

You can type

/search_term

and then

:%s/<ctrl-r>// etc

where you actually press ctrl+r and then / to insert the search term.

It works! No longer do I desire for search and command modes to be unified.

First, using it is simple. It works just like :cd. You type :lcd path/to/directory in command mode. The path is relative to the current directory, so :lcd .., :lcd ~/Desktop, and :lcd /etc/apache2 all work.

The :lcd command changes the path of the current window. The current window is the current frame within the current tab. Now, what’s really neat, is that a new window (created from commands such as :sp and :tabedit) gets the local directory of the window it’s created from.

Most plugins respect it. Both Command-T and ack.vim search within the local current directory. Another plugin, fugitive, bases its commands on the git repository of the current file so it works smoothly with different local directories as well.

Bang commands (like :!cp README.md ~/Desktop) also use the local current directory.

Finally, after using it for a while it gets easy to tell which directory I’m in. The status line displays the path of the current file based on the local current directory, so if I know where the current file is I can quickly figure out where the local current directory is.

I used it the other day with bundle show to dig through the source of two dependencies of my Rails project to better understand them.

If you use vim and haven’t got in the habit of using :lcd I highly recommend it. If you use another editor and wish it handled working in multiple directories gracefully, consider giving vim a shot!

I wrote some code that uses it, just for practice.

$.fn.tree = function(data) {
  var $el = $(this),
      html = [];

  function renderNode(node) {
    if (typeof node === "object") {
      html.push("<ul>");
      for (var key in node) {
        if (node.hasOwnProperty(key)) {
          html.push("<li><strong>" + key + ": </strong>");
          html.push(typeof node[key] == "object" ? renderNode(node[key]) : node[key]);
          html.push("</li>");
        }
      }
      html.push("</ul>");
    }
  }

  renderNode(data);
  $el.html(html.join(""));

  $el.on('click', 'strong', function() {
    $(this).closest('li').find('ul').toggle(200);
  });
};

$("<div>").appendTo("body").tree({
  name: {
    first: "Benjamin",
    last: "Atkin"
  },
  location: {
    city: "Boulder",
    state: "CO"
  },
  accounts: {
    social: {
      twitter: "http://twitter.com/benatkin",
      facebook: "http://facebook.com/atkin",
      gplus: "http://gplus.to/benatkin",
      identica: "http://identi.ca/benatkin",
      rstatus: "http://rstat.us/bat"
    },
    creative: {
      github: "http://github.com/benatkin",
      blog: "http://benatkin.com/"
    }
  }
});

Nothing too fancy.

One area where they work well is for a Heroku-deployable open source app with OmniAuth GitHub OAuth integration. Here is an example initializer (config/initializers/omniauth.rb) from a RailsCast:

Rails.application.config.middleware.use OmniAuth::Builder do
  provider :github, 'CLIENT_KEY', 'CLIENT_SECRET'
end

I would replace CLIENT_KEY and CLIENT_SECRET with my own keys, but I can’t commit them to a public repository, or else another person could use them and I could be held responsible. Rather than leave out my client key and secret using a .gitignore and a sample config, I can use an environment variable:

Rails.application.config.middleware.use OmniAuth::Builder do
  provider :github, ENV['GITHUB_CLIENT_KEY'], ENV['GITHUB_CLIENT_SECRET']
end

With a note in the README about the environment variables used, this should be easy for developers to set this up on their own private apps.

So far I’ve been typing environment variables into the command line when running apps locally and on my VPS, but that’s about to change. I may specify them in my nginx config, which is backed up to a private git repo. I’m not sure yet. Also I’m going to dump out my Heroku environment variables and back them up too.

PeepCode

Pricing: Starts at $12 for one screencast; Twitter: @peepcode and @topfunky

I bought my first PeepCode screencast back in 2008, before Rails and Merb merged. It’s been around for a while, but Geoffrey Grosenbach does a great job of keeping it up-to-date. There are now a number of screencasts about current topics of interest to the Ruby on Rails community, including Rails 3, Backbone.js, PostgreSQL, CoffeeScript, and John Barnette.

Railscasts

Pricing: Free weekly episodes, additional weekly episodes and revised episodes for $9/mo (RailsCasts Pro); Twitter: @railscasts

Railscasts is great for keeping up on Ruby On Rails, and because it’s free, it’s also great for recommending to people who are curious about Rails but not ready to spend money to help them learn it. Ryan Bates is great at explaining things. He covers a wide variety of topics in his screencasts and presents them in a nice format with code snippets.

Update: since I posted this Ryan Bates released Railscasts Pro, which is fantastic!

Ruby on Rails Tutorial

Pricing: $26 (book), $85 (screencasts), $95 (both); Twitter: @railstutorial

Michael Hartl is a physicist who does a number of other things (see his about page) including Web Development with Ruby On Rails. I’ve watched all of his screencasts; they’re fantastic. He builds a web application TDD-style and teaches many different concepts including MVC and how TDD can help with authorization (which is trickier than authentication IMO).

Rails Apps

Pricing: Free; Twitter: @rails_apps

I always thought that the Rails Starter App templates were cool, but they were missing something: in-depth tutorials and a comparison between the many different templates and quick-start tools. Daniel Kehoe has provided both and many more useful resources in his Rails Apps GitHub account.

Code School

Pricing: Free (one substantial free product), $45-55 (single paid products); Twitter: @codeschool

Code School is a set of tutorials designed to help developers to quickly get up to speed building web applications. Gregg Pollack and Envy Labs have been doing podcasts and screencasts for quite some time, and it’s not surprising that they still do it, because they’re very enthusiastic in front of a camera and a microphone.

In order to check that it works, I moved my plugin (installed as a git submodule) outside the wp-content/plugins directory, went to Add Plugin, searched for it, and instructed WordPress to automatically install it. It works! The following CoffeeScript code was highlighted using it:

launch() if ignition is on

If you use WordPress and either CoffeeScript or Clojure, I’d love for you to give it a shot!