Punching firewalls with Mandrill webhooks

(blogarhythm ~ Fire Cracker - Ellegarden)

Mandrill is the transactional email service by the same folks who do MailChimp. I've written about it before, in particular how to use the mandrill-rails gem to simplify inbound webhook processing.

Mandrill webhooks are a neat, easy way for your application to respond to various events, from recording when users open email, to handling inbound mail delivery.

That all works fine if your web application lives on the public internet i.e. Mandrill can reach it to post the webhook. But that's not always possible: your development/test/staging environments for example; or perhaps production servers that IT have told you must be "locked down to the max".

Mandrill currently doesn't offer an official IP whitelist, so it's not possible to use firewall rules to just let Mandrill servers in. Mandrill does provide webhook authentication (supported by the mandrill-rails gem), but that solves a different problem: anyone can reach your server, but you can distinguish the legitimate webhook requests.

I thought I'd share a couple of techniques I've used to get Mandrill happily posting webhooks to my dev machine and servers behind firewalls.

Using HAProxy to reverse-proxy Mandrill Webhooks

If you have at least one internet-visible address, HAProxy is excellent for setting up a reverse-proxy to forward inbound Mandrill Webhooks to the correct machine inside the firewall. I'm currently using this for some staging servers so we can run real inbound mail scenarios.

Here's a simple scenario:

• gateway.mydomain.net - your publicly-visible server, with HAProxy installed and running OK
• internal/192.168.0.1 - a machine on an internal network that you want to receive webooks posted to 192.168.0.1:8080/inbox

Say the gateway machine already hosts http://gateway.mydomain.net, but we want to be able to tell Mandrill to post it's webhooks to http://gateway.mydomain.net/inbox_internal, and these (and only these) requests will be forwarded to http://192.168.0.1:8080/inbox.

Here are the important parts of the /etc/haproxy/haproxy.cfg used on the gateway machine:

global
  #...
 
defaults
  mode http                                # enable http mode which gives of layer 7 filtering
  #...

# this is HAProxy listening on http://gateway.mydomain.net
frontend app *:80                          
  default_backend webapp                   # set the default server for all requests
  # next we define a rule that will send requests to the internal_mandrill backend instead if the path starts with /inbox_internal
  acl req_mandrill_inbox_path path_beg /inbox_internal 
  use_backend internal_mandrill if req_mandrill_inbox_path 

# define a group of backend servers for the main web app
backend webapp                             
  server app1 127.0.0.1:8001             
 
# this is where we will send the Mandrill webhooks
backend internal_mandrill                  
  reqirep     ^([^\ ]*)\ /inbox_internal(.*) \1\ /inbox\2 
  server int1 192.168.0.1:8080  # add a server to this backend

Obviously the path mapping is optional (but neat to demonstrate), and I've left out all the normal HAProxy options like balancing, checks and SSL option forwarding that you might require in practice, but are not relevant to the issue at hand.

Job done! Our internal server remains hidden behind the firewall, but Mandrill can get through to it by posting webhooks to http://gateway.mydomain.net/inbox_internal.

Tunneling to dev machines with ngrok

For development, we usually don't want anything so permanent. There are quite a few services for tunneling to localhost, mainly with developers in mind. Lately I've been using ngrok which is living up to it's name - it rocks! Trival to setup and works like a dream. Say I'm developing a Rails app:

# run app locally (port 3000)
rails s
# run ngrok tunnel to port 3000
ngrok 3000

Once started, ngrok will give you http and https addresses that will tunnel to port 3000 on your machine. You can use these addresses in the Mandrill webhook and inbound domains configuration, and they'll work as long as you keep your app and ngrok running.

Design thinking is not rocket science

OH on the ABC Radio National By Design podcast (00:56): In the field: Paul Bennet

For us the idea of small ideas that people can actually connect together and actually implement are very big ideas.

And I'm sure you've heard people describe design thinking as sort of a combination of rocket science, string theory and calculus.

It isn't. It's not rocket science at all. It's actually very very straight-forward.

It's looking in the world, being inspired by people, co-creating with them, prototyping and then iterating. And it has to be impactful. It has to work.

Writing simple ruby utilities for Google IMAP + OAuth 2.0

(blogarhythm ~ Unpretty/Fanmail: TLC)

There are some good ruby gems available for dealing with OAuth 2.0 and talking to Google APIs, for example:

• google-api-client is the official Google API Ruby Client makes it trivial to discover and access supported APIs.
• oauth2-client provides generic OAuth 2.0 support that works not just with Google
• gmail_xoauth implements XAUTH2 for use with Ruby Net::IMAP and Net::SMTP
• gmail provides a rich Ruby-esque interface to GMail but you need to pair it with gmail_xoauth for OAuth 2 support (also seems that it's in need of a new release to merge in various updates and extensions people have been working on)

For the task I had at hand, I just wanted something simple: connect to a mailbox, look for certain messages, download and do something with the attachments and exit. It was going to be a simple utility to put on a cron job.

No big deal. The first version simple used gmail_xoauth to enable OAuth 2.0 support for IMAP, and I added some supporting routines to handle access_token refreshing.

It worked fine as a quick and dirty solution, but had a few code smells. Firstly, too much plumbing code. But most heinously - you might seen this yourself if you've done any client utilities with OAuth - it used the widely-recommended oauth2.py Python script to orchestrate the initial authorization. For a ruby tool!

Enter the GmailCli gem

So I refactored the plumbing into a new gem called gmail_cli and it is intended for one thing: a super-simple way to whip up utilities that talk to Google IMAP and providing all the OAuth 2.0 support you need. It actually uses google-api-client and gmail_xoauth under the covers for the heavy lifting, but wraps them up in a neat package with the simplest interface possible. Feel free to go use and fork it!

With gmail_cli in your project, there are just 3 things to do:

1. If you haven't already, create your API project credentials in the Google APIs console (on the "API Access" tab)
2. Use the built-in rake task or command-line to do the initial authorization. You would normally need to do this only once for each deployment:
   

   $ rake gmail_cli:authorize client_id='id' client_secret='secret'
   $ gmail_cli authorize --client_id 'id' --client_secret 'secret'

3. Use the access and refresh tokens generated in step 2 to get an IMAP connection in your code. This interface takes care of refreshing the access token for you as required each time you use it:
   

   # how you store or set the credentials Hash is up to you, but it should have the following keys:
   credentials = {
     client_id:     'xxxx',
     client_secret: 'yyyy',
     access_token:  'aaaa',
     refresh_token: 'rrrr',
     username:      'name@gmail.com'
   }
   imap = GmailCli.imap_connection(credentials)

A Better Way?

Polling a mailbox is a terrible thing to have to do, but sometimes network restrictions or the architecture of your solution makes it the best viable option. Much better is to be reactive to mail that gets pushed to you as it is delivered.

I've written before about Mandrill, which is the transactional email service from the same folks who do MailChimp. I kinda love it;-) It is perfect if you want to get inbound mail pushed to your application instead of polling for it. And if you run Rails, I really would encourage you to checkout the mandrill-rails gem - it adds Mandrill inbound mail processing to my Rails apps with just a couple of lines of code.

Ruby Tuesday

(blogarhythm ~ Ruby - Kaiser Chiefs)
@a_matsuda convinced us to dive into Ruby 2.0 at RedDotRubyConf, so I guess this must be the perfect day of the week for it!

Ruby 2.0.0 is currently at p195, and we heard at the conference how stable and compatible it is.

One change we learned that may catch us if we do much multilingual work that's not already unicode is the change that Ruby now assumes UTF-8 encoding for source files. So the special "encoding: utf-8" marker becomes redundant, but if we don't include it the behaviour in 2.0.0 can differ from earlier versions:

$ cat encoding_binary.rb 
s = "\xE3\x81\x82"
p str: s, size: s.size
$ ruby -v encoding_binary.rb 
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin11.4.2]
{:str=>"あ", :size=>1}
$ ruby -v encoding_binary.rb 
ruby 1.9.3p429 (2013-05-15 revision 40747) [x86_64-darwin11.4.2]
{:str=>"\xE3\x81\x82", :size=>3}

Quickstart on MacOSX with RVM

I use rvm to help manage various Ruby installs on my Mac, and trying out new releases is exactly the time you want it's assistance to prevent screwing up your machine. There were only two main things I needed to take care of to get Ruby 2 installed and running smoothly:

1. Update rvm so it knows about the latest Ruby releases
2. Update my OpenSSL installation (it seems 1.0.1e is required although I haven't found that specifically documented anywhere)

Here's a rundown of the procedure I used in case it helps (note, I am running MacOSX 10.7.5 with Xcode 4.6.2). First I updated rvm and attempted to install 2.0.0:

$ rvm get stable
# => updated ok
$ rvm install ruby-2.0.0
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.7/x86_64/ruby-2.0.0-p195.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing requirements for osx, might require sudo password.
-bash: /usr/local/Cellar/openssl/1.0.1e/bin/openssl: No such file or directory
Updating certificates in ''.
mkdir: : No such file or directory
Password:
mkdir: : No such file or directory
Can not create directory '' for certificates.

Not good!!! What's all that about? Turns out to be just a very clumsy way of telling me I don't have OpenSSL 1.0.1e installed.

I already have OpenSSL 1.0.1c installed using brew (so it doesn't mess with the MacOSX system-installed OpenSSL), so updating is simply:

$ brew upgrade openssl
==> Summary
 /usr/local/Cellar/openssl/1.0.1e: 429 files, 15M, built in 5.0 minutes

So then I can try the Ruby 2 install again, starting with the "rvm requirements" command to first make sure all pre-requisites are installed:

$ rvm requirements
Installing requirements for osx, might require sudo password.
[...]
Tapped 41 formula
Installing required packages: apple-gcc42.................
Updating certificates in '/usr/local/etc/openssl/cert.pem'.
$ rvm install ruby-2.0.0
Searching for binary rubies, this might take some time.
No binary rubies available for: osx/10.7/x86_64/ruby-2.0.0-p195.
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.
Installing requirements for osx, might require sudo password.
Certificates in '/usr/local/etc/openssl/cert.pem' already are up to date.
Installing Ruby from source to: /Users/paulgallagher/.rvm/rubies/ruby-2.0.0-p195, this may take a while depending on your cpu(s)
[...]
$ 

OK, this time it installed cleanly as I can quickly verify:

$ rvm use ruby-2.0.0
$ ruby -v
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin11.4.2]
$ irb -r openssl
2.0.0p195 :001 > OpenSSL::VERSION
 => "1.1.0"
2.0.0p195 :002 > OpenSSL::OPENSSL_VERSION
 => "OpenSSL 1.0.1e 11 Feb 2013"

Optimising presence in Rails with PostgreSQL

(blogarhythm ~ Can't Happen Here - Rainbow)
It is a pretty common pattern to branch depending on whether a query returns any data - for example to render a quite different view. In Rails we might do something like this:

query = User.where(deleted_at: nil).and_maybe_some_other_scopes
if results = query.presence
  results.each {|row| ... }
else
  # do something else
end

When this code executes, we raise at least 2 database requests: one to check presence, and another to retrieve the data. Running this at the Rails console, we can see the queries logged as they execute, for example:

(0.9ms)  SELECT COUNT(*) FROM "users" WHERE "users"."deleted_at" IS NULL
 User Load (15.2ms)  SELECT "users".* FROM "users" WHERE "users"."deleted_at" IS NULL

This is not surprising since under the covers, presence (or present?) end up calling count which must do the database query (unless you have already accessed/loaded the results set). And 0.9ms doesn't seem too high a price to pay to determine if you should even try to load the data, does it?

But when we are running on PostgreSQL in particular, we've learned to be leery of COUNT(*) due to it's well known performance problems. In fact I first started digging into this question when I started seeing expensive COUNT(*) queries show up in NewRelic slow transaction traces. How expensive COUNT(*) actually is depends on many factors including the complexity of the query, availability of indexes, size of the table, and size of the results set.

So can we improve things by avoiding the COUNT(*) query? Assuming we are going to use all the results anyway, and we haven't injected any calculated columns in the query, we could simply to_a the query before testing presence i.e.:

query = User.where(deleted_at: nil).and_maybe_some_other_scopes
if results = query.to_a.presence
  results.each {|row| ... }
else
  # do something else
end

I ran some benchmarks comparing the two approaches with different kinds of queries on a pretty well-tuned system and here are some of the results:

Query	Using present?	Using to_a	Faster By
10k indexed queries returning 1 / 1716 rows	17.511s	10.938s	38%
4k complex un-indexed queries returning 12 / 1716 rows	23.603s	15.221s	36%
4k indexed queries returning 1 / 1763218 rows	22.943s	20.924s	9%
10 complex un-indexed queries returning 15 / 1763218 rows	23.196s	14.072s	40%

Clearly, depending on the type of query we can gain up to 40% performance improvement by restructuring our code a little. While my aggregate results were fairly consistent over many runs, the performance of individual queries did vary quite widely.

I should note that the numbers were *not* consistent or proportional across development, staging, test and production environments (mainly due to differences in data volumes, latent activity and hardware) - so you can't benchmark on development and assume the same applies in production.

Things get murky with ActiveRecord add-ons

So far we've talked about the standard ActiveRecord situation. But there are various gems we might also be using to add features like pagination and search magic. MetaSearch is an example: a pretty awesome gem for building complex and flexible search features. But (at least with version 1.1.3) present? has a little surprise in store for you:

irb> User.where(id: '0').class
=> ActiveRecord::Relation
irb> User.where(id: 0).present?
   (0.8ms)  SELECT COUNT(*) FROM "users" WHERE "users"."id" = 0
=> false
irb> User.search(id_eq: 0).class
=> MetaSearch::Searches::User
irb> User.search(id_eq: 0).present?
=> true

Any Guidelines?

So, always to_a my query results? Well, no, it's not that simple. Here are some things to consider:

• First, don't assume that <my_scoped_query>.present? means what you think it might mean - test or play it safe
• If you are going to need all result rows anyway, consider calling to_a or similar before testing presence
• Avoid this kind of optimisation except at the point of use. One of the beauties of ActiveRecord::Relation is the chainability - something we'll kill as soon as we hydrate to a result set Array for example.
• While I got a nice 40% performance bonus in some cases with a minor code fiddle, mileage varies and much depends on the actual query. You probably want to benchmark in the actual environment that matters and not make any assumptions.

My Virtual Swag from #rdrc

(blogarhythm ~ Everybody's Everything - Santana)

So the best swag you can get from a technology conference is code, right? Well RedDotRubyConf 2013 did not disappoint! Thanks to some fantastic speakers, my weekends for months to come are spoken for. Here's just some of the goodness:

• First up, @tenderlove kicked off an #irresponsibleruby meme with some crazy internal fiddles.
• From @gazay: gon is a gem that makes it easy to expose server data to javascript.
• From @flyerhzm and others: convinced me to go try Celluloid and Celluloid::IO for building actor-based asynchronous systems.
• From @zacksiri: transponder an opinionated javascript library for assisting in working with front-end heavy rails app.
• From @netzke: checkout netzke for building complex UIs on Rails with Ext JS.
• From @jimweirich: equator_weight.rb so @tenderlove knows how much more he can eat while in Singapore;-)
• From @dqminh: The RSpec applause_formatter.rb so you too can get applause when your tests go green, just like @jimweirich.
• From @josevalim: awesome talk on concurrency and a tip to checkout thread_safe. And somehow I now have an urge to go play with Elixir too;-)
• From @sausheong: tanks and tanksworld demonstrating Gosu-based games.
• From @sikachu: appraisal for testing against multiple permutations of gem dependencies.
• @a_matsuda convinced us to use Ruby 2.0, spam his friends in Japan on twitter, and blow our minds with Trick 2013 ruby code.
• From @nigelr: fracture an interesting way of co-ordinating expectations for multiple test conditions to things don't fall through the cracks.
• From @steveklabnik: Shoes and Functional Reactive Programming in Ruby with frappuccino. wtf!
• And just for completeness, from @tardate (me): RGovData, sps_bill_scanner, and megar.

Will I still be a Rubyist in 5 years? #rdrc

(blogarhythm ~ Ruby - Kaiser Chiefs)
The third RedDotRubyConf is over, and I think it just keeps getting better! Met lots of great people, and saw so many of my Ruby heroes speak on stage. Only thing that could make it even better next year would be to get the video recording thing happening!

I had the humbling opportunity to share the stage and here are my slides. Turned out to be a reflection on whether I'd still be a Rubyist in another 5 years, and what are the external trends that might change that. Short story: Yes! Of course. I'll always think like a Rubyist even though things will probably get more polyglot. The arena of web development is perhaps the most unpredictable though.

A couple of areas I highlight that really need a bit more love include:

• There's a push on SciRuby. Analytics are no longer the esoteric domain of bioinformaticists. Coupled with Big Data (which Ruby is pretty good at), analytics are driving much of the significant innovation in things we build.
• Krypt - an effort lead by Martin Boßlet to improve the cryptographic support in Ruby. My experience building megar made it painfully obvious why we need to fix this.

Let it never be said, the romance is dead
'Cos there’s so little else occupying my head

I mentioned a few of my projects in passing. Here are the links for convenience:

• RGovData is a ruby library for really simple access to government data. It aims to make consuming government data sets a "one liner", letting you focus on what you are trying to achieve with the data, and happily ignore all the messy underlying details of transport protocols, authentication and so on.
• sps_bill_scanner is a ruby gem for converting SP Services PDF invoices into data that can be analysed with R. Only useful if you are an SP Services subscriber in Singapore, but other wise perhaps an interesting example of extracting postitional text from PDF and doing some R.
• megar ("megaargh!" in pirate-speak) is a Ruby wrapper and command-line (CLI) client for the mega.co.nz API. My example of how you *can* do funky crypto in Ruby ... it's just much harder than it should be!

Amplifying Human Emotion

(blogarhythm ~ Sweet Emotion 相川七瀬)

It all comes back to connectivity. Om Malik (TWiST #327 @00:37:30) has a brilliant characterization of the true impact of the internet:

human emotion amplified at network scale

Rolling the Mega API with Ruby

(blogarhythm ~ Can you keep a secret? - 宇多田ヒカル)

Megar (“megaargh!” in pirate-speak) is a Ruby wrapper and command-line client for the Mega API.

In the current release (gem version 0.0.3), it has coverage of the basic file/folder operations: connect, get file/folder listings and details, upload and download files. You can use it directly in Ruby with what I hope you'll find is a very sane API, but it also sports a basic command-line mode for simple listing, upload and download tasks.

If you are interested in hacking around with Mega, and prefer to do it in Ruby, give it a go! Like this:

# do a complete folder/file listing
session = Megar::Session.new(email: 'my@email.com', password: 'my_password')
session.folders.each do |folder|
  folder.files.each do |file|
    puts file.name
  end
end
# upload a file
file_handle = '../my_files/was_called_this.mp3'
session.files.create( name: 'First.mp3', body: file_handle )

Or from the command line:

$ megar -e my@email.com -p my_password ls
$ megar -e my@email.com -p my_password put *.pdf

I would still call it "experimental" at this stage because it needs more widespread hammering, and of course the Mega API is not fully documented yet. There are many more features of the API that it would be good to support, and I'd love for others to pitch in and help - go fork it on github!

I was keen to get a Mega account and check it out when the launch publicity hit, and was immediately impressed by the web interface. Very slick. Notwithstanding some of the intense analysis and some criticism (for example by SpiderOak and Security Now), the "trust no-one" design approach is very interesting to contemplate and hack around with.

The Mega API is still evolving. The documentation is thin and the main resource we have to work with is the Javascript reference implementation that actually runs the Mega site. But there has been quite a bit of work in the community to hack on the API - particularly in Python (with API analysis and projects like mega.py).

It didn't take me long to realise there was nothing much going on with Ruby. After a bit of messing around, I think the main reason for that is the pretty wretched state of cryptographic support in Ruby. Unlike Python (which has PyCrypto amongst others I'm sure), in Ruby we still on the whole get by with thin wrappers on OpenSSL that look and smell distinctly C-dy. But that's a challenge for another day...

For now I'm pretty happy that Megar has all the main crypto challenges solved (after a bit of low-level reverse engineering supported by a healthy dose of TDD). Now I wonder what I'm going to use it for?

Easy Mandrill inbound email and webhook handling with Rails

(blogarhythm ~ Psycho Monkey - Joe Satriani)

Mandrill is the transactional email service by the same folks who do MailChimp, and I've been pretty impressed with it. For SMTP mail delivery it just works great, but where it really shines is inbound mail handling and the range of event triggers you can feed into to your application as webhooks (for example, to notify on email link clicks or bounces).

The API is very nice to use, but in a Rails application it's best to keep all the crufty details encapsulated and hidden away, right? That's what the mandrill-rails gem aims to do - make supporting Mandrill web hooks and inbound email as easy and Rails-native as possible.

I recently added some new methods to mandrill-rails to provide explicit support for inbound mail attachments (in the 0.0.3 version of the gem).

With the mandrill-rails gem installed, we simply define the routes to our webhook receiver (in this example an 'inbox' controller):

resource :inbox, :controller => 'inbox', :only => [:show,:create]

And then in the controller we provide handler implementations for any of the 9 event types we wish to consume. Here's how we might get started handling inbound email, including pulling out the attachments:

class InboxController < ApplicationController
  include Mandrill::Rails::WebHookProcessor

  # Defines our handler for the "inbound" event type. 
  # This gets called for every inbound event sent from Mandrill.  
  def handle_inbound(event_payload)
    [... do something with the event_payload here, 
         or stuff it on a background queue for later ... ]
    if attachments = event_payload.attachments.presence
      # yes, we have at least 1 attachment. Let's examine the first:
      a1 = attachments.first
      a1.name # => e.g. 'sample.pdf'
      a1.type # => e.g. 'application/pdf'
      a1.content
      # => this is the raw content provided by Mandrill, 
      #    and will be base64-encoded if not plain text
      # e.g. 'JVBERi0xLjMKJcTl8uXrp/Og0MTGCjQgMCBvY ... (etc)'
      a1.decoded_content
      # => this is the content decoded by Mandrill::Rails, 
      #    ready to be written as a File or whatever
      # e.g. '%PDF-1.3\n%\xC4\xE5 ... (etc)'
    end
  end

end

That's nice and easy, yes? See the Mandrill::Rails Cookbook for more tips.

If you love playing with transactional mail and haven't tried Mandrill yet, it's well worth a look!

Designing for Interesting Moments

(blogarhythm ~ Moments Not Words - F.I.B)

Some deep thinking and analysis of how to design for interesting and effective interactions..

Designing for Interesting Moments

View more documents from Bill Scott.

2013: Time for web development to have its VB3 moment

(blogarhythm ~ Come Around Again - JET)

And that's a compliment!

Wow. This year we mark the 20th anniversary of the Visual Basic 3.0 launch way back in 1993.

It's easy to forget the pivotal role it played in revolutionizing how we built software. No matter what you think of Microsoft, one can't deny the impact it had at the time. Along with other products such as PowerBuilder and Borland Delphi, we started to see long-promised advances in software development (as pioneered by Smalltalk) become mainstream reality:

• finally, Rapid Application Development that really was rapid
• simplicity that put the development of non-trivial applications within the realm of the average computer user. It made simple things simple and complex things possible (to borrow from Alan Kay)
• development environments that finally did the obvious: want to build a graphical user interface? Then build it graphically (i.e. WYSIWYG), and build a complete client or client-server app from a single IDE.
• an event-driven programming model that explicitly linked code to the user-facing triggers and views (like buttons and tables)
• perhaps the first mainstream example of a viable software component reuse mechanism (improved and rebranded many times over time: ActiveX, COM, .NET)

In its day, Visual Basic 3.0 was variously lauded (by non-programmers who could finally make the app they always wanted) and loathed (by IT professionals shocked at the prospect of ceding control to the great unwashed). Interestingly, Visual Basic succeeded *despite* the language (BASIC, probably the most widely derided language of all time. Or perhaps it shares that crown with COBOL).

The party didn't last long however, as by the late 90's the internet had fundamentally changed the rules of the game.

VB, PowerBuilder and the like suffered from an implicit assumption of a client-server architecture, and were not prepared for a webified world. They didn't (all) disappear of course, with Visual Basic in particular finding a significant role as Microsoft's mainstream server-side language, and it lives on in Visual Studio. Yet it lost it's revolutionary edge, and had to be content to simply fit in as an "also can do in this language" alternative.

Web Development - a case of one step back and one step forward?

You would think that over the past 20 years, web development would have been able to leap far ahead of what was best practice in client-server computing at the time.

We have certainly come a long way since then, and many advances in practice and technology have become de rigueur. Here are some examples that would not have been considered normal by any stretch in 1993:

• Reliance on open standard protocols at every tier: from client to server, server to database and messaging systems
• Global, well-known repositories of shared, reusable code (Github, Rubygems .. and let's not forget grand-daddy CPAN)
• Version control. There is no argument.
• Automated testing tools and continuous integration.
• Open source is mainstream, and even preferred in many contexts.

Yet it is also salutary to reflect on some of the great innovations we saw back in 1993 that have yet to be re-invented and re-imagined successfully for the web.

I am thinking in particular of the radical productivity that was possible with the event-driven, WYSIWYG GUI programming model. It certainly hasn't gone away (take xcode for example). But why is that not the leading way of building for the web today? After all, the web is graphical and event-driven. A perfect fit one would think.

It has perhaps been the very success of the internet, and the rapid unconstrained innovation it has enabled, that has in turn inhibited major advances in web development.

Those that have come close (such as Adobe Flash) have ultimately failed primarily because they did not embrace the open standards of the web. And others, like Microsoft Visual Studio and Oracle JDeveloper have remained locked in proprietary silos.

On the whole, we still work at levels of abstraction that are no higher, and many times lower, than those embodied by the best tools of 1993. It is, after all, very difficult to build abstractions over a foundation that is in constant flux. And with highly productive languages and frameworks at our disposal (like Ruby/Rails), it makes complete sense for many - myself included - to actively spurn graphical IDEs for the immense flexibility we get in return for working at the coding coalface.

The Tide is Turning

Once the wild west of hackety scripts and rampant browser incompatibilities, the building blocks of the web have been coalescing. HTML5, CSS3 and leading browser rendering engines are more stable, consistent and reliable than ever. Javascript is now considered a serious language, and the community has embraced higher-level APIs like jQuery and RIA frameworks such as ember.js and backbone.js. Web design patterns are more widely understood than ever, with kits like bootstrap putting reusable good practice in the hands of novices.

On the backend, our technology stacks are mature and battle-tested (LAMP, Rails). And we have an array of cloud-ready, open source solutions for just about every back-end infrastructure need you can imagine: from BigData (Hadoop, MongoDB ..) to messaging (RabbitMQ, ØMQ ..) and more.

My sense is that in the past couple of years we have been edging towards the next leap forward. Our current plateau is now well consolidated. Yet despite efforts such as codecademy to open up software coding to all, web development remains as complex as ever. To do it well, you really need to master a dizzying array of technologies and standards.

Time for Web Development to Level Up

What does the next level offer? We don't know yet, but I'd suggest the following as some of the critical concerns for next gen web development:

• a unified development experience: the ability to build a full-stack application as one without the need for large conceptual and technological leaps from presentation, to business logic, to infrastructure concerns.
• implicit support for distributed event handling: a conventional mechanism for events raised on a client or server to be consumed by another client or server.
• event-driven GUI development: draw a web page as you want it to be presented, hook up events and data sources.
• it is mobile: more than just responsive web design. Explicit suport for presenting appropriately on the full range of desktop, tablet and mobile devices
• distributed data synchronisation: whether data is used live on a web page, stored for HTML5 offline, or synchronized with a native mobile application, our tools know how to distribute and synchronize updates.
• (ideally) let's not have to go back to square one and re-invent our immense investments in standard libraries and reusable code (like the extensive collection of ruby gems)

Do we have the perfect solution yet? No.

But we are starting to see enticing inklings of what the future may look like. Perhaps one of the most compelling and complete visions is that provided by the meteor project. It is very close.

Will meteor streak ahead to gain massive mid-share and traction? Or will an established platform like Rails take another giant step forward? Or is there something else in the wings we don't know about yet?

It will be an interesting year. And if the signs are to be trusted, I expect we'll look back on 2013 as a tipping point in web development - its VB3 moment.

Do you think we're in for such a radical shift? Or heading in a different direction altogether? Or will inertia simply carry the status quo... I'd love to hear what others think!

How to make an eBook

(blogarhythm ~ Land of a Thousand Words - Scissor Sisters)

So eBook sales have surpassed hardcover for the first time, and it is no surprise that the rise of the tablets is the main driver.

There's something quite comfortable about having a nice digital bundle of information at your fingertips, like warm buttered toast.

With relatively open standards and the ubiquity of ereaders, the ebook has become ideal packaging for all manner of information, from training manuals to open source project documentation. Or even that book that apparently 81% of us believe we have inside.

So how do you make an ebook? My first thought on searching for options is that we are pretty spoiled for choice. But there are important caveats to note, like how Apple iBooks Author can only publish in full fidelity to iTunes Book Store. And we can't get far before needing to study up on the various formats out there: EPUB is widely used, but not without its criticisms and edge-cases especially when trying to push the boundaries with multimedia and social aspects; the Kindle and other ereaders expect Mobi; and Kindle Fire introduced the KF8 format.

The good news is that producing pretty standard EPUB, Mobi, PDF and HTML variants of your book can be done very easily with a range of commercial and non-commercial tools. It's even possible to build an EPUB by hand with just a text editor if you are game.

I started to experiment with some open source toolchains to see just how well they work in practice. Personally, I'm liking the simplicity of using pandoc to build a range of output formats from markdown source files.

My experiments are in the eBook Toolchains project on github if you'd like to examine the various approaches I've tried. Have you tried something that's not there? I'd love to hear about it - comment below or better yet, send me a pull-request on the github project with your examples added!