Why would you open source a framework?

Interesting to listen to Scott Hanselman and Richard Campbell talking over the rationale for open sourcing Microsoft's ASP.net MVC framework on Hanselminutes show #175.

Part of the answer was a general desire to nudge Microsoft further towards embracing open source: "Why wouldn't you?". Which is admirable.

Partly it is a desire to open up the innovation envelope: Scott talked about his experience releasing TweetSandwich, and then seeing the unexpected derivative applications developed using the source as the base. Designing a framework is a daunting task. By definition, most of the framework's possible uses are not known ahead of time.

Take a listen to The Java Posse #241 which also came out this week, where they discuss the challenges of design as it applies to frameworks. One of the great concepts they talk about is how the best frameworks invariably have well designed escape hatches, to make sure you can overcome that typical problem of '..but the demo worked so well!'

Personally, I think having access to the source code of the framework is one of the most effective 'escape hatches' you can have.

Even if you never plan to fork or modify the framework, the ability to dive in and examine the source when things are not quite working as expected is really the difference between a framework you can work with, and a framework that will be discarded after a couple of projects. It is one of the great things about rails: often the documentation comes up short, but when you look at the api, the source code is but a click away!

Tweeting from your database with short urls

There's something cheekily enjoyable about getting all manner of 'non-human' things to tweet. I've heard of plants tweeting, but until I saw Lewis Cunningham's post announcing ORA_Tweet, I hadn't even thought of getting Oracle Database onto twitter.

Saturdays are good for little projects, so I thought I would add URL shortening service today;-)

Since twitter famously limits you to 140 characters, it is conventional to use a url-shortening service to include hyperlinks in your tweet. So my little play for today was to pair that idea up with ORA_Tweet.

There are a range of URL shortening services available including snipurl and tinyurl (here's a survey of services). I've been using is.gd for a while though, since it sports the simplest GET request 'api' you could imagine, making it great for ad-hoc programmatic use.

So I add an extra package called SHORT_URL which has just two functions:

  FUNCTION encode_url(
    p_url IN VARCHAR2 )
  RETURN VARCHAR2;

  FUNCTION encode_text(
    p_text IN VARCHAR2 )
  RETURN VARCHAR2;

encode_url the main wrapper around the http://is.gd call to get a short url for the one you provide.

encode_text is a more convenient function that takes a block of text, and will replace all the urls it contains with the appropriate shortened versions.

Then there's just one change to the ORA_TWEET package body:

45c45
<       url => 'status=' || SUBSTR( short_url.encode_text(p_string) ,1,140));
---
>       url => 'status=' || SUBSTR(p_string,1,140));

Now you can go wild with URLs in your database tweets:

BEGIN
  DBMS_OUTPUT.ENABLE;
  IF ora_tweet.tweet
    (
      p_user => 'twitter_username',
      p_pwd => 'twitter_password',
      p_string => 'ora_tweet v1.1 is complete! Now with url shortening ... see http://database-geek.com/2009/03/15/ora_tweet-tweet-from-oracle-a-plsql-twitter-api/' )
  THEN
    dbms_output.put_line('Success!');
  ELSE
    dbms_output.put_line('Failure!');
  END IF;
END;

Building on Lewis' original justification for building ORA_Tweet, you could for example include links to a report page or admin screen when your long-running process sends you its "I'm done" tweet.

That's if you need justification;-)

If you are interested, the source is up on my github account now: ORA_Tweet_With_Shorturls.zip

Idea #105: what name babby? (Dugg already pwned)

I just saw namemasher.com mentioned on programmable web.

It's a first step towards addressing one of humankind's biggest challenges: forget about running out of IP addresses, we're going to run out of usernames first!

What kind of handle do you think your children be able to get on Friendfeed? Under what name will your grandchildren be able to tweet? And do you think they stand a chance of getting the same nick across all their services?

There's a mad stampede for names going on, and any self-respecting parent (or prospective parent) who wants to bring up their children right ought to be out there buying up their progeny's place in cyberspace. Along with the tuition fund you need: website domain name, email account, twitter handle, skype, tumblr ... who knows which will survive, so get them all.

You never know: what if you kid gets famous, or even goes into politics? It wouldn't be very presidential if they tweeted as @spaceycasey123456.

Parents need something more than namemasher. In addition to the parents' names, it needs to munge in family and cultural background, existing baby name references like babynames.com (that's the one with the helpful definition of Espn), cross-check against existing accounts with something like namechk.com, and then go out and pre-register all the services for your unborn child. What an 18th birthday present that would make!

In short, the world needs wotnamebabby.com:

Learning not to love Java

Back in 99, I spent about six months procrastinating over a MAJOR decision (or so it seemed at the time).

To stick with the Microsoft camp - where I'd built up substantial experience through ActiveX to DCOM to COM, IIS and Commerce Server 3.0 - or jump onto the Java bandwagon?

Ultimately, Java won out, and along with millions of others, we've made Java the enterprise platform of choice (arguably).

The interesting point upon reflection, is that there was never any doubt in my mind that Java was somehow 'better' and more 'pure', in an academic sense.

Almost ten years later, I shocked myself recently when returning to Java after spending a good part of the past two years infatuated with ruby and rails and other scripting languages.

Where before I saw classical elegance, now my first thoughts were this sux and the language is just getting in my way.

First the verbosity kills me:

   WeLikeLongDescriptiveClassNames myReallyDescriptivelyNamedObject = WeLikeLongDescriptiveClassNamesFactory.getInstance(duh);

How many times do I have to tell the compiler what kind of object I am dealing with? Sure, I understand the benefits of the Factory pattern and the subtleties you can construct by virtue of the inheritance hierarchy. But most of the time, I just want the compiler to do what is most obvious, and don't make me write a novel for the most straight-forward tasks!

Then there's the language-promoted cultural phenomenon of class explosion. Need half a dozen value objects to represent the various information to be passed around in your domain? Sure! With unit tests all, I am sure. Most of the time, I'd now prefer to scream YAGNI!

But the true ephiphany was my run-in with primitives (again ... every year or so). Using JSP sessions as a perfect place to test this out. Take an int and stick it in the session:

int startingInt = 3;
session.setAttribute( "startingInt" , startingInt);

What type is in the session? An Integer object of course. Java has cleverly done a type conversion for you, from a primitive to an object. But woe betide the simple developer who assumes this must be a commutative operation:

int newInt = session.getAttribute( "startingInt" );
// Error!! 

That's what I call a language actively getting in your way. Kind of like if you deposited $10 cash in your bank, but when you went to withdraw the money, the teller threw the shutters down and said you could only get your money back in government bonds.

So do I have a point or is this just a rant (and certainly one that is not as erudite as many have written)?

I guess I must be pretty slow to catch on to what others knew long ago. The true test comes down to being able to realise one's goals. For personal projects, I find I'm able to knock out complete (and impressive) applications in Rails, while I still have ten java-based ideas on the drawing board.

It comes down to whatever works. And it should.

Does that mean I hate Java? Not at all. I still get a bit of a thrill out of its clean lines and voluminous open source library support. Much like seeing an aging and long-since outclassed 512BB. 'Classic' is timeless.

No. I don't hate Java, but neither do I love it the way it used to demand. I look back on my naïve 1999-self in wonder. Building experience is key, but it is a mistake to tie this to a language. Computer languages are the tools of our trade, and the more you know, the better you will be. When people mention Haskell, Scala, Eiffel ... do you know what they are on about, or do you just nod intelligently and pray not to get caught out?

With more tools at our disposal, we are better able to tackle new challenges in the most appropriate way.

To get the job done (which is generally the point, right?).

Crazy Stupid Security Policies #1

So we've all been there .. a super-secure data centre where they ban wireless access while in the server room.


I recently had an acquaintance pulled up for using wireless while in one such place. Twice. I won't name where, because this issue knows no bounds.

It is a nice and reasonable rule from the security policy astronauts' perspective, because usually those responsible for the policy are not also accountable for operations. I presume the main concern is bridging networks and (intentionally or not) providing backdoor access.

They conduct audits and spot-checks, and on the surface everything looks great. If all you are concerned about is the paperwork, your job is done.

But in practice, and from what I have observed over the years, reality is a very different thing.

Unless all you are doing is bolting a server into a rack or plugging the fibre cables in, it is hard to get the job done these days without external network access.

• Maybe that is for research: consulting vendor guides or searching the knowledge bases (because no-one provides all the doc on disk anymore - it's on the web!)


• Or often you need to test the system you are configuring or diagnosing, and that can only be done from "outside" (SSL termination at an external-facing load-balancer for instance).


• Or, in this Web 2.0 world, you need to collaborate with colleagues to get the job done. Skype or IM to get hold of the expertise needed for the task at hand.


• Worse still, you have an "escort" policy, but a simple request to get in or out of the data centre is meet with abuse, recalcitrance or outright hostility from the people who are meant to escort you (like it's not their job!)

All these factors increase the frustration of SEs the world over, in the face of data centre policies that treat IT as if it were like installing an air-conditioner.

Now what happens when the threshold of pain is pushed up and up like this? At some point, the immediate pain (can't get to metalink) exceeds the potential future pain (maybe I'll get caught using wifi).

And, dear data center security experts, happens next?

Human nature takes over. Before you know it, you have a feral group of wifi/bluetooth/3G connected people running around your DC.

The worst part is that you have pushed the behaviour underground, where it is truly uncontrolled. In being secretive, people are breaking the most significant security prohibitions of all: bridging networks. Sorry, you don't know how much it goes on. (personal aside: yes, I admit I have, in the past, used wifi in a non-wifi DC. But being the conscientious and security aware guy that I am, I was always quite anal about disconnecting from the DC network before getting on wifi. Not that anyone knew. And if they did, my reward would have been ... a punishment!!)

So what approach would an enlightened, modern data center manage take? I would sleep much better at night if I:

• Had an open wifi usage policy to bring the practive into the light of day. Maybe tables running between your racks for wifi-connected laptops (bolt them to the desk if you like, with a CCTV overhead), while direct network/server access had to be done rack-side.


• Educate on responsible wifi use. Make sure people understand the risks of bridging nets and make it clear its OK to be on wifi, but not ok to bridge.


• And have a clear understanding with my internal staff that "escorting" is not an interruption to the work they are doing, it is a vital task that will be rewarded.

Fundamentally, this means I would need to reconsider how I defined my job as a data centre manager: I'm not a slave to a policy handed down from upon high, my job is to implement and enforce the best procedures possible that both enforce the policy goals, while providing excellent customer service. Achieving this may mean I need to think a little out of the box, even be a little creative and pragmatic.

Now I should be clear that in writing this I am not condoning anyone who breaks a clearly published policy not to use wifi in a data center. On the contrary, if you have to work in such a place, I'd say stick to the policy, and drive the escorts nuts as you go in, out, in, out and in again to get the job done. Demand a full time escort if you need it.

My real message is to anyone with authority over security policies and their application: recognise that a policy on paper is worth exactly the cost of the paper unless you have taken into consideration the human factors involved and done your best to ensure that your procedures and environment are optimally designed to encourage the very best behaviours, and not the ones you most want to avoid.

Anyone have data centre security horror stories to share? I would love to hear about them! Better yet, how you manage to get around the stupidity, yet stay "legal".

Like you scroll wheel? You want KatMouse!

I just heard about KatMouse on Security Now! #182.

I type fast, but love my mouse ( ... and there has been lot's of controversy about the numeric keyboard getting in the way too).

That means I'm a scroll-wheel addict. Till now, I just accepted the fact that it doesn't kick in all the times I would naturally expect it to work, like old applications, or in controls or windows that don;t have focus.

As soon as I heard Steve talk about KatMouse I knew I just had to get it. Two minutes later, it is installed and I'm loving it!

Best feature: when I have overlapping windows, I can scroll whichever window the mouse is over. Doesn't have to be in focus. Beautiful! This is how the scroll wheel should have worked all along;-)

Wordling your twitter flock (and an aside on sniffing out bad proposals)

Twitter just keeps on attracting people who glob on new and interesting innovations; the latest I've seen is TwitterSheep which does a neat wordle of your followers.

Twitter has got to be the poster-child for YAGNI. By keeping things real minimalist, it not only creates the space but also the yearning for innovation - and we're seeing that in bucketloads (see also MrTweet; proud to say, Singapore compatriots!).

Wordles are simple but interesting semantic analysis toys. How useful can a word frequency count be? You may see a bit of noise, but I read them as a measure of obsession.

I've been using wordle.net for a while to do proposal analysis. It's become a "must-do" step before submission, as the insights can be invaluable. Simply paste in the full text of your proposal, and reflect on the resulting wordle. It's better than therapy!

What should be prominent of course are the concerns and issues that you know your client highly values. But what you will often see in your first drafts are a whole lot of words that are basically synonyms for ME! As in: my company, our product names, our partners, our technology etc etc.

All very well (it is a proposal after all), but just like a good conversationalist, shouldn't good proposals be at least equally weighted towards your listener's needs and desires? I think so, and wordles are quick and easy tools for sniffing out the boilerplate proposal that doesn't give a flying fig for the customer!

Still not convinced? Well, see the wordles that Billy Cripe did of Obama and McCain's convention speeches. Thank god you voted in Obama is all I can say!

What scares me is a seeing an obsession with "Country" "Americans", "fight", "war", "God" and even friggin' "nuclear", "attack" in the one wordle!!!!

So here's my twitter flock. Hmm, we're a pretty boring bunch!



Props to Daryl Tay for the TwitterSheep link.

Best Practices in Web Form Design

I'm humming and hahing over some form designs at the moment. These days you have so many options, especially when you are getting smart with ajax and scriptaculous tricks.

Having options is always a double-edged sword. Yes, they allow you to do amazing things. But they provide a great recipe for procrastination.

.. just the situation where some thoughtful, concise guidance on leading practices from someone who knows their stuff can be a goldmine.

Thankfully I stumbled upon this great presentation on web form design by Luke Wroblewksi. It's a classic, and now I see he has a book out on the topic which instantly went on my "must read" list.

SOA is dead! Was it ever alive?

There's nothing like a "financial correction" to make people wake up and start sniffing the crap that has been shoveled their way.

Waddya know? SOA is dead says Burton.

If I was paying for Burton services, I would be asking for a refund and an explanation as to why it took so long to identify the bleedin' obvious

The irony may have been a bit thick for anyone to realise that my story of Eric the Architect was little more than a lampoon on the generations of IT attempting to find a home within business (true friend - true story - honest!!!).

Guys - pragmatism rules. SOA never had anything to do with the bottom-line. Directly. And the indirect contributions lacked evidence and credibility.

In an excess of pragmatism, Miko Matsumura shut down the SOA center blog on the back of the Burton article. The new blog will be called the Whatever Center. Love the name, but will the dns changes ever propagate? I hope so, for Matsumura-san's integrity. Unlike WFTs transition to WTH, this is not a joke.

Personally, I'm with Justin Kestelyn:

The problems remain with us, whatever we choose to call the solution.

Tripping up on base64 encoding with webcache

Looking at an Oracle Portal installation recently, I wanted to call the WebCache invalidation API from SQL.

$ORACLE_HOME/webcache/examples/invalidation.sql sounds like a good idea, until:

begin
invalidate('hostname.com',9451,'password1','http://hostname.com/page')
end;
Error report:
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at "SCOTT.BASE64_ENCODE", line 51
ORA-06512: at "SCOTT.BASE64_ENCODE", line 57
ORA-06512: at "SCOTT.INVALIDATE", line 38
ORA-06512: at line 2

Uh-oh. The base64_encode function included in the script is having trouble with the password. A quick look at the code...

create or replace function base64_encode
(
   p_value in varchar2
) 
return varchar2 is

    BASE64_KEY constant varchar2(64) := 
        'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
    l_buffer varchar2(32767);
    l_len integer := trunc(length(p_value) / 3);

...

begin
    for i in 0..l_len loop
        l_buffer := l_buffer || encode_chunk(substr(p_value, i * 3 + 1, 3));
    end loop;

    return l_buffer;
end base64_encode;

Note l_len division by 3, then using it in the for loop. Yep, classic 0/1 base offset issue. Any password with a length of 3, 6, 9 etc characters breaks the code. Fixed with a -1:

    for i in 0..l_len - 1 loop
        l_buffer := l_buffer || encode_chunk(substr(p_value, i * 3 + 1, 3));
    end loop;

But that raises more questions. What is this base64 encoding function doing here anyway?

At some point in time it might have been required, but Oracle Database has had the standard function utl_encode.base64_encode for at least a few versions. It encodes RAW so there's a bit of friggin around with types:

select utl_raw.cast_to_varchar2( utl_encode.base64_encode( utl_raw.cast_to_raw('password1') ) ) as B64 from dual;

B64                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
-------------
cGFzc3dvcmQx     

I did note the comment in invalidation.sql to the effect that:

-- this old example is replaced by 2 files, collectively known as the
-- PL/SQL invalidation API.
--
-- the 2 files are
--
-- wxvutil.sql which does similar things as what invalidate.sql did
-- wxvappl.sql which is a wrapper of wxvutil.sql
--
-- both files are located in the same directory.

Well, these files are not in the same directory (they are actually in ../toolkit), and what's the excuse for shipping broken examples anyway, even if they are old and obsolete?

A case of severe backblogarrythmia

My backblog is giving me the irits. And what's worse I join the sad club of people who thought they cleverly invented a new word!

Backblog - Overflow of incidents you intend to write about on your weblog.

However, perhaps I can claim backblogarrythmia as my own:

.. a disruption in the regular flow of weblog posts, often caused by overwork, booze or a new romance. Primary symptom is the backblog, which in turn can exacerbate the problem through increased levels of performance stress. Treatment: JFPS*.

I used to work in a factory where I swear one of the leading hands had a personal goal of inventing a new word everyday. He'd casually call over something like "Hey, pass me the gumlicker would ya?" and take great pleasure in our confusion. He'd be mock-shocked of course at our "ignorance", and delight in explaining the word and how stupid we were for not knowing it. Crazy thing is, he'd always have a great and usually convoluted etymology.

He did this pretty much every day I knew him. Just an average working class bloke, with an average education. But I always secretly admired and envied his creativity with words, wondered where the inspiration came from, and whether he'd use the gift for anything more than keeping his workmates from boredom.

Me? The only words I invent tend to arrive as serendipitous typos. Yesterday's effort:

Museover - particularly thought-provoking and succinct tip or pop-up text that is displayed when you hover your mouse over an icon or word.

恭喜发财!

Update 27-Jan: It's official: museover and backblogarrythmia are in the (pseudo)dictionary

* just fucking post something

Code formatting and line continuations with Javascript and CSS

I've had a question up on stackoverflow recently in an attempt to get a nice solution for adding a glyph with CSS to indicate a wrapped line (e.g. in a command line sample).

With a few good suggestions, I had a reasonable solution but it still required a bit of manual markup to prepare the example. Then I saw Pranav Prakash's post about syntaxhighlighter by Alex Gorbatchev.

With a few tweaks (see the stackoverflow question for the details), I'm now able to combine the best of both worlds: great formatting and syntax highlighting with syntaxhighlighter and also line continuation glyphs. Here's an example of the result:

public class HelloWorld {

  public static void main (String[] args)

  {

    System.out.println("Hello World! But that's not all I have to say. This line is going to go on for a very long time and I'd like to see it wrapped in the display. Note that the line styling clearly indicates a continuation.");

  }

}

NB: if you have javascript disabled, or reading this through a newsfeed (or if I hit a bug) you may not see the full effect. Here's a screenshot of what should be appearing above:


Now I just need to go back and update the code formatting in all my old posts. Basket!

Oracle Shell Scripting

I remember seeing Jon Emmons' announcement on the Oracle News Aggregator and I've had it in my "wanted" list on bookjetty for ages. This week I discovered Jon's Oracle Shell Scripting: Linux and UNIX Programming for Oracle (Oracle In-Focus series) at the NLB and have just enjoyed a good read of it. I wish more DBAs had read this book. In fact it should be mandatory to get an OCP certification! Let's face it, most Oracle installations are running on a *nix variant, and you can't be a DBA if you are not comfortable at both the SQL*Plus and shell prompt. To be a good and efficient DBA in my book, I want to see evidence of thinking smart, and repetitive task automation. When I see so-called DBAs who are happy to type the same "select .. from v$.." query every day of their working life, I doubt their brain is switched on, and I find it really, really scary to think they have the sys/system passwords! They say tool usage is a sure sign of advanced intelligence in birds. And the same applies to all of us in IT. The three examples I look for at an Oracle Database installation are: RMAN Grid Control Shell scripts

If none of these are present, then I tend to presume the real DBA has long left the building. Even if you are using third-party alternatives, do you continue to re-evaluate the Oracle capabilities with each new release?

Jon Emmons' book is of course more focused than this. It perfectly fills a niche, with an approachable, practical and comprehensive coverage of shell scripting from a DBA's perspective.

I can see the ideal audience for this book is people who are reasonable familiar with Oracle administration but are new to shell scripting. This book will rapidly teach you all you need to know on the scripting side (and let you skip alot of stuff you can learn later).

In other words, if you are a DBA who has just been assigned to manage a Unix-based system for the first time in your career: get this book. Forget all the (great) general Linux/Unix/shell scripting books for now. Don't even think the Oracle docs will teach you what you need to know. Oracle Shell Scripting: Linux and UNIX Programming for Oracle (Oracle In-Focus series) is what you need!

If you are coming the other way though - an experienced Linux admin being told that from Monday you also need to manage an Oracle database - I'd say this book probably doesn't have much to teach you. There's much more you'd need to learn about Oracle first (after telling your manager he's crazy), and there are really no scripting tricks in the book that you shouldn't already know. The main benefit you get would probably be a few pages in chapter 6 that cover the tricks of using sqlplus in a shell script - all in one place rather than having to tease it out of the Oracle docs (or see this related question on stackoverflow).

Originally posted on It's a Prata Life.

Synchronising two directory trees - tree-sync 2.4

tree-sync.pl has had a little update thanks to a contribution from Dave Stafford.

• Added option 'diff' to view only changes


• Added option 'ignore' to exclude certain extensions


• Added option 'brief' to remove src/dest directories in report view to make cleaner output

In the 2.4 release I've also added the 'force' option to control read-only file handling.

tree-sync.pl is a bit of a geeky tool, suitable for cases where you want detailed control over a file-based copy/mirror/backup process from the command-line. There are many flashier solutions available these days, but if tree-sync scratches your itch - go for it!

tree-sync.pl continues to be available on CPAN, but from the 2.4 release I have moved the source to github. If you have changes to suggest or bugs to fix, please go ahead and join this project.

Rolling Project Euler on Ruby

I first heard about Project Euler last week on the stackoverflow podcast. Michael Pryor (fogcreek co-founder) makes a quick side reference in discussion with Joel Spolsky, Jeff Atwood and the rest of the SO team.

Well I checked it out last week, got hooked and spent most of the weekend earning my "level 1" badge;-)

Aside from dusting off some long-forgotten and arcane knowledge from my youth, I found it a fantastic opportunity to stretch my fundamental ruby chops. In fact, I'd recommend a few questions at Project Euler as a right-of-passage whenever you are learning a new programming language.

I've only been using ruby for a year or so, and in that time thought I had picked up a fair bit. But I was still able to amaze myself at just how many of the problems were knocked over in just 1 or 2 lines with a bit of duck punching and liberal use of blocks with Enumerables.

I'm late to the Project Euler craze, so you will already find many people posting hints for specific questions if you just google. I thought I'd share some of the "common code" I've been building up as I go through the questions.

I put a recent copy of the source up on github for anyone who is interested (MyMath.rb), but what follows a sampling of some of the more interesting pieces.

First thing you will note is that I have written all these "common" routines as extensions to some of the fundamental classes in the ruby library: Integer, Array, String.

It doesn't have to be this way, and for less trivial activities you might be right to be concerned about messing with the behaviour of the standard classes. Maybe I'm still enjoying my ruby honeymoon period, but I do get a thrill out of being able to write things like

1551.palindrome?
=> true

Integer Extensions

It's just so easy to code up simple calculation and introspection routines..

class Integer
  # @see project euler #15,20,34
  def factorial
    (2..self).inject(1) { |prod, n| prod * n }
  end

  # sum of digits in the number, expressed as a decimal
  # @see project euler #16, 20
  def sum_digits
    self.to_s.split('').inject(0) { |memo, c| memo + c.to_i }
  end

  # num of digits in the number, expressed as a decimal
  # @see project euler #25
  def num_digits
    self.to_s.length
  end
  
  # tests if all the base10 digits in the number are odd
  # @see project euler #35
  def all_digits_odd?
    self.to_s.split('').inject(0) { |memo, s| memo + ( s.to_i%2==0 ? 1 : 0 ) } == 0
  end
  
  # generates triangle number for this integer
  # http://en.wikipedia.org/wiki/Triangle_number
  # @see project euler #42
  def triangle
    self * ( self + 1 ) / 2
  end
end

Prime numbers feature heavily on Project Euler, and I think calculating a prime series was my first lesson on why you can't brute-force everything;-) Enter the Sieve of Eratosthenes and related goodness..

class Integer 
  # http://en.wikipedia.org/wiki/Prime_factor
  # @see project euler #12
  def prime_factors
    primes = Array.new
    d = 2  
    n = self      
    while n > 1
     if n%d==0
        primes << d
        n/=d
      else
        d+=1
      end
    end
    primes
  end
  
  # http://en.wikipedia.org/wiki/Divisor_function
  # @see project euler #12
  def divisor_count
    primes = self.prime_factors
    primes.uniq.inject(1) { |memo, p| memo * ( ( primes.find_all {|i| i == p} ).length + 1) }
  end
  
  #
  # @see project euler #12, 21, 23
  def divisors
    d = Array.new
    (1..self-1).each { |n| d << n if self % n == 0 }
    d
  end

  # @see project euler #
  def prime?
    divisors.length == 1 # this is a brute force check
  end
  
  # prime series up to this limit, using Sieve of Eratosthenes method
  # http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes
  # @see project euler #7, 10, 35
  def prime_series
    t = self
    limit = Math.sqrt(t)
    a = (2..t).to_a
    n = 2
    while (n < limit) do
      x = n*2
      begin
        a[x-2]=2
        x+=n
      end until (x > t )
      begin
        n+=1
      end until ( a[n-2] != 2 )
    end
    a.uniq!
  end

  # @see project euler #23
  def perfect?
    self == divisors.sum
  end

  # @see project euler #23
  def deficient?
    self > divisors.sum
  end

  # @see project euler #23
  def abundant?
    self < divisors.sum
  end
end

Next we visit the Collatz conjecture and an interesting routine to make numbers "speak english"..

class Integer     
  # http://en.wikipedia.org/wiki/Collatz_conjecture
  # @see project euler #14
  def collatz_series
    a = Array.new
    a << n = self
    while n > 1
      if n % 2 == 0
        n /= 2
      else
        n = 3*n + 1
      end
      a << n
    end
    a  
  end

  # express integer as an english phrase
  # @see project euler #17
  def speak
    case
    when self <20
      ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten",
       "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen" ][self]
    when self > 19 && self < 100 
      a = ["twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"][self / 10 - 2]
      r = self % 10
      if r == 0
        a
      else
        a + "-" + r.speak
      end
    when self > 99 && self < 1000
      a = (self / 100).speak + " hundred"
      r = self % 100
      if r == 0
        a
      else
        a + " and " + r.speak
      end      
    when self > 999 && self < 10000
      a = (self / 1000).speak + " thousand"
      r = self % 1000
      if r == 0
        a
      else
        a + ( r <100 ? " and " : " " ) + r.speak
      end      
    else
      self
    end
  end
end

Calculating integer partitions is one of my favourites ... a nice, super-fast recursive algorithm. For problems like "how many ways to make $2 in change?"

class Integer 

  # calculates integer partitions for given number using array of elements 
  # http://en.wikipedia.org/wiki/Integer_partition
  # @see project euler #31
  def integer_partitions(pArray, p=0)
    if p==pArray.length-1
      1
    else
      self >= 0 ? (self - pArray[p]).integer_partitions(pArray ,p) + self.integer_partitions(pArray,p+1) : 0
    end
  end   
end

Finally, rotations and palindromes (base 2 or 10): methods that rely on some underlying String routines that come later...

class Integer 
  # returns an array of all the base10 digit rotations of the number
  # @see project euler #35
  def rotations
    self.to_s.rotations.collect { |s| s.to_i }
  end

  # @see project euler #4, 36, 91
  def palindrome?(base = 10)
    case base 
    when 2
      sprintf("%0b",self).palindrome?
    else
      self.to_s.palindrome?
    end
  end
end

Array Manipulations

Array handling is particularly important. Start with some simple helpers, then move onto greatest common factor and a couple of least-common multiple implementations. My favourite here - lexicographic permutations.

class Array

  # sum elements in the array
  def sum
    self.inject(0) { |sum, n| sum + n }
  end
  
  # sum of squares for elements in the array
  # @see project euler #6
  def sum_of_squares
    self.inject(0) { |sos, n| sos + n**2 }
  end
  
  # @see project euler #17
  def square_of_sum
    ( self.inject(0) { |sum, n| sum + n } ) ** 2
  end
    
  # index of the smallest item in the array
  def index_of_smallest
    value, index  = self.first, 0
    self.each_with_index {| obj, i | value, index = obj, i if obj<value  }
    index
  end

  # removes numbers from the array that are factors of other elements in the array
  # @see project euler #5
  def remove_factors
    a=Array.new
    self.each do | x | 
      a << x if 0 == ( self.inject(0) { | memo, y | memo + (x!=y && y%x==0 ? 1 : 0)  } )
    end
    a
  end

  # http://utilitymill.com/edit/GCF_and_LCM_Calculator
  # @see project euler #5
  def GCF
    t_val = self[0]
    for cnt in 0...self.length-1
      num1 = t_val
      num2 = self[cnt+1]
      num1,num2=num2,num1 if num1 < num2
      while num1 - num2 > 0
        num3 = num1 - num2 
        num1 = [num2,num3].max
        num2 = [num2,num3].min
      end
      t_val = num1
    end
    t_val
  end

  # http://utilitymill.com/edit/GCF_and_LCM_Calculator
  # @see project euler #5
  def LCM
    a=self.remove_factors
    t_val = a[0]
    for cnt in 0...a.length-1
      num1 = t_val
      num2 = a[cnt+1]
      tmp = [num1,num2].GCF
      t_val = tmp * num1/tmp * num2/tmp
    end
    t_val  
  end

  # brute force method:
  # http://www.cut-the-knot.org/Curriculum/Arithmetic/LCM.shtml
  # @see project euler #5
  def lcm2
    a=self.remove_factors
    c=a.dup
    while c.uniq.length>1
      index  = c.index_of_smallest  
      c[index]+=a[index]
    end
    c.first
  end

  # returns the kth Lexicographical permutation of the elements in the array
  # http://en.wikipedia.org/wiki/Permutation#Lexicographical_order_generation
  # @see project euler #24
  def lexicographic_permutation(k)
    k -= 1
    s = self.dup
    n = s.length
    n_less_1_factorial = (n - 1).factorial # compute (n - 1)!
    
    (1..n-1).each do |j|
      tempj = (k / n_less_1_factorial) % (n + 1 - j)

      s[j-1..j+tempj-1]=s[j+tempj-1,1]+s[j-1..j+tempj-2] unless tempj==0
      n_less_1_factorial = n_less_1_factorial / (n- j)
    end
    s
  end
  
  # returns ordered array of all the lexicographic permutations of the elements in the array
  # http://en.wikipedia.org/wiki/Permutation#Lexicographical_order_generation
  # @see project euler #24
  def lexicographic_permutations
    a=Array.new
    (1..self.length.factorial).each { |i| a << self.lexicographic_permutation(i) }
    a
  end
    
end

String Helpers

Last but not least, some String methods that just make things so much easier...

class String

  # sum of digits in the number
  # @see project euler #16, 20
  def sum_digits
    self.split('').inject(0) { |memo, c| memo + c.to_i }
  end

  # product of digits in the number
  # @see project euler #8
  def product_digits
    self.split('').inject(1) { |memo, c| memo * c.to_i }
  end
  
  #
  # @see project euler #4, 36, 91
  def palindrome?
    self==self.reverse
  end 

  # returns an array of all the character rotations of the string
  # @see project euler #35
  def rotations
    s = self
    rots = Array[s]
    (1..s.length-1).each do |i|
      s=s[1..s.length-1]+s[0,1]
      rots << s
    end
    rots 
  end
    
end

With all the above in place - and with the aid of a few brain cells - some deceptively complicated questions (like "How many different ways can £2 be made using any number of coins?") are essentially one-liners:

require 'benchmark'
require 'MyMath'

Benchmark.bm do |r|
  r.report {
    answer = 200.integer_partitions([200,100,50,20,10,5,2,1])
  }
end

Love it;-)

Best Oracle OpenWorld 2008 quote so far..

WSJ blogger Ben Worthen on the HP Oracle Exadata Storage Server:

Oracle now has a device that is kind of like an iPod except that it is a lot bigger and a lot more expensive and not as cool.

"... priced at US$4,000 per terabyte of storage, plus the database license costs" Oh my.

On context paths in Java EE

I was recently involved (tangentially) in an exercise to migrate a tomcat-based JSP application to Java EE packaging, which had me taking a fresh look at the concept of context paths and considering best practices for handling them.

When you package and deploy a Java EE web application, it has a context-root which effectively becomes the path on your application server that the application is available under. For example, the following module definition would setup MyApp to be available under http://server.domain/myapp/

<module>
<web>
<web-uri>MyApp.war</web-uri>
<context-root>myapp</context-root>
</web>
</module>

Context paths make it possible to host many applications on the one server as long as you keep the paths unique. As advised in Build to Spec! Part II:

Always specify a unique context root for every Web application you deploy to avoid naming collisions with applications already deployed.

It is possible to install an application with a context root of "/" but there are two considerations to bear in mind:

1. Applications servers will usually have a default application already installed under "/" which would need to be removed first.


2. The reason why the default application exists is to host resources that are not packaged as a web application (which may or may not be a concern, depending on what you are serving).

Problems arise when applications are coded with implicit assumption or explicit reference to the context path they will run under.

This is often the case - as I discovered - when migrating simple JSP applications to Jave EE packaging. Either the app assumes it will run from "/", or it has hard-coded paths under the root.

It can also occur in applications designed to be packaged as a .war or .ear file, if the developer assumes that the context-path will remain the same and does some hard-coded shortcuts. This breaks the Java EE separation of duties design, and prevents the system administrator from chosing to deploy the application on another context path (if, for example, there is a conflict with another application).

What the Specs Say

The JSR 53: JavaTM Servlet 2.3 and JavaServer PagesTM 1.2 Specifications define the context path concept, and some relevant API features.

Firstly there is getContextPath() which allows you to obtain the context path.

There's always been some doubt as to how sendRedirect() should behave though, but now that is cleared up in the 2.3 spec. From Servlet 2.3: New features exposed

And finally, after a lengthy debate by a group of experts, Servlet API 2.3 has clarified once and for all exactly what happens on a res.sendRedirect("/index.html") call for a servlet executing within a non-root context. The issue is that Servlet API 2.2 requires an incomplete path like "/index.html" to be translated by the servlet container into a complete path, but doesn't say how context paths are handled. If the servlet making the call is in a context at the path "/contextpath," should the redirect URI translate relative to the container root (http://server:port/index.html) or the context root (http://server:port/contextpath/index.html)? For maximum portability, it's imperative to define the behavior; after lengthy debate, the experts chose to translate relative to the container root. For those who want context relative, you can prepend the output from getContextPath() to your URI.

How to Retrofit Correct Context Path Handling

Migrating an application to Java EE packaging can be a it of a nightmare if url references are not nicely relative, and avoid any assumptions about the full path to the application.

Obviously, in this situation you probably can't avoid going in to clean up the code at some point.

But there are some tricks that can be used to delay that activity.

I've experimented with using servlet filters to do rewrites on the oubound HTML, and that seems to work fine. The filter intercepts all the output of the application, and can be used to fixup text/html or css using regex replacements, and even change the sendRedirect behaviour if desired. But it does introduce some overhead, and I wouldn't see it as a permanent solution. (see EnforceContextRootFilter-1.0-src.zip for all the lowdown on the servlet filter approach, including code you can use and adapt if it meets your needs)

restful_authentication and OpenID with Rails 2

Note to self: Yes! managed to navigate the various OpenID resources for rails and managed to successfully setup OpenID with restful_authentication on Rails 2.1.

There are a few tricks to be aware of. Prompted by a question on stackoverflow.com, I thought I would post my notes here.

Installing restful_authentication

Usually, you will be plugging open_id_authentication into your app's identity management framework. One of the most popular - and the one I've been using of late - is restful_authentication.

Getting restful_authentication working seems to require a few tweaks however - at least for the latest version found up on github.

Firstly, I've had no success directly installing it using the plugin script like this:

ruby script/plugin install git://github.com/technoweenie/restful-authentication.git

Rather, I've had to manually clone the github repository and manually setup the plugin directory:

$ git clone git://github.com/technoweenie/restful-authentication.git
$ mv restful-authentication/ myapp/vendor/plugins/restful_authentication
$ rm vendor/plugins/restful_authentication/.gitignore
$ rm -fR vendor/plugins/restful_authentication/.git

Note that I'm cleaning up some of the git bits, and also changing the plugin directory name to use an underscore rather than a hyphen (Rails usually complains to me otherwise).

Setting up restful_authentication is then pretty straight-forward:

$ mkdir lib
$ ruby script/generate authenticated user sessions
$ rake db:migrate

Note: if you don't already have the 'lib' directory in your application, it must be created first, otherwise the generator will fail.

Installing open_id_authentication

Ryan Bates' Railscast on Openid is the best thing I've found to follow. Even though it was recorded with Rails 1.2.3, I've been able to successfully follow the tutorial with Rails 2.1.0. The only point to note is that for:

gem install ruby-openid

I installed 2.1.2, rather than the 1.1.4 used in railscast.

Installing open_id_authentication is then a doddle.

$ ruby script/plugin install open_id_authentication

Follow the Railscast from this point to integrate OpenID with restful_authentication.

Using restful_authentication on heroku

The plugin installation problems mentioned above also mean that you need to use a few tricks to get it working in a heroku-hosted application. I've found it best to clone your heroku app and add the restful_authentication plugin locally, and then git push it back to heroku when done.

Show the Whale!

Adam Keys and Geoffrey Grosenbach introduced the term for 2008 on the rails podcast: show the whale.

I think it's perfect (meaning site down/build broke etc), and in my lexicon already!

==> No, this is not the official fail whale logo! The real one was done by Yiying Lu, a talented young designer from China/Sydney, who is now world famous thanks to twitter's stability problems.

Code Like a Rocket Scientist

Do. Or do not. There is no try. -- Jedi Master Yoda The Seven Secrets of How to Think Like a Rocket Scientistby Jim Longuski is a great book. It's almost a pattern language for innovation and getting things done. (I reviewed it in a little more detail here) Longuski's intent is clearly to crystallize some useful patterns from the life of a "rocket scientist" and recommend them for general use. He successfully pulls it off and in a way that doesn't come over as elitist or sycophantic. As a little thought experiment, I was interested to see how one might apply his ideas to the arena of coding or software development. The "seven secrets" actually encompass 50 lessons in all, so this post soon got a bit out of hand! Anyway, this is what I came up with ... so enjoy, and let me know your own thoughts!

1. Dream

Imagine It

You'll never achieve what you can't even imagine. That's good general advice for life. When it comes to our software efforts though, how much time do we really spend letting our imaginations run wild before shifting to a convergent thinking mode and start drilling down on the solution?

Work on the Big Picture

Every journey begins with the first step; a cathedral gets built brick by brick. But without the big picture, we are lost with a pile of bricks.

Arguably the technology world is too obsessed with the big picture. Think frameworks to end all frameworks, and our constant reinvention of the perfect computing paradigm: client-server to network-computer to EAI and SOA.

But perhaps that is a sign that we don't spend enough time deeply thinking out the big picture and thus we are stuck leapfrogging from one 80% solution to the next. Who is the programming equivalent of Einstein with his general theory of relativity and lifelong search for a unified theory of everything? Or are we still waiting for that person to be born?

Aim High

It goes without saying that you must aim high in order to achieve great things.

In software, I think it's actually rather common to have high goals in terms of functionality. Often our problem is that the goals are found to be too high at the end of the day. Massive scope slash, pulled features and death-marches result. But remember this is the "Dream" stage. If there is a failure, it is not in the dream, it is skipping straight to "Stage 7. Do"!

One area where our dreams tend to lack ambition is quality. Peopleware has a lot to say on this subject!

BS!

Encouraging a bit of BS is a good way of knocking the stiffness and formality out of the process. Which is what you need if you are looking for creativity, even if your code is "serious stuff".

There's also good logic here. When you invent the most outrageous porkers, you are probably using the technique of inversion - you are purposely searching for inspiration outside the nominal constraints of your problem. In other words, the bullshit artist naturally lives outside the box. Which is where you would expect to find the breakthrough idea of course, right? Make sense, or is that just a prime example of good BS itself!?

Brainstorm

Create Desire

Tell a Story

Sleep on It

Literally. As in: work on a problem. Go to bed. Wake up in the morning and find the solution is within your grasp.

"Sleep on It" is common folklore, but Longuski's own experience lead him to believe it held some truth.

In Brain Rules, John Medina presents more fascinating research that demonstrates the effect. Not only are problems more easily solved, but having slept on a problem, you are more likely to make a creative leap to a better solution.

Interestingly, Medina's work also demonstrated proved the truth another bit of folklore: the natural orientation of some people to be early birds, and others to be night owls.

So in a software development setting, we would be wise to take advantage of this knowledge:

• If you are trying to crack a hard problem, pulling an all-nighter is a dumb thing to do. Work it, get some sleep, then come back to it in the morning.


• Know who your night owls and early birds are. If you want the worst productivity possible, making night owls struggle in at 7am and early birds work late into the night.

Think JFK

Who is the JFK, the visionary, for your software project? The one person you all turn to, who helps you believe the impossible might actually be possible, and more importantly nurtures the desire to go for it?

Not all projects merit "man on the moon" scale leadership of course. But personally, I'm not interested in working on a project that doesn't have a clear goal. And all goals require at least a little vision to be tangible.

So whatever the project, its worth thinking about whether you need a 2 foot, 5 foot or 6 foot JFK. And who will that be?

Is it you? Are others waiting for you to stoke up the courage to play that role? Carpe diem!

2. Judge

Get Real

DeMarco & Lister coined the brilliant term "Management By Hyperbole" in Peopleware. Which I think hits too close to home in many software development shops. Everyone is going to be the next Google, right?

But sometimes you just need to get real, toss the crackpot ideas, and get on with it.

Play Games

.. and then get back to work please. You are not Matt Broderick.

Simulate It

The tooling in IT is getting to the stage where simulation is a realistic possibility in routine development. Take Oracle's Real Application Testing for example.

Run a Thought Experiment

The cheapest prototype of all.

Know Your Limits

I'm an optimist without illusions -- JFK

Postscript: Michael clued into the fact that this quote is wrong; it should be "I'm an idealist without illusions". Either I mis-typed or the book is wrong. The sentiment remains however, or we could now quote Obama with "I'm An Optimist, Not A Sap" ;-)

Weigh Ideas

3. Ask

Ask Dumb Questions

Ask Big Questions

Ask "What If?"

Ask "Animal, Vegetable or Mineral?"

Ask Just One More Question

4. Check

Prove Yourself Wrong

This immediately brings testing to mind. Make sure you put on the black hat (in both the hacker and deBono sense) when unit testing. Don't just write tests that prove things work, write tests for the failure modes and boundary conditions too.

But we know that already, and usually try for some level of rigorous testing.

Where we probably need to take this advice more to heart is at the front-end of the process: requirements, architecture and design.

Have you ever seen a new architecture, specification or design get approved with only cursory review? And perhaps reviewed by people who were not really motivated or perhaps even qualified to do a thorough job?

I have. And you know who will suffer for it: the brave team charged with implementing and supporting the system.

As we play the role of analyst, architect or designer we should make time to do a thorough critique. Try and break what you have conceived. Admittedly, that can be a hard thing to do. So invite a colleague to help you to try and break the design - an invitation not many would refuse! Do it, even if it's not in the project plan.

Inspect for Defects

The IT industry has adopted the quality mantra from manufacturing, but unfortunately our adoption is still highly selective.

The typical software testing method is akin to classical "quality by inspection".
Poke it, prod it, and raise a ticket if it breaks.

That's about as advanced as weaving textiles with steam-driven looms.

We are seeing more practices that emphasise "quality by design" however, and that's a good thing. For example, the test-first and test driven development movements try to help us to assure quality right at the first gate.

A big area for improvement seems to be a general focus on addressing systemmic factors in quality. Fix a bug? What about getting to the root cause of why that bug existed in the first place? Simple techniques like 5 whys can help get us out of a rut.

Have a Backup Plan

I learned this lesson good while doing IT support. Desperate users, desperate situations. You soon discover the importance of always knowing how to retrace your steps, and always having a contingency when you are about to do something that could smoke the users' data (or the hardware).

When the walls are buring, there's nothing like the confidence that comes from having backup plans well-ingrained as your standard practice. It can mean the difference between cool, calm success and landing a bloated fail whale.

It is the same idea I adopt when using source code control. I'm no fan of the school of thought that says only commit at the end of the job. Commits are my breadcrumbs that mark every step, and prevent bad changes from invalidating a whole coding session.

Do a Sanity Test

A good technique to apply to the next social-network-crowdsourcing-prosumer-2.0 business plan you see.

But equally valuable for more technical work. Prime candidates for a shrewd eye are server sizing, that convoluted custom framework design, and integration architecture.

Check your Arithmetic

Know the Risks

Most projects have some form of risk analysis process. Unfortunately, in my experience, most are cursory point-in-time assessments, with a half-hearted consideration of mitigations. You might do just as well getting a tarot reading.

Risks are like enemies ... hold them close, and know them well.

Thinking back over the recent projects I've managed, I realise that they all have some aspect of the plan that has been specifically arranged because of the understanding of the risks we faced. My three favourites:

• Scheduling specific proof-of-concept activities at the very front of the project to eliminate risks or put mitigations to the test.


• Including a performance test, whether the client required it or not. (every well-run performance test I have been involved with has identified at least one issue worth fixing. An issue that would otherwise have gone undetected prior to launch)


• Re-sequence project activities to address high risk activities as soon as possible.

Question Your Assumptions

5. Simplify

Keep It Simple, Stupid

Get agile.

Draw a Picture

Make a Mock-up

Rapid prototypes are good. Probably the single biggest reason why Visual Basic 3 rocked the world. Suddenly, anyone could mock-up a working GUI app in no time at all.

Now we're seeing some fantastic examples of this same capability for the new internetworked world. WaveMaker may be the new PowerBuilder for the web. And heroku is making Ruby on Rails a hosted no-brainer.

It's an interesting co-incidence that the original Visual Basic was derived from a programmable form system, code name Ruby.

Name the Beasts

Look at the Little Picture

Do the Math

Apply Ockam's Razor

6. Optimize

Minimize the Cost

Minimize the Time

Be Mr Spock

Make it Faster, better, Cheaper (but not all three)

Isn't the paradox that almost defines the software industry and the bane of the fixed price contract?

In fact, classical project management long held that projects are defined by the triple constraints of scope, cost, and schedule. What was unsaid of course is that a fourth variable - quality - was assumed to be constant.

Which is of course a great joke.

But the even bigger joke is that most organizations still tend to behave as if all four variables can be dictated at will.

Kent Beck delivered the seminal piece (in the software context) addressing this issue with his rather radical proposal for Optional Scope Contracts. This is eminently sensible and usually the best reflection of the real objectives on the ground ("we only have this much money to spend on the project; must deliver something within 3 months; and can't compromise on this level of quality. So what we can adjust along the way is: scope!")

This is a key topic covered in the excellent book, Extreme Programming Explained.

Know When Bigger is Better

We have Moore's Law to blame for making this a perennial issue in IT.

Our systems comprise hardware and software. In crude terms, we seek the optimal balance between throwing hardware at the problem and choosing the most productive development tools.

The most exciting trend we are seeing now of course is the mainstream adoption of dynamic scripting languages (python, ruby and php) in problem domains that were once the exclusive province of "proper" (compiled) languages.

Let Form Follow Function

I'll resist making a joke on designers with their iMacs, photoshop and flash.

In truth, the leading ideas in graphic/web/interaction design are spot on. Check out Robert Hoekman's Designing the Moment for example (highly recommended).

But let's talk about coding for a sec. Have you ever spent time polishing code, refactoring, or building a really cool bell or whistle ... before you've got even the most basic features working properly?

Test-driven development, and an agile planning method like scrum are good anti-dotes for this!

Pick the Best People

Fred Brooks highlighted the huge variation in individual programmer productivity in the classic The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition).

For one of the best treatments of how to work with this reality, Peopleware is the canonical read:

The final outcome of any effort is more a function of who does the work than of how the work is done.
..

• get the right people


• make them happy so they don't want to leave


• turn them loose

Make Small Improvements

7. Do

Learn by Doing

Real programmers know that what the docs say can be a universe away from how things actually work.

The only way you get to find this out (and hopefully make it all work) is to roll up your sleeves and do stuff.

Similarly, I believe that the only good architect is a practicing architect. If you are not taking every opportunity to work on your craft and keep current, you are ossifying.

The non-practicing architect is typically engaging in what I call "retrospective architecture": looking back at what worked in the past, codifying it as an "architecture" and promulgating it as a "standard". That's putting innovation in the deep freeze, and needs to be shunned!

Sharpen Your Axe

The best, smartest and most creative developers I know are really lazy. In a good way: work smart, not hard.

So while your "good" developer may spend 3 hours cutting and pasting SQL statements around to generate a test date set, the "lazy" developer has whipped up some sql/perl/awk script of Excel sheet that does the same in 5 minutes.

They love their tools, keep them good and sharp, and always on the lookout for a new one. Sharp tools make quick work of repetitive grunt work.

Correct It on the Way

It is salutary to learn that for the first US planetary mission to Venus, the rocket scientists actually argued about whether course corrections would be needed on the way. Thankfully, to avoid missing Venus by a million miles, trajectory correction maneuvers (TCM) became standard practice. It is of course how "smart" bombs work too.

And the same applies to software development. No matter how perfectly you have things planned out, there will always be some unpredictable external influences along the way. If you are not able to respond, then the outcome is no longer under your control.

This is a core belief of the agile movement, and enshrined in the agile manifesto.

Do Something

JFDI aka NIKE!

Doing software is a natural procrastinator's heaven on earth (or hell, depending on how firm those deadlines are).

As a well-practiced procrastinator and perfectionist myself, I've learned that "Do Something" is best advice ever.

The strange thing is ... it doesn't matter so much what you do (even to the extent of temporarily switching to another project or task). [this technique is discussed in another book I've read, but I forget the reference for now]

The important thing is you build some momentum and get moving. The more you do, the more opportunity you will have to explore alternatives and backtrack if necessary.

Don't Ignore Trends

Work on Your Average Performance

Software development is well known for its reliance on crash schedules and bucketloads of overtime to get things done.

For short periods of time, it may be a necessary and effective course of action to get you out of a desperate hole.

But over the course of a year, how does the productivity of such a shop compare with one that is operating at a more steady, controlled pace?

More than likely, the shop that just focuses on the bursts of peak performance, has a terrible long term average performance. Not only is average performance out of control because no-one is looking, but it's taking a pummeling from the burnout, turnover and very deep troughs in performance between peaks.

In software development, we should be much more concerned with improving our average performance than maximising our peak. "Situation nominal" as Houston would say.

Look Behind You

In software we conventionally think of using project postmortems and knowledge sharing initiatives to capitalise on our history. Everyone pays lip service to this idea, but in my experience few companies effectively practice it.

This is especially true in organisations driven by the quarterly numbers. It is easy to understand why: there's an inherent disincentive built into the system. The project is delivered and the value is captured, so a postmortem is just a cost that may not have a return for who knows how long (and you are not even sure who the beneficiary may be).

But even if the organisation doesn't see the value, as professionals you would expect we would take the initiative, right? Unfortunately not, because these project-level retrospectives tend not to make a great deal of intuitive sense. Have you ever delivered the same project twice? Isn't technology changing so rapidly that there's more value in learning the new, than tilling the soil of the old?

All these factors conspire to work against our ability to learn from the past at the organisational and individual level. Which is really bizarre when you think about it, given that being successful in the software industry almost demands a focus on continuous learning.

Personally, I think we have two powerful weapons at hand to battle this tendency:

1. The patterns movement, which allows us to understand generic concepts and package them for reuse. We may never do the same project twice, but we will use the same approaches many times.


2. The agile movement, which emphasises incremental effort, and an ever-increasing range of collaborative tools (all that "Web 2.0" stuff). Do your learning and sharing all the time. If you leave it to the end of the project, you already know when it will get done ... never!

Learn From Your Mistakes

Never! Or get the book and you might have a chance.