my recent reads..

Atomic Accidents: A History of Nuclear Meltdowns and Disasters; From the Ozark Mountains to Fukushima
Power Sources and Supplies: World Class Designs
Red Storm Rising
Locked On
Analog Circuits Cookbook
The Teeth Of The Tiger
Sharpe's Gold
Without Remorse
Practical Oscillator Handbook
Red Rabbit

Tuesday, June 02, 2009

Java Puzzlers. Be Afraid of the Dark.


Apparently, Joshua Block and Neal Gafter started the "Java Puzzlers" idea at Oracle Open World 2001. I wish I was there.

Subsequently, they've turned it into a book
, and a website.

If you program in Java, you must read this book. It covers the kind of traps in your code you wouldn't even imagine could be there.

I can guarantee you have written at least one of these issues into your code. And QA never caught it. And it is out in the wild RIGHT NOW. Arrgh!!

These days, I think (and hope) that most professional developers are relatively atuned to coding security issues and the rise of opinionated testing methodologies (Unit testing, TDD, BDD etc).

But this book is a real eye-opener to the range of issues that you wouldn't even think worthy of a unit test case.
public static boolean isOdd(int i) {
return i % 2 == 1;
}
Seems like a reasonable test for odd numbers? Except it is wrong a quarter of the time.

And that is just puzzle #1.

The book takes you through issues with strings, loops, classes, libraries, all the way to puzzle #95, where you learn why you shouldn't program like your brother.

Get a flavour from this presentation on Scribd from JavaOne 2007... Java Puzzlers me java puzzlers from javaone 2007

Sunday, May 31, 2009

Lessons in Re-branding: My Aquarium and SpeedDate's Agressive Acquisition Strategy

The My Aquarium Facebook application will soon become .. a dating app??? WTF!


At first I thought it must be a joke, or someone hacked the developer's facebook account.

But amazingly, it seems for real. SpeedDate have apparently been acquiring quite a number of Facebook applications, and My Aquarium is just one of the latest.

I don't know what on earth they are thinking though. Do they seriously expect to just buy users like this? Isn't there a fundamental demographic and motivational mismatch between users of a cute aquarium app and the dating crowd (except by pure coincidence)?

Rather than endearing people to SpeedDate, I'd expect the reaction is more like this:
Get the hell of my Facebook page. First you buy up and kill off one of my apps, then you expect me to use your totally unrelated app? Get real!


Kind of like if Microsoft came along and bought up Adobe then sent an email to all Photoshop users saying they must all upgrade to Excel. Can you imagine the consumer revolt that would cause?

I don't know anything about SpeedDate, but this behaviour just makes me want to see them fail big time. Not a good PR position to be in...

Tuesday, May 26, 2009

Hyperwords - fact-checking the web at a glance

Two things I find myself doing oh so frequently when on the net:

  1. Getting referred to wikipedia after googling

  2. Checking word spellings and definitions with one of the online dictionaries

With the Firefox add-on Hyperwords, both these activities just got incredibly easier. Just select text in your browser and you have instant access to the related wikipedia entry, check the dictionary and more (stock quote lookups etc).

Even better, the results pop-up in the browser so you are not left with a cascade of windows or tabs to get lost in.

It joins firebug as one of the top two "must-have" add-ons for my Firefox install!



Hat tip to blankanvas for putting me onto this..

Saturday, May 23, 2009

TDD and BDD is old school. Make the jump to HDD (Humour Driven Development)

SlashWeb just posted their list of the 25 Best Programmer Comics. I wonder ... seems like it could have been inspired by the stackoverflow question What’s your favorite "programmer" cartoon?.

xkcd's Proper User Policy apparently means Simon Says (sudo make me a sandwich) comes #1 in the SlashWeb list, versus the stackoverflow community voting xkcd's Little Bobby Tables to #1.

Conclusion? Either way, xkcd rocks.

But how's this for cool: xkcd's "antigravity with python" actually made it in as a patch to the python source code!

Sunday, May 17, 2009

The Software Architect's Professsion. Or Delusion?


That was a happy age, before the days of architects, before the days of builders. -- Seneca c.4BC-65AD

I hesitated as I reached for The Software Architect's Profession: An Introduction (Software Architecture Series) on the library shelf.

Did I really want to read another treatise on the role of the software architect? Hasn't the term architect been so abused as to now be worthless, even downright counter-productive? In this, I think I am one with Jeff Atwood and Joel Spolsky who discussed the questionable value of the title "Software Architect" on StackOverflow podcast #44.

Nevertheless, my hand followed through. I think I was persuaded by the unimposing nature of this concise little 100-page book.


I was pleasantly surprised; this is a great little book for stimulating some thinking around the role of an architect for the advanced reader. But I worry that it attempts to position itself as "An Introduction". A novice, unprepared to read the text critically, may easily be mislead by the book's definitive statements about what a software architect is and what they do.

Personally, I believe the book is fundamentally flawed in three important aspects:

1. Are we really in Crisis because we lack a Software Architecture Profession?


Firstly, the premise that today's Crisis in Software -
[the] parade of failures and half-failures that has grown over the years as a result of a world without an established profession of software architecture

- is wholly unsupported by any direct evidence. The authors' central argument is flawed by asserting an apparent causal relationship when in fact only coincidence had been established beyond doubt. A number of well-known software runaways and failures are mentioned, but I don't know of any where the original case studies attributed the failure primarily to the lack of "an established profession of software architecture". The authors get around this problem by redefining the conclusions and suggesting that all faults may eventually be explained by architecture. It seems to me self-serving and circular.

2. A Flawed Analogy with Building Construction


Second, the authors attempt to reinforce their argument with the proposition that the analogy with building architecture is self-evident. Buildings need architects. Software is like building. Therefore software needs architects. Hmmm. I am reminded of Bernard Rudofsky's book "The Prodigious Builders" which celebrates the history of vernacular architecture. That is, architecture without Architects (unfortunately a stunningly boring book for what ought to be a highly inspirational subject).

I particularly disagree with the authors' contention that software is not developed: it is built (with a sense of finality). The Google-inspired trend towards the perpetual beta is the most visible evidence to the contrary. The authors object to the notion that to develop implies to unfold, uncover, and make known. On the contrary, I find this a most apt description of what we do within the software profession: the youth and continuing innovation within the field does mean that software development and the architecture it requires is more akin to exploration, invention and discovery than to a formalised application of the tried and true.

Strike two.

3. Premature Specialisation


I began to renew my hope for the book as it explored the historical foundations of architecture. Michelangelo can truly lay claim to the title of Architect ("master builder"); his work on St Peter's Basilica epitomizes the unltimate balance between function, beauty, and structure,

Vitruvius is famous for asserting in his book De architectura circa 50BC that a structure must exhibit the three qualities of firmitas, utilitas, venustas — that is, it must be strong or durable, useful, and beautiful. A sense of proportion and harmony is represented in Leonardo Da Vinci's famous illustration of Vitruvian Man.

Such ideas begin to shape the conventional definition of an architect. A master who not only understands structure, utility, and beauty in order to successfully render a design into plans, but has the practical experience to supervise their realisation through construction.

At this point, I think the authors are getting onto the right track. However they stumble at the last post by then inexplicably turning this into an argument for a limited and specialised concept of a "Software Architecture Profession", where the architect only retains responsibility for venustas (design/beauty). Utilitas (function/utility) is the client's problem, and firmitas (form, materials, logistics) is the province of the engineers, scientists and code monkeys.

Time for the Renaissance?


The authors' call for the codification and ossification of a software architecture practice is I think at least 50 years premature.

What an "Architect" needs to be concerned with is still going through successive waves of tumultuous change. Up to the green-screen era, computer system architecture necessarily had a strong hardware component. Come the GUIs and increasing processing power in the 90s, it seemed a singular focus on "software architecture" as a technical discipline was a valid vocation. Now the waves of web-driven innovation and the emergence of the "Rich Internet Application" is again challenging our notions of what architecture entails. And again, the "real world" is encroaching the pure software realm with the rise of increasingly powerful and widely available mobile computing platforms (think iPhone, Android), and the revolution in pervasive automation (think Arduino).

I think the Java Posse were spot on when they discussed the growing need for cross-fertilisation and collaboration between designers and developers on podcast #247 - Design and Engineering. This is a time of divergence, not convergence, in the business of producing software & technology-based systems.

In truth, I question how appropriate both words are in the term "Software Architect":
  • Software - it is perhaps only in the last 10-20 years that it has been possible to construct computer software at the level of complexity that warrants the existence of an architect in the classical sense. And I suspect that in another 10 years it will seem ludicrous to suggest that you can be an Architect of only software ("just a turn-of-the-century fad"). Software is just one component of a "built environment" that encompasses everything from the information infrastructure and systems technology to the psychology, art and design of human interaction; ultimately leading to a desired collaboration between people and machines in the context of real-world objectives.
  • Architect - the common use of the term in the computing field has stripped this word of it's more noble dimensions. No longer is the architect "the person with the vision and skill to make dreams a reality". They are more likely to be the person in the corner who produces nothing but paper, leaves no fingerprints on the pages of history, and is generally ignored by those who are really making things happen.


I don't know what you should call the people who have the experience and ability to lead others to do amazing things with the information technology we have at our disposal.

I'm just pretty sure that "Software Architect" doesn't even come close to being adequate. And building a "profession" around a woefully inadequate definition is a one-way ticket to irrelevance and obscurity.

Saturday, April 11, 2009

Why would you open source a framework?

Interesting to listen to Scott Hanselman and Richard Campbell talking over the rationale for open sourcing Microsoft's ASP.net MVC framework on Hanselminutes show #175.

Part of the answer was a general desire to nudge Microsoft further towards embracing open source: "Why wouldn't you?". Which is admirable.

Partly it is a desire to open up the innovation envelope: Scott talked about his experience releasing TweetSandwich, and then seeing the unexpected derivative applications developed using the source as the base. Designing a framework is a daunting task. By definition, most of the framework's possible uses are not known ahead of time.

Take a listen to The Java Posse #241 which also came out this week, where they discuss the challenges of design as it applies to frameworks. One of the great concepts they talk about is how the best frameworks invariably have well designed escape hatches, to make sure you can overcome that typical problem of '..but the demo worked so well!'

Personally, I think having access to the source code of the framework is one of the most effective 'escape hatches' you can have.

Even if you never plan to fork or modify the framework, the ability to dive in and examine the source when things are not quite working as expected is really the difference between a framework you can work with, and a framework that will be discarded after a couple of projects. It is one of the great things about rails: often the documentation comes up short, but when you look at the api, the source code is but a click away!

Tweeting from your database with short urls

There's something cheekily enjoyable about getting all manner of 'non-human' things to tweet. I've heard of plants tweeting, but until I saw Lewis Cunningham's post announcing ORA_Tweet, I hadn't even thought of getting Oracle Database onto twitter.

Saturdays are good for little projects, so I thought I would add URL shortening service today;-)

Since twitter famously limits you to 140 characters, it is conventional to use a url-shortening service to include hyperlinks in your tweet. So my little play for today was to pair that idea up with ORA_Tweet.

There are a range of URL shortening services available including snipurl and tinyurl (here's a survey of services). I've been using is.gd for a while though, since it sports the simplest GET request 'api' you could imagine, making it great for ad-hoc programmatic use.

So I add an extra package called SHORT_URL which has just two functions:
  FUNCTION encode_url(
p_url IN VARCHAR2 )
RETURN VARCHAR2;

FUNCTION encode_text(
p_text IN VARCHAR2 )
RETURN VARCHAR2;
encode_url the main wrapper around the http://is.gd call to get a short url for the one you provide.

encode_text is a more convenient function that takes a block of text, and will replace all the urls it contains with the appropriate shortened versions.

Then there's just one change to the ORA_TWEET package body:
45c45
< url => 'status=' || SUBSTR( short_url.encode_text(p_string) ,1,140));
---
> url => 'status=' || SUBSTR(p_string,1,140));
Now you can go wild with URLs in your database tweets:
BEGIN
DBMS_OUTPUT.ENABLE;
IF ora_tweet.tweet
(
p_user => 'twitter_username',
p_pwd => 'twitter_password',
p_string => 'ora_tweet v1.1 is complete! Now with url shortening ... see http://database-geek.com/2009/03/15/ora_tweet-tweet-from-oracle-a-plsql-twitter-api/' )
THEN
dbms_output.put_line('Success!');
ELSE
dbms_output.put_line('Failure!');
END IF;
END;
Building on Lewis' original justification for building ORA_Tweet, you could for example include links to a report page or admin screen when your long-running process sends you its "I'm done" tweet.

That's if you need justification;-)

If you are interested, the source is up on my github account now: ORA_Tweet_With_Shorturls.zip

Friday, April 10, 2009

Idea #105: what name babby? (Dugg already pwned)

I just saw namemasher.com mentioned on programmable web.

It's a first step towards addressing one of humankind's biggest challenges: forget about running out of IP addresses, we're going to run out of usernames first!

What kind of handle do you think your children be able to get on Friendfeed? Under what name will your grandchildren be able to tweet? And do you think they stand a chance of getting the same nick across all their services?

There's a mad stampede for names going on, and any self-respecting parent (or prospective parent) who wants to bring up their children right ought to be out there buying up their progeny's place in cyberspace. Along with the tuition fund you need: website domain name, email account, twitter handle, skype, tumblr ... who knows which will survive, so get them all.

You never know: what if you kid gets famous, or even goes into politics? It wouldn't be very presidential if they tweeted as @spaceycasey123456.

Parents need something more than namemasher. In addition to the parents' names, it needs to munge in family and cultural background, existing baby name references like babynames.com (that's the one with the helpful definition of Espn), cross-check against existing accounts with something like namechk.com, and then go out and pre-register all the services for your unborn child. What an 18th birthday present that would make!

In short, the world needs wotnamebabby.com:

Thursday, April 09, 2009

Learning not to love Java

Back in 99, I spent about six months procrastinating over a MAJOR decision (or so it seemed at the time).

To stick with the Microsoft camp - where I'd built up substantial experience through ActiveX to DCOM to COM, IIS and Commerce Server 3.0 - or jump onto the Java bandwagon?

Ultimately, Java won out, and along with millions of others, we've made Java the enterprise platform of choice (arguably).

The interesting point upon reflection, is that there was never any doubt in my mind that Java was somehow 'better' and more 'pure', in an academic sense.

Almost ten years later, I shocked myself recently when returning to Java after spending a good part of the past two years infatuated with ruby and rails and other scripting languages.

Where before I saw classical elegance, now my first thoughts were this sux and the language is just getting in my way.

First the verbosity kills me:

WeLikeLongDescriptiveClassNames myReallyDescriptivelyNamedObject = WeLikeLongDescriptiveClassNamesFactory.getInstance(duh);

How many times do I have to tell the compiler what kind of object I am dealing with? Sure, I understand the benefits of the Factory pattern and the subtleties you can construct by virtue of the inheritance hierarchy. But most of the time, I just want the compiler to do what is most obvious, and don't make me write a novel for the most straight-forward tasks!

Then there's the language-promoted cultural phenomenon of class explosion. Need half a dozen value objects to represent the various information to be passed around in your domain? Sure! With unit tests all, I am sure. Most of the time, I'd now prefer to scream YAGNI!

But the true ephiphany was my run-in with primitives (again ... every year or so). Using JSP sessions as a perfect place to test this out. Take an int and stick it in the session:


int startingInt = 3;
session.setAttribute( "startingInt" , startingInt);

What type is in the session? An Integer object of course. Java has cleverly done a type conversion for you, from a primitive to an object. But woe betide the simple developer who assumes this must be a commutative operation:


int newInt = session.getAttribute( "startingInt" );
// Error!!

That's what I call a language actively getting in your way. Kind of like if you deposited $10 cash in your bank, but when you went to withdraw the money, the teller threw the shutters down and said you could only get your money back in government bonds.

So do I have a point or is this just a rant (and certainly one that is not as erudite as many have written)?

I guess I must be pretty slow to catch on to what others knew long ago. The true test comes down to being able to realise one's goals. For personal projects, I find I'm able to knock out complete (and impressive) applications in Rails, while I still have ten java-based ideas on the drawing board.

It comes down to whatever works. And it should.

Does that mean I hate Java? Not at all. I still get a bit of a thrill out of its clean lines and voluminous open source library support. Much like seeing an aging and long-since outclassed 512BB. 'Classic' is timeless.

No. I don't hate Java, but neither do I love it the way it used to demand. I look back on my naïve 1999-self in wonder. Building experience is key, but it is a mistake to tie this to a language. Computer languages are the tools of our trade, and the more you know, the better you will be. When people mention Haskell, Scala, Eiffel ... do you know what they are on about, or do you just nod intelligently and pray not to get caught out?

With more tools at our disposal, we are better able to tackle new challenges in the most appropriate way.

To get the job done (which is generally the point, right?).

Thursday, April 02, 2009

Crazy Stupid Security Policies #1

So we've all been there .. a super-secure data centre where they ban wireless access while in the server room.


I recently had an acquaintance pulled up for using wireless while in one such place. Twice. I won't name where, because this issue knows no bounds.

It is a nice and reasonable rule from the security policy astronauts' perspective, because usually those responsible for the policy are not also accountable for operations. I presume the main concern is bridging networks and (intentionally or not) providing backdoor access.

They conduct audits and spot-checks, and on the surface everything looks great. If all you are concerned about is the paperwork, your job is done.

But in practice, and from what I have observed over the years, reality is a very different thing.

Unless all you are doing is bolting a server into a rack or plugging the fibre cables in, it is hard to get the job done these days without external network access.

  • Maybe that is for research: consulting vendor guides or searching the knowledge bases (because no-one provides all the doc on disk anymore - it's on the web!)

  • Or often you need to test the system you are configuring or diagnosing, and that can only be done from "outside" (SSL termination at an external-facing load-balancer for instance).

  • Or, in this Web 2.0 world, you need to collaborate with colleagues to get the job done. Skype or IM to get hold of the expertise needed for the task at hand.

  • Worse still, you have an "escort" policy, but a simple request to get in or out of the data centre is meet with abuse, recalcitrance or outright hostility from the people who are meant to escort you (like it's not their job!)


All these factors increase the frustration of SEs the world over, in the face of data centre policies that treat IT as if it were like installing an air-conditioner.

Now what happens when the threshold of pain is pushed up and up like this? At some point, the immediate pain (can't get to metalink) exceeds the potential future pain (maybe I'll get caught using wifi).

And, dear data center security experts, happens next?

Human nature takes over. Before you know it, you have a feral group of wifi/bluetooth/3G connected people running around your DC.

The worst part is that you have pushed the behaviour underground, where it is truly uncontrolled. In being secretive, people are breaking the most significant security prohibitions of all: bridging networks. Sorry, you don't know how much it goes on. (personal aside: yes, I admit I have, in the past, used wifi in a non-wifi DC. But being the conscientious and security aware guy that I am, I was always quite anal about disconnecting from the DC network before getting on wifi. Not that anyone knew. And if they did, my reward would have been ... a punishment!!)

So what approach would an enlightened, modern data center manage take? I would sleep much better at night if I:

  • Had an open wifi usage policy to bring the practive into the light of day. Maybe tables running between your racks for wifi-connected laptops (bolt them to the desk if you like, with a CCTV overhead), while direct network/server access had to be done rack-side.

  • Educate on responsible wifi use. Make sure people understand the risks of bridging nets and make it clear its OK to be on wifi, but not ok to bridge.

  • And have a clear understanding with my internal staff that "escorting" is not an interruption to the work they are doing, it is a vital task that will be rewarded.



Fundamentally, this means I would need to reconsider how I defined my job as a data centre manager: I'm not a slave to a policy handed down from upon high, my job is to implement and enforce the best procedures possible that both enforce the policy goals, while providing excellent customer service. Achieving this may mean I need to think a little out of the box, even be a little creative and pragmatic.

Now I should be clear that in writing this I am not condoning anyone who breaks a clearly published policy not to use wifi in a data center. On the contrary, if you have to work in such a place, I'd say stick to the policy, and drive the escorts nuts as you go in, out, in, out and in again to get the job done. Demand a full time escort if you need it.

My real message is to anyone with authority over security policies and their application: recognise that a policy on paper is worth exactly the cost of the paper unless you have taken into consideration the human factors involved and done your best to ensure that your procedures and environment are optimally designed to encourage the very best behaviours, and not the ones you most want to avoid.

Anyone have data centre security horror stories to share? I would love to hear about them! Better yet, how you manage to get around the stupidity, yet stay "legal".

Friday, February 27, 2009

Like you scroll wheel? You want KatMouse!

I just heard about KatMouse on Security Now! #182.

I type fast, but love my mouse ( ... and there has been lot's of controversy about the numeric keyboard getting in the way too).

That means I'm a scroll-wheel addict. Till now, I just accepted the fact that it doesn't kick in all the times I would naturally expect it to work, like old applications, or in controls or windows that don;t have focus.

As soon as I heard Steve talk about KatMouse I knew I just had to get it. Two minutes later, it is installed and I'm loving it!

Best feature: when I have overlapping windows, I can scroll whichever window the mouse is over. Doesn't have to be in focus. Beautiful! This is how the scroll wheel should have worked all along;-)

Saturday, February 21, 2009

Wordling your twitter flock (and an aside on sniffing out bad proposals)

Twitter just keeps on attracting people who glob on new and interesting innovations; the latest I've seen is TwitterSheep which does a neat wordle of your followers.

Twitter has got to be the poster-child for YAGNI. By keeping things real minimalist, it not only creates the space but also the yearning for innovation - and we're seeing that in bucketloads (see also MrTweet; proud to say, Singapore compatriots!).

Wordles are simple but interesting semantic analysis toys. How useful can a word frequency count be? You may see a bit of noise, but I read them as a measure of obsession.

I've been using wordle.net for a while to do proposal analysis. It's become a "must-do" step before submission, as the insights can be invaluable. Simply paste in the full text of your proposal, and reflect on the resulting wordle. It's better than therapy!

What should be prominent of course are the concerns and issues that you know your client highly values. But what you will often see in your first drafts are a whole lot of words that are basically synonyms for ME! As in: my company, our product names, our partners, our technology etc etc.

All very well (it is a proposal after all), but just like a good conversationalist, shouldn't good proposals be at least equally weighted towards your listener's needs and desires? I think so, and wordles are quick and easy tools for sniffing out the boilerplate proposal that doesn't give a flying fig for the customer!

Still not convinced? Well, see the wordles that Billy Cripe did of Obama and McCain's convention speeches. Thank god you voted in Obama is all I can say!

What scares me is a seeing an obsession with "Country" "Americans", "fight", "war", "God" and even friggin' "nuclear", "attack" in the one wordle!!!!

So here's my twitter flock. Hmm, we're a pretty boring bunch!



Props to Daryl Tay for the TwitterSheep link.

Saturday, February 07, 2009

Best Practices in Web Form Design

I'm humming and hahing over some form designs at the moment. These days you have so many options, especially when you are getting smart with ajax and scriptaculous tricks.

Having options is always a double-edged sword. Yes, they allow you to do amazing things. But they provide a great recipe for procrastination.

.. just the situation where some thoughtful, concise guidance on leading practices from someone who knows their stuff can be a goldmine.

Thankfully I stumbled upon this great presentation on web form design by Luke Wroblewksi. It's a classic, and now I see he has a book out on the topic which instantly went on my "must read" list.

Sunday, February 01, 2009

SOA is dead! Was it ever alive?

There's nothing like a "financial correction" to make people wake up and start sniffing the crap that has been shoveled their way.

Waddya know? SOA is dead says Burton.

If I was paying for Burton services, I would be asking for a refund and an explanation as to why it took so long to identify the bleedin' obvious

The irony may have been a bit thick for anyone to realise that my story of Eric the Architect was little more than a lampoon on the generations of IT attempting to find a home within business (true friend - true story - honest!!!).

Guys - pragmatism rules. SOA never had anything to do with the bottom-line. Directly. And the indirect contributions lacked evidence and credibility.

In an excess of pragmatism, Miko Matsumura shut down the SOA center blog on the back of the Burton article. The new blog will be called the Whatever Center. Love the name, but will the dns changes ever propagate? I hope so, for Matsumura-san's integrity. Unlike WFTs transition to WTH, this is not a joke.

Personally, I'm with Justin Kestelyn:
The problems remain with us, whatever we choose to call the solution.

Tuesday, January 27, 2009

Tripping up on base64 encoding with webcache

Looking at an Oracle Portal installation recently, I wanted to call the WebCache invalidation API from SQL.

$ORACLE_HOME/webcache/examples/invalidation.sql sounds like a good idea, until:
begin
invalidate('hostname.com',9451,'password1','http://hostname.com/page')
end;
Error report:
ORA-06502: PL/SQL: numeric or value error
ORA-06512: at "SCOTT.BASE64_ENCODE", line 51
ORA-06512: at "SCOTT.BASE64_ENCODE", line 57
ORA-06512: at "SCOTT.INVALIDATE", line 38
ORA-06512: at line 2

Uh-oh. The base64_encode function included in the script is having trouble with the password. A quick look at the code...
create or replace function base64_encode
(
p_value in varchar2
)
return varchar2 is

BASE64_KEY constant varchar2(64) :=
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/';
l_buffer varchar2(32767);
l_len integer := trunc(length(p_value) / 3);

...

begin
for i in 0..l_len loop
l_buffer := l_buffer || encode_chunk(substr(p_value, i * 3 + 1, 3));
end loop;

return l_buffer;
end base64_encode;

Note l_len division by 3, then using it in the for loop. Yep, classic 0/1 base offset issue. Any password with a length of 3, 6, 9 etc characters breaks the code. Fixed with a -1:

for i in 0..l_len - 1 loop
l_buffer := l_buffer || encode_chunk(substr(p_value, i * 3 + 1, 3));
end loop;

But that raises more questions. What is this base64 encoding function doing here anyway?

At some point in time it might have been required, but Oracle Database has had the standard function utl_encode.base64_encode for at least a few versions. It encodes RAW so there's a bit of friggin around with types:
select utl_raw.cast_to_varchar2( utl_encode.base64_encode( utl_raw.cast_to_raw('password1') ) ) as B64 from dual;

B64
-------------
cGFzc3dvcmQx

I did note the comment in invalidation.sql to the effect that:
-- this old example is replaced by 2 files, collectively known as the
-- PL/SQL invalidation API.
--
-- the 2 files are
--
-- wxvutil.sql which does similar things as what invalidate.sql did
-- wxvappl.sql which is a wrapper of wxvutil.sql
--
-- both files are located in the same directory.


Well, these files are not in the same directory (they are actually in ../toolkit), and what's the excuse for shipping broken examples anyway, even if they are old and obsolete?

Monday, January 26, 2009

A case of severe backblogarrythmia

My backblog is giving me the irits. And what's worse I join the sad club of people who thought they cleverly invented a new word!
Backblog - Overflow of incidents you intend to write about on your weblog.

However, perhaps I can claim backblogarrythmia as my own:
.. a disruption in the regular flow of weblog posts, often caused by overwork, booze or a new romance. Primary symptom is the backblog, which in turn can exacerbate the problem through increased levels of performance stress. Treatment: JFPS*.

I used to work in a factory where I swear one of the leading hands had a personal goal of inventing a new word everyday. He'd casually call over something like "Hey, pass me the gumlicker would ya?" and take great pleasure in our confusion. He'd be mock-shocked of course at our "ignorance", and delight in explaining the word and how stupid we were for not knowing it. Crazy thing is, he'd always have a great and usually convoluted etymology.

He did this pretty much every day I knew him. Just an average working class bloke, with an average education. But I always secretly admired and envied his creativity with words, wondered where the inspiration came from, and whether he'd use the gift for anything more than keeping his workmates from boredom.

Me? The only words I invent tend to arrive as serendipitous typos. Yesterday's effort:
Museover - particularly thought-provoking and succinct tip or pop-up text that is displayed when you hover your mouse over an icon or word.

恭喜发财!

Update 27-Jan: It's official: museover and backblogarrythmia are in the (pseudo)dictionary

* just fucking post something

Sunday, December 07, 2008

Code formatting and line continuations with Javascript and CSS

I've had a question up on stackoverflow recently in an attempt to get a nice solution for adding a glyph with CSS to indicate a wrapped line (e.g. in a command line sample).

With a few good suggestions, I had a reasonable solution but it still required a bit of manual markup to prepare the example. Then I saw Pranav Prakash's post about syntaxhighlighter by Alex Gorbatchev.

With a few tweaks (see the stackoverflow question for the details), I'm now able to combine the best of both worlds: great formatting and syntax highlighting with syntaxhighlighter and also line continuation glyphs. Here's an example of the result:

public class HelloWorld {

public static void main (String[] args)

{

System.out.println("Hello World! But that's not all I have to say. This line is going to go on for a very long time and I'd like to see it wrapped in the display. Note that the line styling clearly indicates a continuation.");

}

}


NB: if you have javascript disabled, or reading this through a newsfeed (or if I hit a bug) you may not see the full effect. Here's a screenshot of what should be appearing above:


Now I just need to go back and update the code formatting in all my old posts. Basket!

Sunday, November 30, 2008

Oracle Shell Scripting


I remember seeing Jon Emmons' announcement on the Oracle News Aggregator and I've had it in my "wanted" list on bookjetty for ages.

This week I discovered Jon's Oracle Shell Scripting: Linux and UNIX Programming for Oracle (Oracle In-Focus series) at the NLB and have just enjoyed a good read of it.

I wish more DBAs had read this book. In fact it should be mandatory to get an OCP certification!

Let's face it, most Oracle installations are running on a *nix variant, and you can't be a DBA if you are not comfortable at both the SQL*Plus and shell prompt. To be a good and efficient DBA in my book, I want to see evidence of thinking smart, and repetitive task automation. When I see so-called DBAs who are happy to type the same "select .. from v$.." query every day of their working life, I doubt their brain is switched on, and I find it really, really scary to think they have the sys/system passwords!

They say tool usage is a sure sign of advanced intelligence in birds. And the same applies to all of us in IT. The three examples I look for at an Oracle Database installation are:
  • RMAN
  • Grid Control
  • Shell scripts


If none of these are present, then I tend to presume the real DBA has long left the building. Even if you are using third-party alternatives, do you continue to re-evaluate the Oracle capabilities with each new release?

Jon Emmons' book is of course more focused than this. It perfectly fills a niche, with an approachable, practical and comprehensive coverage of shell scripting from a DBA's perspective.

I can see the ideal audience for this book is people who are reasonable familiar with Oracle administration but are new to shell scripting. This book will rapidly teach you all you need to know on the scripting side (and let you skip alot of stuff you can learn later).

In other words, if you are a DBA who has just been assigned to manage a Unix-based system for the first time in your career: get this book. Forget all the (great) general Linux/Unix/shell scripting books for now. Don't even think the Oracle docs will teach you what you need to know. Oracle Shell Scripting: Linux and UNIX Programming for Oracle (Oracle In-Focus series) is what you need!

If you are coming the other way though - an experienced Linux admin being told that from Monday you also need to manage an Oracle database - I'd say this book probably doesn't have much to teach you. There's much more you'd need to learn about Oracle first (after telling your manager he's crazy), and there are really no scripting tricks in the book that you shouldn't already know. The main benefit you get would probably be a few pages in chapter 6 that cover the tricks of using sqlplus in a shell script - all in one place rather than having to tease it out of the Oracle docs (or see this related question on stackoverflow).

Originally posted on It's a Prata Life.

Monday, October 27, 2008

Synchronising two directory trees - tree-sync 2.4

tree-sync.pl has had a little update thanks to a contribution from Dave Stafford.
  • Added option 'diff' to view only changes

  • Added option 'ignore' to exclude certain extensions

  • Added option 'brief' to remove src/dest directories in report view to make cleaner output

In the 2.4 release I've also added the 'force' option to control read-only file handling.

tree-sync.pl is a bit of a geeky tool, suitable for cases where you want detailed control over a file-based copy/mirror/backup process from the command-line. There are many flashier solutions available these days, but if tree-sync scratches your itch - go for it!

tree-sync.pl continues to be available on CPAN, but from the 2.4 release I have moved the source to github. If you have changes to suggest or bugs to fix, please go ahead and join this project.

Tuesday, October 21, 2008

Rolling Project Euler on Ruby

I first heard about Project Euler last week on the stackoverflow podcast. Michael Pryor (fogcreek co-founder) makes a quick side reference in discussion with Joel Spolsky, Jeff Atwood and the rest of the SO team.

Well I checked it out last week, got hooked and spent most of the weekend earning my "level 1" badge;-)

Aside from dusting off some long-forgotten and arcane knowledge from my youth, I found it a fantastic opportunity to stretch my fundamental ruby chops. In fact, I'd recommend a few questions at Project Euler as a right-of-passage whenever you are learning a new programming language.

I've only been using ruby for a year or so, and in that time thought I had picked up a fair bit. But I was still able to amaze myself at just how many of the problems were knocked over in just 1 or 2 lines with a bit of duck punching and liberal use of blocks with Enumerables.

I'm late to the Project Euler craze, so you will already find many people posting hints for specific questions if you just google. I thought I'd share some of the "common code" I've been building up as I go through the questions.

I put a recent copy of the source up on github for anyone who is interested (MyMath.rb), but what follows a sampling of some of the more interesting pieces.

First thing you will note is that I have written all these "common" routines as extensions to some of the fundamental classes in the ruby library: Integer, Array, String.

It doesn't have to be this way, and for less trivial activities you might be right to be concerned about messing with the behaviour of the standard classes. Maybe I'm still enjoying my ruby honeymoon period, but I do get a thrill out of being able to write things like
1551.palindrome?
=> true


Integer Extensions


It's just so easy to code up simple calculation and introspection routines..

class Integer
# @see project euler #15,20,34
def factorial
(2..self).inject(1) { |prod, n| prod * n }
end

# sum of digits in the number, expressed as a decimal
# @see project euler #16, 20
def sum_digits
self.to_s.split('').inject(0) { |memo, c| memo + c.to_i }
end

# num of digits in the number, expressed as a decimal
# @see project euler #25
def num_digits
self.to_s.length
end

# tests if all the base10 digits in the number are odd
# @see project euler #35
def all_digits_odd?
self.to_s.split('').inject(0) { |memo, s| memo + ( s.to_i%2==0 ? 1 : 0 ) } == 0
end

# generates triangle number for this integer
# http://en.wikipedia.org/wiki/Triangle_number
# @see project euler #42
def triangle
self * ( self + 1 ) / 2
end
end


Prime numbers feature heavily on Project Euler, and I think calculating a prime series was my first lesson on why you can't brute-force everything;-) Enter the Sieve of Eratosthenes and related goodness..
class Integer 
# http://en.wikipedia.org/wiki/Prime_factor
# @see project euler #12
def prime_factors
primes = Array.new
d = 2
n = self
while n > 1
if n%d==0
primes << d
n/=d
else
d+=1
end
end
primes
end

# http://en.wikipedia.org/wiki/Divisor_function
# @see project euler #12
def divisor_count
primes = self.prime_factors
primes.uniq.inject(1) { |memo, p| memo * ( ( primes.find_all {|i| i == p} ).length + 1) }
end

#
# @see project euler #12, 21, 23
def divisors
d = Array.new
(1..self-1).each { |n| d << n if self % n == 0 }
d
end

# @see project euler #
def prime?
divisors.length == 1 # this is a brute force check
end

# prime series up to this limit, using Sieve of Eratosthenes method
# http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes
# @see project euler #7, 10, 35
def prime_series
t = self
limit = Math.sqrt(t)
a = (2..t).to_a
n = 2
while (n < limit) do
x = n*2
begin
a[x-2]=2
x+=n
end until (x > t )
begin
n+=1
end until ( a[n-2] != 2 )
end
a.uniq!
end

# @see project euler #23
def perfect?
self == divisors.sum
end

# @see project euler #23
def deficient?
self > divisors.sum
end

# @see project euler #23
def abundant?
self < divisors.sum
end
end


Next we visit the Collatz conjecture and an interesting routine to make numbers "speak english"..
class Integer     
# http://en.wikipedia.org/wiki/Collatz_conjecture
# @see project euler #14
def collatz_series
a = Array.new
a << n = self
while n > 1
if n % 2 == 0
n /= 2
else
n = 3*n + 1
end
a << n
end
a
end

# express integer as an english phrase
# @see project euler #17
def speak
case
when self <20
["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten",
"eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen" ][self]
when self > 19 && self < 100
a = ["twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"][self / 10 - 2]
r = self % 10
if r == 0
a
else
a + "-" + r.speak
end
when self > 99 && self < 1000
a = (self / 100).speak + " hundred"
r = self % 100
if r == 0
a
else
a + " and " + r.speak
end
when self > 999 && self < 10000
a = (self / 1000).speak + " thousand"
r = self % 1000
if r == 0
a
else
a + ( r <100 ? " and " : " " ) + r.speak
end
else
self
end
end
end


Calculating integer partitions is one of my favourites ... a nice, super-fast recursive algorithm. For problems like "how many ways to make $2 in change?"
class Integer 

# calculates integer partitions for given number using array of elements
# http://en.wikipedia.org/wiki/Integer_partition
# @see project euler #31
def integer_partitions(pArray, p=0)
if p==pArray.length-1
1
else
self >= 0 ? (self - pArray[p]).integer_partitions(pArray ,p) + self.integer_partitions(pArray,p+1) : 0
end
end
end


Finally, rotations and palindromes (base 2 or 10): methods that rely on some underlying String routines that come later...
class Integer 
# returns an array of all the base10 digit rotations of the number
# @see project euler #35
def rotations
self.to_s.rotations.collect { |s| s.to_i }
end

# @see project euler #4, 36, 91
def palindrome?(base = 10)
case base
when 2
sprintf("%0b",self).palindrome?
else
self.to_s.palindrome?
end
end
end


Array Manipulations


Array handling is particularly important. Start with some simple helpers, then move onto greatest common factor and a couple of least-common multiple implementations. My favourite here - lexicographic permutations.
class Array

# sum elements in the array
def sum
self.inject(0) { |sum, n| sum + n }
end

# sum of squares for elements in the array
# @see project euler #6
def sum_of_squares
self.inject(0) { |sos, n| sos + n**2 }
end

# @see project euler #17
def square_of_sum
( self.inject(0) { |sum, n| sum + n } ) ** 2
end

# index of the smallest item in the array
def index_of_smallest
value, index = self.first, 0
self.each_with_index {| obj, i | value, index = obj, i if obj<value }
index
end

# removes numbers from the array that are factors of other elements in the array
# @see project euler #5
def remove_factors
a=Array.new
self.each do | x |
a << x if 0 == ( self.inject(0) { | memo, y | memo + (x!=y && y%x==0 ? 1 : 0) } )
end
a
end

# http://utilitymill.com/edit/GCF_and_LCM_Calculator
# @see project euler #5
def GCF
t_val = self[0]
for cnt in 0...self.length-1
num1 = t_val
num2 = self[cnt+1]
num1,num2=num2,num1 if num1 < num2
while num1 - num2 > 0
num3 = num1 - num2
num1 = [num2,num3].max
num2 = [num2,num3].min
end
t_val = num1
end
t_val
end

# http://utilitymill.com/edit/GCF_and_LCM_Calculator
# @see project euler #5
def LCM
a=self.remove_factors
t_val = a[0]
for cnt in 0...a.length-1
num1 = t_val
num2 = a[cnt+1]
tmp = [num1,num2].GCF
t_val = tmp * num1/tmp * num2/tmp
end
t_val
end

# brute force method:
# http://www.cut-the-knot.org/Curriculum/Arithmetic/LCM.shtml
# @see project euler #5
def lcm2
a=self.remove_factors
c=a.dup
while c.uniq.length>1
index = c.index_of_smallest
c[index]+=a[index]
end
c.first
end

# returns the kth Lexicographical permutation of the elements in the array
# http://en.wikipedia.org/wiki/Permutation#Lexicographical_order_generation
# @see project euler #24
def lexicographic_permutation(k)
k -= 1
s = self.dup
n = s.length
n_less_1_factorial = (n - 1).factorial # compute (n - 1)!

(1..n-1).each do |j|
tempj = (k / n_less_1_factorial) % (n + 1 - j)

s[j-1..j+tempj-1]=s[j+tempj-1,1]+s[j-1..j+tempj-2] unless tempj==0
n_less_1_factorial = n_less_1_factorial / (n- j)
end
s
end

# returns ordered array of all the lexicographic permutations of the elements in the array
# http://en.wikipedia.org/wiki/Permutation#Lexicographical_order_generation
# @see project euler #24
def lexicographic_permutations
a=Array.new
(1..self.length.factorial).each { |i| a << self.lexicographic_permutation(i) }
a
end

end


String Helpers


Last but not least, some String methods that just make things so much easier...
class String

# sum of digits in the number
# @see project euler #16, 20
def sum_digits
self.split('').inject(0) { |memo, c| memo + c.to_i }
end

# product of digits in the number
# @see project euler #8
def product_digits
self.split('').inject(1) { |memo, c| memo * c.to_i }
end

#
# @see project euler #4, 36, 91
def palindrome?
self==self.reverse
end

# returns an array of all the character rotations of the string
# @see project euler #35
def rotations
s = self
rots = Array[s]
(1..s.length-1).each do |i|
s=s[1..s.length-1]+s[0,1]
rots << s
end
rots
end

end


With all the above in place - and with the aid of a few brain cells - some deceptively complicated questions (like "How many different ways can £2 be made using any number of coins?") are essentially one-liners:
require 'benchmark'
require 'MyMath'

Benchmark.bm do |r|
r.report {
answer = 200.integer_partitions([200,100,50,20,10,5,2,1])
}
end

Love it;-)