drop.io - cool & effective "media" sharing

..I hesitate to say "file sharing" because it gets funkier than that. Record somthing on your phone, mail to your drop.io drop, and it shows up as an enclosure on an rss feed = instant podcast!

The basics are solid - dead simple file sharing. First heard about this on net@nite; definitely worth checking out.

If - for some reason I can't think of right now - you wanted to send me a file, you can now drop it on my drop.io widget:

drop.io: simple private sharing

Using OVD Filtered Directories for LDAP Authentication

Oracle Virtual Directory (OVD) is one of the little-known or understood hero products in the Oracle suite of technology offerings [I put OEM Grid Control in the same class].

In this post I'm going to share a few thoughts on OVD, and present a few approaches for using OVD to present a restricted view of information from another directory, and how that can be used to limit access to applications that use an LDAP authentication mechanism.

When I was first learning about OVD back in early 2007, after the Oracle acquisition, it immediately grabbed my attention. Simple, easy to use, but so powerful - a swiss army knife for anyone working in the directory management space. Maybe that is the wrong analogy, because the greatness in OVD is that it doesn't try to boil the ocean - it just does one thing, but does it really well.

Simply put, it allows you to combine directory-related information from disparate sources (LDAP, AD, database etc) and present an LDAP-compliant view in real-time. And the virtual bit is real (if that makes any sense) - OVD doesn't store anything, unlike a meta-directory; it just passes through the directory requests according to the rules you setup.

The virtual nature makes OVD ideal in large enterprise situations, where control of directories may be distributed. Another group may have a directory that contains some information you want to use as part of your "directory view", but are not going to cede any ownership or agree to any changes anytime soon, like adding some new attributes. Wheel in OVD!

Likewise, affiliated companies may want to share directory information, but not handover control. And if the business relationship comes to and end, the directory owners want to know that they can turn off access in a moment, without needing worry about cached or replicated data left on the other side of the corporate divide. OVD to the rescue!

Case in Point: you need a subset of an existing directory

The inspiration for this post is a small challenge I was involved with recently. The company was deploying a new web application - just happened to be Oracle WebCenter Wiki, but the same applies to any application that supports LDAP authentication.

The only directory available contained a mix of users - some who should be able to access the wiki, and some who shouldn't. Configuring the wiki authentication mechanism at the directory is simple - but it is an all or nothing proposition. And of course, we couldn't go change anything the directory itself.

Sounded like a job for OVD!

Here's the basic setup - OVD is deployed between our application and the main directory, like a proxy server. We want to OVD to effectively "filter" requests from the application.

Configuration of OVD is done using the OVD Manager client, which connects to the administration port of the OVD server.

Approach #1: DN Matching

If you can define the distinction between included/excluded entries in terms of an entry's DN, then a very simple solution is to use the "DN Matching" property of the source adapter. This is fund in the Routing configuration.

Say for example, we only wanted our OVD directory to include entries that are in the ou=ITGroup or ou=Management containers. In this case, we would set the DN Matching property with a regular expression that will match on the DN string:

m/(.*)ou=(ITGroup|Management)(.*)/

Approach #2: ACL Restrictions

In practice, the DN may not provide enough information to distinguish items to include and exclude, and it is necessary to discriminate on the basis of an attribute, such as "departmentName". In this case, access control in the OVD engine may be configured to restrict the directory view based on a suitable filter.

Filter: departmentName=ITGroup

It is important to note with this approach that ACLs can be applied to all LDAP operations, except bind.

As a result, the directory view we have created with OVD appears to only contain the filtered subset of information: we cannot browse, serach, get or modify anything else. However, if you present a fully-qualified DN and associated password, it will authenticate and bind any entry that exists in the source database.

On spec, that seems to blow the whole approach out of the water. That's what I was thinking too, until Mark Wilcox helpfully nudged me along with a neat insight...

If our requirement is to use OVD to restrict the set of users that can authenticate via an application, we only need to consider the application authentication mechanism. In most cases, the process is similar to the one illustrated below. The user enters an id or username, which is used by the application to lookup the user's DN, which is then used to bind along with the user-supplied password. If the application can't find the DN in the first place, then no bind is possible.

Of course, the acknowledged security "exposure" in this case is that a user can bypass the application and directly bind via OVD if they know their DN. However this is probably a false risk, because the user would have always had a similar capability with the source directory itself (assuming that the source directory and OVD are equally accessible to the user over the network).

Bottom line? Using ACLs to restrict the search effectively controls the set of users that the application can authenticate.

In Practice: Oracle WebCenter Wiki

Oracle WebCenter Wiki is thye example application, but you can think of it as any old J2EE application packaged as an EAR that supports Java SSO. By default it will use JAZN XML file-based storage for user accounts.

When deployed in OC4J, the security provider used for the Wiki application can be easily changed via the Enterprise Manager web interface.


Switching to OVD as the authentication source is a simple matter of selecting the Oracle Security Provider for 3rd Party LDAP Server and configuring it with some simple directory details:


Almost done. There are two assumptions that I think the wiki makes about the directory. Just need to make sure these are setup:

• wiki users must be members of the group called "users"
• administrators are members of the group called "oc4j-administrators"

Now you are done. The wiki authentication is being performed against the limited set of users visible through OVD.

Caveat: selecting the Oracle Security Provider for 3rd Party LDAP Server causes the site to revert to basic authentication (i.e. popup a username/password dialog instead of using a web form). Not a big deal, but you will find the "logout" feature in the wiki now fails because it assumes form-based custom authentication. So once you have people lured into your wiki, they are trapped! ;-)

Wrapping Up

I've covered two techniques for restricting the set of information published via OVD: DN Mapping, and ACL Filters.

There are other approaches that I've not covered here. For one, Java or Python plug-ins (a.k.a. mappings) can achieve the same result, as well as more complex behaviours of course.

These techniques allow OVD to be used to restrict overall access control for applications that use LDAP authentication mechanisms.

Once again, hat-tip to Mark Wilcox for his help when I thought I'd hit a wall while researching this topic!

Desktop Keyboards Stuck in Design Limbo

Keyboards are terrific examples of how bad design can get stuck in a rut, unable to overcome inertia. Everyone says qwerty is a bad idea, yet I couldn't imagine using anything else now since it's use is so ingrained.

But another aspect of keyboard design that has me really grumpy is the whole numeric keypad appendage on desktop keyboards. It is a holdover from the days when users were "data entry clerks". But we are stuck with it (Microsoft only have two keyboard models without it, while ALL Logitech models are saddled with this cancer Postscript: Dean Chu corrected me here; Logitech's diNovo models don't have the numeric keyboard).

This started to really annoy me of late, because I've been switching between a laptop during the day, and a desktop at night.

Working with a desktop keyboard again was feeling really strange and difficult, but after some reflection I realised the problem. My right-hand is used to shifting all the time between jkl; and the mouse. On the laptop, this is a subtle and effortless gesture. On the desktop, its like playing table tennis.

The fact that virtually all laptop designs eschew the separate numeric keypad should be proof that it is evolutionary dead wood.

So this is my grumpy call for all keyboard manufacturers to wake up their snoozing product managers/designers and actually innovate for once. Fix this ergonomic nightmare! At least give us some choice ... integrate it with function keys like laptops; use separate USB numeric keypads; even consider sticking it on the left-hand side of the keyboard.

And for all those poor souls who really are still data entry clerks, I'm sure there will be no-brand outfits from China knocking out standard 102-key designs for years to come.

Is it just me? Did I get up on the wrong side of the bed today, or do others feel this way too?

Postscript 9-Feb-2009 ... hat tip to mqt for linking Trevor Blackwell's solution: just chop it off! If you gotta take a bandsaw to a product to make it fit-for-use, then something's wrong, right?!!

Appcelerator - bringing down the wall between RIA and SOA?

I wonder if the Appcelerator guys have finally cracked the RIA and SOA dichotomy? I first came across them on Coté's RIA Weekly #008 RedMonk Radio podcast.

I've presented my views before on what I see as the three megatrends in IT:

• Web 2.0 - or more generally, RIA
• SOA
• Grid - although today I'd probably update this to be "Cloud Computing"

But the distinction between RIA and SOA has always felt forced; unrelated working metaphors, owing more to the historical segregation of the communities addressing each than strict architectural principles.

While industry lines have been drawn very clearly around these two domains (take OpenSOA v. OpenAJAX for example), there have been many attempts to nibble away at the distinction. AJAX toolkits like SAJAX and SWATO strive to make calling back-end resources more convenient. And frameworks like ADF approach the problem from the other end, by "hiding" AJAX rendering in their server-side, SOA-aware paradigm.

So what has Appcelerator got to do with this?

From what I understand so far, the key is that they have unified the event/messaging model both within the browser and the "SOA-sphere", and done so in a very elegant way. There are three parts to their solution:

• Web Expression Language
• RIA Widget Framework
• RIA Message Broker

All components in an Appcelerator application communicate via simple lightweight messages using the RIA Message Broker. On the server-side, Appcelerator provides a set of SOA Integration Points that enable service creation in Java, PHP, Ruby, .NET, Python and Perl.

On the client, the Web Expression Language message-enables HTML elements. The RIA Widget Framework is a Javascript-based API that enables you to create new widgets and wraps existing third-party widgets like scriptaculous.

The end result is a very clean, lightweight and seemless development approach. HTML attributes define behaviour: what messages to send, and what to do when a message is received. And the real magic: when you send a message, you do not know or care if it handled by another HTML element on the same page, or a SOAP Web Service somewhere out on the net.

Here's how straight-forward it gets. An example of an input button messaging a calendar widget to show itself..

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:app="http://www.appcelerator.org">
...
<app:calendar title="Pick a Date" on="l:show.calendar then execute" inputId="mydate">
</app:calendar>
<input type="text" id="mydate" value="click me" on="focus then l:show.calendar"/>
...

Or an input button sending a message...

<input type="button" value="submit" on="click then r:login.request"/>

.. that is handled by a Java service:

import org.appcelerator.annotation.Service;
import org.appcelerator.messaging.Message;

public class LoginService
{
@Service(request = "login.request", response = "login.response")
protected void processLogin (Message request, Message response)
{
// get request data
String username = request.getData().getString("username");
String password = request.getData().getString("password");

User user = userDAO.login(username,password);

// format response
if (user != null)
{
response.getData().put("success",true);
response.getData().put("user",user);
return;
}
response.getData().put("success",false);
}
}

Appcelerator looks like one to definitely watch closely and investigate further..

Proven Enhancement

Never has Rails Trac been more entertaining than this. The change history is just too funny;-)

Quick NLB Catalogue Lookups

Postscript: I've given up all ambitions of hacking NLB now that I have discovered Bookjetty - a bookshelf site with library integration. Check it out!

OK, so this post won't mean diddly unless you live in Singapore, or have a local library using the CARLweb system.

But say you are browsing one of the book retailers on the web - like amazon - and you want to know if the book is available in your local library?

Here's a little trick to help you find that out in one-click (not patented;-). It is inspired by some posts over at The Box Factory.

Below is a link that you save to your bookmarks (Firefox)/favorites (IE). Once saved, when you are on a book page at amazon etc, just select the bookmark/favorite and it will pop-up a window to Singapore's National Library Board catalogue system and try and find the book in the library for you.

NLB Book Lookup <-- Bookmark/Favorite this!

Simple steps: How to Use It

1. Right-click on the link above, and select Bookmark This Link.. (Firefox) or Add to Favorites.. (IE)


2. Go thru the steps to save the link. you may get a warning about "javascript:" links not being recognised. Ignore/proceed .. trust me!


3. Check you Bookmarks/Favourites menu - should now find NLB Book Lookup in the list.


4. Navigate over to a book retailer and find a book (try the Ruby Cookbook (O'Reilly) at Amazon for example)


5. While viewing the book page, select NLB Book Lookup from your Bookmarks/Favorites menu


6. A new window will open to display the NLB Library Catalogue record (if there is one)

Grumpy Stuff - But Looking Forward

The NLB is still using CARLweb, which is an ancient product from TLC.

Once of the really nasty features of this system is that it uses session tracking in the URL query string of your request. Not only does this prevent deep-linking (so you can't easily email a catalogue link to a friend for example), it makes the catalogue un-indexable by web search engines like google.

It also means that the Bookmark/Favorite link I gave you above has to go in the front-door, and creates a new session each request. So unfortunately that means that if you use this a few times in quick succession, you may get a maximum sessions exceeded error. Sorry, for now you'll just have to close the windows and wait a while for sessions to expire.

Yes, CARLweb sux. It truely belongs to an earlier era of computing!

The good news: I've heard whispers that the NLB have a project running to upgrade their catalogue search facilities. Hopefully we'll soon have a much better service to use. My top wishes:

• Support for deep-linking. No user/session-related information in URLs.


• A nice robots.txt that allows the search engines to fully index the catalogue


• Flexible RSS feeds - as well as the usual "latest acquisitions" etc, a flexible and published approach to parameterising the feed e.g. by library location, by subject and any other search criteria. Or another way to express this would be "get any search result as RSS"


• Getting ambitious now ... a published Web Service/REST API


• A mashup developer's guide - encourage people in Singapore to exploit the NLB catalogue! A competition even?

Explaining Regular Expressions

Unless you get to work with regular expressions everyday, the details can easily blur. You remember enough to know when a regex could be useful, but not enough to write it without reaching for a reference manual.

There are a few tools out there to help you write and understand regular expressions, including some IDEs that can provide assistance (like Komodo).

Sometimes you just want a quick explanation of a regular expression you might have seen in some code. The YAPE::Regex::Explain perl module is one tool that helps you do just that. Turning the module into a simple command-line tool is a simple one-liner:

#!/usr/bin/perl -w
print YAPE::Regex::Explain->new($ARGV[0])->explain;

I've also turned it into a simple CGI utility for those times when my regex memory fails me. You can use it here: regexplainr (sorry, you may find that site offline but here's the source code).

In a previous post, On Parsing CSV and other Delimited/Quoted Formats, I used the following regular expression to parse a whitespace-delimited string:

"([^"]+?)"\s?|([^\s]+)\s?|\s

Regexplainr produces the following commentary:

The regular expression:

(?-imsx:"([^"]+?)"\s?|([^\s]+)\s?|\s)

matches as follows:

NODE EXPLANATION
----------------------------------------------------------------------
(?-imsx: group, but do not capture (case-sensitive)
(with ^ and $ matching normally) (with . not
matching \n) (matching whitespace and #
normally):
----------------------------------------------------------------------
" '"'

----------------------------------------------------------------------
( group and capture to \1:
----------------------------------------------------------------------
[^"]+? any character except: '"' (1 or more
times (matching the least amount
possible))
----------------------------------------------------------------------
) end of \1
----------------------------------------------------------------------
" '"'
----------------------------------------------------------------------
\s? whitespace (\n, \r, \t, \f, and " ")
(optional (matching the most amount
possible))
----------------------------------------------------------------------
| OR
----------------------------------------------------------------------
( group and capture to \2:
----------------------------------------------------------------------
[^\s]+ any character except: whitespace (\n,
\r, \t, \f, and " ") (1 or more times
(matching the most amount possible))
----------------------------------------------------------------------
) end of \2
----------------------------------------------------------------------
\s? whitespace (\n, \r, \t, \f, and " ")
(optional (matching the most amount
possible))
----------------------------------------------------------------------
| OR
----------------------------------------------------------------------
\s whitespace (\n, \r, \t, \f, and " ")
----------------------------------------------------------------------
) end of grouping
----------------------------------------------------------------------

(Permalink to this regexplanation)

Book tip: (thanks to Tony) O'Reilly's Mastering Regular Expressions. Available on google books, and also from Amazon.

Popflying the Oracle Community

Back last year I got all excited about Yahoo Pipes ( Web 2.0 Wake-up Call for BPEL? ) and its graphical approach to building mashups. In fact I still use a Pipes mashup as my mega-aggregated Oracle newsfeed.

Now I've discovered a new toy - Microsoft Popfly - and finally a reason to install Silverlight! I remember popfly getting mentioned in Bex Huff's roundup of mashup tools last year, but at the time it was still in restricted beta.

Popfly's still in beta, but now open to one and all. Put aside your biases for a minute and go play ... if you are like me, it will blow your mind. It may not be the final word in mashup tools, but it sure opens your mind to what should be possible.

Here's a little mashup I created 5 minutes up the learning curve. It's a cute display of the latest photos posted to the Oracle Community site:


Postscript: yes, you do need Silverlight installed for this to work of course. Silverlight is available for "all major browsers running on the Mac OS or Windows". If you are not prompted automatically for the Silverlight install, visit the Silverlight site.

Supporting Singapore's bid to host the Youth Olympic Games 2010

I just discovered the online campaign to support Singapore’s bid for the Youth Olympic Games 2010. It would be truely great to see the event held here, and Signapore's certainly got the location, facilities, environment and culture to do a great job. And to be fair, better chances than ever hosting the full Olympic Games! So I've registered my support, and you'll see the logo flying on my blog from now on;-) If you want to support the bid on your blog, just register here. Postscript 21-Feb-2008: Singapore Wins!

On Parsing CSV and other Delimited/Quoted Formats

Parsing delimited text that may have quoted elements is a perennial requirement. Quick-and-dirty parses can be achieved with regular expressions, but for more flexible and encapsulated parsing I've been checking out the opencsv java library. Hat tip to Jakub Pawlowski for highlighting the library on his blog

A Regular Expression Approach
Just recently I released and blogged about a JDeveloper Filter Add-in, and it contains a class called ExecShell [API, source] which needs to know how to break a command line into its component arguments. The command line is of course space-delimited, but may use quotes to group an argument with embedded spaces (so a simple split on spaces won't do).

The salient code below uses the REGEX to chop theCmdLine String into theCmdArray Vector of arguments:

Vector<String> theCmdArray = new Vector<String>(0);
String REGEX = "\"([^\"]+?)\"\\s?|([^\\s]+)\\s?|\\s";
Pattern p = Pattern.compile(REGEX);
Matcher m = p.matcher(theCmdLine);
while (m.find())
{
theCmdArray.add( m.group().trim() );
}

The regular expression bears a little explaining, and is inspired by this example. Here's how it breaks down:

\"([^\"]+?)\"\\s?	Matches a group within double-quotes. Group is a lazy match on one or more characters except double-quote. Optionally followed by some whitespace
|([^\\s]+)\\s?	or Matches a group delimited by whitespace, optionally followed by some whitespace
|\\s	Discards a pure whitespace match

In this case, we are using whitespace as the delimiter (appropriate for command lines). The regex can be adapted for other delimiters by replacing \\s with the delimiter. For example, to handle a comma-separated format:

String REGEX = "\"([^\"]+?)\",?|([^,]+),?|,";

Using OpenCSV
The same space-delimited parsing requirement can be met with a couple of lines and the opencsv library:

CSVReader reader = new CSVReader(new StringReader(theCmdArray), ' ');
String[] s = reader.readNext();

Simple, yet currently not so robust. Since we define the delimiter to be a single space (over-ridding the default comma), other whitespace characters (like a tab) will not be recognised. Further, repeated spaces will not be coalesced, but will each be treated as the delimiter for a new element.

Internally, CSVReader parses the input character-by-character and so adapting to handle repeated delimiters as one would be reasonably straight-forward.

The Right Way To Do Wrong - a good read for security buffs

The Right Way To Do Wrong - An Exposé of Successful Criminals is a very old book, published in 1906. I was intrigued since it was written by Harry Houdini, and I hadn't realised he was also an author. Houdini's motive for writing the book is to warn off the righteous by educating them in all forms of devious frauds and scams, and to cause those less well intentioned to give pause before taking up a life of crime. Reading the book over 100 years after publication, I am amazed - but perhaps on reflection not surprised - that Houdini manages to describe in great detail just about every Internet-related scam in existence (allowing of course for a transposition of technology)! When he talks of Begging Letter Swindles, think Nigerian Letter or "419" Fraud. For Tricks of Bunco Men, see Advance Fee Scheme. The ease in which Impersonation/Identity Fraud was practiced in a pre-IT age... and just about every other gambit you can find on the FBI Common Fraud Schemes site. If you are into IT Security, I think you'd enjoy reading this and mulling over the relevance to your day-to-day work. It is salutary to realise there is nothing new in the Evil that Men do, just new ways of doing it! I listened to The Right Way To Do Wrong in audio from LibriVox. It is also available in print from Amazon.

iTunes and DRM - Destroy Real Music

Ignatius Low gets nostalgic over music in it's physical form in an article for The Straits Times today (I still want a CD I can hold - Feb 3, 2008 - article is only available to online archive subscribers. ST hasn't quite caught up with the NYT yet!).

ABOUT a month ago, I made my first purchase from the Apple iTunes Store. It was a tedious and deliberate process, given that the iTunes Store hasn't been launched in Singapore yet, but it had to be done..

He goes on to question what would be lost if the world does move wholesale to digital downloads .. nostaligic memories of browsing for music with friends in the local record shop .. the personal stories forever tied to each CD or LP on your shelf.

Travelling and the Real Music Store Experience
Thankfully, we haven't totally lost the music store experience yet, although it is dominated by the major chains like HMV. I hope we never lose it. It would deprive me of one of the most enjoyable travel activities - whenever in a new city, I always try to make time for a few hours at a local music store to browse and buy. It is a great way to get a lock on the local music culture and discover some amazing artists.

DRM - The Worst Idea Ever Foistered on the Music Industry
There is a more insidious side to the whole digital download approach to purchasing music. Its called DRM - Digital Rights Management (or Destroy Real Music if you prefer). Still in use for most music on iTunes, DRM restricts your use to iTunes and iPod, and you only get 5 chances to register with another computer (like if you re-install or upgrade). Apple of course are not exactly forthright in telling you all this, prefering to regale you with all the benefits of digital downloads.

To make a LOTR analogy, its as if DRM is the poison holding Théodred under the spell of the wicked Wormtongue (read RIAA).

When I look at my record collection, I see the albums I inherited from my parents - even grandparents - and remember the thrill of exploring and learning to appreciate all this old and unusual music. I was the new generation rediscovering the musical gems of my elders, and I think a critical step towards musical maturity. It is how I found a place in my collection for Les Paul & Mary Ford, Duke Ellington and Fats Domino along side Iron Maiden, Madonna and Regurgitator.

If DRM and digital downloads become the way of the future, then this is one formative experience I will be unable to bequeath to any grandchildren I may be lucky enough to have.

The Dawn of a DRM-free Golden Age?
Thankfully, DRM seems to be on the way out - at least for music if not movies for the time being. And anyone who really cares about music should add their strength to kill it good and proper as soon as possible. EMI were perhaps the first major label to signal the trend, and break away from Wormtongue's spell.

The EMI move significantly bolstered Amazon's move on itunes with their DRM-free MP3 store (although in a perverse turn of logic, I can order physical CDs from Amazon yet because I live in Singapore I still cannot purchase their mp3 download versions. Go figure!)

Amazon's move was likely no insignificant factor in Apple now moving to dump DRM in their new iTunes Plus service (probably more correctly called iTunes Minus). Aple are still extremely coy in telling their users what's really going on (try searching for DRM on the iTunes site). I think it is scandalous how little information is provided to iTunes users about the DRM restrictions that are still applied to the majority of songs you can purchase on iTunes.

To put it simply: beware!

• When you purchase music on iTunes, it is still DRM-locked, unless you are clearly given the iTunes Plus purchase option.
• If you buy DRM-locked music on iTunes which is later made available unlocked in iTunes Plus, you can upgrade but you will have to pay for the privilege of getting you music purchases switched over (currently 40c per song or 30% of the album price)

Personally, I plan to stick to purchasing CDs, especially with so many available today at a nice price (like S$11.95 and up for recent chart albums). When amazon finally make the mp3 store available to me, I'll probably use that for the odd purchase (especially for individual tracks).

But iTunes? Sorry Apple. I love your iPods, but your support for DRM leaves such a bad taste that even now that you are reforming I will run a mile before willingly purchasing from your store.

JDeveloper Filter Add-in Now on Sourceforge

I introduced the JDeveloper IDE External Filter Add-in project last weekend. Happy to announce that the project's registration on sourceforge is complete, so full source and downloads are now available from the sourceforge jdevfilter project page.

Tux gets his break with OS/2

Staying with my OS/2 reminiscence for a moment, there's a trove of old gems up on YouTube (including the famous quote from Bill "we believe OS/2 is the platform for the 90's").

While IBM ads in the US defied the limits of corny, in Japan it is all so kawaii.

Of course! The correct way to market an operating system is with a TV star/model (Yamaguchi Tomoko 山口智子), a bear and a penguin.

At least we know what Tux was up to before he got his big break with Linux in 96...taking gigs playing "DOS" and living it up in Ropongi!

OS/2 Still Kicking..!?

I was pretty amazed to see IBM responding as recently as 16-Jan-2008 to the requests by the OS/2 online community to open source OS/2.

I've admitted before to being an OS/2 fanatic for a few years, but seriously, I think we should let it die and just savour the memories!

One of the great things about the OS/2 community way-back-when is that it was a total geek-out (memories of the OS/2 Devcon in 1992 - still have the bag tag!). Linux has I think long taken that mantle.



It's a bit of a laugh to find some of my OS/2 detritus still kicking around the Internet, like:

• My article in EDM/2: REXX - The Developer's Best Friend
• A few tools still up in the os2 collection at hobbes:
• xeol for text file format conversion
• A Workplace Shell Command Shell Starter
• INISED - a utility to manipulate Windows-style profile (INI) files
• varc - Visual archive utility front-end, requires VREXX
• dsize - Display file/directory space usage on drives (REXX)
• shlong - Lists non-FAT names on an HPFS drive (REXX)
• .. mostly developed using REXX or my favourite compiler of the time: Borland C++ for OS/2!

I had fun with a couple of other projects that never made it to release (in both cases beaten to the post by the big boys. But dang it, no M&A windfalls in those days!):

• LNDW - Lotus Notes Doorway. I wrote a web server in C++ that provided gateway access to Lotus Notes database under OS/2. Tricky adaptation of the Notes C API to C++ and getting it to work with BC++. Then came Domino...
• OS2IP - a project with a long-lost collegue Bryan Ryan (if you are out there Bryan - drop me a line!). At the time, there was no free TCP/IP stack for OS/2 Warp, so we set about writing one. Got as far as an NDIS packet driver, and a user mode frame interface before IBM finally got their act together and made us obsolete! Nevertheless, good fun hacking around with the Driver Dev Kit in assembler, and bridging to a C++ user-mode API. I don't think it was until then that I really understood Comer in detail. When you are working at that level, it is amazing how exciting exchanging just a few bytes of data in a chat application can be!

JDeveloper Filter Add-in

I hinted at a project to develop a simple filter add-in in my last post on Code Generation with JDeveloper. So that turned into my latest weekend project;-)

I'm registering the project on sourceforge, but in the meantime you can get the full source kit here, or if you prefer just the add-in jar here (which just needs to be dropped into your ${jdev.home}/jdev/extensions folder). [30-Jan-2008: the project is now available on sourceforge]

What does it do?
It is a simple idea. Allow you to invoke an external program to filter any text you have selected in the JDeveloper Editor. Of course, it is up to you to define what filter means - limited by your imagination only. And by externalizing the filter process, it means you can use the tool or scripting language of your choice to implement and make on-the-fly changes to your filter.

To be precise, the filter operates like operating system pipes:

• Selected text is sent to the standard-input of the filter process.
• The standard-output of the filter process is written back to replace the selected text.

Configuring the Filter
Configuration is done via the JDeveloper Tools | Preferences menu, and is very simple at present. It only supports a single filter process definition. As you can see in the screenshot, you just need to enter a valid command line to invoke the filter. A nice enhancement would be the support for multiple filter definitions.


This is where the flexibility comes in. As an example, I've provided a rot13 encoder written in ruby [samples/myrot13filter.rb in the source kit].

puts $stdin.readlines.to_s.tr("A-Za-z", "N-ZA-Mn-za-m")

Ruby, Perl, php, vbscript/WSH, bash ... use whatever you prefer. Rather than needing to develop and deploy an add-in in Java, you can just script the filter.

Applying the filter is simply a matter of selecting some text in the IDE, and choose the Custom Text Filter item from the right-click pop-up menu

The Development View
The truth is that I developed this add-in as more an investigation of the Extensions SDK, and the filter idea appeared to be unexplored territory.

As Brian Duff writes, there are some major improvements in the support for JDeveloper Extensions in 11g. Since it is still in preview, I decided to stick with Extension SDK 10.1.3 however.

There are four classes of significance. since this was an educational exercise, I spent a bit of time to make sure I really understood how it all worked, and tried to reflect that with liberal comments in the source.

• CodePaneAddin - implements the pop-up menu hook and applies the filter
• ExecShell - wraps the invocation of the filter process iwth stdin/out handling. It is independent of the extension API and may be of interest in its own right
• ConfigPanelData - handles preferences panel configuration
• ConfigData - bean to wrap the configuration data

The source kit comes with a JDeveloper project and also ant build file. If you want to build and deploy the add-in to JDeveloper, simplest is to first check the environment-specific settings in build.properties and then simply run

C:\MyDocs\MyDev2\JDevFilter>ant deploy
Buildfile: build.xml

init:

prepare:

compile:

jar:

deploy:
[copy] Copying 1 file to C:\oracle\jdevstudio10131\jdev\extensions

BUILD SUCCESSFUL
Total time: 0 seconds

Interested? Head on over to sourceforge to download the kit.

Code Generation with JDeveloper

I've written about Jack Herrington's Code Generation in Action before; it is a great book for stimulating your thinking on how you go about producing code.

JDeveloper has evolved into one of the strongest IDEs when it comes to code generation. It's not explicitly mentioned in the book however. That's not a criticism of the book, but it does beg a review of how JDeveloper stacks up against the concepts covered.

Integrating code generation into your IDE

Writing a code generator can be difficult-but convincing your fellow engineers to use [it] can be even more difficult. The key .. is to make it as simple as possible to use.

IDE integration is one approach to address this problem, and the common techniques supported are external tool invocation and filtering.

JDeveloper only directly supports external tool invocation (see the Tools menu). For instance, you can call simple Ruby scripts like the following to perform file-level processing (this example simply comments each line of the file that is currently open in the editor pane).

fh = File.open( ARGV[0] )
text = fh.read()
fh.close
text.gsub!( /^/, "// " )
File.open( ARGV[0], "w" ).write( text )

NB: in the 11.1 preview, it seems the "Availability" conditions are still a little broken and it is not possible to make commands available only when a file (or certain type of file) is open.

Filters are not directly supported by JDeveloper, however implementing a filter is a fairly straight-forward project using the JDeveloper Extension API.

JDeveloper Extension API

Herrington does not directly address the topic of IDE APIs, which is understandable given the scope of his book. However if you are looking to implement complex and very specific code generation approaches you will probably want to look into this topic further.

JDeveloper comes with a comprehensive Java-based extension API which was significantly revamped in (I think) 10.1.3 to bring it into line with the standardisation efforts of JSR-198.

I must say documentation of the extension development process is a little thin. And it is only in 11.1 that JDeveloper has explicit project support for extension development (although I find it a little broken and mysterious in the preview edition). The best approach at present seems to be good ol' copy-and-modify the extensive set of samples provided by Oracle at the Extensions Exchange. There is a growing number of 3rd party open source extensions that are even available through the JDeveloper Check for updates menu.

Still, no filter add-in is available, and you are of course stuck with developing in Java. It of course puts me in mind of a little project: a simple filter add-in that then allows you to invoke a filter process developed in the language of your choice such as perl, ruby or python. Maybe I'll be able to follow-up this post with an example;-) PS: yes I did; you can read about it here.

JDeveloper's Generation Credentials

I've been hedging around the fact that JDeveloper is very significantly geared towards code generation itself. It is one of the main drawcards for why you would use it over your favourite text editor.

In the back cover of the book, Herrington includes a very illustrative Generator Tree which provides a taxonomy of generators, with of course references back to what is covered in the book. I thought I'd experiement with mapping JDeveloper into the taxonomy. I doubt if I'll manage to cover all of JDeveloper's features (or get it all right), but here goes:

Class	Platform	Technology	Herrington	JDeveloper
Database Access	MS	ASP	ASP Generator	n/a
		ASP.NET	ASP.NET Generator	n/a
	Java	EJB	EJB Generator	EJB Entities from Tables
		JDBC	JDBC Generator	ADF Business Components
		JDBC	n/a	Toplink O-R Mapping
	PHP	PHP	PHP Generator	n/a
	Perl	DBI	Perl DBI Generator	n/a
User Interface	Java	JSP	JSP Generator	JSP/struts/JSF wizards
		Swing	Swing Generator	Swing wizards; ADF Swing wizards
	MS	MFC	MFC Generator	n/a
Documentation	SQL	SQL	SQL Documentation Generator	n/a
	Java	Java	JavaDoc	JavaDoc
Unit Tests	C	Tests	Augmented C Unit Test Generator	n/a
		Ordered Tests	Ordered Test Generator	n/a
	Test Data	n/a	Test data generator	n/a
	Test Robots	n/a	Test robot generator	n/a
File Formats	Java	CSV	CSV file Reading Java Generator	File Adapter
	Java	Data Adapter	Data Adapter Generator	(arguably) File Adapter; Toplink
	Java	Binary	Binary File Reader Generator	n/a
Web Services	Java	XML-RPC	XML-RPC Generator	n/a?
	Java	SOAP	SOAP Generator	Java Web Service from WSDL
		PL/SQL	n/a	PL/SQL Web Service
		JMS	n/a	JMS Web Service
		Various	n/a	Technology Adapters
Business Logic	Java	Math	Equation Generator	n/a
	Java	Reports	Report Logic and Interface Generator	n/a
External Libraries*	C++	DLL	DLL Wrapper Generator	n/a
	Scripting Languages	C++	External Language Wrapper Generator	n/a
Configuration*	Firewall	any	Firewall Configuration Generator	n/a
	Reference Data*	any	Lookup Function Generator	n/a
		any	Macro Lookup Table Generator	n/a

*I've reclassified these items from the original table in the book.
NB: I generated the table above using an OpenOffice Calc spreadsheet [generator-taxonomy.ods]. Good example of dog-fooding perhaps;-) I've always found Excel/Calc to be really useful tools for rapidly generating scripts, SQL or HTML fragments like this.

With so much capability in JDeveloper, you might expect instant adoption. But there are of course two classic issues still to be addressed:

1. There's quite a steep learning curve until you get comfortable with what features are available, and when they are appropriate to use. The rapid enhancement of JDeveloper and all the technology over the past few years means that your average dev shop probably doesn't have the luxury of many gurus who can coach the team either
   
2. It seems to be an inherent trait of developers (arguably a good one), that generators and wizards are not fully trusted and accepted until you've had the chance to dig down under the covers. That's simply a lot of work for all of JDeveloper's features, and there isn't a great deal of information available to help your short-cut the process.

The natural conclusion is that to optimise adoption, I'd exhort the JDev team (and the community of users) to put as much effort as possible into producing resources for JDeveloper's code generation (wizards etc) and extension SDK. Importantly, to put the stress on explaining the how and why these things work (not just step-by-step tutorials).

Postscript:

• Shay Shmeltzer just posted a useful summary of How Do I Start Learning JDeveloper and ADF?


• Olaf Heimburger has started a new blog series demonstrating ADF In Action
• Eduardo Rodrigues has a great post on his favourite JDeveloper extensions with some good recommendations of what you should install or at least have a closer look at.

2007 In Review

Picking up the idea to present "my blog-year in review", by taking just the first complete sentence each month's post (remembering not to make this the first post for Jan;-)

January	Apache Axis2 is the core engine for Web services.
February	I've been working a bit with Oracle's OCCI (the C++ API for Oracle Database), and stateless connection pools in particular.
March	Recently wanted to track down the details of the process that had a specific port open.
April	"Where's the alert log?" .. usually the first thing you want to know when looking at a new database.
May	Say you have a connection to a remote Oracle Database server and want to get the ORACLE_HOME setting.
June	A little while ago I got to dust off my C++ skills for a project that was to use Oracle Database (via OCCI) and also Websphere MQ
July	This week its my pleasure to host Log Buffer #54, the Carnival of the Vanities for DBAs.
August	I've just spent the past few days at Oracle OpenWorld Asia Pacific in Shanghai.
September	I've blogged previously about playing with the reCAPTCHA service in Perl.
October	I had the privilege of listening to Kevin Walsh's "30/30" presentation last week.
November	Well, today is my first as a member of the Oracle Alumni
December	Laurent Schneider just asked what LinkedIn is good for?.

Hmm, very geek. What's not listed is the post that generated the swiftest, strongest reaction: Ballmer Peak and the Programmers' Paradox, which probably goes to show that frat humour is the surest way to internet fame;-)

During the year I also started blogging quick reviews of all the books I read over at PrataLife. So as a variation on the meme, here's my Book-Year In Review:

July	The Extraordinary Adventures of Arsène Lupin, Gentleman-Burglar (audiobook)
August	The Ambler Warning
September	A Different Point of View (audiobook)
October	The Warden (audiobook)
November	Code Generation In Action
December	Quarter Share (audiobook)

The number of audiobooks is a bit misleading, as I'm sure I actually read at least as many on paper (although Neal Stephenson is slowing me down a bit at the moment!). And ye gads, only one tech book! But this is to be expected: with fiction its easy to read and do a quick blog entry, whereas technical non-fiction is generally not suited to blogging an instantaneous reaction. They'll tend to stay on my desk/shelf for a probationary period;-)

Backup your WebCenter Wiki. Yikes!

Previously I wrote about installing Oracle WebCenter Wiki, and also had my 2c worth.

One of the surprises is that it uses file storage by default. Database storage seems to be a work-in-progress (Toplink, hibernate and HSQLDB are all involved somehow), but not a documented option at this point.

Now if you are going to run WebCenter Wiki for real, that means as a dilligent system admin you want to be pretty sure you really understand how the file storage is working so you can have good backup and recovery procedures in place.

The documentation on Backing Up and Restoring Wiki Content sounds straight-forward. Just make a backup of the OC4J_HOME/applications/application_name/owc_wiki/pages folder.

But BEWARE!

Gotcha #1: Don't Forget Your Toothbrush!
A little poking around, and it is clear that this is NOT all you need to backup. I've noted content modifications in at least the following locations also:

OC4J_HOME/applications/application_name/owc_wiki/attachments [any uploaded attachments]
OC4J_HOME/applications/application_name/owc_wiki/templates [added/changed page templates]

The jzwiki backup/restore instructions also tell you to backup WEB-INF/classes/yawikiDB.script (WebCenter wiki is based on jzwiki)

So to be sure you haven't left anything behind, I'd suggest you probably want to be backing up the entire OC4J_HOME/applications/application_name/owc_wiki directory structure no matter what the doc says, and practice some restores!

Gotcha #2: Beware of EAR Re-deployments!
If you've worked with OracleAS/OC4J deployment before, and also noticed that the wiki content is stored in the application directory structure, I hope it rang an alarm!

Yes, if you touch or replace the owc_wiki.ear, then say goodbye to your wiki content and pray your backup is good and not too old! OC4J will redeploy the application and give you a nice fresh install, deleting all your wiki content and customisations;-)

I have always had a sneaking suspicion that there can be other circumstances in which OracleAS/OC4J may choose to redeploy an EAR. I can't really substantiate this, but I have had situations in production where EARs are redeployed for no immediately apparent reason.

This is of course very hairy for any non-trivial production deployment. The obvious first step would be to configure wiki content storage to some other location. This isn't documented however, and I'm not sure if its even possible with the product as it is.

So for now, it seems just backup or rsync deep and often, and enjoy the view as you walk the edge of the abyss;-)

WebCenter Wiki Future?
So far, there's no real news on what is planned for WebCenter Wiki in 11g. If you've read my posts, it will come as no surprise that my view is the 10g wiki is far from production-ready.

Which makes the question of what is in store for 11g so interesting. Maybe a significant revamp of the 10g (jzwiki) wiki? Maybe an internally developed replacement based on Content DB or UCM?

Or maybe something completely different? A collegue pointed me to Jive Clearspace today .. "discussions, blogs, files, instant messaging, and wiki documents under one unified umbrella" .. and since Oracle already have Jive Forums integrated in WebCenter it raises interesting possibilities!

Ringing in the New Year with a timely reminder from xkcd

I've refrained from reposting xkcd for a while, but this is just too good! So where did you leave your key-ring last night, huh?


Happy New Year!