Why SOX won't keep your feet dry

I had the privilege of listening to Kevin Walsh's "30/30" presentation last week. Kevin is Oracle's CTO for Asia Pacific and has a long and illustrious career which makes him uniquely qualified to deliver this talk - the theme of which is to look 30 years back in IT history to get a perspective on how far we have come, and then attempt to look 30 years into the future (with all due disclaimers). Take a 30 year view, and things get distinctly freaky - nanotech, quantum computing and so on.

It's actually the second time I've heard the presentation, but this time I happened to have just finished Tim Flannery's The Weather Makers (my review is here).

Now, Kevin's theme was technology so I can't fault his focus, but I got to thinking that within the 30-year timeframe one of the biggest challenges facing the world is climate change, and this must surely have an impact on what we know today as "business software".

Now, before you say "pish!" and dismiss this as an eco-nut rant, consider three facts of importance:

1. that climate change is upon us - we have reached the tipping point [by 2005 we had refereed proof of global warming]


2. that the change is for the worse - the only disagreement being about the specific effects our planet will need to endure


3. that while human activity is not the only contributor, we know that it is a factor, and more importantly we know ways in which our impact can be lessened (Bex Huff just had a great post about the greenest cars; you may be surprised)

So the question is really no longer should we act but will we?

Green-thinking Enterprise Software

Of course in 2007, the prevailing mentality is that enterprise software hasn't got alot to do with the environment. It's the power companies, OPEC and so on that matter, right? Take a look at the OpenWorld session agenda, and it's hard to avoid the conclusion that the environment is ... well, off the agenda. On the Oracle corporate website you will find an environmental statement much along the lines of any "non-energy related" company. We turn our lights off at night etc etc.

I couldn't find an environmental statement on the SAP site, which is strange since I thought the Europeans were renowned for their green intelligence. SAP Environmental Compliance is covered under GRC, but looking further at the SAP Mining or Oil&Gas industry marketing sites and the promise is simply to help you lower costs, increase profitability, and improve competitiveness.

I reckon this is something that needs to change.

Fundamental to the health of the planet (and future of the human race) is the efficiency and carbon load firstly of the power grid, and secondly transportation. While governments and consumers have a significant role, on the whole it is business that is either the direct power user, or are the mediator of our demand. And what do businesses around the world rely on to operate efficiently and profitably? Why, that wonderful class of enterprisey software of course - which for the majority of companies means SAP or Oracle, the behemoths of the industry.

Enterprise software has three extremely important potentials when it comes to the envirnmental agenda:

• It provides unprecendented access to manage and report on end-to-end operations - supply chain, manaufacturing, distribution, retail and so on


• It facilitates the reduction of data to financal metrics


• Good enterprise software done well has the ability to promulgate best practice across industry and across the globe.

In other words, why shouldn't the CEO's dashboard be displaying the company's envirnomental performance alongside traditional measures, and most importantly (if you want to see behavioural changes), show the cost impact to the company as a result? Completed your energy audit and started to implement recommendations? Great - just watch the benefits start flowing to the bottom line.

Or to put it another way - enterprise software has a critical role to play in ensuring that our ability to execute on the environmental agenda can keep up with our save-the-planet rhetoric.

The Tao of SOX and what it has to do with the Environment

Unfortunately SOX won't keep your feet dry as water levels rise, nor may you need them to keep warm.

But it is instructive how SOX has swept the world of enterprise software, and arguably stems from just a couple of cases of massive corporate fraud in the US and one in Europe. Within a few years, the legislative impact has spread around the globe, and all enterprise software companies worth their salt have a SOX or Basell II position.

It has spawned a new "industry" called Governance, Risk and Compliance (or variations thereof). While this can and does incorporate safety, health and environmental aspects of corporate governance, the SOX pendulum is still swinging and the focus remains squarely fixed on financial aspects only. Checkout the Oracle GRC materials and you'll find (with some effort) a scant 1 or 2 indefinite references to the environment. SAP's Environmental Compliance material does seem to read better. RedMonk's James Governor has been giving some good coverage of GRC offerings, including the environmental aspect.

Just like SOX has had a major impact on our economic environment in such a short time, I think we need a similar revolution concerning our actual environment.

I hate to say it, but in the US it may just be a matter of a few more years of class 4-5 hurricanes to trigger a change. A government facing continued multi-billion relief budgets and a population clamouring for "something to be done", is a government ready to act.

So we're not just talking about a salve for the conscience. The insurance industry will be heard also, and I'm sure that carries quite a bit of weight on Capitol Hill. From the Wall Street journal 7 May 2003:

With all the talk of potential shareholder lawsuits against industrial emitters of greenhouse gases, the second largest re-insurance firm, Swiss Re, has announced that it is considering denying coverage, starting with directors' and officers' liability policies, to companies it decides aren't doing enough to reduce their output of greenhouse gases.

Why China could be the turning point

Across the globe, there is nothing conceptual about the environmental challenges facing China, and so much stems from the thirst for power to drive its growth.

As usual, any stats about China are boggling: 120-odd new coal-fired power stations in the current decade, and more to come each year (despite the fact that they also project to commission 2 nuclear power stations a year for the next few decades).

With typical PRC pragmatism, there are signs that China knows it needs to act and has already adopted the goals of reducing its national energy intensity by 4 percent each year and obtaining 15 percent of the country's energy from renewable sources by 2020. Expect ever-tougher environmental laws being executed in China.

So while the China market for Enterprise Software booms, I think it is also fair to expect that within a few years software that just offers to implement efficient and profitable business processes will not cut it. The products that sell will be those that also help companies to meet their environmental obligations.

This trend could produce some interesting outcomes. The expectation will of course be that the western products (notably Oracle and SAP) must surely incorporate all the environmental best practice. If this is found not to be the case (hmmm?), it would open an opportunity for one of the Chinese ERP vendors to develop and sell a product properly configured for the local market. After which, we find the vendor turn around and sell their environmentally-best-practice enterprise software back into the US and European markets, a la Lenovo?

Why the Enterprise Software vendors need to wake up

Well, to stay in business and make alot of money seems a pretty good reason.

We could be talking as little as 5 years for the worm to turn - a few more hurricanes hitting the US, and China rapidly awakening to the fact that it's ecomonic success is so heavily dependent upon the environment.

At that point, you will not be the most important enterprise software company if you do not have adequate environmental, health and safety capabilities baked into your offering. You will not be welcome, useful or relevant.

So where is the Oracle/SAP Environmental Management System, Carbon Ledger or the Energy Audit and Reporting System? The problem is that most of the enterprise software has not been designed with an environmental agenda in mind.

So what's an enterprisey company to do?

• Launch your environmental mission statement at your OpenWorld/Sapphire/etc. The mission? To be the most important enterprise software partner helping business meet their environmental commitments across the globe.


• Short-term, beef up the environmental focus of your Governance, Risk & Compliance-type offering. It should be the easiest quick-fix.


• Commit say 20% of R&D to baking environmental capabilities into core business services (application components)

Bonus marks may be awarded if you want to point out that for all it's good work, the Gates Foundation will be pretty irrelevant should humanity face the worst outcomes predicted by those modelling climate change. So Larry could help fix the planet, give Bill a bloody nose ... and make money doing it;-) Perfect!

Through the looking glass

If my rant hasn't been kookie enough already, I'll go out on a limb and humbly make some 30-year predictions ....

• The world survived to 2037. So far. But the jury is still out as to whether we have acted just-in-time or just-too-late.


• By 2037, value investors on the stock markets pay as much note of CPS (carbon-per-share) as EPS (earnings-per-share).


• Some years earlier, Oracle donated an installation of Database 20q (quantum edition) to globally consolidate carbon emission, energy distribution and planetary metrics, and drive the climate change models used for the daily briefings to the leaders of all UN member states.


• Oracle released Fusion Applications 20e (environmental edition), which as the marketing material states is "used by 9 out of 10 Global Fortune 1000 companies to meet or exceed the environment-neutral operating standards as required by law while maximising benefits for their shareholders, employees and community".

SOA/Enterprise 2.0. The Rebel Alliance takes on Deathstar-2.0

What is the driving force for *2.0 in the enterprise? Deathstar-2.0 jokes aside, Billy (Fusion ECM) and Jake (AppsLab) have been nutting this out. Billy's in the "its all about the data" camp, and Jake's taking the "power to the people" stance. Bex has chipped in with a view that "its about knowledge".

You may think its a cop-out, but I think its a case of you're all right. Depends on what perspective you want to take.

Yes, its all about me, but in the same way that driving around town is all about me. Until a semi wants to cut into my lane. On the road we have established ways of collaborating to keep things moving nicely (signals, lane markers, the odd toot of a horn). It's all about me in the sense that I want to safely get to where I'm going. But the roads department has a broader objective.

I'd suggest that the real difference with Web-2.0 in the wild is that there really is no broader objective to speak of. I blog. I like. That's good enough for me.

But in an enterprise, there's always an over-riding agenda that's bigger than any individual: profitability, customer satisfaction, market share etc. If that's not front of mind, you deserve a pink slip .. give you lots of time for facebook!

With that in mind, I'd like to share a visualisation of the enterprise I call the "SOA Sphere". Not perfect, by tries to cram a number of key concepts together:

• Information (data/knowledge) is the core asset of an organisation


• Business Services/processes/applications collaborate with users and act upon the knowledge base


• Security is an all-encompassing capability; a key enabler of innovation


• People live, breath and work together in the environment that these services create. No distinction really between employees, customers, partners or the great unwashed

Proof! Oracle Development have a funny bone

Recurity Labs GmbH have a very interesting post on their investigation of the new password algorithms in Oracle Database 11g.

I did a double-take when I saw the set of hashing algorithm identifier values (used as a parameter to the ztv2ghashs hashing function)..

0xf00d means: Use MD4 
0xdead means: Use SHA1 
0xbeaf means: Use MD5 

Ah! It's an oldie but a goodie. Brings back schoolboy memories of getting your LCD calculator to spell out well-known petroleum companies.

And good to see that even in the depths of Oracle Development there's a willingness to do something a little special, for no reason other than because they can.

Pipes - Web 2.0 Wake-up Call for BPEL?

Recently I've spent some time playing with Yahoo Pipes. I was tipped off by comments to Chris Muir's blog on Drowning in Oracle Blogs Aggregators.

So I've had some fun with Pipes, using it to aggregate all my Oracle news for example. Pipes is still beta and has a fairly modest objective of giving end-users a way to craft their own data feeds on the net, but I think there are some exciting implications.

My Little Experiment

I did a simple experiment with a Perl CGI that issues a "quote of the day" in JSON format, consumed it in Pipes, and then subscribed to an RSS version of the result in FeedReader.

I could have used an existing data source supported by Pipes (RSS, JSON, CSV or XML) but I wanted to experiement with formats. Hence the little bit of coding ..

use strict;
use CGI;
use JSON;

print header();

my $obj = {
  title   => 'Quote of the Day',
  quote   => 'the only dependency we have on the proof of concept is whether it works or not'
};

print objToJson($obj);
1;

(btw, that's one of my favourite quotes from a collegue during a particularly long and weary evening;-)

The Pipes editor makes it a piece of cake to get tricky ... like add a flickr lookup that will roll in a suitable photo to go along with the quote:

All fairly straight-forward, but what grabbed my attention is how Pipes provides an extremely intuitive graphical editor that makes it simple for an end-user to transform and consume data in a hosted model that would conventionally require server-side programming to arrange. It even has an integrated "debugger" that lets you see the output at any stage of the pipe you select.

Pipes - a Web 2.0 Wake-up Call for BPEL?

There are a few things that Pipes can't do (yet), like handle SOAP Web Services, incorporate human workflow or asynchronous events, secure published pipes, and of course reach data sources or systems that are not accessible from the Internet.

But I'm thinking it wouldn't take too much enhancement along these lines and we'd see Pipes rapidly encroaching the BPEL domain (albeit without the standards support). This would be a perfect example of the Web 2.0 challenge to Enterprise IT.

• Pipes - the funky UI that lets anyone create and publish their "flow"


• BPEL - a robust standards-based engine that runs "processes" that have been designed, tested and deployed by experts

If these two extremes don't come crashing together soon, I think we are not trying hard enough!

I'm not sure if Yahoo could do this without a rip'n'replace of the Pipes back-end, and rethink the whole hosting model. More likely that we see someone re-think our approach to BPEL, and the set of use cases it assumes.

I don't hear or see much movement in this direction from the existing BPEL players however. That includes the commercial offerings like Oracle BPEL Process Manager and also the free/open source kind like ActiveBPEL and OpenBPEL.

I suspect we need to challenge a few key assumptions implicit in current BPEL offerings. That processes are (a) largely a conversation between systems, and (b) built by a developer (or at least a skilled business analyst).

While that can and will remain true for core business processes, I think Pipes is beginning to demonstrate that there is a bigger picture we have yet to address.

Put it this way. I'd like to take a robust BPEL engine with...

• BPEL standards support of course


• Web Services (WSDL + SOAP) and WSIF adapter model


• Sophisticated human workflow and asynchronous task support


• Something I can deploy behind the firewall if needed (to access enterprise goodies)


• A decent security model

Then add some of the Pipes goodness:

• Oh-so-Web-2.0 hosted design tools - made for end-users.


• "Publish" instead of "Deploy". Implies automatically maintained (and accessible) registry of published pipes/processes.


• Support for RSS, JSON, CSV and plain XML "web sources"


• Inheritance by chaining or cloning existing pipes


• Smart output rendering. Need RSS, JSON or SOAP? If it makes sense, ask for it and the correct format should be delivered. Transformation taken care of by the infrastructure, not the process definition.

As an enterprise user, this means I'd expect to be able to launch my company's BPELPipes site and see the pipes published by IT. If I need something a bit different, I just go ahead and create it (or clone an existing service). And once I've done creating my pipe, I can use it immediately as maybe an RSS feed or somthing I can embed in my portal. And share it with others if I like..

Perhaps here lies the future of Enterprise 2.0 Workflow (WikiWorkflow?!), finally bringing SOA to the people rather than having it left stuck somewhere in the middle tier.

BPEL + Pipes. Workflow for all?

(Post moved to here..)

Adding reCAPTCHA to Oracle SSO

I've blogged previously about playing with the reCAPTCHA service in Perl. Seriously cool! Not because it's foolproof - it isn't - but the side-effect of helping to digitize old documents and books is a truely great idea.

I'm starting to see reCAPTCHA more often now. Bex Huff put it in his comment form, and blogged about it (though I can't find his posting anymore. Update: link from Bex, thanks!). But I haven't seen it used with Oracle SSO yet ... sounds like an interesting weekend project!

So I had a poke around, and like to share the solution. Although I am going to integrate the recaptcha service, you could use the same approach to add any 2nd or 3rd factor to the SSO authentication process. End result is the reCAPTCHA appearing and working in the Oracle SSO login page. The sample here is based on the Oracle Collaboration Suite 10g branding:


The sources for my example are available as OssoRecaptcha-1.0-src.zip. See readme.txt in the zip for more detailed instructions and discussion.

There are basically two things we need to take care of to integrate reCAPTCHA. First, customise the login page to render the captcha challenge. Secondly, we need to insert a custom authenticator to handle the captcha validation before the standard authentication.

I've used the ReCaptcha Java Library released by Tanesha Networks to simplify things.

Customising the Login Page

This is the simplest part, and pretty well documented in "Creating deployment-specific pages".

The following code renders the captcha challenge and just needs to be included in the login page at an appropriate point.

<%
// create recaptcha
ReCaptcha captcha = ReCaptchaFactory.newReCaptcha(RecaptchaConf.RECAPTCHA_PUBLIC_KEY, RecaptchaConf.RECAPTCHA_PRIVATE_KEY, false);
String captchaScript = captcha.createRecaptchaHtml(request.getParameter("error"), null);
out.print(captchaScript);
%>

RecaptchaConf is a class included in the sample to hold your site-specific reCAPTCHA keys that you can easily get by registering at http://recaptcha.org.

Customising SSO Authentication

We have a simple task: intercept and evaluate the catpcha response before allowing standard SSO authentiation to proceed. Simple, yet not exactly documented unfortunately. The documentation for "Integrating with Third-Party Access Management Systems" is almost what we need to do, but not quite.

The approach I have taken is to sub-class the standard authenticator (oracle.security.sso.server.auth.SSOServerAuth) rather than just implement an IPASAuthInterface plug-in.

The only method of significance is "authenticate", where if the captcha response is present, we evaluate it prior to handing off to the standard authentication.

public IPASUserInfo authenticate(HttpServletRequest request)
throws IPASAuthException, IPASInsufficientCredException
{

SSODebug.print(SSODebug.INFO, "Processing OssoRecaptchaAuthenticator.authenticate for " + request.getRemoteAddr());
if (request.getParameter("recaptcha_challenge_field") == null) {
throw new IPASInsufficientCredException("");
} else {
// create recaptcha and test response before calling auth chain
ReCaptcha captcha = ReCaptchaFactory.newReCaptcha(RecaptchaConf.RECAPTCHA_PUBLIC_KEY, RecaptchaConf.RECAPTCHA_PRIVATE_KEY, false);
ReCaptchaResponse captcharesp = captcha.checkAnswer(request.getRemoteAddr(),
request.getParameter("recaptcha_challenge_field"),
request.getParameter("recaptcha_response_field"));
SSODebug.print(SSODebug.INFO, "ReCaptcha response errors = " + captcharesp.getErrorMessage());
if (!captcharesp.isValid()) {
throw new IPASAuthException(captcharesp.getErrorMessage());
}

return super.authenticate(request);
}
}

A couple of things to note:

• This method is first called prior to the login challenge to see if you are already authenticated, hence the check for a captcha response before boldly going ahead to authenticate
• The specific exception messages raised in this class seem to get "lost" by the time the handler returns to the login page (at which point you always seem to have a generic failure message). In other words, users will basically just get told to try again. I haven't found a way around this yet.
• See the example usage of SSODebug to log messages which will appear in the SSO log (as configured in ORACLE_HOME/sso/conf/policy.properties)
• We'll deploy the custom class into the OC4J_SECURITY container, rather than to $ORACLE_HOME/sso/plugins since it seems plugins get a limited environment that does not include all of the required support classes. Deploying to OC4J_SECURITY avoids this problem.

Deployment

The most robust approach to deployment is to explode, modify and the rebuild the OC4J_SECURITY EAR file ($ORACLE_HOME/sso/lib/ossosvr.ear) once you are confident everything is working fine. I haven't covered how you do that here however.

Rather, I'm deploying the sample directly into an existing OC4J_SECURITY container. Note that with this approach, if you ever redeploy the OC4J_SECURITY application (which can happen during an upgrade or patch for example), then your changes
would be destroyed.

There's an Ant build script included in the sample that takes care of the details, but is pretty straightforward...

Firstly, two copy operations:

1. Copy the login page to $ORACLE_HOME/j2ee/OC4J_SECURITY/applications/sso/web/
2. Copy the supporting jar files to $ORACLE_HOME/j2ee/OC4J_SECURITY/applications/sso/web/WEB-INF/lib/

Second, the authenticator configuration is governed by $ORACLE_HOME/sso/conf/policy.properties.

MediumSecurity_AuthPlugin = oracle.security.sso.server.auth.SSOServerAuth
# replaced with:
MediumSecurity_AuthPlugin = com.urion.captcha.OssoRecaptchaAuthenticator

Finally, we are ready to restart the OC4J_SECURITY container

opmnctl restartproc process-type=OC4J_SECURITY

and test out the customised login. Try...

http://you.site:port/oiddas

Give it a go! Love to hear from anyone who deploys reCAPTCHA on a production Oracle Portal or Applications site for example.

Postscript: Patrick Wolf obviously had a weekend free also, and has now posted a solution for adding reCATPCHA to APEX ;-) Cool!

Postscript 2008-06-03: I finally got around to setting this up with its own sourceforge project.

Revisiting 11g Native Web Services

I've just moved from 11.1.0.5 to 11.1.0.6, and updated my post First Tests of 11g Native Web Services in a few areas:

• The XDB_* role names have been corrected in 11.1.0.6 to match the documentation.
• Confirmed that the auto-generated WSDL still assumes 'orawsv' as the URL pattern (see below). So for now, it's clear that you should stick with 'orawsv' as the servlet name and URL pattern for your native web services configuration.

Thanks to Marco Gralike and Christopher Burke for the chats we've had in the comments thread. Here's a summary of the links that came up in discussion:

• Oracle 11g - How to enable native WSDL services
• Oracle 11g Security - XMLDB is here to stay...
• Mark Drake's example in the forums

+0.9

Launch is over, software's released, and now I've actually posted a real 11g blog entry, so I've decided "Tardate 10.2" deserves to grow up by +0.9. Now to be known as "Tardate 11.1"!

First Tests of 11g Native Web Services

I mentioned in Log Buffer #54 that "Patrick Wolf stumbled across an 11g feature that means DBA's may put Java/SOA guys out of work".

Well I finally got myself setup with an 11g test system and the Native Web Services was the first thing I jumped at testing.

Conclusions first?

• Although perhaps not a blockbuster new feature, Native Web Services provide an easy solution for exposing data services into a SOAP-oriented application stack.


• For 11.1.0.5 (only), see Metalink Note 444191.1 for updated installation instructions.


• The auto-generated XMLSchema and WSDL have some goofy features (like hyphens in method names) that may break SOAP client toolkit smarts.


• The is no real control over the generated XML payload, which will possibly limit the usefulness of Native Web Services for Enterprise SOA initiatives.


• The security model is simple but effective. The underlying database user role model provides authentication and access control.


• The ability to call arbitrary packages/procedures/functions is perhaps the most powerful feature. No incremental coding or configuration is required for each method. A good service interface is thus just a matter of careful procedure design and construction.


• I am less convinced that the ability to execute arbitrary SQL via Native Web Services is a good thing. It's not SOA, and it's not a good replacement for JDBC, ODP etc. Seems like just an invitation for bad hacks and shortcuts..

So, after a couple of hours of playing, I think maybe the Java/SOA guys don't have to panic just yet.. ;)

Here's the blow-by-blow, where I describe the setup gotchas I hit, and my simple SOAP::Lite testing with Perl ..

Initial Scan..

It took me a few moments to orient myself in the documentation; rather than being a distinct major new feature, Native Web Services are an enhancement of XML DB, and so the requisite documentation is mainly found in the XML DB Developer's Guide.

The architecture of Native Web Services is quite simple. The main moving part is a servlet called 'orawsv' which brokers SOAP 1.1 requests and handles automatic WSDL generation. Out of the box, it is not enabled.

The feature works in two main ways:

1. An arbitrary SQL DML request can be sent to the main 'oawsv' endpoint, and the results are returned as XML in the SOAP response.


2. Procedures and functions may be invoked directly over SOAP.

First Up - Configuration

Configuring Native Web Services is simply a matter of enabling the orawsv servlet, and then granting user access through role assignment.

This is covered in Chapter 33 of the docs.

I first tried this with 11.1.0.5, and found that the role names mentionedd in the docs do not match (XDB_WEBSERVICES instead of XDBWEBSERVICES and so on). For an 11.1.0.5 install, refer to Metalink Note 444191.1 for corrected installation instructions. In 11.1.0.6, role names correctly match the docs.

The second thing I discovered (actually, a problem I created for myself then had to fix!) is that the servlet name does matter! The SERVLET_NAME (as in the setup code below) must be 'orawsv'.

Thirdly, although you can set the URL pattern to anything you wish (and calls work), the auto-generated WSDL always assumes 'orawsv' when generating endpoint addresses. Effectively this means you must use 'orawsv' as the URL pattern also.
NB: thanks to Christopher Burke for alerting me to the WSDL situation.

DECLARE
SERVLET_NAME VARCHAR2(32) := 'orawsv';
BEGIN
DBMS_XDB.deleteServletMapping(SERVLET_NAME);
DBMS_XDB.deleteServlet(SERVLET_NAME);
DBMS_XDB.addServlet(NAME => SERVLET_NAME,
LANGUAGE => 'C',
DISPNAME => 'Oracle Query Web Service',
DESCRIPT => 'Servlet for issuing queries as a Web Service',
SCHEMA => 'XDB');
DBMS_XDB.addServletSecRole(SERVNAME => SERVLET_NAME,
ROLENAME => 'XDB_WEBSERVICES',
ROLELINK => 'XDB_WEBSERVICES');
DBMS_XDB.addServletMapping(PATTERN => '/orawsv/*',
NAME => SERVLET_NAME);
END;
/
GRANT XDB_WEBSERVICES TO SCOTT;
GRANT XDB_WEBSERVICES_OVER_HTTP TO SCOTT;
GRANT XDB_WEBSERVICES_WITH_PUBLIC TO SCOTT;

I couldn't quite figure out exactly why the 'orawsv' SERVLET_NAME is significant. It apparently just is, providing an explicit reference to the required servlet. The DBMS_XDB.addServlet docs are unfortunately terse and unenlightening.

So that all took me an hour more than it should. But not to worry - I am now up and running!

Now What Can It Do? Testing a Function Call by Web Services..

I was particularly interested to see a procedural interface exposed via Web Services, so my first little test was to call a database function from Perl (using SOAP::Lite). I created this very simple function in SCOTT:

CREATE OR REPLACE FUNCTION empcount
RETURN NUMBER IS
emp_count number;
BEGIN
SELECT count(*) INTO emp_count FROM emp;
RETURN emp_count;
END;
/

This becomes available at the endpoint 'http://server:port/orawsv/SCOTT/EMPCOUNT', and the auto-generated WSDL is available at 'http://server:port/orawsv/SCOTT/EMPCOUNT?wsdl'.

Next I knocked up a bare-bones Perl client. Note the redefined get_basic_credentials method to provide the basic authentication credentials that are required.

#!/usr/bin/perl -w
#
use SOAP::Lite +trace => 'debug';
sub SOAP::Transport::HTTP::Client::get_basic_credentials {
return 'scott' => 'tiger';
}
my $serviceNs = 'http://xmlns.oracle.com/orawsv/SCOTT/EMPCOUNT';
my $soap = SOAP::Lite
->proxy('http://localhost:8080/orawsv/SCOTT/EMPCOUNT');
my $som = $soap->call( SOAP::Data->name('SNUMBER-EMPCOUNTInput')->attr({'xmlns' => $serviceNs}) );
print "The response from the server was:" . $som->result . "\n";

And the answer is .. 14;) [see the end notes below for the request/reply transcript].

As you can see, pretty simple, but it should be simpler. One liners should be possible, like this:

print SOAP::Lite-> service('http://scott:tiger@localhost:8080/orawsv/SCOTT/EMPCOUNT?wsdl')->EMPCOUNT();

... but you'll notice that the 11g generated method names (like 'SNUMBER-EMPCOUNTInput') have annoying hyphens, which subverts the smart typing and dynamic binding that kits like SOAP::Lite are capable of. Doh!

Closest I can get is this:

print "The response from the server was: ";
print SOAP::Lite
->uri('http://xmlns.oracle.com/orawsv/SCOTT/EMPCOUNT')
->proxy('http://localhost:8080/orawsv/SCOTT/EMPCOUNT')
->call ( 'SNUMBER-EMPCOUNTInput' )
->result;

Endnotes - EMPCOUNT SOAP Request and Response

{The request ... }
SOAP::Transport::HTTP::Client::send_receive: POST http://localhost:8080/orawsv/SCOTT/EMPCOUNT HTTP/1.1
Accept: text/xml
Accept: multipart/*
Accept: application/soap
Content-Length: 461
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://xmlns.oracle.com/orawsv/SCOTT/EMPCOUNT#SNUMBER-EMPCOUNTInput"

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<SNUMBER-EMPCOUNTInput xmlns="http://xmlns.oracle.com/orawsv/SCOTT/EMPCOUNT" xsi:nil="true" />
</soap:Body>
</soap:Envelope>

{The response ... }
SOAP::Transport::HTTP::Client::send_receive: HTTP/1.1 200 OK
Server: Oracle XML DB/Oracle Database
Content-Type: text/xml; charset=UTF-8
Client-Date: Mon, 13 Aug 2007 16:06:28 GMT
Client-Peer: 127.0.0.1:8080
Client-Response-Num: 1
Client-Transfer-Encoding: chunked
DAV: 1,2,<http://www.oracle.com/xdb/webdav/props>
MS-Author-Via: DAV

<?xml version="1.0" ?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<EMPCOUNTOutput xmlns="http://xmlns.oracle.com/orawsv/SCOTT/EMPCOUNT">
<RETURN>14</RETURN>
</EMPCOUNTOutput>
</soap:Body>
</soap:Envelope>

The BI/EPM Revolution (..needed, please!)

Frank Buytendijk makes a good case for why generic BI/EPM frameworks are most probably pretty useless in practice.

What puzzles me deeply is that so much of the focus of BI remains concerned with the technology infrastructure and dashboard bells and whistles. This is seen most clearly when a new enterprise application or business process is being introduced, and you hear the almost dismissive assertion made that "it will integrate with you DWH", or that it will "plug in to your management dashboard". And "we'll scope for, let's say, 20 reports to be built".

I think we are all missing the point.

I'm going to suggest that on the whole, business is still struggling with two symptoms of IT growing pains ...

• The IT profession has lost sight of the fact that reports, dashbords and the like are not end products in their own right. They typically have no intrinsic value. The value lies purely in the decisions and actions that may be taken as a consequence (the exception perhaps being for reports that are required by an external party such as shareholders, government or regulatory authority).


• Equally, I think business management is still adapting to a world where information is plentiful. Even when fact-based decision making is practiced, it is all too common to find it be based on spreadsheets (which may or may not have had their original source data pulled from an enterprise system).

Every time I hear the ERP vendors boasting about enterprise adoption, I cynically imagine a Microsoft advertisement that could (probably truthfully) claim

"99.9% of Fortune 500 companies run their business on Excel!!"

I think its easy to understand how we got to this position. I remember working for a reinforcing mesh ("fabric") manufacturing company in the late 80's. We had factories filled with fabric machines like the one in the picture. They weren't networked or anything. The only information that the production manager had to work with were the production sheets filled out by the operator each shift, and the maintenance manager had to send his technicians around to check out the machines periodically.

We were working in an environment where information was scarce, yet the business imperitives were very much the same as today: production manager was concerned with productivity and profitability; and the maintenance manager cared about the maintenance costs and machine performance (netting to the "TCO" of the plant). We managed through a combination of MWA (Management by Walking Around, aka gemba / genchi genbutsu), MGF (Management by Gut Feel! .. some would say experience;-), and also selected special projects to capture data related to problem hypotheses [what I was most involved in].

But the world has turned in the last 20 years. Today, most organisations have more information sitting in various databases and enterprise systems than they know what to do with. I believe the challenge these days is firstly to make that information accessible, and then - most critically - to ensure that our management practices can make effective use of that information.


We actually have great technology these days for cracking that first challenge - making information accessible and actionable. I like to think in terms of three tiers (management, operational, infrastructure), as in the diagram.

Unfortunately, that's often as far as our thinking (and implementations) go.

However as Frank points out in his post, we need to address the strategic framework for BI. How BI impacts, changes or even transforms the business.

For me, the implications are clear for all decision-makers (which may or may not just mean "management")...

• In the past we have excelled at managing in an information-poor environment. Now we need to go back to school, unlearn some of our assumptions and practices, and start exploiting the information that (should be) at our fingertips.


• This means a renewed relevance of management science, particularly quality management (e.g. 6 Sigma) and key concepts such as kaizen/continuous improvement.


• In true "Flat Earth" style - if you don't, someone else will and then you are history, mate.

And for IT, I think a few things to consider...

• First, get real and get connected to your business. You are not delivering 20 reports, you are delivering information that will hopefully help your business users may really good decisions, impact the bottom line and keep your job safe.


• Avoid the temptation to dumb-down and de-scope BI and reporting. It could be the most business-critical aspect of the whole project ... Promote monitoring and management to first-order use cases.


• Consider all tiers of monitoring and management: from corporate performance, to operational management, to infrastructure.

So maybe this is an aspect of what Enterprise 2.0 is really all about?

Take the 2007 Perl Survey

Take part in the 2007 Perl Survey! The Perl Survey is an attempt to capture a picture of the Perl community in all its diversity. No matter what sort of Perl programmer you are, they'd love to hear from you.

The survey can be found at: http://perlsurvey.org/

It only takes about 5 minutes to complete, and will be open until September 30th, 2007.

My Reading Feed

Just a quick note to highlight a relatively new enhancement to blogspot .. the ability to get an RSS feed for a given label. I stumbled across the instructions while trying to figure out how to get a feed of all the book reviews I decided to put up at my Prata Life blog. The basic URL formulation is like this:

http://blogname.blogspot.com/feeds/posts/default/-/labelname
e.g. my reading feed is:
http://pratalife.blogspot.com/feeds/posts/default/-/Read

So now I have a little side-panel on the tardate blog showing the most recent books I've read, using the RSS widget. Cool!

Top of the list right now is The Ambler Warning by Robert Ludlum, which had some interesting, coincidental cross-overs to other books I'd read recently (blink, Softwar). Includes an interesting cameo(?) as the book reaches a climax, action centers on the World Economic Forum meeting at Davos..

"A couple of yards away from him, an older, rangy American billionaire - someone whose 'enterprise software' was an industry standard across the globe..." Hmmm, ring any bells? If you don't think Bill makes 'enterprise software', then I reckon it can only be one person ...

OTN in China and the importance of aggregators

Justin reported the stunning success of the OTN lounge at OpenWorld Shanghai 2007. Although I never made it to the lounge, the buzz in the halls and corridors was great testament to the enthusiasm of the Oracle community in this part of the world.

My trip up to Shanghai also provided a pointed reminder of China's on-again-off-again dance with blogspot (currently "off-again") when I tried to update my blog.

Now I'm not going to argue that China needs to cease its attempts to censor the net since I respect it's authority to exercise such control, just as we do in the west but with different values. However I must admit I had never thought of the collateral damage inflicted on the Oracle community of users in China, since it seems quite a large number of bloggers on Oracle topics are using one of the blocked platforms.

Fortunately it seems that the various Oracle-topic aggregators have not been hit by any of the blocks (and presumably won't be because of the generally non-political content).

So I'd encourage the aggregators to note the importance of aggregating full-text feeds with attachments ... it may be the only chance that some people may have of reading the content (also works better for anyone with an offline reader anyway).

For Oracle, I'd suggest this is another reason why it would be great to open up to hosting non-employee blogs.

Innovation on show at OpenWorld Shanghai

I've just spent the past few days at Oracle OpenWorld Asia Pacific in Shanghai. Although of course a smaller scale to the one coming up in San Francisco in November, I think there were 8000 or so attendees and in true OOW style, each session slot presented the challenge "which of these 10 concurrent sessions do I want to attend?!"

I also got to present a session myself, which was great fun (thanks to those who filled up my session on the last afternoon of the last day!) More of that in future posts maybe ...

Innovation was one of the key themes of the conference and Charles Phillips' keynote. And I think there was no better illustration of this than the demo that the OARDC team were showing in the demogrounds. It got my "best in show" vote!

Their Second Life "Innovation+ Village" show-cased a solution concept bridging the real and the virtual worlds, and they brought some interesting ideas into play.

The scenario was a Second Life wine store. In the room you can examine the products and get some gratuitous live-video. But then you can make purchases (using Linden$), which are captured in iStore for delivery in "real life" (I hope .. still waiting for my case of Bollinger;-) BI dashboards track order volumes and status. Then back in Second Life, you provide feedback on the wine purchased. The feedback is not only collated in Oracle Database for analysis (more dashboards), but also fed into a blog of comments on the particular wine (WebCenter).

Overall, a great example of an "Enterprise Mashup". I don't think the team created any new technology pieces, just a novel solution using a combination of available components and APIs, and provided the content.

While you may doubt the financial viability of doing business in Second Life, it does emphasise the fact that creating Enterprise 2.0 value for your organisation does not necessarily involve a huge development investment. Just as a great artist can produce a masterpiece with a child's paintbox, real solutions are possible today if you have the imagination and creativity to combine the youthful "Web 2.0" component palette with your existing IT infrastructure.

Congrats to the crew from OARDC that put the solution concept together - Lennard Low, Teo Kian Hui, Wang Rong Rong, Zhang Rong (I believe this is a group photo below) and thanks Rong Rong for the demo;) Hope you get to show this in San Francisco too ... make sure you get a more prominent space in the DemoGrounds though, and don't have the screen facing away from the entrance!

Playing with CAPTCHAs

Security Now! #101 with Steve Gibson and Leo Laporte covered the deceptively simple challenge of differentiating human from non-human automated clients (great podcast as always .. check it out). Commonly this is done with distorted text like the sample on the right.

The Official CAPTCHA Site has a wealth of information about this field, including discussion of the relay attack that has the potential to defeat any "are you a human?" test (because it enrolls unwitting human accomplices to do the work).

The coolest piece of work though is the reCAPTCHA project. This is a project of the School of Computer Science at Carnegie Mellon University, and it provides a public service for plugging-in a CAPTCHA to your site. But unlike other systems that are just wasting 10 seconds of your time, this system is actually digitizing public domain archives at the same time (getting you to fix the translation of words that have defied the best OCR software)!

Perl is one language that you can easily use reCAPTCHA with, using Andy Armstrong's Captcha::reCAPTCHA module.

After installing the module, it just takes a few minutes to register your site and setup a test page. I built a quick reCAPTCHA cgi in perl .. you can try it out here (sorry, currently may find it offline but get the script source here: myReCaptcha.pl. I think I spent 10 minutes setting this up, and then an hour playing with it;) All in the name of digitizing historical works...

But I'm afraid the humour archive has the best captcha ever ..

Fellow travellers on the road to Fusion..

At one point, the cynical would have said that Fusion is just a con to placate all the PeopleSoft, Siebel and JD Edwards customers. Others just complained about the con-fusion.

Of course, Fusion Applications are still a way off in the future, and I'm not sure if even Oracle Development really know in full cinematic detail what shipping the products will really entail. Its all still fairly big-picture stuff .. although in classic Oracle style, the user/user experience seems to have got lost along the way. But there's enough pressure and creativity that I think we will see the current Fusion formula grow an additional term at some point. i.e.

Fusion = Grid Computing + SOA + Enterprise Information Architecture
will become
Fusion = Grid Computing + SOA + Enterprise Information Architecture + Web 2.0

In other words, potentially the best darn definition of "Enterprise 2.0" in the market so far!

In the meantime however, some things are coming into clear focus. One is the critical importance of SOA and Security Fusion Middleware components (a.k.a. Oracle Application Server). This struck home for me when listening to the recent AppCast interview with Cliff Godwin who is now heading Oracle's Fusion Upgrade Program Office. What is the world coming to.. Oracle Applications folks selling technology?!!

The true test of fusion will be the extent to which it is embraced by customers and the user community. It was great to see Floyd Teter kick-off a new series on his blog where he will be covering his company's Detailed Roadmap to Fusion Applications. Hats off to Floyd for the courage to do this in the open, and to provide the necessary detail that will actually make this useful for others considering the same path.

Log Buffer #54: a Carnival of the Vanities for DBAs

This week its my pleasure to host Log Buffer #54, the Carnival of the Vanities for DBAs. Thanks to Dave Edwards at Pythian for the invite to put this together. Dave must be a very chilled sorta guy to be able to survive each Friday, not knowing what on earth his guest editor is going to churn out!!

Of course 11g continues to be a big topic this week.

Dan Norris adds his views on the best 11g new features to the flurry that came out last week. Christian Bilien's having a party to celebrate ASM fast mirror resync. Syed Jaffar Hussain discovers the alert log will now be XML. Personally I shudder; still not convinced that XML is appropriate for log files.

It strikes me that in this period between launch and release, there's more eyeballs on docs that at any other time in the product lifecycle;)

NB: There still seems to be some confusion whether we are meant to know when 11g will arrive or not. Steven Chan's older post almost convinced me there's a good reason why we don't hear dates;)

In the SQL Server world, Kalen Delaney blogs in wonderment having discovered the dialog box that time forgot. And Bob Beauchemin has some good SQL Server book reviews.

While on book reviews, Stewart Smith is impressed by O'Reilly's Backup & Recovery because it covers just about every way to back up and recover systems (I trust not this way).

So did PostgreSQL trounce Oracle or not? Kevin Closson exposed the lazy reporting that seems to have sparked this "non-story". But you know how pointless it is to try and correct the facts once they are out on the net, right? Its a pity for the controversy, because as Jay Pipes writes, the benchmark achievement is no mean feat for the PG dev team.

Lots of interseting stuff on mysql this week. Mats Kindahl takes mysql proxy for a decent test drive, concludes it has great potential and offers some constructive comments on gaps to be filled. The MySQL 5 HA with DRDB and Heartbeat guide is very well put together by Mark Schoonover (who blurs the line between blog post and technical reference guide!)

What do I like most about open source? Take something like Jay Pipes internals of MyISAM Concurrent insert (part 1 posted this week). Great example of hard core geek writings. Can you ever imagine Oracle, IBM or Microsoft getting down and dirty on internals like this? I love it!

Moving on to deeper thoughts...

K. Brian Kelley mulls over what it means to be a dba, but might be surprised that Patrick Wolf stumbled across an 11g feature that means DBA's may put Java/SOA guys out of work.

Doug Burns ponders just how many blogs a person should have, and in the process gave quite a few pause for thought.

Over in the Oracle Forums, activity seems to be at a high but one wonders about the signal to noise. I think Sidhu was remarkably restrained when I think my response may just have been jfgi!

... and that's pretty much a wrap for this week. Thanks for reading my first attempt at editing the Log Buffer. It just remains for me to leave you with my lame attempt at a bit of 11g humour. Keep on blogging!

Check LOCAL_LISTENER if you run RAC!

Had a case recently with a 10gR2 RAC install. Everything seemed to have been setup to spec, but we were seeing clients occasionally getting ORA-12545 errors and failing to connect, and things getting even worse during failover testing.

After investigating and solving this, it was painfuly obvious how easy the configuration issue can sneak into a RAC install, which prompts me to blog about it now. Bottom line: if you are installing RAC or are responsible for managing a RAC database, I strongly suggest you swivel over right now and 'show parameter LISTENER'!

So back to our case.... looking at a sqlnet client trace, it was apparent that the client was being redirected to the server hostname, not one of the RAC virtual addresses. Two problems:

1. the client couldn't resolve the server hostname since it wasn't in DNS or the client hosts file, and
   
2. the client shouldn't be connecting to the server hostname anyway!
   

The client tnsnames.ora and the servers' tnsnames.ora and listener.ora files had all been checked and were setup with only references the the RAC virtual addresses, so where was the reference to the 'physical' hostname coming from? Well the answer is the LOCAL_LISTENER server parameter.

You would think however that if your client connection descriptor (taken from tnsnames.ora for example) only referenced virtual addresses, you would be safe, right? Not the case, and having a solid understanding of how the listener works is critical to knowing why.

The problem stems from the way that in 10g instances automatically register with the listener, and it is very easy to fall into this trap if you haven't paid very close attention to section "9 Understanding the Oracle Real Application Clusters Installed Configuration" in the platform-specific cluster installation guides.

If you have a DEDICATED server config, then the LOCAL_LISTENER parameter is used for the instance registration with the listener. If you are using a default listener on port of 1521, then DBCA will not automatically set the LOCAL_LISTENER. Section "9.8 Configuring the Listener File (listener.ora)" describes how to manually set a correct LOCAL_LISTENER value, but if you haven't done that, it will default to a connection string that refers to the physical host address (not virtual address).

But you might think "connecting to the physcial host address can't be too bad, can it?". Well yes, there are two problems you can see:

1. Clients may not be able to resolve the host address if you don't have that in DNS, and more importantly
   
2. In a failover situation, clients will not follow the virtual IP.
   

Even then, you might think this would be a very rare problem, because client's tnsnames or other naming is always telling them to connect to the virtual address anyway.

Again, not so. It can be very common for the client to get a connection to the physical host address even if the tnsnames tells them to connect to the virtual address, because of RAC workload management and listener redirects.

Lets take an example of a RAC service called SVC with two instances SVC1 and SVC2 running on host1 and host2 (with virtual addresses host1_vip and host2_vip). The client tnsnames would look something like this:

SVC = 
  (DESCRIPTION = 
    (ADDRESS = (PROTOCOL = TCP)(HOST = host1_vip)(PORT = 1521)) 
    (ADDRESS = (PROTOCOL = TCP)(HOST = host2_vip)(PORT = 1521)) 
    (LOAD_BALANCE = yes) 
    (CONNECT_DATA = 
      (SERVER = DEDICATED) 
      (SERVICE_NAME = SVC)
      (FAILOVER_MODE=(TYPE=select)(METHOD=basic)(RETRIES=10)(DELAY=1)) 
    ) 
  )

The diagram shows how a connection is handled through the listener in the case where the LOCAL_LISTENER is not set correctly. The flows go like this:

1. The client selects a virtual host from the address list and attempts a connection, in this case to the listener on host1_vip.
   
2. The listener selects a preferred instance to handle the session for the service SVC. If the local SVC1 instance is down, or if it thinks instance SVC2 is better able to service the request it sends a listener redirect to the client. This redirect will be to the physical hostname (host2)
   
3. If the client is not able to resolve "host2", you will see an ORA-12545 at this point. If it can resolve the address, then the client establishes a connection to the listener on the host2 address. If SVC2 is running, you should now have a good connection to the database. However, consider now what happens if host2 fails. CRS will ensure that the host2_vip will shift over to host1, but the client is connected to host2 address and you will get into a TCP timout situation. Maybe your client will eventually detect the dead connection and attempt to reconnect (using VIPs), but at best the user application will have stuttered for a significant period of time (depending on your tcp and sqlnet settings).
   

Now consider what should happen, with a correctly configured LOCAL_LISTENER.

1. The client selects a virtual host from the address list and attempts a connection, in this case to the listener on host1_vip.
   
2. The listener selects a preferred instance to handle the session for the service SVC. If the local SVC1 instance is down, or if it thinks instance SVC2 is better able to service the request it sends a listener redirect to the client. This redirect will be to the virtual hostname (host2_vip)
   
3. We get a good connection established to SCV2 on host2_vip. Again, consider now what happens if host2 fails.
   
4. CRS will ensure that the host2_vip will shift over to host1, and the client connection follows this VIP. There will be a slight interruption to communication (depending on how quickly the VIP take-over occurs, and also other factors such if you are connecting through a NAT router, the NATting tables will need to update). Depending on how you have configured FAILOVER_MODE (session, select), you should find that very soon your database connection is alive and you can continue working.
   

Fortuntely there is an easy fix: update the LOCAL_LISTENER parameter to reference the virtual address.

Oracle have a couple of notes on the issue (342419.1, 333159.1) and how to setup LOCAL_LISTENER. Note however, a the time of writing this (and I'm trying to get it fixed), note 342419.1 does not exactly describe the fix correctly.

It mentions to set the LOCAL_LISTENER using the command like this (where server tnsnames.ora has an instance-connect string using VIP address called 'LISTENER_LXDB0036' ):
Alter system set LOCAL_LISTENER= 'LISTENER_LXDB0036' scope=both;

However, this sets a server parameter that would be picked up on all instances. So there are actually two choices:

1. change the tnsnames.ora file on each server to have a different, instance-specific definition of 'LISTENER_LXDB0036' , or
   
2. I think the preferred way: set a SID-specific parameter by adding the SID parameter to the alter system command (assuming the instance name is SVC1 in this case):
   Alter system set LOCAL_LISTENER= 'LISTENER_LXDB0036' scope=both SID='SVC1';
   

Note that the listener registration is is only a problem if the database was created with a lister on default port. Haven't seen it myself, but apparently if you use DBCA to create a database with a non-default listener, then the LOCAL_LISTENER entry is explicitly set with correct reference to VIP.

So the message for today. If you are installing RAC or are responsible for managing a RAC database, I strongly suggest you swivel over right now and 'show parameter LISTENER'. Make sure you don't have an HA problem lurking in your system just waiting to bite you and the very worst moment!

Postscript: I submitted a request for change, and note 342419.1 has now been updated to reflect a more correct solution as of 25-Jun-2007.

Postscript 2: This issue is apparently addressed in 11g. Thanks for the report on this karlarao.
Postscript 2b: As Mark Twain would have said, "The reports of a solution are greatly exaggerated" - apparently 11gR2 still does not set a FQDN by default. Time to crack out a test...

iX2007 day 2 pm - SOA track

Ended up catching the SOA track which was moderated and introduced by Ng Beng Lim from NCS. Oracle, Microsoft and BEA all had a failrly standard pitch with a heavy emphasis on governance (although slightly different ways of modelling the overall journey).


Jurgen Coppens (Accenture) did a refreshingly different approach with a focus on overall transformation (to be expected;) but in particular value realization. Presented some good models, like my sketch to the right, and also advocated a pragmatic approach of focusing on the quick wins in the early stage.

One impression I got from all the speakers however is that they all called for a rational architecture and governance process that is strongly tied to business strategy. In fact this is a fundamental assumption. I have this impression however that for many companies are not really ready to address the question of whether to adopt SOA or not, since they are still have not established the foundation architecture and governance competency. Baby steps.

Anyway that's a wrap for the conference. As I mentioned in my first post, this was the first that had caught my eye for some time, and it lived up to the promise.
ix2007

iX2007 day 2 am - some nice provocative points of view

Morning sessions for day 2 of iX2007 have wrapped. Some highlights...

Douglas Merrill, Google, delivered a provocative point of view on the implications on Enterprise IT. Talked of the changing expectations of users and what they will demand and use in the enterprise. Yes its Web 2.0, but also more broadly the impact of an increasingly IT-literate workforce.
For IT, it means embracing new ways of conceiving enterprise solutions. And if SMEs are running their own infrastructure and applications, they are probably wasting their money; they will be exploiting Software as a Service.

Bit disappointed in David Willis' (Gartner) presentation. Did a good job of summarising the trends but nothing earth-shattering.

Louis Broome gave us great insight into the benefits Microsoft are getting run production (podcasts, video) inhouse. Cost-savings are one aspect (75% or so). Amazing to hear that their latest studio only cost US$13k to setup, and their 4.5 headcount are producing close to 1000 hours of video a year (thats with post production, compositing and the like).

Jeremiah Owyang argued that the corporate website is rapidly becoming irrelevant, being sidelined with the rise of 'disruptive' social network tools (from a marketeer's perspective). Go to the corporate website for product specs etc, but if you want buying advice and opinion, you are more likely to trust your peer networks. Corporate employees are in a way contributing to this with the rise of blogging etc, where they blur the corporate boundary with informal communication (bidirectional).

A good morning. Now looking forward to the tracks; looks like I'll hop between SOA and Security, although I'd also like to catch eGovernment!
ix2007

iX2007 Singapore - day 1 deep thoughts

iX2007 started today here in Singapore. The first conference that really caught my eye for quite some time. Today was keynote day.

iN2015 Update - Optimism in the air?

Interesting to see Mr Chan Yeng Kit (iDA) presenting his update on Singapore's iN2015. When you look at the progress being made in key areas like digital infrastructure upgrade and undergraduate admissions in ICT programmes, I think it reflects accurately the "mood" in the industry, which I'd summarise as "optimistic, with increasing confidence". SARS, dot.bomb and the Asian financial crisis are still recent memories but slowly receeding into history ...

Business Forum - a Truely Interactive Panel Discussion!

The Business Forum ran as a panel discussion involving Pek Chew Yai (SiTF), Lim Chin Hu (Frontline), Craig Gledhill (Cisco) and Phey Teck Moh (Pacific Internet). Robert Chew (Accenture) did a great job as moderator by feeding the discussion with comments coming from the chatroom. I've got to say that this resulted in the most interactive and interesting panel discussion I've ever experienced in this part of the world. Amazing to see the realtime flow from keyboard to chatroom then having a direct impact on the discussion. I think The Digital Movement cashed in on a really good bet by setting up the chat (running 37 Signals Campfire) and providing SMS and free wireless access. We may be too paisei here to step up to a microphone, but in the chatroom the speech runs free and wild!

Great to see the conference assuming a web persona; Jeremiah Owyang has already blogged on his Day 1 thoughts.

Singapore - Producers or Consumers?

All this enthusiasm to use the technologies at hand to enhance the conference experience really underscored for me one of the topics that got a bit of an airing during the panel: are we clear on our aspirations as both consumers and producers? I think so much of the focus in iN2015 is geared towards the consumption ... improving our already-great infrastructure, ensuring we have an ICT-savy workforce. I think we are still struggling with the production aspect, the irony being that there appears to be a huge amount of energy and creativity pent up in Singapore just waiting to be unleashed on the production side. How many of the grads from SMU, NUS, NTU and the like are destined to face the choice of (a) stay in Singapore but end up in a less exciting role than they had hoped for, or (b) head overseas to pursue their dreams? I guess they can take comfort that going overseas is at least a realistic option once again!

Meaning no disrespect to any of the pioneers doing amazing work in Singapore, ICT and Digital Media production is still niche here. And I don't see too many signs this is changing. There seems to be a defacto assumption that its good enough to be a "hub". After all, that is what has made Singapore so successful in logistics, corporate and financial services. But I wonder if that necessarily has to be the case for ICT? Its almost like we are afraid to dream the big dream.

I'll throw a hypothetical out there:

• Singapore has a skilled workforce, an attractive base for foreign workers, great infrastructure, stable and transparent government (in most things;), cultural ties to India, China and the West, high proficiency in English in addition to many other languages, and lower salary costs than most western countries.
  
• So why don't we see, for example, large software companies basing development centres here?
  
• Sure, the prevailing view is that if its not in China or India, you must be crazy. But what would be so crazy, for example, of a US company maintaining corporate HQ in the US, development centres in China or India, but running R&D, product management and lead development out of Singapore?
  
• Personally I think the net result would be a huge benefit for the company concerned. I think you would also find managing offshore development in India and China from Singapore surprisingly effective.
  
• And as a result, all these creative, innovative technopreneurs being groomed in Singapore would find much more opportunity to do big things in their own backyard.

Solutions? Well of course, in true Singaporean style, we could say the gahmen needs to do more to set a bold vision, attract and support ventures of this nature. That means EDB, SiTF and iDA. But I think the real change needs to come from within the ICT community in Singapore (inlcuding those working in the multinationals). Be big, be bold, be best. Believe.

Tomorrow...

Anyway, looking forward to the track sessions tomorrow. My only problem is that I want to be in 3-4 places at the one time. Hard to decide between the 6 tracks: digital media, eGovernment, security, SOA, wireless and eLearning. I think I want to participate in them all ... now is there a technology solution for that?

ix2007

Diving for SOAP Perls

Antony Reynolds' recent Diving for Perls with WSIF post gave a great example of how you can use HTTP bindings to call perl CGI scripts from Oracle BPEL Process Manager.

If your perl code is not already available to be called in this way, then what to do? Certainly the "ideal" would be make it available as a native Web Service and do away with any special binding. Thanks to the SOAP::Lite module, this is actually quite easy to do.

I'm going to walk through an example of how to take some aribitrary perl code, wrap it as a Web Service, and then call it from a BPEL process. See the diagram:

The Perl Code

In this example, there's really only one bit of code that "matters" ... a helloWorld function. I'm going to start with this wrapped in a perl class module called HelloWorld.pm. As you'll see shortly, wrapping the business functionality in a class is a good idea because it allows automatic dispatching from the Web Services interface.

$ cat HelloWorld.pm
#!/usr/bin/perl -w
use strict;
package HelloWorld;
our (@ISA, @EXPORT, $VERSION);
use Exporter;
$VERSION = 1.00;
@ISA = qw(Exporter);
@EXPORT = qw( helloWorld );

sub helloWorld {
my ($self,$foo) = @_;
return 'Hello ' . $foo;
}
1;

Important to note that while the code here contains some of the module niceties, it doesn't make any reference to SOAP, CGI or BPEL. It's plain perl. We can prove that with a little perl test program:

$ cat helloWorld.pl
#!/usr/bin/perl -w
use strict;
use HelloWorld;
print HelloWorld->helloWorld( 'Sunshine' );

$ perl helloWorld.pl
Hello Sunshine
$

The SOAP Interface

The dynamic typing of perl and flexibility of the SOAP::Lite module really live up to the make simple things easy motto. In three lines of code we have a SOAP CGI server for our HelloWorld class (that's why I made it a class;)

$ cat HelloWorld.cgi
#!/usr/bin/perl -w
use HelloWorld;
use SOAP::Transport::HTTP;
SOAP::Transport::HTTP::CGI
->dispatch_to('HelloWorld')
->handle;

That was so easy, there must be a catch right? Well yes, one comes to mind: the reply message elements will necessarily have some generated names (like "s-gensym3") since there is nothing in our code to provide any guidance for things like the "name" of function return value elements.

Testing SOAP Client-Server

After dropping HelloWorld.cgi and HelloWorld.pm into my apache cgi-bin, I'm ready to test the SOAP service over HTTP. We can whip up a client in no time:

$ cat HelloWorldWSClient.pl
#!/usr/bin/perl –w
use SOAP::Lite;

my $soap = SOAP::Lite
->readable(1)
->uri('urn:HelloWorld')
->proxy('http://localhost:8000/cgi-bin/HelloWorld.cgi');

my $som = $soap->helloWorld(
SOAP::Data->name('name' => 'Sunshine')
);
print "The response from the server was:\n".$som->result."\n";

$ perl HelloWorldWSClient.pl
The response from the server was:
Hello Sunshine
$

If we sniff the network or route this request via a tool like org.apache.axis.utils.tcpmon, we can see the outbound request and incoming reply:

Creating a WSDL file

Alas, perl's flexibility means that automatically generating a WSDL for our SOAP service is easier said than done. Unlike in strongly-typed languages, perl methods can take an arbitrary number of parameters of arbitrary type ... whereas of course a Web Service should have a very clearly defined interface.

I think one of the best approaches at present for generating WSDL in perl is the Pod::WSDL module. I'll perhaps leave that for another blog entry. For now lets just assume we'll manually create a WSDL for our service:

$ cat HelloWorld.wsdl
<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions targetNamespace="http://localhost:8000/HelloWorld" xmlns:impl="http://localhost:8000/HelloWorld" xmlns:wsdlsoap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:tns1="http://localhost:8000/HelloWorld">

<wsdl:message name="helloWorldRequest">
<wsdl:part name="name" type="xsd:string" />
</wsdl:message>

<wsdl:message name="helloWorldResponse">
<wsdl:part name="s-gensym3" type="xsd:string" />
</wsdl:message>

<wsdl:portType name="HelloWorldHandler">
<wsdl:operation name="helloWorld" parameterOrder="name">
<wsdl:input message="impl:helloWorldRequest" name="helloWorldRequest" />
<wsdl:output message="impl:helloWorldResponse" name="helloWorldResponse" />
</wsdl:operation>

</wsdl:portType>

<wsdl:binding name="HelloWorldSoapBinding" type="impl:HelloWorldHandler">
<wsdlsoap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http" />

<wsdl:operation name="helloWorld">
<wsdlsoap:operation soapAction="" />
<wsdl:input name="helloWorldRequest">
<wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8000/HelloWorld" use="encoded" />
</wsdl:input>
<wsdl:output name="helloWorldResponse">
<wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8000/HelloWorld" use="encoded" />
</wsdl:output>
</wsdl:operation>

</wsdl:binding>

<wsdl:service name="HelloWorldHandlerService">
<wsdl:port binding="impl:HelloWorldSoapBinding" name="HelloWorld">
<wsdlsoap:address location="http://localhost:8000/cgi-bin/HelloWorld.cgi" />
</wsdl:port>
</wsdl:service>

</wsdl:definitions>

Invocation from a BPEL Process

Now you have all the bits in place to invoke your Perl code as a fully-fledged Web Service from within BPEL. I won't go into this in detail here because it is the standard Web Service invocation process. Just add an "invoke" activity in your process and point it to a partner link defined based on the WSDL generated above.

Once you have deployed your process, you can test it from the BPEL Console. Here's an example of the invoke activity in one of my tests:

Conclusion?

Hopefully I've shown that exposing perl code as a Web Service is actually pretty simple. Once done, the code is then available for use by standards-based tools like Oracle BPEL Process Manager.

There are a couple of consideration to bear in mind though:

1. SOAP::Lite provides some great hooks for automatically generating a SOAP interface, however these come with the caveat that reply message elements will necessarily have some "generated" names
   
2. Automatic WSDL generation is confounded by perl's dynamic typing. Modules like Pod::WSDL provide some good solutions though.
   

An MQ and OCCI Demo

A little while ago I got to dust off my C++ skills for a project that was to use Oracle Database (via OCCI) and also Websphere MQ. Oracle and IBM already make a range of demos available, but they are mostly all very closely scoped on one feature only. Since I didn't find anything that included all they key concepts in a full working demo, I put together a combined OCCI/MQ demo to do the job (available for download as a tar/gzip file here: occidemo.tgz, see the readme.txt for details).

A couple of key things demonstrated:

1. C++ (OCCI) Oracle database access
   
2. Transparent Application Failover (TAF) notifications in C++ (OCCI)
   
3. Building a C++ application with MQ and OCCI support
   
4. Using makefile flags to build either with full or a "stub" database library class
   

The demo is written for Linux (32 or 64 bit) and has been tested with Oracle Database 10g Server, Oracle 10g Instant Client, and IBM WebSphere MQ 6.0.

The diagrams below give a simple exposition of how the demo is structured. The executables "mqproducer" and "mqconsumer" are MQ clients shuttle messages back-and-forth via queues. For each message sent by "mqproducer", a reply is expected from "mqconsumer". The readme.txt in the archive contains fairly detailed coverage of how to run the demo.

If the sample is built with full database support, then a "dblibrary" is linked in that will persist each message to database (and the dblibrary_test program can be use to test the operation).
If the sample is built with a database "stub", then a dummy database library is substitued, and the programs a built without any Oracle Database support linked in at all. This can be useful when just wanting to focus on the MQ aspects in isolation.