Pipes - Web 2.0 Wake-up Call for BPEL?

Recently I've spent some time playing with Yahoo Pipes. I was tipped off by comments to Chris Muir's blog on Drowning in Oracle Blogs Aggregators.

So I've had some fun with Pipes, using it to aggregate all my Oracle news for example. Pipes is still beta and has a fairly modest objective of giving end-users a way to craft their own data feeds on the net, but I think there are some exciting implications.

My Little Experiment

I did a simple experiment with a Perl CGI that issues a "quote of the day" in JSON format, consumed it in Pipes, and then subscribed to an RSS version of the result in FeedReader.

I could have used an existing data source supported by Pipes (RSS, JSON, CSV or XML) but I wanted to experiement with formats. Hence the little bit of coding ..

use strict;
use CGI;
use JSON;

print header();

my $obj = {
  title   => 'Quote of the Day',
  quote   => 'the only dependency we have on the proof of concept is whether it works or not'
};

print objToJson($obj);
1;

(btw, that's one of my favourite quotes from a collegue during a particularly long and weary evening;-)

The Pipes editor makes it a piece of cake to get tricky ... like add a flickr lookup that will roll in a suitable photo to go along with the quote:

All fairly straight-forward, but what grabbed my attention is how Pipes provides an extremely intuitive graphical editor that makes it simple for an end-user to transform and consume data in a hosted model that would conventionally require server-side programming to arrange. It even has an integrated "debugger" that lets you see the output at any stage of the pipe you select.

Pipes - a Web 2.0 Wake-up Call for BPEL?

There are a few things that Pipes can't do (yet), like handle SOAP Web Services, incorporate human workflow or asynchronous events, secure published pipes, and of course reach data sources or systems that are not accessible from the Internet.

But I'm thinking it wouldn't take too much enhancement along these lines and we'd see Pipes rapidly encroaching the BPEL domain (albeit without the standards support). This would be a perfect example of the Web 2.0 challenge to Enterprise IT.

• Pipes - the funky UI that lets anyone create and publish their "flow"


• BPEL - a robust standards-based engine that runs "processes" that have been designed, tested and deployed by experts

If these two extremes don't come crashing together soon, I think we are not trying hard enough!

I'm not sure if Yahoo could do this without a rip'n'replace of the Pipes back-end, and rethink the whole hosting model. More likely that we see someone re-think our approach to BPEL, and the set of use cases it assumes.

I don't hear or see much movement in this direction from the existing BPEL players however. That includes the commercial offerings like Oracle BPEL Process Manager and also the free/open source kind like ActiveBPEL and OpenBPEL.

I suspect we need to challenge a few key assumptions implicit in current BPEL offerings. That processes are (a) largely a conversation between systems, and (b) built by a developer (or at least a skilled business analyst).

While that can and will remain true for core business processes, I think Pipes is beginning to demonstrate that there is a bigger picture we have yet to address.

Put it this way. I'd like to take a robust BPEL engine with...

• BPEL standards support of course


• Web Services (WSDL + SOAP) and WSIF adapter model


• Sophisticated human workflow and asynchronous task support


• Something I can deploy behind the firewall if needed (to access enterprise goodies)


• A decent security model

Then add some of the Pipes goodness:

• Oh-so-Web-2.0 hosted design tools - made for end-users.


• "Publish" instead of "Deploy". Implies automatically maintained (and accessible) registry of published pipes/processes.


• Support for RSS, JSON, CSV and plain XML "web sources"


• Inheritance by chaining or cloning existing pipes


• Smart output rendering. Need RSS, JSON or SOAP? If it makes sense, ask for it and the correct format should be delivered. Transformation taken care of by the infrastructure, not the process definition.

As an enterprise user, this means I'd expect to be able to launch my company's BPELPipes site and see the pipes published by IT. If I need something a bit different, I just go ahead and create it (or clone an existing service). And once I've done creating my pipe, I can use it immediately as maybe an RSS feed or somthing I can embed in my portal. And share it with others if I like..

Perhaps here lies the future of Enterprise 2.0 Workflow (WikiWorkflow?!), finally bringing SOA to the people rather than having it left stuck somewhere in the middle tier.