Yahoo has continued its recent trend towards remaining relevant today and launched a BETA of a game changing service called Yahoo Pipes and the response from the early adopters has been so good that they had to shut down the service for the better part of the day to increase capacity. I will explain my take on the service shortly but first a quick overview of what the service aims to be as it evolves towards its 1.0 release.

Much of the web today is a hodge-podge of data mixed in with the visual representation of that data. Web pages look good and present value to users but do not interact easily with other web pages or services. There has been a movement towards making web content more structured and the ultimate goal for some visionaries like Tim Berners-Lee is the idea of the sematic web in which different types of content and services on the web can interact intelligently, and with a minimal amount of human intervention.

Standards such as RSS have helped free up some types of content into a structured and streamable format. This, combined with web services such as FLICKR, Google Maps and others offering SOAP and REST API formats have helped lead to the rash of mashups web users have been beneficiaries of over the past couple of years. While mashups have been relatively easy to create from a programmers perspective, they have not been easy to create as an internet savvy but non-technical end user. Yahoo Pipes aims to change that. Though it is named after UNIX pipelines, from where the Yahoo service gets its inspiration, the approachability of the service, with its visual assembly environment aims for a different crowd than the obscure and often obtuse command line interfaces of such UNIX stalwarts as vi and shell scripting. Pipes allows its users to visually connect the input and output nodes of different pre-built modules together to create mashups with significantly less effort than is currently required. Not that mashups are difficult to create, but now they have gone from requiring a little bit of a programmer’s time to requiring only the end users efforts. The output of the Pipes mashup is pushed out in RSS, RDF, JSON, or ATOM format so that it can then be reused again by someone or something else. Toss in a user content sharing on Pipes and suddenly every user can feel like a king.

My take on Yahoo Pipes after briefly being able to use the service is that it is a harbinger of the next stage of the democratization of data. By empowering users to mix RSS feeds, repurpose content, and define their own complex flows of data and services it helps bring power to the true users of the data. The marketing data, sales data, and financial data of an organization are controlled not by their respective departments but primarily by the technology department. This model will remain true for some time, but as elegant and powerful services such as Yahoo Pipes come into their own, and as the web continues to expose data feeds via standardized formats the end users will be able to expose, repurpose, blend, and consume the data in original and meaningful ways by themselves.

Great programmers will always be needed, and I believe they are the true artistic masters of our time. As great programmers create not just great applications and services, but great tools to allow the true end users to repurpose those applications and services we will all be better off. Web savvy end users want to stand on the shoulder of giants not wait in line for their IT department to respond to their requests. Just as programmers today sling code more efficiently using Rails instead of slogging through Assembler, non programmers are now being exposed to elegent and evolving services such as Yahoo Pipes to enable their own ideas and meet their own needs.

To be fair IBM announced an early stage approach towards end user created flows today in the form of QEDWiki - work on the name guys, but otherwise a great service as well. Marc Andreesen has built a platform over at Ning to allow for the easy creation of social networking sites. Yahoo Pipes is more generalized and likely to catch on versus those platforms. Tim O’Reilly calls Yahoo Pipes a milestone in the history of the internet. In my view its one of the biggest steps towards the democritization of data, de Tocquiville would blog relentlessly about this if he were alive today.