The end of aggregation?

The last 8 (or 800 arguably) years of media consumption for me have been marked by aggregation. In 2001, before RSS was in any sort of wide use, I had built an aggregator called Blocktrack.com. Because of the unstructured nature of the data (individual web pages), I had to create a learning algorithm that would, over time (2 to 5 page loads) identify relevant new content vs irrelevant new content (rotating banner ads, text ads, etc). This wasn’t foolproof, but it worked well enough. It was a lot of work though. The blogosphere was also small enough back then that you could try to capture its current state through some fun experiments.

A few years later, Weblogs.com emerged as a ping server. Now, instead of going out and actively scraping and looking for new content, blogs were proactively sending out signals that they had new content. Combine this with the now well established use of RSS and content movement was coming out of the stone ages. You still had to scrape Weblogs.com however, and eventually the model did not scale well.

That was over 6 years ago, and here we still sit today, scraping for content. Sure Google Reader does a beautiful job of making everything seem realtime, and people are now often tweeting their content updates, but it is questionable if the model we have developed is one that will stand the test of time.

Aggregation as a model is destructive, but we have chosen to ignore that fact, and it continues to be the dominant model today.

The News Feed

In September 2006 Facebook launched the News Feed and changed a lot about the way we view content on the web. Twitter’s launch soon followed and services like FriendFeed continued to launch and gain visibility. The News Feed was coming in to it’s own and it had obvious advantages over straight aggregation for information management.

News Feeds have been especially useful in helping users understand the flow when multiple types of information are being presented. Calendar Dates, Photos, Comments, Wall Postings and Profile Updates were all pieces of information that were essentially impossible to communicate before the News Feed Model. There was no way to “aggregate” profiles and understand how they were changing.

One thing we have struggled with at my new company is to understand how to connect existing enterprise systems to new tools. I wrote about this in June on the FastForward Blog.

The end of Aggregation

The question that sticks out in my mind is: Will streams of pushed signals replace aggregated feeds? And if they do: will we even notice?

We could very well be seeing the end of almost a decade of innovation. Aggregation has brought us an incredibly rich set of experiences and concepts, both inside the enterprise and in the world of public Social Media. It has helped people go from being unknowns to celebrities is no time and it has enabled the creations of an astonishing amount of content in the blogosphere.

The streams of data we are now creating have a lot to live up to, and I believe that it will. We are now moving beyond simply blogging content and hoping people find it to now creating smaller but more manageable pieces of information which may be much more naturally shared and repurposed.

What we need to move forward?

Streams are currently walled gardens. Twitter, Facebook, Friendfeed and others are all a single point of failure. It is imperative that we develop distributed feed models that leverage an existing messaging protocol such as XMPP (thinking has been going on in this area for some time) and we need personal feed builders which can subscribe to, and listen for, these updates. NoseRub is currently developing a really interesting distributed feed service, and it is all open source.

Inside the enterprise we need to re-factor existing information systems to contribute to the enterprise Dynamic Signal and we need to provide users with the tools to build their own customized feeds of that data. This means that ERP systems will exist at the same level of priority in the stream as status updates by your co-workers. Designing and developing these feeds is a strategic job that will require insight in to the data sets and systems that are creating value behind the firewall.

So, say goodbye to aggregation. You may not even notice when it is gone, but the same technology that has taken us this far and has given us so much will be done for, and that is something worth paying attention to.