The end of aggregation?

The last 8 (or 800 arguably) years of media consumption for me have been marked by aggregation. In 2001, before RSS was in any sort of wide use, I had built an aggregator called Because of the unstructured nature of the data (individual web pages), I had to create a learning algorithm that would, over time (2 to 5 page loads) identify relevant new content vs irrelevant new content (rotating banner ads, text ads, etc). This wasn’t foolproof, but it worked well enough. It was a lot of work though. The blogosphere was also small enough back then that you could try to capture its current state through some fun experiments.

A few years later, emerged as a ping server. Now, instead of going out and actively scraping and looking for new content, blogs were proactively sending out signals that they had new content. Combine this with the now well established use of RSS and content movement was coming out of the stone ages. You still had to scrape however, and eventually the model did not scale well.

That was over 6 years ago, and here we still sit today, scraping for content. Sure Google Reader does a beautiful job of making everything seem realtime, and people are now often tweeting their content updates, but it is questionable if the model we have developed is one that will stand the test of time.

Aggregation as a model is destructive, but we have chosen to ignore that fact, and it continues to be the dominant model today.

The News Feed

In September 2006 Facebook launched the News Feed and changed a lot about the way we view content on the web. Twitter’s launch soon followed and services like FriendFeed continued to launch and gain visibility. The News Feed was coming in to it’s own and it had obvious advantages over straight aggregation for information management.

News Feeds have been especially useful in helping users understand the flow when multiple types of information are being presented. Calendar Dates, Photos, Comments, Wall Postings and Profile Updates were all pieces of information that were essentially impossible to communicate before the News Feed Model. There was no way to “aggregate” profiles and understand how they were changing.

One thing we have struggled with at my new company is to understand how to connect existing enterprise systems to new tools. I wrote about this in June on the FastForward Blog.

The end of Aggregation

The question that sticks out in my mind is: Will streams of pushed signals replace aggregated feeds? And if they do: will we even notice?

We could very well be seeing the end of almost a decade of innovation. Aggregation has brought us an incredibly rich set of experiences and concepts, both inside the enterprise and in the world of public Social Media. It has helped people go from being unknowns to celebrities is no time and it has enabled the creations of an astonishing amount of content in the blogosphere.

The streams of data we are now creating have a lot to live up to, and I believe that it will. We are now moving beyond simply blogging content and hoping people find it to now creating smaller but more manageable pieces of information which may be much more naturally shared and repurposed.

What we need to move forward?

Streams are currently walled gardens. Twitter, Facebook, Friendfeed and others are all a single point of failure. It is imperative that we develop distributed feed models that leverage an existing messaging protocol such as XMPP (thinking has been going on in this area for some time) and we need personal feed builders which can subscribe to, and listen for, these updates. NoseRub is currently developing a really interesting distributed feed service, and it is all open source.

Inside the enterprise we need to re-factor existing information systems to contribute to the enterprise Dynamic Signal and we need to provide users with the tools to build their own customized feeds of that data. This means that ERP systems will exist at the same level of priority in the stream as status updates by your co-workers. Designing and developing these feeds is a strategic job that will require insight in to the data sets and systems that are creating value behind the firewall.

So, say goodbye to aggregation. You may not even notice when it is gone, but the same technology that has taken us this far and has given us so much will be done for, and that is something worth paying attention to.

5 thoughts on “The end of aggregation?

  1. The future of aggregation (or the death of) for me is people powered. Instead of sorting info feeds, I sort people feeds. I do this already in Google reader. And of course Twitter and Facebook embody the idea wholly. Smart people who deliver good info on certain subjects, certain niches, are becoming invaluable.

  2. Hi J
    I am with Jason – I have been seeing in practice the emergence of the Human “Editor/aggregator/curator who uses the aggregator tools to offer them a pool to fish in.

    I suspect that in a world of infinite content the human and the personal selection in fields will be where the value will be found – for instance think of say the Classical Music Mavens that exist – not only can they add value by talking about Beethoven but can also help us navigate through 500 recordings of the 5th. Think of any topic and there will be a few people at the centre who know so much – who can form the “sun” that holds the system together

  3. Aggregation will turn into curation. People will put these things together.

    And to be a stickler regarding something you said, streams are NOT walled gardens. Twitter has open API’s. You can put anything on top of it to make it something different. Twitter feeds can come into any environment, I’d argue nobody has done a good job of it yet.

    Facebook, MySpace, sites that require you to log in and oh-by the way-keep your information and have a right to use it because of how they worded the term and conditions-those are walled gardens.

    I don’t like it when people mix these two together, they are VERY different. It’s an important distinction.

  4. Late to the game on this one, but I’m not quite sure I get this. Isn’t it all of the above? All of this is content and signals and the winners will be those who can make sense of it. See: Social Filtering for some thoughts on how curation at a couple of levels and social signals combine to form a more interesting stream. Of course, that’s in one domain at a time. Much more challenging when you try to define “interesting” across everything. Still, it’s all of the above.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s