Beginning last year, Michael Arrington and many others identified content farms as the great disruptor, an innovation that will eventually eclipse traditional and social media. With their vast pools of cheap distributed labor, farms would glut the market with low quality content, overwhelming older approaches.

Demand Media is the lightning rod for content farming. But with the information disclosed in their recent IPO filing, it’s clear that content farms are not the great disruptive innovation that people fear.

Figure 1: Demand Media is the lightning rod for content farming
(Image source: Demand Media)

To declare an innovation truly disruptive, it needs to embody some rare qualities. High on the list are productivity and cost advantages. Disruptive innovations bring massive improvements in productivity. The cost advantages allow the innovation to permeate to entirely new markets and usage scenarios, profoundly shaking the incumbents and delighting new customers.

Content farms don’t exhibit these productivity advantages. Demand Media, purportedly among the most sophisticated farms in terms of their operational processes, are reporting underwhelming results. While farms are automating aspects of the content manufacturing process, they remain labor intensive, hand rolled operations. When compared against older models, farms are not demonstrating the productivity advantages that would be needed to qualify as a disruptive innovation.

Figure 2: Content farms employ large numbers of writers, often freelancers, to create content
(Image source: Flickr, http://www.flickr.com/photos/bcostin/366629702)

Another crucial aspect of disruptive innovations is the element of surprise. Disruptive innovations are disruptive because incumbents often deny or ignore them until it’s too late. These innovations challenge our fundamental assumptions about what’s possible. As such, it’s difficult for those outside the innovators’ circle to appreciate the impact.

Consider social media, exhibited in projects like Wikipedia, as truly disruptive. When Wikipedia began, even those close to the project didn’t think it was possible to build a world class encyclopedia through a massive collaboration of strangers and amateurs. It was only through these types of grand experiments that the disruptive implications of Web 2.0 and social media became widely understood.

Again, content farms fail to exhibit this quality of disruptive innovation. Even in the earliest days, few were denying that content farming is possible. When you parse the attributes of content farms, they are understood by all as incremental innovations. Their main differentiation is in the commercial aspects: applying distributed and collaborative approaches to content manufacturing is old news; marrying those aspects to a sophisticated commercial enterprise is the incremental innovation.

Demand Media may disagree. In their SEC filing, Demand describes their differentiation as follows:

While traditional media companies create content based on anticipated consumer interest, we create content that responds to actual consumer demand. Our approach is driven by consumers’ desire to search for and discover increasingly specific information across the Internet. By listening to consumers, we are able to create and deliver accurate and precise content that fulfills their needs.

A fine purpose, but again, on closer examination, it’s hardly disruptive. The difference between anticipated consumer interest and actual consumer demand even sounds incremental, graduating along the same axis of consumer demand. There’s certainly no aspect of fundamentally challenging our assumptions of what’s possible. Publishers have been eking out incremental improvements in market segmentation forever. Even the specific approaches of Demand Media are exhibited across the activities of domainers and arbitrageurs, even reverse engineered in the patents of the most powerful search company in the world.

Content farms like Demand Media are important. They’ve demonstrated the scale under which social media projects can be conducted as commercial publishing ventures. They’re important, but in a very incremental way. Unlike truly disruptive innovations, they won’t put content industries on a new growth trajectory.

None of this is to suggest that the content industries should rest easy. There’s certainly a fundamental disruption in content on the horizon, but you won’t find it on the farm. Rather, the magnitude of productivity advantages and assumption-challenging capabilities needed are found only in factories, in far more industrialized and automated approaches. And although most will deny or ignore their presence until it’s too late, content factories will make farming seem quaint in comparison.

We’ll look at content factories in more detail in a future post. Follow us on Twitter or subscribe to our RSS feed.

Figure 3: Content factories create machines that automate content manufacturing
(Image source: Primal, http://about.primal.com/2009/11/are-web-factories-stealing-your-job/)

Machines and godless industrialists are running amok, polluting the Web with low rent content. Oh the humanity! Where will we find a little human redemption in this scourge of machine generated content?

Why, “News For You!” say some industrialists. Farms and factories may find redemption by adding social and personalization enhancements. Recently Google News, that granddaddy of content factories, unveiled its most significant overhaul since its launch in 2002. Features such as a personal news stream, where consumers can edit categories and content sources to suit their needs, bring that human touch to machine automation.

Google News: News for you

“We’ll save you!” say some experts. Farms and factories may redeem themselves through partnerships with legitimate organizations. Begrudgingly, traditional news organizations are partnering with farms like Demand Media, AOL Seed, and the Examiner.com. Even Google News is bringing in a select group of human editors as partners.

While all interesting developments, this reporter remains unconvinced. Such discussions and industry moves are suspiciously old school, overwhelmingly focused on the quality of content produced. Content farms and factories are often written off as sub-human attempts at journalism. It echoes earlier criticisms of blogging and citizen journalism.

Then as now, market disruptions cannot be evaluated against old standards. Content farms and factories need a new value assessment, one that has nothing to do with journalistic quality (or the lack thereof). Quality has already grossly overshot the expectations of many consumers, which is precisely why the industry is so ripe for disruption. And however glamorous it may seem to incumbents, most consumers don’t want to edit and curate the news any more than they want to report and write it. This self-absorbed supply-side perspective is leaving traditional journalists ill equipped to adjust to these changes.

Maybe there is a deeper, more pervasive humanity supporting the rise of content farms and factories. Consumers want content that is supportive of their immediate needs. They want convenience and simplicity, to be treated as markets of one. Content farms and factories succeed in this fine grained segmentation. Any perceived shortcomings in quality are more than overcome by the convenience benefits: content that is more individually task-oriented and crafted to the readers’ unique requirements.

“Create by consuming”, says Primal. Last week, our factory took another step towards our vision of consumer-directed content manufacturing, with the release of Primal Pages. We don’t expect consumers to act like journalists, but we do want to empower them as the ultimate architects of the content they consume. Under our model, consumers direct the automated content manufacturing in real-time — as they’re consuming it — through the topics they specify and their unique click-paths.

Primal Pages: Create content by browsing

By journalistic standards, this may be publishing at its most basic. But that’s the entire point: a drive towards ultimate simplicity and convenience for consumers as consumers.

The sad truth is that most incumbents are failing to examine the true humanity in content farms and factories in the motivations of the consumers. Eric Schmidt, describing the “cultural phenomenon” that is Google, captures it succinctly: “It’s a way in which consumers consume information.”

First and foremost, consumers consume information, preferring the most tailored and convenient information possible.

Posted by Peter Sweeney

The intelligent design hypothesis

Professional content creators are adopting the posture of intelligent designers. They’re unquestionably intelligent; no one knows more about high quality content and how to create it. Here’s their perspective:

The skills involved in creating artful prose or insightful journalism has an irreducible complexity. Machines simply cannot create quality content. Content that is manufactured by machines is, at best, an unintelligently cobbled together hodgepodge of regurgitated information.

The reason content factories are so economically successful is mass production. They compete not on quality, but on quantity. Search engines and aggregators are complicit in this abuse of the Web, force feeding a glut of low quality content to unsuspecting consumers.

Professional content creators are under siege because content factories are stealing our jobs and glutting the market with low quality content. The market for hand crafted content is eroding as the value of content depreciates, making it economically impossible for professionals to practice our trade.

Inevitably, our hand crafted content could be wiped out of existence, and all of us will suffer for it. Everyone should be concerned.

The evolution hypothesis

Content factories are adopting the posture of evolutionists: provide for the basic elements of an evolutionary process—reproduction, mutation, and selection—and allow the content to evolve and adapt to meet the needs of consumers. Here’s their perspective:

Quality is subjective; good enough is good enough. Consumers care mostly about the time and effort needed to complete their tasks. They have an unapologetically demand-side bias. Consumers would rather have low-quality content that is more specific to their immediate tasks than high-quality content directed to a more general audience.

Content factories are successful due to our adaptive approach. We encode memes in content. Tiny mutations in these memes may seem grotesque on the surface, but in the aggregate they provide consumers with many variations on a theme. From this diversity, search engines allow consumers to make their own selections, which are fed back to the content factories to inform the next iteration.

Professional content creators are under siege only because they are slow to adapt. Their value assessments of the content are self-serving, out of step with their markets.

But it’s not the end of the world for content creators. While formulaic content will continue to move from human creators to machines, markets for high quality content will survive this industrial revolution, just as hand crafted goods survived mass production. Those that adapt will thrive, those that don’t will be marginalized.

Evolution or intelligent design?

Content professionals need to face the realities of Web 3.0 industrialization, just as they had to confront the social revolution of Web 2.0. While technological change is disruptive, it also brings new opportunities.

For their part, the content factories face the same competitive pressures as any other industry. No amount of productivity will protect those that fail to find markets that value their content. And these competitive pressures will also force factories to continue to ascend the quality ladder, often by incorporating the knowledge and best practices of content professionals.

Unlike the biological world, on the Web, evolution and intelligent design are reconcilable.

Back in May 2009, I argued that industrialization is the most transformative force on the Internet. Human tasks are rapidly being displaced by machines. Factories of advanced technologies are being constructed to automate the manufacture of information and content. Predictably, there is much hand-wringing and righteous indignation expressed about this economic sea change.

Primal Fusion just returned from the Defrag conference where industrialization was a focus of the conversations. Andy Kessler’s keynote discussed how industrialization will destroy jobs. It was lambasted as dystopian and ridiculous. Stowe Boyd spoke of social media leading to a post-industrial world; my talk (slides, coverage) discussed the present reality of Internet industrialization.

A recent Wired article on Demand Media was another kick to the industrial hive. This “fiendishly clever” company is automating aspects of the editorial process and manufacturing workflow, creating prodigious amounts of content to fill the nooks and crannies of the online world. Predictably, the response has been overwhelmingly negative.

The concerns are real. Industrialism affects lives. Jobs are being lost and venerable institutions are being eradicated. Unfortunately, gauging by the reaction in the blogosphere, and through my own travels venturing in this area, people are awash in misconceptions and half-truths about this disruptive force.

Myth #1: Industrialism is about automating jobs

The criticisms of industrialism can largely be boiled down to one main indictment: factories destroy jobs. They marginalize skilled professionals and subjugate their roles.

While there is some truth in this criticism, it is also superficial and misleading. There is a naïve view of a “sweat shop” migration, where formerly highly paid professionals end up toiling away at a pauper’s wage creating the drivel that editorial machines assign to them.

The reality is far more nuanced. The technologists who enable this transformation are not automating whole jobs or professions, but rather discrete tasks.

For example, consider the process of manufacturing online content. It’s an extremely complicated process that involves a large number of highly specialized tasks: workflow management, technical infrastructure, editorial and creative direction, content aggregation, layout, information architecture, and many others. As these tasks are automated, fewer people need to be involved in the process. Costs decrease and new economies of scale are realized.

Productivity increases with automation

Forward-looking professionals adapt to technology opportunistically. They evolve new roles that incorporate these tools.

Myth #2: Internet industrialism is about the future

Some take comfort in automation as science fiction, far removed from our current economy.

In reality, the drive towards these productivity advantages is relentless and in full force, right now. Billions are flowing into industrialized Internet industries. (note 1) Giants like Google and Amazon have already emerged. A large number of well-financed and industrious start-ups are rushing in to fill needs for specialized machinery. All are leveraging the network of the open Internet to drive this industrial revolution forward.

Denying this present economic reality, or imagining that concurrent social revolutions will displace this economic drive towards productivity, would be a risky response.

Internet Industrial Revolution

Myth #3: Internet factories create spam

Content professionals take pride in the quality of their hand-crafted content. They imagine machines will only create spam, polluting the Internet with a glut of low-quality content that consumers will reject in droves.

It’s painful to watch critics of industrialism pulling out their quality knives in this quantity gun fight. While artisans decry the death of professional content, the factories quietly find markets where the quality of their content is valued as “good enough”.

There are no absolute measures of quality. It needs to be assessed in the context of our insatiable demand for information. People value content that makes their tasks simpler and more convenient; content that is evermore customized to their needs. Assessments of quality need to reconcile these fast-changing dynamics of supply and demand.

Is there room for abuse? Of course; we will struggle with content spam just as surely as we struggle with spam in all its existing forms. But quality, as an argument against industrial productivity, is a misdirection you’ll want to avoid. Just ask the publishing industry.

Are Web factories stealing your job?

The answer is a resounding no. Jobs and professions are more than collections of tasks. But factory builders are gunning for formulaic and repetitive tasks that lend themselves to automation. If you wear tasks like comfortable old shoes, your job will be overrun. If you embrace these technologies as just another tool in your professional toolbox, you’ll run these factories.

If you’d like to check out some examples of Web factories, the slides from my Defrag presentation are online. If you want to get involved with Primal Fusion in building one of these factories, check out our alpha and careers.

Notes

1. Semantic technologies are a key enabler for automation. Project10X
   projects that the market for semantic technologies worldwide will exceed
   US$50-billion in 2010.

Cory’s article is a tight encapsulation of widely held concerns over search. We share too much private information with search engines. They hold too much power over what is relevant and important. And there is a troubling lack of transparency and accountability into how search engines weigh these decisions.

His bottom line: Vesting this kind of power with a handful of companies is a “terrible idea”.

Fair enough, but when he ventures into possible solutions, Cory, like so many other Internet watchers, makes a serious wrong turn. “Put that way, it’s obvious: if search engines set the public agenda, they should be public.”

He’s correct in pointing out that we need a “transparent, participatory solution”. But he’s taxing search as both the villain and the liberator, when neither label holds.

This is the crux of a seemingly universal misconception: that search will always be the beginning and the end of the Internet. It seems impossible for many to imagine a next-generation Internet that doesn’t have searching at its core.

But dynasties rise and fall, and so will search as our primary online experience.

It’s happening now. There are complementary approaches to validate that search is not the beginning and the end of the Internet. Social networking is the most obvious example. Many people now prefer to tap their social network rather than searching for the information themselves.

Looking further out, companies like Primal Fusion and Siri are advocating for new consumer interaction models based in computer agency. Rather than using computers as tools, we can task them like personal assistants, without involving people at all.

The answer to the problems cited above isn’t in reinventing open versions of successful commercial initiatives, but rather relentless innovation forward. We need to broaden the online environment with new networks and new consumer interaction models.

More concretely, Google knows nothing of my search when it’s brokered through my social networks and computer assistants. A distributed Internet experience mitigates the risks of centralized services.

So the next time you hear a suggestion for a better, faster, smarter search, repeat after me: Search, baby, search! Then go imagine a bigger, broader, richer way to experience the Internet.

Primal Fusion is pleased to support this discussion by sponsoring a lecture by Cory Doctorow, Copyright versus Universal Access to All Human Knowledge and Groups Without Cost. It will take place on Thursday, October 22 at 4:00 PM at the Perimeter Institute in Waterloo, Ontario, Canada, as part of Quantum to Cosmos (Q2C): Ideas for the Future. We hope to hear from you, at the event and online.

Challenges with current approaches

Most of the current approaches to building the Semantic Web focus on content. We create semantic representations of existing assets such as databases, documents, and social media. Machines “read” this knowledge and execute tasks on behalf of consumers. In the Semantic Web world, the approach is content first, consumers second.

Unfortunately, semantifying content is proving to be an extraordinarily daunting task. When we expand the scope of the problem beyond our existing content assets to include knowledge generally, in all its subjective and boundless glory, the challenges of a content-first approach becomes clear. We need alternative strategies, and more importantly, many hands on the problem. (Note 1: Problems)

Some of the content-first datasets published in the Linking Open Data community project.

The consumer-first approach

One such alternative approach reverses the order of operations. It puts the focus on consumers before content. On the surface, this may seem outlandish but, in many ways, it provides a more semantic and scalable roadmap.

Under a consumer-first approach, we build systems that let consumers create the semantics that reflect how they approach, interact with, and experience content. Rather than describing the content, we represent aspects of the consumers’ thoughts, perspectives, and intentions. These consumer knowledge models can then impose a structure on the mass of unstructured content on the Web.

The major difference here is that the semantics and structure are discovered through the expectations that consumers have for the content, rather than being imposed by knowledge engineers in advance.

Note also that consumer-first is not Web 2.0. While Web 2.0 collaborative processes are obviously consumer driven, they are often framed within the task of annotating content as opposed to annotating the mental models of the consumers themselves. Further, the complexity of semantics and knowledge representation demands a Web 3.0 industrial approach to simplify things for consumers.

Example of consumer-first

As an example, let’s take a real estate application. Under a content-first approach, developers would provide some structure to organize the real estate properties in the database. To be effective, they would attempt to anticipate the needs of the consumers, and semantically annotate the property listings accordingly. Finally, consumers are asked to search through these structured listings to find a property that meets their needs.

Within a consumer-first approach, the system would begin by creating a mental model of the consumer’s requirements, a representation of their “dream home” if you like. Once the consumer articulates their “dream home”, it is used as a lens through which the property listings could be evaluated.

Given a formal structure of the consumer’s input, the property listings can remain in a largely unstructured form. However, in the process of filtering the listings through these mental models, each property could be annotated with the semantics identified by consumers.

In other words, semantic webs evolve as a by-product of this consumer-first process, instead of a bottleneck at the start. Technically, the problem shifts from the knowledge representation of existing content to the knowledge representation of abstract thought.

Technical feasibility and testing

To the question of technical feasibility, there are many existing technologies that would support this consumer-first approach: NLP to translate natural language; Web 2.0-style workflow such as semantic wikis; ontology editors; and many others. But critically, it requires a break with the past, putting the focus of the knowledge modeling unambiguously on the consumer wants instead of the producer wares. (Note 2: Technical comparisons)

At Primal Fusion, we’re testing the productivity of this consumer-first approach through our thought networking service. You can begin exploring this approach through our current alpha release (demo video, product walk-through, free registration).

Brainstorming with the Primal Fusion service generates semantic data.

Thought networks are semantic data structures that capture the mental models that people create as they work with content to complete their tasks.

Consumers brainstorm using the Primal Fusion service to create thought networks of the ideas they want to express. A semantic synthesis technology builds data structures in response to the consumers’ interactions. Once these mental models are created, software agents interact with Web services such as search engines to accomplish tasks such as searching, filtering, harvesting, and organizing information. (Note 3: User interactions)

As software agents interact with the semantic data and the Web, formerly unstructured sources are annotated with the semantics provided by consumers. In our alpha, a text filtering process categorizes and describes the content within expansive knowledge models generated by the consumers.

The result is a significant production of semantic data with minimal effort from consumers; the software provides the heavy lifting in both the construction of the thought networks and the tasks of the software agents.

Unstructured content becomes annotated with consumer semantics.

Consumers create this valuable semantic data as a by-product of their activities, costing them little in terms of their time or the skills required. At the moment, we are working to express the output in open vocabularies and plan to publish these semantic representations as Linked Data.

Scalability

In terms of scalability, this approach may seem counterintuitive. Our content assets are very tangible, while our thoughts and intentions are very abstract and seemingly boundless.

However, this focus on the consumer reduces the unbounded nature of knowledge to two very tractable dimensions: How many consumers do you need to serve and how many interactions (or tasks) do you need to support at any given time?

On closer inspection, both aspects are quite finite, and intuitively so. It is far easier for a consumer to provide a snapshot of their knowledge directed to a specific task, than for a producer to try to anticipate all the possible perspectives on their content.

Obviously, on a global scale, the Semantic Web remains an extremely large undertaking, but consumer-first offers an important scalability benefit. The factors of production—the consumers driving this process—move in lockstep with their consumption.

As consumers begin to capture their mental models and tasks in this formal way, there is a decreasing incremental cost due to the potential for individual reuse. We can reapply aspects of the same mental models to many tasks.

Billions of semantic generators

There is so much focus on the complexity and intricacies of making our content semantic and machine-readable that a simple observation is often overlooked: All of this effort is aimed at meeting the needs of people.

Creating knowledge models of content is valuable, but only one side of the coin. We also need to create knowledge models for people as the consumers of content, to give a voice to their perspectives. But most importantly, we need to recognize that every person is a semantic generator; collectively, we are the ultimate meaning-making machines. How can we better devise solutions to put those billions of semantic generators to work on building the Semantic Web?

Notes

1. Problems with content-first: The problems of scale in a production-centric model are well documented. Creating semantic representations for existing digital content is prohibitively expensive due to the amount of online content and the compounding effect in the volume of data needed to create machine-readable semantics. Further, there is a need to incorporate personal semantics, representing the interpretations and viewpoints of the consumers of the content that further complicates the problem. Lastly, as compared with the models, standards, and protocols that preceded it, the Semantic Web has placed a much higher barrier to entry on producers, a stark change from the very accessible WWW.
2. Technical comparisons: While various technologies have an affinity with a consumer-first approach, they should not be confused as implementations of a consumer-first approach. For example, NLP may be used to transform a natural language statement into a formal, machine-readable structure. Under a content-first model, it may be used as a gateway into the exploration of existing knowledge models, while under a consumer-first model, it may be used in the creation of knowledge models. Similarly, tools such as ontology editors may be used by knowledge engineers or simplified for use by laypeople, depending on the overarching implementation strategy.
3. User interactions: One of the key challenges of this consumer-first approach in the field of knowledge representation is that tactics are needed to manage the complexity of the activity. This is a key differentiator from existing Web 2.0-style approaches, in which consumers are collaborating directly and with full knowledge of the activity. Here, we must leverage implicit and indirect means to relate consumer activity to knowledge representation.

This story reads much like the first Industrial Revolution. Artisans and skilled tradesman used to create everything by hand. Then, through the emergence of a handful of technical innovations, came the age of mass production. It was a profound turning point in human history, affecting every aspect of daily life.

Today, most content is still created by hand, the best of it by highly skilled artisans drawing on centuries of scholarship and experience. Recently, we’ve seen significant innovations in social approaches to content creation. But Web 3.0 industrialization takes content manufacturing to an entirely different level. Instead of users manually creating content, machines automate the heavy lifting. Consumers simply push the buttons and get stuff done. Think spinning wheels versus textile mills.

We’re in the midst of this Industrial Revolution right now. Billions are being spent worldwide on semantic technologies to create the factories and specialized machinery for manufacturing content. Railways of linked data and standards are being laid to allow these factories to trade and co-operate. And the most productive information services in the world are those that leverage Web 3.0 industrial processes and technologies.

Our company, Primal Fusion, is building one of these industrial Web 3.0 services. As an entrepreneur, I’ve ventured in Web 1.0, 2.0, and now 3.0. When I started in 1996, it took a team of highly skilled artisans many weeks to create a single website. Today, Primal Fusion enables individual consumers to build personal websites not in weeks or days, but in minutes, merely by brainstorming their interests. Accordingly, we’re measuring orders-of-magnitude productivity gains over Web 2.0 user-generated content models. (Note 1: Measurement)

This time, the Luddites among us won’t break the machines, but they will express skepticism about the quality of the content produced. Are machine-manufactured creations of a lower quality than those created by artisans? Of course, just like a mass-produced garment is of a lower quality than one that’s handcrafted. But who can afford handcrafted goods? The right question is whether consumers and producers can complete their tasks faster and cheaper than they can now. (Note 2: Examples)

And that’s the crux of this argument: The industrialization of content manufacturing is driven by the relentless pursuit of productivity advantages, not quality improvements. Once the industrial methods for manufacturing content reach a good-enough quality from the consumer’s perspective, the cost advantages will drive widespread adoption among consumers and producers alike.

This isn’t an apocalyptic vision, it’s just business. Productivity gains are steadily moving us towards consumer-directed, machine-powered content manufacturing. What many fail to realize is that Web 2.0 is far from the height of this model. In fact, it’s only the start. At what point will any consumer be able to push the buttons on the content-creation machines all by themselves? At what point will producers embrace industrial approaches and hand their customers the wheel?

So if not apocalyptic, this is certainly a disruptive transformation. Of course, highly skilled artisans are not going away, nor will their fine, handcrafted content. But demand will be affected, potentially marginalizing aspects of their services. However, this industrialization brings new opportunities, as well. Content professionals may choose to direct their skills to industrial methods, helping technologists design tools and developing innovative ways to include these tools in their services.

My colleague, Robert Barlow-Busch and I went to the Information Architecture Summit in Memphis last March to really dig into this topic. We presented an overview of how Web 3.0 technologies are disrupting the practice of information architecture (download slides, audio). There was also some excellent coverage of Web 3.0 topics by Chiara Fox from Adaptive Path and by Chris Thorne from the BBC.

So do people in the business of manufacturing content agree that we’re facing a fundamental change? Overall, I would say no. Many practitioners see Web 3.0 technologies as only tangentially relevant to what they do. They don’t view their work as manufacturing, at least not in an industrial sense, even as industrialists embrace these disciplines in the development of automated systems. (Note 3: Approaches)

But of course, if everyone saw this transformation coming, it wouldn’t be disruptive.

At the upcoming Web 3.0 Conference in New York, we’ll be keeping this discussion going. I’ll be participating in a panel alongside Eghosa Omoigui, Alex Iskold, and Greg Boutin, to discuss the state of Web 3.0 and Semantic Web technologies. If you’re in the area, I hope you can attend. And as always, please share your thoughts and ideas below.

Notes:

1. We’re measuring these productivity gains across a broad number of areas, including consumer participation rates in content creation, the amount of content created per contribution, customer value in terms of eCPM pricing for content created, and consumer value in terms of page views of content created.
2. A few examples of machine-synthetic approaches to content creation are listed below. Automated tasks include finding, selecting, describing, organizing, assembling, combining, and calculating content, all under the direction of consumers.
   • Google News: A computer-generated news site where the articles are selected and ranked by computers.
   • Kosmix: A “categorization engine” that organizes the Internet into magazine-style topic pages.
   • Wolfram Alpha: A demonstration of a “knowledge computation engine” that dynamically calculates facts in response to questions.
   • Primal Fusion: Example of a “thought networking” website created through a semantic synthesis technology.
3. There are many disparate fields that may intersect in the development of a Web 3.0 industrial system, including computer science, library and information science, cognitive science, interaction design, information architecture, and more.

My previous venture builds and manages large-scale communities. There we witnessed a constant churn of community members into smaller cliques. Even though the communities are focused on very specific interests, namely individual recording artists, cliques form around every topic imaginable, most having nothing to do with music at all.

Large networks like Facebook or LinkedIn face this fragmentation on a massive scale. But even the narrowest social network is not immune. Any service that’s organized around a static activity or interest will become fragmented as its membership grows. The reason is that the very organizing bases for social networks, the foundations for their existence, are constantly changing from within.

Sociologists have a good understanding of these forces. People want to form temporary social networks for specific purposes. We don’t keep friends and loved ones as a pool of network labour. Instead, we look to weak ties – acquaintances, friends-of-a-friend, and co-workers – to help us get things done. Underlying our “antisocial” resentment towards social networks is the conflict between strong, persistent connections and the true nature of our social relationships.

Today’s social networking systems were not built to support informal, ad hoc relationships. Large lists of “friends” or “connections” are much more a scorecard for one’s networking ability than any purposeful network of weakly associated people. The role of a social networking service is in mediating connections and providing a supportive structure. But if the organizing foundations for the network are constantly shifting, how do you provide the organization to support it?

The solution adopted by the large players is to become social networking infrastructure: focus not on the individual needs of each clique, but rather allow them to define their own organization and tools. But in so doing, they are necessarily ceding their role as a social networking service for the weakly tied, task-oriented networks that represent the bulk of our needs. This gap offers considerable market opportunities for any service that can reconcile the needs of dynamic networks with the benefits of structure and organization.

I’ve argued that thought networks, as a class of semantic networks, don’t need social networks to operate effectively. The corollary to this argument is that semantic networking illustrates one possible manifestation of extremely weakly tied and dynamic social networks. But whatever services and networks emerge from antisocial networking, the winners won’t be managing lists of “friends” or specific topics. Instead, they’ll learn to organize large numbers of highly specialized, task-oriented and fluid networks that don’t hang around long enough to become awkward.

Past: Connecting People

Jon Udell was addressing the social dimension of the Web and its powerful influence on knowledge acquisition. We don’t just interact with this “global encyclopaedia”, he explained. People discover each other through the intersections of documents they create. “As people expose aspects of their thought process in tangible form as documents, human connections are made.”

“Absolutely perfect,” replied Howard Bloom, but unfortunately, a terribly protracted process. “When we try to find each other, and try to find the knowledge we get from each other, these days it’s as difficult as getting from New York to California in 1848.”

Howard offered researchers some suggestions to address the problem, such as personal agents, completely under our individual control, connecting people and their knowledge. These agents would work continuously, whether we’re engaged as active participants in the process or spending our time elsewhere.

Speaking with the conviction of his personal experience, Jon claimed that this vision is well on its way to becoming reality. “I feel like the distance [between people] has been shortened dramatically already, and those agents are already out there working on my behalf all the time. The corpus of documents I’ve extruded to the Web creates a surface area for interaction. Any time another document touches one of mine, I notice it; my sensors fire.”

Jon was alluding to the asset that his years of publishing on the Web has produced: a network of his thoughts. But as Howard put it, “It takes us years to write the stuff that we write. We create condensed knowledge in a form that we could never give to you on the fly…We need to make sure that the connect time between you and me goes down to seconds, not months.”

Future: Connecting Thoughts

Note how these perspectives on knowledge-sharing networks are tightly, almost inextricably tied to connecting people. To connect thoughts, we must connect thinkers. This is a pervasive meme: “While some social network Web sites are becoming destinations, the next wave will thread social networking into a wide variety of applications,” according to Chris Shipley, product analyst and executive producer of the DEMO conferences.

However, for knowledge acquisition, the goal isn’t connecting people, it’s about connecting knowledge. Today, the social dimension acts as an intermediary in the process. It’s like going to a friend to get their input into a task and ending up spending the rest of the day hearing about their vacation. As social animals, it’s a welcome distraction to be sure, but also extremely time-consuming.

Thought networks are created by people and are ultimately in the service of people. But to interoperate, thought networks don’t need people. It’s this capability of semantic interoperability that promises to reduce the friction between our thoughts. For a really fast and fluid exchange of ideas, you can’t have people in the middle, slowing things down. People should perform those functions that only people can accomplish: creating truly new thoughts and making complex decisions.

This is reminiscent of a common pitfall with any disruptive technology: The technology is first applied to an existing process, rather than allowing the technology to augment and even displace the process itself. To explain, let me take you back to the 19th-century as Howard did. When the telephone was first invented, many assumed it would inherit the modus operandi of the telegraph. They imagined that people would dictate their messages to a telephone operator, for the operator in turn to verbally pass along the message to the recipient. At the time, the telephone seemed confusing, prone to failure, and needlessly complex.

Of course, the telephone didn’t need that antiquated process. Its new technology demanded a leap in imagination. Researchers and entrepreneurs in semantic technology need to leap, as well. If the goal is connecting knowledge rather than socializing, it is wasteful to apply the “telegraph” model of social networking to the “telephone” model of semantic technology. The two are complementary and easily integrated, but they are also independent.

How do we shorten the distance between our thoughts? By recognizing that thought networking is not semantically-enhanced social networking. This is the essence of pure and frictionless semantic networks. Semantic representations, the language of thought, provide a medium that makes thoughts tangible. And unlike simple documents, these highly structured models provide the ability for computers to “read” our thoughts and mediate across the thoughts of different people. Decoupled from documents and social networks, the connect time between our thoughts can go from months to seconds.

Present: What’s Next?

I’m hoping this document, this static representation of my thoughts, manages to touch Jon’s network and the networks of all who may be thinking along these lines. But I certainly don’t expect it to happen immediately. The process is presently slow and plodding. In the meantime, Primal Fusion and others are working on more dynamic, more sophisticated, and increasingly frictionless systems of thought networking.

If you’re interested in exploring these ideas with us, I encourage you to subscribe to our blog and register for our alpha project. We’ll make every effort to get you involved as quickly as possible.

Thanks to the team at Primal Fusion for their help with this post.

What would happen if you could instead make your thoughts tangible and concrete? What if you could collect your thoughts as readily as you can search online? What if your thoughts could self-organize around your tasks while you’re off doing other things? Thought networking is the idea that’s driving our efforts at Primal Fusion to explore these big questions.

What is Thought Networking?

Thought networks provide a concrete semantic representation of our thoughts, ideas and interests. Encoded as data, our thoughts are accessible to the power of computing. Semantic synthesis provides the means to expand and connect these thoughts in entirely new ways. Equally important, as structured data, thought networks may be used as inputs to software “agents” to automate much of the drudgery of our online experience.

Semantic representation. Semantic technologies are ideally suited to the task of thought networking. To borrow a phrase from Steven Pinker, semantic representations are the “language of thought”. Technically, a thought network is a type of semantic network. It represents thoughts as interconnected concepts. This lattice-like structure is how thoughts are represented as data and made concrete. Ideally, people will remain blissfully unaware of this deep structure as they interact with their thought networks.

Semantic synthesis. Semantic representation is only one aspect of a thought network. We don’t merely form our thoughts, we connect them. The word “thought” signifies a very elemental cognitive unit. We manipulate and compound these primitive structures to form ideas, arguments, and perspectives. Primal Fusion uses a semantic synthesis technology to fuse primitive concepts into these higher order constructs. Synthesis places thoughts in meaningful, task-oriented contexts.

Automation and tasking. This is where thought turns to action. With our thoughts collected and organized, we can put them to work: our thoughts may be collated with content and expressed as documents; they might traverse the Web to find related information; they might interface with social networks to connect like-minded individuals. Organized within semantic networks, thoughts have the power to direct computers on our behalf.

Benefits of Thought Networking

Ultimately, thought networking is intended to provide two main benefits. First, it will enhance our cognition. Our ability to store and recall information is extremely limited. Thought networks can help by expanding the number of thoughts at our disposal and organizing them effectively. This isn’t a comment on our cognitive abilities, any more than a calculator is a comment on our math skills. It’s merely illustrative of the way computing complements thinking.

Secondly, thought networks will save time and effort. Consider how much time you spend collecting and organizing your thoughts. Whether you’re writing a paper for school, planning a trip, or researching a medical condition, thoughts are the necessary precursor to action. If we can migrate some of that arduous process online, we can put computers to the task of simplifying your online experience.

Perhaps the best way to summarize these benefits is by way of an analogy to social networks. In social networks, we put our identity online. Our digital identities live and interact in social networks even when our attention is elsewhere. Other people interact with us virtually while we’re off doing other things. And similarly, we can keep tabs on the people we care about, regardless of barriers of time or geography.

Such are the benefits of thought networking. Once we digitize our thoughts and put them online, our thoughts may interact with the world even when we’re not attending to them. They become both an independent embodiment of our thinking as well as a powerful knowledge asset. Thought networks won’t displace thinking any more than social networks displace our socializing. But they will augment our ability to think and get stuff done.

Does this notion of thought networks seem fanciful? I expect for some it will. But think back five years: how many expected we’d push our socializing onto the network? Wasn’t Wikipedia an outrageous idea, as a collective representation of our knowledge? So while semantic technology provides for better search, better content management, and better data integration, it also provides an opportunity to make our thoughts tangible. In this, thought networking is simply another facet of ourselves – in our digital world.

Thanks to the team at Primal Fusion for their contributions to this post.