SAN FRANCISCO – The unthinkable has happened: Twitter has decided to make money. Longtime users of the microblogging service, which for years has operated without a viable business model, are anguished at the prospect of paid ads appearing among their tweets. But advertising is just the tip of the iceberg. Twitter’s vast and ever-growing data store will be the true profit center, say company execs — both for Twitter and for independent developers.

Exactly to what extent Twitter plans to make its data available to outside parties remains unclear, but the company’s APIs are already accessible for developers to access its services, and last October it signed deals with Google and Microsoft to allow tweets to appear alongside search results. Now Twitter is reportedly developing “analytical products” aimed at marketers who want to mine the Twittersphere for insight into public opinion about companies, products, and brands — and it’s encouraging others to do the same.

[ Keep up with app dev issues and trends with InfoWorld's Fatal Exception and Strategic Developer blogs. | Follow the latest news in software development with InfoWorld's Developer World newsletter. ]
In so doing, Twitter joins a growing number of online companies in creating a new kind of software platform. In the past, tool vendors have offered developers languages and code libraries that gave them access to computing functions in simple, standardized ways. In this new paradigm, however, a platform consists of more than just frameworks and APIs. It also comes prepackaged with a complete, rich data set, and often that data is the platform’s most valuable aspect. These new “data platforms” are creating exciting new opportunities for developers, though they are not without their challenges.

Data Mining Like Never Before

It’s easy to scoff at the idea of Twitter as a source of valuable data. Among consumers, the divide between the tweets and the tweet-nots is deep, and those who do not partake see little of interest in a service that allows users to post 140-character status updates. But Twitter is not alone in thinking it’s sitting on a goldmine. The U.S. Library of Congress recently announced plans to archive all public tweets since March 2006, presumably as an aid to future anthropological research.
The Twitter archive will add to the growing collection of data the U.S. and local governments have begun offering to the public. The most prominent example is Data.gov, a clearinghouse for data feeds from federal government agencies that was launched by the Obama administration last year. The site has already spawned a small commercial ecosystem as vendors offer number-crunching and analytical services to help companies and lobbyists make sense of the raw data.
Of course, Web apps that draw from online public data sources are nothing new. So-called mashups combine disparate data sources in novel ways — using Google Maps to plot sightings of the McDonald’s fast-food chain’s elusive McRib sandwich, for example. But companies are only now beginning to realize that high-volume Websites, such as Twitter, present opportunities to conduct data mining on a new and unprecedentedly massive scale.

Previously, only large retailers such as Amazon and Wal-Mart or portal providers such as Google and Yahoo have had access to such large data sets. So far these companies have held their cards close to their vests, and for good reason. Their customer data is a key component of their competitive advantage. Now Twitter and others — particularly, social networks such as Facebook and MySpace — are banking that smaller vendors are willing to pay to level the playing field, and they’re developing data platforms to make that happen.

Selling Out Users’ Privacy

Selling access to raw user data is not without peril, however. Privacy concerns are growing among users of online services. Earlier this week, regulators from 10 countries sent a letter to Google CEO Eric Schmidt urging his company to collect only a minimum of personally identifying information and to ensure that the data was protected by adequate security measures. Failure to do so, the letter implied, could threaten Google’s ability to legally conduct business in the co-signers’ countries.
Data platforms will likely make enforcing security more difficult. Twitter believes its data store contains information valuable to marketers, such as how much buzz a new movie premiere is getting or how successful a viral ad campaign has been. But Twitter users who assume nobody is listening except a few friends and family might also unwittingly tweet information of a more sensitive nature — their children’s names and where they go to school, for example. Recently, an activist site called PleaseRobMe demonstrated how easy it was to scan for tweets announcing that Twitter users had left their homes. Developers will need to tread carefully to avoid potential liability for similar searches.
Equally worrisome to many consumers is the ability to cross-reference multiple databases to produce a much more thorough profile of an individual than any one source could provide. Even if a data platform’s API makes only a limited amount of information available, there’s little to prevent developers from querying multiple sources and matching corresponding fields. If these and similar privacy concerns cause users to quit online services en masse, plans to launch new data platforms will have backfired.

Where Do Developers Fit In?

Data platforms present other concerns for developers, however. Even more than mobile platforms such as Apple’s iPhone, a data platform like Twitter’s is a walled garden. If Twitter cuts off a developer’s access to its data sources for any reason, that developer’s business is sunk. Rather than risking public exposure, then, the safer bet for independent code shops may be to develop “cloud middleware” around data platforms, including tools that facilitate unique kinds of analysis not foreseen by the original data vendors.
One problem with this approach, however, is that it inevitably pits developers in competition with their data platform vendors. For example, Twitter’s “analytical products” could overlap those built by its developer community, and Twitter may be in a better position to reach potential customers.

But this risk might not be as great as it sounds. Like many companies that have only recently begun reaching out to developers, Twitter’s relationship with its developer community is evolving. For example, rather than create its own iPhone client, earlier this month Twitter acquired Tweetie, a leading independent iPhone app for the platform. Just as Microsoft has purchased many startups over the years, being acquired may prove to be a viable strategy for developers offering novel products built on data platforms. The field of data mining social networks is yet very new; it’s a good bet there’s money to be made helping companies sort it all out.