But that is just the eye-candy. The substance is buried deep in the prose, under the heading “Data Management and Personalization Technologies.” Get a load of this:

“loading a user’s home page typically requires accessing hundreds of servers, processing tens of thousands of individual pieces of data, and delivering the information selected in less than one second. In addition, the data relationships have grown exponentially and are constantly changing.”

And then there is this:

“We use a proprietary distributed system that is able to query thousands of pieces of content that may be of interest to an individual user to determine the most relevant and timely stories and deliver them to the user in milliseconds.”

And this:

“We store more than 100 petabytes (100 quadrillion bytes) of photos and videos.”

And this:

“We use an advanced click prediction system that weighs many real-time updated features using automated learning techniques. Our technology incorporates the estimated click-through rate with both the advertiser’s bid and a user relevancy signal to select the optimal ads to show.”

But my favorite is this:

“Our research and development expenses were $87 million, $144 million, and $388 million for 2009, 2010, and 2011, respectively.”

So R&D expenses grew almost five-fold in three years. Considering Facebook had $1 billion in profit on $3.7 billion of revenue last year, the company’s research budget came to 10% of sales. This is very healthy (albeit natural, perhaps, with a company boasting such hefty profit margins). According to the OECD, the top 100 R&D-inteisve companies in the IT and telecoms sectors spend an average of nearly 7% of revenue on R&D.

Most of the fruits of the R&D is probably kept internal and covered under trade secrets. But for that generous sum, the prospectus informs us:

“As of December 31, 2011, we had 56 issued patents and 503 filed patent applications in the United States and 33 corresponding patents and 149 filed patent applications in foreign countries relating to social networking, web technologies and infrastructure, and related technologies. Our issued patents expire between May 2016 and June 2031.”

But the most interesting thing is how much was not exposed in the prospectus. In a section were Facebook purported to explain its analytics, with an example of how it uses elements on a webpage to determine what ads to show (page 87), the example was so juvenile as to be meaningless.

It is actually funny the way Facebook keeps quiet on analytics, considering that the first time the word appears is on page 12, when Facebook cites it as one of the “risk factors” that could ruin the business:

“our inability to improve our analytics and measurement solutions that demonstrate the value of our ads and other commercial content”

Though it is loath to make too much of it, since it is its main source of value, Facebook is an analytics company before anything else. Google might have been the world’s first big-data IPO. Facebook may be the first analytics one. But you wouldn’t know it from its IPO prospectus.

The reason is that the idea of quantification is really a phenomenon of the Middle Ages in Europe (laying to rest the old canard that they were “dark ages” devoid of progress). On the other hand, the period of antiquity is typified by man describing his world as one of qualities. (Remember Socrates’s “forms?” And Aristotle’s taxonomy on just about everything?)

To be sure, in the area of money we can talk about quantification and thus data as we think about it today. But in many of Pete’s terrific examples of how the Sumerians recorded their world — in the “fixed media” of clay tablets and the like — I am unsure if the term data fits.

Ought “writing” be considered data? If so, how about caveman paintings? Surely the Egyptian hieroglyphs imparted information — but should we call it “data” per se? The only way to answer that question is to define data.

The word data is the plural of datum, neuter past participle of the Latin dare, “to give”, hence “something given,” instructs Wikipedia. “1. Facts and statistics collected together for reference or analysis. 2. The quantities, characters, or symbols on which operations are performed by a computer, being stored and transmitted in the form of…” reports a Google definition.

Building on the idea that data may be something different than just recording information, at what point does something go from being simply info to data?

I have a few ideas on how to answer this — I am scribbling away on a large work that looks at this topic among others. But I’m not quite ready to share it with the world, since the thoughts are still fermenting. In the meantime, Pete’s post is a wonderful look at how an early society recorded and used information. Among my favorite points:

* “Written records remove the problem of fallible memories, but replaces it with a second-degree question of provenance. How do you know the data accurately reflects what happened?”

* “We still have a disturbing tendency to trust anything that’s recorded, without understanding the subjective process that went into creating the record.”

* “The main way Sumerians protected the integrity of their data was through curses. This may seem laughable to a modern audience, but I don’t think we’re so different. Do you expect the FBI to actually raid your house if you copy that VHS tape?”

* “In the absence of real answers, we’ll take bogus ones painted with a veneer of data, just like the Sumerians.

* “If there’s any way you can, please think about how to open up data you control, it’s the best way to pass it on to posterity.”

Having pointed out what I enjoyed most, let me close on a final quibble. Pete writes:

“The Sumerians recorded everything on stone or clay tablets … This data exhaust gives a rich view into trade, worship, life, death, medicine and almost every other aspect of the Sumerian’s world.”

It is absolutely not “data exhaust” in the way that the term has come to be known (and how I helped popularize it in a report a few years ago). The idea was information provided as a byproduct of interacting with information that itself could be collected and analyzed. The simplest example is tracking readers activities to reveal to website visitors the most-read articles, as a simple heuristic to indicate what might interest them.

What Pete describes, and what the Sumerians recorded, was information (or perhaps data) pure and simple. No “exhaust” about it — other than that the tablets had been thrown away by the Sumerians before modern archeologists dug them up.

But all this ranting is only meant to add momentum to my appreciation for Pete’s splendid work in this post and others!

According to a new US government policy, lawyers representing Guantanamo prisoners are allowed to read Wikileaks’ classified US documents — but not print or save them. The actual policy “guidance” is here (from Politico) and an analysis by Politico’s Josh Gerstein is here.

Are the US officials that devised this policy out of their minds? How could anyone rationally adopt such an inherently inconsistent policy?

If the lawyers cannot read the material, they are blocked from accessing pertinent information that is already in the public domain, which could help them prepare a defense. Allowing access is only sensible. To do otherwise would be to deny reality (that the material is widely available), and might deny justice too.

However, crippling that access by placing arbitrary restrictions on its use make no sense whatsoever. Why? On what basis is one allowed to read but not print or save? Surely the US does not mean for the frailty of a person’s memory to govern how material is put to use. But that is the policy’s effect.

The irony is that the current policy is actually a slightly more rational shift from previous rules that forbid any access at all. It underscores the fact that the government has no clue how to respond to the new world we’re in regarding BigData leaks.

And it is a longstanding problem. Just this month, the US officialy released the trove of documents known as the Pentagon Papers — 40 years after they appeared in the New York Times. (The AP’s story is here) The Economist, in an article last month about it (“The open society and its ostriches“) argued that the way to think about these cases is that “the illegal disclosures in effect declassify the information.”

When the contradiction between futile policies and the reality on the ground grow so wide as to be preposterous — as it is now — something has to give. It will be the rules, of course, that go. But with government, this takes a long time.

Here’s a compilation of the tweets (followed by a quick analysis relating it to Sony’s Stringer on security):

* * *

Top 10 signs you no longer have privacy and should get over it:
10. The guy behind the McDonalds counter greets you with, “Would you like a salad to help you with your constipation?”
9. A Google search on “white only clubs” has just one result: TaylorMade.
8. Your soon to be ex-spouse produces your iPhone GPS database in settlement hearings.
7. The TSA stops molesting and radiating your 82 year old mom because she is clearly not going to hijack that plane.
6. 20 neighbors show up at same Groupon inspired Spearmint Rhino happy hour in Vegas.
5. IRS starts auditing folks who don’t pay income taxes, not the folks who pay the most.
4. Local police become largest purchaser of camera equipped UAV’s.
3. Your parents require your Facebook, laptop, and phone passwords and actually review your online activity regularly. And you are 40.
2. The UPS driver delivers your small package to your door and, with a smile and wink, asks if you would like batteries with that.
1. Twitter starts suggesting Tweets for you, and they are perfect and better than your own.

* * *

As in 1999, McNealy is right on fact, wrong on what to do about it (as critics argued at the time). Not ensuring some protections is irrational. But whether he’s right or not is beside the point. It is refreshing when a top executive calls it as he sees it — and a bit silly when people quibble with the wording rather than the larger point itself.

Here, I’m thinking of Sony’s boss, Howard Stringer, who recently described the PlayStation Network hack is words that was sure to eviscerate him among tech journos. “Nobody’s system is 100 percent secure,” he said in a conference call. “This is a hiccup in the road to a network future.” (in Bloomberg’s piece). “It’s not a brave new world; it’s a bad new world,” he said (in the WSJ piece).

Stringer has been pounced on by some in the press. He shouldn’t be. Though the point he raises we’ve known for a long time, it is still quite right.

An upcoming EU report will say that “geo-location data has to be considered as personal data… The rules on personal data apply,” an EU official tells the Wall Street Journal.
The implication is that data collected by cellphones, twitter, Facebook and others must be handled like names, birth dates, and other personal information: requiring user consent, deletion after a certain period, and kept anonymously.

This is absolutely preposterous. Yes, rules — better, tougher rules — are desperately needed. But to simply drop the data into a pre-existing regulatory bucket (as the EU is doing, of calling it “personal information” which has sweeping regulatory burdens) is asinine. It will hold back the amazing innovations and services that are just starting to emerge, and future ones that we can scarcely imagine today.

Calling something new (geo-location data) something old (personal identifiable information, or PII in the trade) is a far too blunt way to go about upholding legitimate public interest concerns that need to be addressed. It avoids the more humble — and probably more effective — task of trying to figure out the new properties of this type of data, and thus devise appropriate ways to balance personal privacy with innovative services. It’s harder to do this, but sounder.

This of course will happen, but over time, and probably in a different regulatory jurisdiction. Possibly America? Perhaps China? Maybe Brazil? But European geo-loco firms will suffer in the meantime, since they’ll crammed into a regulatory straitjacket. And to be clear: this is not to say that better rules aren’t needed — they definitely are. But they ought be sensible ones.

Failing to take a more cautious and reflective regulatory approach results in things like the EU’s 1998 privacy directive. It did an excellent job of getting governments into the privacy arena, but it had lots of silly parts too. For one thing, it required an international “safe-harbor” provision in order to do innocuous things like allowing a US firm in France to send its payroll data to headquarters in Detroit. The rules are already out of date, and although it boasts strong enforcement provisions, they’ve barely ever been used.

In fairness, the US has miserable privacy legislation — no country does it well — but the piecemeal approach and building up of a body of regulatory experience is looking like a better way forward. There is no “privacy kommissar” in America, but that hasn’t stopped the FTC from taking serious action often.

A far better way to proceed is the way the US is moving. Sen. Al Franken’s opening statement to hearings on May 10th on cellphone privacy was a paragon of wise policymaking: he wants to find the right balance. He was scorching in his condemnation of current practices:

“Once the maker of a mobile app, a company like Apple or Google, or even your wireless company gets your location information … these companies are free to disclose your location information and other sensitive information to almost anyone they please-without letting you know. And then the companies they share your information with can share and sell it to yet others-again, without letting you know. This is a problem. It’s a serious problem.”

But at the same time, he understood the risks of regulating too soon:

“I just want to be clear that the answer to this problem is not ending location-based services. No one up here wants to stop Apple or Google from producing their products or doing the incredible things that you do. You guys are brilliant. When people think of the word “brilliant” they think of the people that founded and run your companies.”

If this gap in regulatory approach is not settled, the result may well be another round of the privacy wars. Companies like Apple, Google, Facebook, Twitter, Foursquare and others will have to tailor their operations depending on jurisdiction, down to their very code base. The EU will argue that they have to do this any way for language and law. But this still fractures and debilitates the service. And it is hypocritical: the idea behind the EU’s common market and common currency is about the gains from harmonization.

The best way to ward off bad public policy are good case studies of excellent services. Industry basically has its head in the sand hoping this issue will go away (it won’t) or is in hiding, hoping it doesn’t need to disclose how the services work (it does). Their actions are shortsighted. Geo-location services are interesting and useful, and if people really knew what was happening, many would be fine with it, provided a backstop of basic protections exist.

The case must be made publicly. So what are the amazing new services that are emerging that show why the EU’s approach is not quite right? Share your stories here.

The problem is that the amount of information has expanded so much that it has become almost impossible to work with or comprehend in its totality. But new techniques that actually rely on the huge scale makes that huge scale manageable and indeed useful. And though it is certainly an overstatement to say it “solves the problem,” I’d argue this the right way to think about it.

Examples abound, if we look at things the right way. For example, computer translation was hard until the “test data” went from the billions to one trillion words — and then the machines got talking (as Google’s Peter Norvig explains here and Steven Levy tell here). Likewise, Jeff Jonas of IBM recounts a situation years ago when by adding more information to a database of people, the number of records on individuals actually shrunk: he was able to identify and consolidate duplicate entries.

But the inspiration for these musings is an article in last week’s The Economist, “The science of science: How to use the web to understand the way ideas evolve.” Researchers came up with a clever way to identify and classify texts by grasping meaning from their content, outside of what the authors felt the classification ought be.

This lets machines parse huge volumes of text that people can’t do, or can’t do well. Academic authors label the subject areas of their papers, but sometimes use far too many as a way to trick people into reading it, or are limited to just five labels which may be too narrow. Sometimes they are required to use pre-determined labels from library science, which fails to account for emerging areas of scholarship. So for example, Adam Smith never regarded himself an economist — the term didn’t exist in that context — rather, he was a moral philosopher. This system would place him alongside Malthus, a pastor by trade and demographer by study, who incontestably wrote on economics.

Moreover, the system enables one to see how ideas molt and meld over time — just as Smith and Malthus seemed out of step with their “professions” in their time, but were foundational for the new field of economics. And it bears repeating: the reason the technology described in the article works is because there is enough data to make inferences about meaning. As the article states:

“Citation indices, which work only where publications refer to their sources explicitly, form a tiny nebula in the digital universe. News articles, blog posts and e-mails often lack a systematic reference list that could be used to make a citation index. Yet they, too, are part of what makes an idea influential.”

This opens up new areas for researches to amass sources. For instance, the huge area of “gray literature” (as it’s called in library science) that is slightly outside the mainstream publication world is now more easily retrievable and citable.

It also indirectly overcomes Google’s inherent shortcoming. Google’s PageRank algorithm, at its most basic level, counts inbound link akin to academic citations and presumes that a page with more is more relevant. But basing relevance on link structure invites imperfection because ordinary people are themselves imperfect and may not link to the ideal content, thus creating suboptimal search results. The technique described in the article may help remedy this.

The upshot is that we are generally familiar with the idea that a characteristic of big-data is it seems to exhibit “inverse scaling features”: the more data you add, the better the system gets (rather than deteriorates, as most systems do when under more load). But another step ahead of this point is that “big-data solves the problem of big-data.” With so much info around, the only way to tackle it is to use its huge size to sort itself. This idea sounds like a serpent eating its tail — but it may be more than that.

I recently scribbled a few thoughts about the data dimensions of responding to Japan’s crisis for The Economist’s website: “The information equation” on April 24th. I was impressed that a private-sector company was playing the role that a governmental organization or NGO might play. (It’s a Google.org project, to be exact.)

Among the things I learned was that Google collected $5.5 million in donations through its crisis-response page. A small but not insignificant haul. But it got me thinking. The world of BigData is about learning new things from information that is otherwise invisible to the naked eye. What could the donation data tell us about how to more effectively solicit charitable contributions? Specifically, as I wrote in the penultimate paragraph of the article:

The donation data may offer a chance to learn new things about how people contribute. For example, what is the average amount? Does it follow a standard normal deviation (ie, a “bell curve”) in which a few give a little and a lot, with the majority donating around $15? Or is it a power-law distribution, in which there are two or three extremely rich donors, a handful of generous ones, followed by a long tail of $2 contributions? Did they donate using PayPal or credit cards? What time of day do people give? Is it after they have read a news story or clicked a link within an e-mail? The information would help fundraisers tailor how to make their appeals. And the data can be broken down by country or even city via Internet Protocol addresses.

I’ve asked Google’s hyper-helpful PR team to run the idea past their number-crunchers, to get access to the findings so I can write a story about this. It’s sort of like Google Flu Trends, but for charities. It would be highly valuable information for NGOs to know — particularly one that is dear to my heart, International Bridges to Justice (where I proudly serve on the board).

This is a fictitious example (and one imagined by Mike Driscoll of Metamarkets). But it helps to concentrate the mind on what is new about the BigData revolution taking place, and how information cleverly reused can create new sources of economic value. And this leads me to thinking about the Wall Street Journal’s excellent article “The Really Smart Phone” by Robert Lee Hotz, which is part of the paper’s impressive series “What They Know.”

There’s much to praise in the piece. Instead, I want to put forward some vital distinctions that industry needs to consider, when thinking about some of the trends happening.

First, we need to separate the process of BigData from its output. The article — like the industry — doesn’t really do this. For example, sometimes we talk about being able to track 100 million cellphone users (but don’t note the substance of what is being tracked: calls? location? bills?) And sometimes we talk about what we learn, such as a person’s susceptibility to obesity. But is it because location data shows they’ve been sedentary? Or because they bought lots of ice-cream from their iPhone?

These distinctions are crucial. In one instance, it is anonymized metadata, in another it is individual information. The ways that entities are allowed use these different types of data perhaps ought be different too.

Most people, and most articles in the press, approach the BigData issue from the negative: “if you only knew what they know about you!” But I believe that industry ought be far more transparent because if people did know, they’d probably be more impressed than alarmed. (It is a point that I made in my special report “The data deluge” in The Economist last year.) Specifically, what is so unsettling: what they collect? Or what they know? On the surface, we bristle at both. But when we look deeper, it gets fascinating to see what new things can be learned from a big body of data.

Hence, the transition from “Ick!” to “Wow!” But I think the failure of industry to be open about its practices will hold it back. Thus: the public will cry: “Wait!”

People are antsy. Regulators are uneasy. And business is barricading itself. Amazon never discusses it. Google does — and thus invites abuse, alas. Apple is characteristically silent. “Google Inc. defended the way it collects location data from Android phones, while Apple Inc. remained silent for a third day,” the WSJ wrote in a separate article in April 23.

I think with the right outreach, BigData firms can make the case for collecting and processing the information. It will change the debate to the more essential questions: who owns the information, who gets to benefit from it, how it is valued, how it is protected and what are the penalties if this trust is abused?

This further establishes the distinction I identified at the outset: separating the process from the substance. In fact, we are talking about so many records — millions of people, zillions of data-points of locations or calls — that the practical effect seems to be anonymization, even if is not done effectively in practice.

To overcome the “Wait!,” I’d urge that we make rules. As for a starting point to think about them, I’ll discuss another time. For now, a look at what the WSJ piece did a nice job of highlighting. The large numbers associated with the research was interesting, but actually unimportant — they’re just big numbers; the process. The actual output of what is to be learned is far most interesting. Specifically, cellphone BigData lets us:

- pinpoint “influencers,” the people most likely to make others change their minds

- forecast where people are likely to be at any given time in the future

- predict which people are most likely to defect to other cellphone carriers

- reveal subtle symptoms of mental illness

- foretell movements in the Dow Jones Industrial Average,

- chart the spread of political ideas as they move through a community

- expose a cultural split that is driving a historic political crisis (in Belgium)

- deduce that two people were talking about politics

- detect flu symptoms before the students themselves realized they were getting sick

A final note: all of these insights were gleaned by parsing two types of data: location and interconnections among users — metadata. There is a lot more mobile data to collect; we’ve barely scratched the surface. Also, nota bene that none of the data relates to specific content from the phone or user. And it is not clear that the data collected can be traced back to a specific user, other than in cases of academic research in which consent was granted.

In some ways, the data collected looks like the “pen register” information that is less spooky than an outright wiretap: eg, who calls whom and when, but not what they said. It has a lower standard for law enforcement to obtain.

My point is this: the BigData issues we’re confronting now are the easy ones. So this is the moment to start thinking about seriously debating them, and arriving at answers — as a precursor to the harder issues coming down the pike.

As I said this, his look changed from one of polite indifference (and mild pity), to genuine curiosity. Now, I had to explain myself. What I blurted out what this:

The world is becoming data-ized as digital information and numerical measurement is being applied to all aspects of what people do, particularly things that couldn’t be measured before because it was impractical or impossible. (Think: using wireless and GPS in cars to base insurance premiums on where and when people actually drive, as has been possible since 2007.)

The impact will be as profound as the scientific method in the 18th century — which quickly moved past the sciences and left its mark on all areas of human endeavor. For instance, what is “quantitative decision making” in management, if not the scientific method applied to business…. Likewise, the BigData revolution is plowing through the sciences, and also jumped into mainstream areas, such as business and government.

My dinner companion got it.

It is not easy to describe what is happening and why it matters. But parallels with the past, albeit imperfect, are usually very useful. Hence, trotting out the scientific method to explain the here and now.

One possibility comes to mind: perhaps shopping patterns are so statistically consistent, routine and as personal as DNA, that information about a person’s previous purchases — or even non-shopping activities — enables an algorithm to know if the customer is truly the person he or she says.

That is interesting. But, alas, the report looks at more prosaic things, ie:

PayPal, Amazon, and Google have all developed sophisticated analytical tools and infrastructure to identify patterns of fraudulent activity. Paypal, for example, has a series of Fraud Management Filters that screen payments and sort out transactions that warrant review because of their amount, their origin, or other factors that can be set by a merchant. [...] PayPal and Amazon have developed fraud detection tools that depend on massive datasets containing not only financial details for transactions, but IP addresses, browser information, and other technical data that will help these companies refine models to predict, identify, and prevent fraudulent activity. PayPal and Amazon have had years to amass databases of the transaction details for hundreds of millions of customers across thousands of merchants.

The sort of filtering and checking described above (bold emphasis mine) involves no conceptual shift in how to use data. All that is being described is doing the same intutive techniques that one would have long done in a world of “small data.” The only thing “big” about it is that there’s a lot more data to sift through. But the firms are not using the size and depth of data to do anything novel per se.

This is a pity. The revolution that is taking place in other dimensions of the Internet industry is that companies can do entirely new things with a big data set that they cannot do with a small one. A former top Google executive once told me that Google Checkout was created in part because the firm realized that learning about a customer’s shopping pattern could better detect fraud, which is the key e-commerce stumbling block.

Likewise, at the O’Reilly Strata conference in February, hallway chit chat was about how a financial services firm might be able to more accurately predict whether someone will repay a loan using Facebook’s social graph than a FICO score, since best predictor if person will repay is if their friends repay their loans. (Actually, the example was told to me as if it were already being done, though not with Facebook’s data). Yet I think I’m safer considering it apocryphal until I hear it first hand.

Does anyone know of incredible stories of how “big data” is being used in new ways to reduce financial fraud? If so, comment here or email me directly.