In my analysis I claimed that the NSA had to buy lots and lots of servers. Something like 1 million.
What friends of mine, correctly, pointed out is that most servers render data and do not storie data.
Which, of course, reveals my bias. At Zynga we don’t render the data for the user, we just process the data. . At other web companies most servers render the thing the user sees.
Mea Culpa.
The reality is that of the 1 million servers, for many web properties, only a fraction stores data. So let’s say 10% which is probably fair. That reduces the problem to 100k servers.
Except …
FB, Yahoo and Google are probably just one of the interesting places people store data.
They also store data on Box, DropBox, S3, EBS, tumblr, etc, etc, etc. Any application that stores data for sharing is a target for the NSA.
They also store data in Hotmail (now known as Outlook… Really … )…
The point is that you can easily shrink the problem down, and then I can easily grow it.
And then the interesting problem that the folks at the NSA have to solve isn’t just storing the data but finding connections across the data. Just the size of the data motion and data indexing boggles the mind.
The point is that this is a huge infrastructure.
And that the problems of management, scaling, operations remain real even before we get to the really interesting question of data analysis.
Now it’s entirely possible that there are researchers in the NSA that have solved all of big-data’s problems that the rest of us are working on. It’s possible.
And unicorns might exist.
Look if this is real, it means that my understanding of where the state of the art is, is about 10 years behind the curve. And if the US government has sat on this kind of advanced software, then the entire decade we spent figuring this shit out was … wasted.
And if they are that good, that means that entire areas of human endeavor could be accelerated if they gave that software away. Just think about what we could do with the kind of real-time analysis. What would we do if we could sift through all the data about all of humanity in real-time …
At the end of the day, I am having a hard time believing that the rest of the planet is 12 years behind some mysterious dark organization.
Is it possible? Absolutely. Likely, no.