One of my favorite South Park episodes of all time is “The Simpson’s did it”. In that episode the creators of South Park enumerate every stupid asinine plot device the Simpsons ever used.
In the tech industry in the 1990’s we had a saying “IBM did it”. In other words, no matter how great an idea you think you might have had, IBM had already done it.
So I was tremendously gratified when a buddy of mine pointed out that NetApp did it in response to the Xorbas paper.
Here’s the patent and the key elements of the paper:
“Specifically, a single diagonal parity block is calculated across a series of row parity groups. Upon the failure of any data blocks, each in an independent row parity group, the invention enables recovery of the data blocks using local row parity. Upon the failure of any two blocks within a single parity group, the invention facilitates recovery using a combination of local row parity and global diagonal parity calculations. Advantageously, the present invention allows more efficient (and easier) recovery of single failures in an array configured to enable recovery from the concurrent failure of two storage devices (disks) within a sub-array of the array. “
The good news is that the all of the the patent claims are written in terms of row and diagonal parity, protecting against exactly two failures.
Intriguingly when NetApp was looking at this technology we were focused on how do you make disks more reliable. What makes the Xorbas work fascinating is that they focused on how to make bricks (disk+compute) more reliable.
One of the objections to bricks, at the time, was that it was a very expensive product given the cost and performance of disk, CPU and memory. That kind of architecture would make sense if the compute was significantly more intensive than just file system operations. Otherwise you were spending a lot of money to add a small amount of disks compared to adding a shelf.
And the reality is that Isilon wasn’t quite as big a success as NetApp feared.
As a modest reflection on the past, at the time, I remember folks wondering how we could bring compute closer to the disks. And we had these ideas about vmware and virtualization and missed this whole big data thing entirely.
What we really missed was that the database layer was going to change radically to a scaleout architecture where a clustered file system that made disk more cost-effective and reliability more cost-effective would be tremendously valuable.
But we didn’t. And I was there, involved in those discussions and I was as blind to the opportunity as anybody else.
We were too focused on disk drives, not on the changing nature of compute. And missed how our technology could apply to that changing nature of compute.
So like IBM, NetApp did it and someone else will capitalize on it.