"...since we were using low-end commodity storage that was by no means enterprise-quality."
If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? And more importantly, why? Why do "enterprise grade" products even exist if the largest and most deep-pocketed corporations don't find them to be good value propositions?
Choosing between "enterprise" vs "consumer" products is a lot like choosing between getting an Dell desktop vs building you own - a choice many of us made during our early IT years. The former gives you a reliable and predictable product, but it might be slightly lagging in tech or come at a significant premium. The latter gives you full control over your specs, but you carry the burden of fixing it and making drivers work together and that requires high tech skills and overheads.
Enterprise hardware aims to be predictable: large businesses have to minimise risks so they value stability and clarity in performance, costs, and technical know-how. Enterprisey products are never cutting edge or best-in-breed or efficient, but they are predictable and that helps large businesses to have a reliable foundation for planning and executing IT projects over 3-5 years with known budgets, risks, support contracts etc. This predictability is important for businesses that are not based on IT (for example, making frozen foods or assembling tractors).
Facebook/Google/Amazon and other software-centered businesses depend on agility and finely-tuned machines - they need their "custom-spec" equivalent, and they have the skills required to manage it. But many other companies have known needs and are happy with standard enterprise products - they are the equivalent of an office worker that needs a machine to run MS Office, and a standard machine from Dell/IBM/HP is a more appropriate choice for them (rather than tinker with building their own).
The original poster was asking, if I can reword properly: Who is using enterprise-grade hard drives instead of consumer-grade ones, if FB doesn't (and why)?
Both consumer- and enterprise-grade hard drives use standard interfaces (SATA, etc) so all this "drivers" stuff you talked about doesn't apply to this particular hardware. Since even enterprise-grade hard drives will fail one day, one needs to have a proper backup strategy, which would backstop the needs of supporting of consumer-grade HDDs. So in light of this, why buy enterprise HDDs?
(I'm not saying one shouldn't, I'm just not buying the explanation above :])
It is a lot like the pets vs cattle comparison. If your underlying structure can tolerate larger amounts of downtime the extra cost of enterprise equipment. For many applications it is easier to buy more expensive hardware that has exceptional uptime than it is to hire people to rewrite and take advantage of cloud based options.
Enterprise hard drives use SAS connectors to allow redundant paths and a full scsi command set. Enterprise hard drives have a bit error rate an entire order of magnitude better than consumer hard drives. The rated powered on hours and mean time between failure will be better, but not more cost effective with enterprise gear unless you take into account the labor cost and downtime cost of replacing that hardware.
I feel that explanation answers the question perfectly.
It's not about the fact that the hardware looks and interfaces the same, it's that it lasts longer, performs reliably and predictably and comes with support. Even if you design for failure, it's much better to have 1 drive fail than 3 in a system.
The enterprise stuff will just be more durable and usually more performant in every way so it's worth the cost, especially when most companies are not tech giants with all the engineering talent to build something like cold storage from the ground up. FB/Google/Amazon/Microsoft/Apple are very specialized high-tech companies who know how to do this and not representative of most "enterprise" customers.
If you are doing RAID, especially on a hardware RAID card consumer drives generally don't support TLER. This means a consumer drive will sit and try to read a bad sector again and again for long periods of time. You'll notice this on desktops when a drive is getting bad sectors that it will pause 30 or 40 seconds at a time. In a RAID setup it's better to fail fast and mark the drive bad for replacement, but the consumer drives don't send the hardware/OS the information it needs to do it quickly.
Most enterprise drives come off the exact same production line too.. Some have a different external interface but the important parts are all nearly the same, they might test them differently
Lets say you enterprise division/company wants to store 100 Terabytes of data. You ring up Netapp and buy a solution for say $100,000 and plug it in. You get support, lots of documentation and it is usually fairly reliable.
Now if you are Google or Facebook you want 100 Petabytes of storage but instead of paying Netapp $100 million you employee 20 really smart guys to build you something, you put it on cheap hardware and you build exactly what your software will talk to and what it needs.
It costs you $10 million up front but the hardware cost is just $15 million because you use cheaper drives and build (thousands) of your own servers.
Of course this doesn't mean that "Enterprise Storage" isn't going to be the next premium product hit by Open Source type solutions.
I guess it depends how close to bare metal you want to get.
Google/Facebook/AWS (and maybe a Backblaze) have architected their data center setup to not require the functionality that Netapp provides. If you buy a NetAPP and plug it in your netork it is because you cant architect a datacenter from the ground up...
Facebook was a founding member of the Open Datacenter Alliance. They have a cool githup repo and put out interesting whitepapers like:
It is not just cheaper but architected better. They would use Cisco and other enterprisey stuff if it were better but it is a case of "il y a moins bien mais c'est plus cher" as the French Linux motto goes...
Which is not to say NetAPP or Cisco is bad - just that you can engineer the datacenter to make it work better without enterprise functionality.
As an object lesson in failure, I had an experience with my first buildout of Hadoop back in 2011. Not so successful because I bought all the wrong components. Raid cards, 10G networking, HP and Bladenetworking rack switches, dual power supplies.
It was hard to unlearn how I was used to setting up servers.
Wow - still a bit painful to remember my first go around with this. What i am about to admit is damning enough to keep me unemployed for the forseeable future so hopefully no hiring managers are reading this...
I was a sysadmin and did the datacenter buildouts for the company. The task was to take the dev setup in Softlayer (on bare metal servers) and reproduce it in our racks.
The error was to take this somewhat literally. The bare metal servers that we leased from Softlayer were 2u boxes with redundant power supplies, Raid controllers and 10g networking.
I found a Supermicro configuration that almost identically matched and ordered 30 datanodes; 2 name nodes (this was back when Hadoop used the curiously named primary and secondary name nodes); and a server to launch jobs from.
We already had a Netezza in place and since that used Bladenetwork switches I decided to use the same as TOR switches with a beefier on as the Aggregation switch.
We decided to use Ubuntu as the OS and the Cloudera packages - but without the support or the console.
Every single one of those choices were mistakes. It is remarkable in hindsight that it worked at all.
The mistakes:
1) starting with the power supplies. Since they were redundant it dictated an A/B power setup in the racks. What this means is that you cannot use more than 50% of the power density because the entire rack is set to fail over. Each PDU has to be able ot keep the whole rack up so it alarms at 40% capacity. My using redundant powersupplies I was more than halving the amount of power i had at my disposal.
2) RAID. I hadnt yet read the excellent Hadoop Operations O'Reilly book which tells you why RAID is a bad idea for hdfs. Further buildouts included ripping out the RIAD card and JBODing the drives allowing Hadoop to properly use the raw disks.
3) 10G networking - because more is better right? Later after we hired talented networking folks I learned a bit about how important caching is in switches - particulary for Hadoop. My expensive monster Bladenetwork switches were falling over because the bursty traffic would saturate the switches. In addition I had the networking setup for a much more conventional network and we were doing the switch aware stuff in the config but had a wide open /16 as the network. I learned about properly segmenting to avoid uneccesary cross talk.
4) Ubuntu - never listen to developers.
;)
Actually it was our default OS. CentOS works better for Hadoop I would later find out.
5) Lack of support. Cloudera is expensive but - unless you are already an expert - worth it. We should have gotten help early on.
6) Hardware choices. Supermicro was a bad choice. Penny wise / Pound foolish. They changed hardware config and it was hard to get replacemnts, etc. Cant say enough good things about working with PSSClabs for hardware though.
I already mentioned the switches. I learned about Arista switches which are amazing for this application.
By the time I left that company, I learned an enormous amount from really smart engineers that knew gobs more about Hadoop and netwrking than I did.
I think the number of datanodes was up over 200 and we had a lean 1u datanode from PSSCLabs with single powersupplies 12 drives JBOD with onboard flash for the OS. Bonded 1G networking running up to Arista TORs and mutliple Arita Agg switches in a leaf / spine topology.
Because Facebook, Google, and Amazon have the engineering talent to build their own.
No "ordinary" enterprise (basically everyone else) can just say "oh, mounting and unmounting is an issue? let's use a raw disk instead!" -- that is just well beyond their capabilities.
Also, only Facebook / Google / Microsoft / Amazon have the scale that designing your own cold storage makes sense.
This is very true. I work in the public sector and while we often need to store petabytes of data we have a hard time finding and keeping engineering talent to do stuff like this. Instead we buy expensive enterprise stuff. I find it frustrating.
I agree. Public dollars should, as much as possible, be used to implement and advance open software and hardware. It's no longer just about "free" as in no cost, it's free as in no contract, no lock-in; and not "no one owns it" but rather "everyone owns it." There's a certain obligation and clear benefit for everyone to make it better, including easier to implement.
Not really. Skipping the filesystem layer and using the raw disk has been an optimization for databases (ms-sql, mysql, oracle db) has been a practice for quite a while.
If you're in the on-premises storage industry (aka NetApp, EMC), you will be amazed by the number of companies still buy networked storage arrays. They are ranging from Fortune 500 firms to extremely smaller ones like dental offices, school district, hospital... They all have one thing in common: they don't trust their data with public cloud infrastructure.
If you ask that way, I'd guess the answer is: For any network storage array there are at least a few on-premise buyer that trust this particular product.
I guess that there are an awful lot of companies that, out of desire to save a few bucks, will go with home-NAS-types.
Ultimately the main difference between the vendors is performance and availability - you pay a lot more to increase either of those. If you don't need either even a consumer style synology nas can work.
It's actually starting to go that way in some enterprise environments, especially in storage where ditching the overhead of EMC/IBM/NetApp has a potentially high reward.
But more conventionally, there is a different thought process in enterprise IT and what I call industrial IT. Enterprise IT implements systems around minimizing failure events and create bespoke environments to meet solution requirements. Industrial orgs acknowledge that these events are a fact of life build process and software to deal with them. They also tend to limit the options available to consume IT.
Also, "at scale" cannot be achieved by an single entity outside of US Federal .gov. I work for a massive organization... 150k+ people. Facebook serves 10000x more users, and have the engineering resources to do amazing things as a result.
It depends on the applications you're going to use them for, and how much you're willing or able to invest in making your applications be able to take advantage of using a larger quantity of cheaper hardware. A lot of big companies run applications that are not nearly as resilient to failure or nearly as linearly scalable as the systems routinely built by the internet giants.
I realized this before with regards to the ridiculously overpriced top-top-of-the-line CPUs. Let's say you're running an existing application at, say, a moderately small scale: 20 servers. Each server is only handling a couple dozen requests per minute, so the load is not very high, but the requests still take too long. You've hired a whole team of engineers to work on optimizing the code to make it run faster, and that costs you, say $10 million a year. You'd be happy to buy another hundred servers if that made it run any faster, but that's just not how it works -- the application is all single-threaded code, and maybe you can make that better, but that's what your $10 million engineering team is working on (maybe). So in the meantime, you spend whatever amount of money you can on getting the absolute fastest hardware you can find to throw in those servers -- fast CPUs, fast hard drives, fast network... and even if those servers cost $50k each, that's still just a fraction of what you're spending on the software engineering work.
No I think that they are using drives with a specific purpose and do not need enterprisey functions.
I think they mean drives that are the quality of the WD black drives, it looks like them in the one picture. The types of drives that you might build a Hadoop cluster with knowing that you will be replacing 1-2 drives per 100 every month.
These drives (unless I misread) do not need to be hotswappable nor any other specialized setup. I would bet they are JBOD and directly attached with minimal controllers. If one drive per tray is powered on at a time then there is no RAID running across that tray...
As a point of comparison an expensive enterprise drive that you might see in a Netezza appliance costs many times more and has specialized firmware to work in a specific application, with certain RAID controllers at specific firmware setings. From experience I do not think that they are much more reliable - they are specific to the appliance, or in some cases like Aerospike required SSDs specific to a software requirement.
This storage design is stripped bare - the HDDs are off for most of their lifetime. I guess the biggest stress on them is spinning up and spinning down. They must focus on precise temperature/humidity control as it might be as much of a problem to have the drives too cold as too hot.
Because when using commodity hardware, strength is in numbers. Very few organizations need the computing power of Google, FB or Amazon, so they can't benefit from the "reliability at scale" that those companies can. Therefore, they need to buy enterprise-grade HW which fails far less often, even if disproportionately more expensive.
Facebook used to use Dell C series servers and HP servers, tons of Dell Custom Solutions C1100/C2100/C6100 and DL160 G6 servers showed up off lease years ago (DL160 racks were shown in a time magazine article of a FB datacenter). I'm fairly certain I've seen pictures of Dell C6220 servers at Facebook as well.
If Facebook (and I believe it's similar with other big players like Google and Amazon) are not using "enterprise grade" hardware (because it doesn't make economic sense at scale), who is? And more importantly, why? Why do "enterprise grade" products even exist if the largest and most deep-pocketed corporations don't find them to be good value propositions?