CGTalk > Technical > Technical and Hardware
Login register
reply share thread « Previous Thread | Next Thread »  
 
Thread Tools Search this Thread Display Modes
Old 02-28-2014, 04:41 PM   #1
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,918
File Servers Project

At work I setup a pair of file servers and I would like to share the results because the topic of file servers has come up a few times in the past on this forum. Prior to the project there were a lot of prosumer NAS devices from LaCie and Seagate and a few hardware RAID 5/6 arrays on Windows Server. As needs grew more devices were purchased and thrown into the mix (another NAS here, another NAS there). This was a problem because stuff was hard to find, most stuff wasn't backed up nor would it be easy to backup, and most of the devices had fairly bad performance like the LaCie and Seagate NAS products.

So the goal was the opposite of that - high performance, scalable, backed up, and easy to manage storage. I picked ZFS on Linux because ZFS provides all of those things (and a lot more) and I'm familiar with Linux. Had I not been familiar with Linux I would've opted for ZFS through NAS4Free or FreeNAS which provide a web interface for management.

http://zfsonlinux.org/

For the hardware I picked the SuperStorage Server 6047R-E1R36L from Super Micro since it's designed for software defined storage like ZFS. There are two, one for production and one for backup. It has 36 drive bays that are connected to a SAS controller and backplanes with switches but there's no RAID functionality (don't need it with ZFS). They have a few other useful features like redundant power supplies and hotswapable fans. From there you add processors, memory, disks, and network adapters.

http://www.supermicro.com/products/...047R-E1R36L.cfm

No frills desktop drives were chosen since there's no hardware RAID controller and thus no need for error recovery control like TLER/CCTL/ERC which you find on "RAID" or "enterprise" edition drives so they don't drop out of arrays when trying to read bad sectors. At the time that meant a 4TB drive for $165 each, or a little less than half the cost of equivalent drives with error recovery control. When disks account for the largest single line item in a storage system that's a big deal.

Some other hardware details, went with Intel X540-T2 10GbE adapters since they work with Linux out of the box and have dual ports (one for the switch and one direct for backups). There are four SATA connected SSD, one for the OS (Ubuntu 12.04 LTS) and three for caching which ZFS calls ZIL (sync write caching) and L2ARC (read caching). There's 128GB of ECC memory which ZFS uses for the ARC (read caching). There's also a Netgear M7100-24X which is a 24 port 10GbE switch and threw in the optional redundant power supply. Clients use the Intel X540-T1 10GbE adapter which is a single port adapter and is widely supported in terms of operating systems (support on Mac through Small Tree).

In the end what does that get you? It gets you about 90TB of storage that can lose up to 12 disks, can easily saturate a 10GbE pipe, backs up automatically to an identical setup every hour, and has automated rolling snapshots so you can revert to older versions or recover accidentally deleted files. Once a week it scrubs itself for errors and automatically corrects any that it finds. External JBOD arrays can expand the system to 144 drives (about half a petabyte with 4TB disks) and it would continue to be in the same namespace and could even happen without rebooting the system.

The cost was around $35,000 for the pair of servers, a dozen client adapters, 10GbE switch, cabling, and eight spare disks. A few vendors bid on it and they ranged from $120,000 to $150,000 for similar setups though with less disk space (24 drive bays instead of 36). The setup has been running for more than a month now and has about 10TB of data on it. Clients typically see 300MB/s (2.4Gb/s) transfers which is limited by the disks on the clients.

Hopefully this is helpful information for somebody. For what it's worth ZFS scales down too, like an 8 bay or even smaller server. For scaling up I'd look at Lustre or GlusterFS on ZFS and move to a distributed cluster. If you have any questions or want more specific information let me know.
__________________
http://www.whenpicsfly.com
 
Old 03-05-2014, 01:11 AM   #2
tswalk
Expert
 
tswalk's Avatar
portfolio
Troy Walker
USA
 
Join Date: Jan 2012
Posts: 717
very cool... can you provide us a pic? I'ld love to see it.

I also like the throughput... must be nice!
__________________
-- LinkedIn Profile --
-- Blog --
-- Portfolio --
 
Old 03-05-2014, 09:54 PM   #3
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,918
There are some photos on my Google+ page of when I was assembling them and also when they were installed in the racks.

https://plus.google.com/10750604733...sts/cGbrbQVsczR
__________________
http://www.whenpicsfly.com
 
Old 03-05-2014, 10:02 PM   #4
tswalk
Expert
 
tswalk's Avatar
portfolio
Troy Walker
USA
 
Join Date: Jan 2012
Posts: 717
is that a bag from Fry's?

.. reminds me of some of the HP arrays I use to work on... looks very pro.. nice job! I bet they're crazy loud too.
__________________
-- LinkedIn Profile --
-- Blog --
-- Portfolio --
 
Old 03-05-2014, 10:36 PM   #5
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,918
Hahaha, yes, it is indeed a Fry's Electronics bag. I had to get a few SATA cables and power adapters for the SSD which I didn't order with the rest of the parts (oops!). They are surprisingly quiet for what they are. You can hear the fans go full tilt when it boots but after that they calm down quite a lot. Either way it doesn't matter though because I'm far away from the racks!
__________________
http://www.whenpicsfly.com
 
Old 03-06-2014, 03:11 AM   #6
tswalk
Expert
 
tswalk's Avatar
portfolio
Troy Walker
USA
 
Join Date: Jan 2012
Posts: 717
lol.. ya i find myself running over there on occasion too.

would be nice if Mouser Electronics had a "pickup" place here.. right down the street from me, but in a pinch Fry's is good for a quick run!
__________________
-- LinkedIn Profile --
-- Blog --
-- Portfolio --
 
reply share thread


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 05:18 PM.


Powered by vBulletin
Copyright ©2000 - 2016, Jelsoft Enterprises Ltd.