Building a 32 TB Server: a thought experiment.

August 31, 2008

Daniel Gimpelevich and Holden Aust built a 16 TB server for Christian Einfeld and his Digital Tipping Point Project.  See Linux Journal, Issue 173, September 2008.  I am impressed that these gentlemen built a server with four times the capacity of anything that I have ever attempted.  It’s funny because in Einfeld’s article he mentions it almost in passing.  My jaw was on the floor.  Also I think it’s cool that he’s a lawyer who is also very much into technology.  Moreover, his philanthropic efforts in San Francisco are admirable.

The server they built motivates me to try to build a 32 TB server.  There are three problems that I have not worked out.  One: fitting 16 drives in one box.  I would wait to build the server when 2 TB drives are obtainable.  I am assuming that I can find a case somewhere that will hold 16 drives.  If I can’t, I would have to have some sort of an external enclosure and run SATA cables to it.  Two: I don’t know if FreeNAS can handle 32 TB of storage.  If not, I’d have to use some other platform, but I suspect it could do it or could be made to do it.  Three: I don’t know if you can put three or four SATA cards on one motherboard.  Obviously these gentlemen figure that part out.  It must be possible, I just don’t know how to do it.  It may be as easy as plugging them in.

If I were to succeed, a RAID 5 FreeNAS server would provide 20.8 TB of usable space out of the 32 TB available.  Since you have to do backups anyway, it almost makes sense to have two raid zero 32 TB servers as you would get 27.73 TB of usable space each and faster performance.

At this point, this is just a thought experiment.  In terms of money, when the drives become available, we’re not talking about that much compared to other types of extreme computing.  For example some people will spend in excess of 14 or $15,000 buying an overclocked “ultimate” machine.  A 32 TB server would probably only cost $2-$3000 to build.

I have to admit it is exciting.  I don’t know what I would use it for.  I still have plenty of space on my 4 TB server that only has 2.6 TB of usable space.  Even with an HD TiVo and pulling HD content off of it and putting it on the server, I don’t think I would need anything close to 32 TB of space.  But it would be fun to build.


11 Responses to “Building a 32 TB Server: a thought experiment.”

  1. Gary Sims Says:

    Wow, what a great idea! Thinking about the limits of FreeNAS, if the normal UFS file system can’t cope with 16TB then you could try ZFS (which is in version 0.7) as that has a theoretical limit of 16 Exabytes!

  2. ArcSine Says:

    Just built one up for ~$6000 on NewEgg (mostly). One SuperMicro SC846E2-R900B 24-bay enclosure, 24 Barracuda 7200.11s, two 250GB OS disks, one 3Ware 9690SA, SuperMicro board/Xeon LV/yadda yadda. Etherchannel the dual GigE, put the disks in RAID5 with two hot spares. Yeesh, what a monster.

  3. thaed Says:

    Whoa. That’s some serious equipment. Do you have pictures? What OS are you running? What are you going to use it for?

  4. huma200 Says:

    Don’t use raid 5 nor raid 0, use raid-z, you will get more usable space and it has a better feature set than the standard raids.
    Hope to see pics of that beats soon!!

  5. huma2000 Says:

    damm!! I wrote my nickname bad!! xD is Huma2000 instead Huma200 xD

  6. nytefall Says:

    We are building several 50+TB systems using FreeNAS and 1/2TB SATA drives in LSI 16-slot enclosures with Fiber Channel host connects to the FreeNAS server. Should be interesting here in a few weeks.


    • Minsik Says:

      Well done ideas i see, How is the project going and do you have a blog or something that gets updated with progress?

      good luck from a freenas supporter and devotee

  7. Gpshelzw Says:

    Three years teen fashion bbs LOL, they are acting good. Guy reminds me of Will Ferrell and Patrick Swayze at the same time. And the girl is just so fucking sexy

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: