Infiniband file transfer experience

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

Thread Tools Display Modes
  07 July 2013
Infiniband file transfer experience

I saw that the price for infiniband card has dropped dramatically. It is possible to find two cards and a cable for less than $100 to connect two PC at 40Gb/s.

I don't have any experience with these and I'm wondering how fine it works.

Does anyone work on remote files using infiniband ? Is it fast ? Is there some latency ?
  07 July 2013
The simple answer is you don't want InfiniBand. Applications have to be written specifically to take advantage of it, that's partially why the latency is so low because there's no networking software stack at the operating system level that applications would normally have to go through.

Why are you trying to get that much bandwidth between two machines anyway? Unless you're running a couple dozen SSD in a stripe array you'll never saturate the link or come anywhere close to it.
  07 July 2013
There will be a RAID array providing more than 1GB/s troughtput on the file server. Much more than Gb ethernet.

10Gb ethernet is very expensive. It seems easy to setup a TCP/IP network over infiniband (it is installed with the drivers according to some tutorials).

We handle lot of videos. This year we have stabilized 600 x 40min footage from GoPro camera. That's 60TB of data flowing through our system. Once stabilized by the render farm, each video is 100GB made of 3MB still images. Our poor videographer has to copy this during night to his workstation then he can process the file next day.

I want to give him a way to work directly on file server.
  07 July 2013
In my opinion 10GbE would be cheaper, easier, and more than sufficient bandwidth. Look at the Intel X540-T1 adapter. Two of those and some CAT-6a cable will get you a 10Gb/s link for less than $1,000.

Yes, it's possible to do IPoIB but that depends on the platform and it still goes through the network stack like any other adapter so the advantages of using InfiniBand are a moot point.

Also I think 10GbE hardware is a better investment than InfiniBand hardware. Chances are the network at the facility will eventually be upgraded to 10GbE and utilize most of the existing infrastructure, you can't say the same for InfiniBand. The InfiniBand would be a one off kludge because InfiniBand is really only used for high performance computing (i.e. supercomputers).
  07 July 2013
Something else to consider is the maximum cable length. InfiniBand 4X QDR is limited to 5 meters on copper while 10GbE is limited to 100 meters on copper. If using InfiniBand the machines would basically have to be in the same room (or right next door).
  08 August 2013
It could work. I've heard of people running IP over IB but never tried it. Please try it and report back.
Fix it in pre.
  08 August 2013
Thread automatically closed

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.
CGTalk Policy/Legalities
Note that as CGTalk Members, you agree to the terms and conditions of using this website.
Thread Closed share thread

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Society of Digital Artists

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump

All times are GMT. The time now is 10:44 PM.

Powered by vBulletin
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.