CGTalk > Technical > Technical and Hardware
Login register
Thread Closed share thread « Previous Thread | Next Thread »  
 
Thread Tools Search this Thread Display Modes
Old 07-25-2013, 01:20 PM   #1
bokkan
New Member
portfolio
Seb
toulouse, France
 
Join Date: Nov 2009
Posts: 7
Infiniband file transfer experience

I saw that the price for infiniband card has dropped dramatically. It is possible to find two cards and a cable for less than $100 to connect two PC at 40Gb/s.

I don't have any experience with these and I'm wondering how fine it works.

Does anyone work on remote files using infiniband ? Is it fast ? Is there some latency ?
 
Old 07-25-2013, 03:14 PM   #2
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,923
The simple answer is you don't want InfiniBand. Applications have to be written specifically to take advantage of it, that's partially why the latency is so low because there's no networking software stack at the operating system level that applications would normally have to go through.

Why are you trying to get that much bandwidth between two machines anyway? Unless you're running a couple dozen SSD in a stripe array you'll never saturate the link or come anywhere close to it.
__________________
http://www.whenpicsfly.com
 
Old 07-25-2013, 03:26 PM   #3
bokkan
New Member
portfolio
Seb
toulouse, France
 
Join Date: Nov 2009
Posts: 7
There will be a RAID array providing more than 1GB/s troughtput on the file server. Much more than Gb ethernet.

10Gb ethernet is very expensive. It seems easy to setup a TCP/IP network over infiniband (it is installed with the drivers according to some tutorials).

We handle lot of videos. This year we have stabilized 600 x 40min footage from GoPro camera. That's 60TB of data flowing through our system. Once stabilized by the render farm, each video is 100GB made of 3MB still images. Our poor videographer has to copy this during night to his workstation then he can process the file next day.

I want to give him a way to work directly on file server.
 
Old 07-25-2013, 03:44 PM   #4
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,923
In my opinion 10GbE would be cheaper, easier, and more than sufficient bandwidth. Look at the Intel X540-T1 adapter. Two of those and some CAT-6a cable will get you a 10Gb/s link for less than $1,000.

Yes, it's possible to do IPoIB but that depends on the platform and it still goes through the network stack like any other adapter so the advantages of using InfiniBand are a moot point.

Also I think 10GbE hardware is a better investment than InfiniBand hardware. Chances are the network at the facility will eventually be upgraded to 10GbE and utilize most of the existing infrastructure, you can't say the same for InfiniBand. The InfiniBand would be a one off kludge because InfiniBand is really only used for high performance computing (i.e. supercomputers).
__________________
http://www.whenpicsfly.com
 
Old 07-25-2013, 04:05 PM   #5
olson
Houdini|Python|Linux
portfolio
Luke Olson
Dallas, USA
 
Join Date: Jan 2007
Posts: 2,923
Something else to consider is the maximum cable length. InfiniBand 4X QDR is limited to 5 meters on copper while 10GbE is limited to 100 meters on copper. If using InfiniBand the machines would basically have to be in the same room (or right next door).
__________________
http://www.whenpicsfly.com
 
Old 08-02-2013, 08:34 PM   #6
dmeyer
raaaaaerrrrrrrrr!
 
dmeyer's Avatar
designer
de-zy-nur
USA
 
Join Date: Nov 2002
Posts: 1,270
It could work. I've heard of people running IP over IB but never tried it. Please try it and report back.
__________________
Fix it in pre.
 
Old 08-02-2013, 08:34 PM   #7
CGTalk Moderation
Lord of the posts
CGTalk Forum Leader
 
Join Date: Sep 2003
Posts: 1,066,480
Thread automatically closed

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.
__________________
CGTalk Policy/Legalities
Note that as CGTalk Members, you agree to the terms and conditions of using this website.
 
Thread Closed share thread


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 12:23 AM.


Powered by vBulletin
Copyright ©2000 - 2016, Jelsoft Enterprises Ltd.