I saw that the price for infiniband card has dropped dramatically. It is possible to find two cards and a cable for less than $100 to connect two PC at 40Gb/s.
I don’t have any experience with these and I’m wondering how fine it works.
Does anyone work on remote files using infiniband ? Is it fast ? Is there some latency ?
The simple answer is you don’t want InfiniBand. Applications have to be written specifically to take advantage of it, that’s partially why the latency is so low because there’s no networking software stack at the operating system level that applications would normally have to go through.
Why are you trying to get that much bandwidth between two machines anyway? Unless you’re running a couple dozen SSD in a stripe array you’ll never saturate the link or come anywhere close to it. :rolleyes:
There will be a RAID array providing more than 1GB/s troughtput on the file server. Much more than Gb ethernet.
10Gb ethernet is very expensive. It seems easy to setup a TCP/IP network over infiniband (it is installed with the drivers according to some tutorials).
We handle lot of videos. This year we have stabilized 600 x 40min footage from GoPro camera. That’s 60TB of data flowing through our system. Once stabilized by the render farm, each video is 100GB made of 3MB still images. Our poor videographer has to copy this during night to his workstation then he can process the file next day.
I want to give him a way to work directly on file server.
In my opinion 10GbE would be cheaper, easier, and more than sufficient bandwidth. Look at the Intel X540-T1 adapter. Two of those and some CAT-6a cable will get you a 10Gb/s link for less than $1,000.
Yes, it’s possible to do IPoIB but that depends on the platform and it still goes through the network stack like any other adapter so the advantages of using InfiniBand are a moot point.
Also I think 10GbE hardware is a better investment than InfiniBand hardware. Chances are the network at the facility will eventually be upgraded to 10GbE and utilize most of the existing infrastructure, you can’t say the same for InfiniBand. The InfiniBand would be a one off kludge because InfiniBand is really only used for high performance computing (i.e. supercomputers).
Something else to consider is the maximum cable length. InfiniBand 4X QDR is limited to 5 meters on copper while 10GbE is limited to 100 meters on copper. If using InfiniBand the machines would basically have to be in the same room (or right next door).
It could work. I’ve heard of people running IP over IB but never tried it. Please try it and report back. 
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.