I really need to get around to writing on here more often…
Anyway, the NVIDIA GPU Grant Scheme has re-opened after a little break over Christmas and is now giving away Titan V (Volta) GPUs to academics. I was lucky enough to bag one of these earlier this week, so I’m eagerly awaiting the delivery! All that’s required is a 2000 word explanation of why it would be helpful to your research, so it’s well worth your time.
The Titan V is a significant upgrade to their previous offering in that the Titan V has 640 “Tensor cores” in addition to the usual CUDA cores. The tensor cores are dedicated to performing matrix multiplies using half precision arithmetic (though the input and output matrices can be single precision). There is a more detailed explanation of their use here. Using half precision on dedicated hardware components means a dramatic increase in speed for these operations, which are at the heart of neural network training.
The downside to using half precision arithmetic is the potential loss of accuracy in the result, depending upon the accumulation of rounding errors. Whilst this isn’t a huge problem for small matrix multiplications, using half precision for other matrix problems such as inversion will – essentially – multiply this error by the condition number of the matrix. At this point we can quickly run into problems! Nick Higham recently touched upon these topics in this blog post.
Sam Relton's Blog 