docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/ is a good place to start for Linux.

Ubuntu 14.04 is strongly recommended since 14.10 does not support xorg-video-abi-11 that the nvidia-340 driver package in the .deb depends. If you want to use Ubuntu 14.10 then you have to use the runfile.

(Ubuntu 14.04 instructions)
Get and install the Ubuntu 14.04 deb. This adds some repos to your repo list so that you can get up-to-date prebuilt binaries for both the CUDA drivers and the rest of the stuff.

then

apt-get update
apt-get install cuda cuda-toolkit

The installation is done!

(Ubuntu 14.10)
You have to use the runfile. Download that package only.

do 

./{runfile}.run --override 

to force the modules to build with gcc-4.9. If you try to change your system gcc to 4.7, since that's what the module officially supports, your kernel will complain since it was probably built with gcc-4.9 and will not load modules built with 4.7. Using 4.9 for everything seems to be ok.

If this goes ok, the installation is done.

(Post-install stuff)
You then need to update your PATH so that the cuda compiler can be found.
edit ~/.profile to contain the following:

PATH=/usr/local/cuda/bin:$PATH

then do

source ~/.profile
nvcc -V

and you should see some nice version strings.

also you need to add the CUDA libraries to the system LDPATH so the linker can get at them. DO NOT FOLLOW THE WEBSITE'S INSTRUCTIONS. LD_LIBRARY_PATH is deprecated on Ubuntu (I think) and you will get errors finding libcudart.so.6.5 or something if you try to do it that way. Instead, do

sudo cat /usr/local/cuda/lib64 > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

to update all the linker paths. Then, you can do

sudo ldconfig -p | grep -i cuda

to see that your system knows where all the cuda libraries are. 

To check if everything is working, you can build the stuff in /usr/local/cuda/samples and run bandwidthTest.

(Bonus CUDNN install)
If you were ever curious about the useCUDNN flag in Caffe's Makefile.configure, there's an easy way to install that too.

First, get the CUDNN-R1 stuff from NVIDIA. https://developer.nvidia.com/cuDNN

Then
tar -xzvf cudnn-6.5-linux-R1.tgz
cd cudnn-6.5-linux-R1
sudo cp lib* /usr/local/cuda/lib64/
sudo cp cudnn.h /usr/local/cuda/include/

do another

sudo ldconfig

to get the library symlinks set up and then rebuild caffe.

Then you're done!

-- Chris