Compile times can be affected by the following:
CPU horsepower. VBSP, VVIS and VRAD can use up to 16 threads however a 16-core Xeon would be outperformed by a 8-core higher clocked CPU like a i7-5960X. Hell, I'm willing to bet that an overclocked "consumer" i7 (like a 6700K or 4790K) will outperform a more-than-8 core Xeon. Don't use AMD CPUs for these kinds of tasks. Their single-core performance is really sub-par compared to Intel's lineup (although the upcoming Zen architecture promises to make that issue go away).
Less-than-optimal optimization of the map. VVIS can literary never finish in some cases if it can't figure out what's visible from where.
Tiny lightmap grid on all the surfaces. The smaller the grid, the more detail you'll get in your lighting but the longer it takes for VRAD to figure out your lighting. Also increases map file size.
Leaks will cause compile times to shorten, actually. If you have a leak VVIS will run as if it's set to -fast, and VRAD will only do basic lighting. However, compile times should be the least of your problems when you have a leak.
RAM speeds and capacity aren't that big of a deal. If you pay attention to the RAM usage of the compiler you'll see that VRAD is the biggest user of RAM, but it will never even come close to 500-550 MB usage, unless you have a map the size of the entire Hammer grid with a 1x lightmap grid of course (don't do this would ya?).
Compile settings of course can influence compile times, for better or for worse. Think of running VVIS on -fast which doesn't actually test visibility, or running VRAD with the whole lot active, like LDR and HDR compiling, static prop lighting, you know the deal.
Make sure the compiler is the only program running (apart from Hammer obviously). Running things like TF2 in the background takes power from the CPU. This should be common knowledge though.
If I think of something else later I'll edit this post.