Well, it can be caused by a couple of things.
1: You have a lot of very small lightmaps on a lot of surfaces. While it looks pretty, it will also take a lot longer or in semi-rare cases cause the compiler to crash. In the Material Selection menu you have an option to change the lightmap grid size. Default is 16, in areas with higher contrasts between dark spots and lit spots you can go with 8 or 4.
2: Lighting calculations take a lot of time and it often eats up 100% of your CPU time. It's entirely possible for Hammer to become unresponsive during a compile, especially during the VRAD stages. Just let it do its thing. It can take 10 minutes to over 2 hours depending on map size and compiler settings. RAM usage indeed jumps up during VRAD, it needs to store all of its calculated lightmaps in there before it writes them to the bsp file when VRAD's finished.
3: You have a brush or displacement that's made in such a way that VRAD can't figure out for the life of it how it should calculate the lighting for it causing it to hang and eat up your RAM in the process. Look for brushes with weird geometry (they're often invalid as well) or displacements with abnormal geometry like it intersecting itself. I remember when I made a brush about 256x512 HU big, then made a displacement out of it and compressed it down to something like 64x64 with all sorts of up and down geometry on that displacement (I was new to Hammer at that point, I know better now).