Does your map has complex geometry so the visleafs get cut some crazy way? I had this problem myself (see here:
https://tf2maps.net/threads/cutllinkedlist-overflow-exhausted-memory-allocator.27307/) with a map, that was very complex an unoptimized as I didn't knew how to use any optimaziation tools or even func details.
I ended up abandoning the map, because I couldn't find any answer for the problem. Only thing that helped was hiding half of the map (it was a mirrored koth map) and just compiling the other. So I'm guessing that the map was just too much to handle for my computer.
I know that this isn't really an answer for your problem, but I hope it gives you some hints and ideas and what could be wrong.