I would suggest researching random dungeon generation, especially related to roguelike games. Like, actual old school roguelikes, not the modern "anything with permadeath is a roguelike" interpretation.
This implementation, for example, makes some really interesting results:
https://www.reddit.com/r/gamedev/comments/1dlwc4/procedural_dungeon_generation_algorithm_explained/
Obviously that won't make a TF2 map, but if you massage the rules to follow the general design features of a 5CP map you might be able to get close enough to produce a playable map. The 2D part is simple, I imagine making it 3D with height variations will be a bit of a challenge.
This implementation, for example, makes some really interesting results:
https://www.reddit.com/r/gamedev/comments/1dlwc4/procedural_dungeon_generation_algorithm_explained/
Obviously that won't make a TF2 map, but if you massage the rules to follow the general design features of a 5CP map you might be able to get close enough to produce a playable map. The 2D part is simple, I imagine making it 3D with height variations will be a bit of a challenge.