Depending on what map I'm working on, it's starting that's the hardest. It's hard to do the little details when you have this awesome idea, but have to get the spawns and filters set up first. Once you get into the swing of things though, then it gets really fun as you shape it into the cool idea you had.
Then there's the flipside, where you start off really hyped and exited for this map that you're working on, but once you have the basic layout set up and get into "crap what should go here" part. Then things start to go downhill. I cannot tell you how many maps that I've started and now they just sit. (This isn't just for TF2. I find myself constantly setting up Half-Life 2 and Portal 2 maps, only to give up on them partway.)
I find what I end up doing most is either setting up a level in Dev Textures, or creating this really nice looking scenic thing, or creating a map solely for testing something that I want to learn how to use. (e.x. ropes, new skybox, prefabs/npcs, etc.)
Wow that was longer than expected. I'm just going to stop talking now.