Map file size help

When you say fully custom, what do you mean exactly?
Are these assets made by yourself? If so, make sure you optimise them to reduce file size.

Otherwise, make sure that the lightmap scale in your map is at a reasonable value (if you have all surfaces set to 1, for example, that will hugely boost your file size)

If it's not either of those, i'm assuming there's some kind of compile error occurring. Try using compilepal.

And if even that fails you... then I got no clue.

If you are serious with using a TON of custom assets: You don't need every wall to be a custom texture, or every corner to have a different crate model.
Check that only the stuff you actually need is being packed. Your packing program might have done an oopsie. Use GCFScape to check what's inside the BSP.

As fun and rewarding as it is to use custom assets, remember that TF2 is an old game and the Source engine is even older. You will have to replace as many custom textures, models and sounds as possible in order to get it to a decent size. I would suggest learning Unity if you want more freedom with your creation.

For models, consider reducing the polycount and making certain models or details just sprites or textures.
Same models can have different color variations and bodygroups (if the armature is the same).
For materials, remember that most TF2 textures are 512x512, so aim for that or lower. Creative texturing can make certain textures redundant.

1.56 GB is insanely large. Are the assets you are using actually made for the game? Make sure that any materials you're using have appropriate vtf compression (ex. they are imported with dxt1 or dxt5 rather than RGBA8888). Make sure sounds are compressed mp3s rather than .wav files, they can bloat the filesize. If you open the map in gcfscape you can use the properties option on folders to look at where all of the filesize bloat is coming from.

try to turn some of the brushes into a model (I think it will do as a weight optimization, although I might be wrong.)