Performance packed vs not packed?

I have not tested no. But as a general game development rule, it's usually better to have assets that will be loaded together to be contiguous on your storage medium. This allows games to load faster when seek times are long (for example, on a console game disk).

Following that logic, there's potential for faster loadings when packing assets but as far as tf2 is concerned, it's always loading slow no matter what

Having assets packed into a map or into a vpk has the advantage of compression through repacking. This makes them better for taking up less disk space compared to putting everything into the custom folder. It's also one less step for someone trying to download the map when all of the required assets are already there. I don't think any slight performance boost would offset these very big advantages.