The switch was made because every game that were moved to steampipe (also known as Steam3, a content distribution system) required their own depot. Previously, games were just mounted with files on a different launcher.
The only reason the switch from GCF to VPK appears to have been absolutely necessary for pre-L4D titles is because Valve seemed to have banned binary patching in updates on the SteamPipe delivery system. I mean, I guess it makes sense since they switched from a proprietary download system to using HTTP.
If you've ever looked at how VPK works, you'll notice that to add new things to a VPK in TF2, vpk.exe just adds 1 or more new VPK files (such as tf2_textures_068.vpk) with the new contents, then updates the _dir.vpk file (such as tf2_textures_dir.vpk).
Anyway, getting back to the point, I requested this back before they finished moving TF2 over to SteamPipe. TF2 was (as usual) Valve's test game for switching over to the new system and they had a SteamPipe beta for it for like 3 months before the actual switchover.