Select the overlay in the 3D window and then scale it in the graph panes. Most of the overlays are square but with stains and such you can play around with them as much as you like.
In Hammer, the overlays will have a really annoying choppy/blinky effect but don't worry, once in game they look fine. Sometimes if you're using a small or translucent overlay they are hard to see, but don't worry, they're there.
The only problem I've encountered is when you have two brushes and you're trying to get the overlay to appear on both. Even if they're lined up, the overlay only appears wherever the Overlay icon is. If anyone has a fix for this it would be much appreciated. If not... I will find a way.