The vertex tool is powerful, but it's also really easy for someone who doesn't know what they're doing to create an invalid structure, which is why it's best to avoid it where possible until you're comfortable with Hammer and you're ready to learn advanced stuff. I'll try my best to explain what I've learned from playing around with it, but I can't get any screenshots ATM because I'm not on my computer, and my knowledge may be incomplete or wrong, so feel free to take what I say with a grain of salt.
There are two ways to create an invalid solid with the vertex tool: either creating a concave solid, or creating a solid with an invalid face. A concave shape is a shape with a part that collapses into itself. Here's a picture:
The shape on the left is a concave shape because it has a part of it that collapses in on itself. The shape on the right is convex, which is the opposite of concave- it has no parts that collapse in on itself. In geometric terms, concave shapes have at least one internal angle that is greater than 180 degrees, while every internal angle in a convex shape is less than or equal to 180 degrees. If you translate these shapes into the 3D world of Hammer, the shape on the left would be invalid, while the shape on the right would be perfectly valid. Luckily, this problem can usually easily be remedied by cutting your concave solid into 2 or more convex solids. This problem is also easier to avoid in the first place.
Creating a solid with an invalid face is the more difficult problem to avoid. An invalid brush face is one that Hammer cannot figure out what plane it is on (if you've studied planes in geometry you'll know what I mean); in other words, the brush face is not flat. When I say flat, I don't mean it has to line up to the grid. I mean that if you were to take the brush face and line it up to the grid, it would be entirely flat, with no curves or anything. This problem can usually be avoided with two techniques: selecting and moving multiple adjacent vertices together, and using caution and foresight when using the vertex tool. Always ensure your brush faces are flat. It can be difficult to find or fix an invalid brush face without looking closely at each brush face individually, which is why it may be better to just delete the invalid solid and recreate it with the above tips in mind. Additionally, triangular brush faces cannot be invalid, so you may find those useful if you can't seem to vertex manipulate a brush without making it invalid.
This page on the Valve Dev Wiki talks about what I've just talked about in more complex terms.