The problem is that you're trying to make a face into something other than a flat plane.
Brushes in hammer work like having a set of planes where the only practical (non-texture) information is their angles and position, and then the brush's solid volume is defined by the shape inside of all of them. So, every face needs to be flat, and they can only make a convex structure.
What you're trying to do is take a face and make it something other than a flat plane, which is plain impossible. You probably want to use a displacement or cut your brush up some more.