There are three things that come to mind:
- Your custom material's VMT file may not be configured correctly.
- That brushwork may be invalid. Do you only see the problem here? If so, try deleting and recreating it.
- Your VMT may be using a transform to manipulate the rotation of the texture. Hammer has trouble displaying materials with transforms in the 3D viewport. An example is the security glass material.
Would you mind copying your custom material's VMT file contents into the thread?