For vertex animations, you have to use FBX, not the SMD from Wallworm. It compiles fine with the workshop tools inside TF2. I haven't tried it manually but since GUIStudio would compile with the same exe, it should work. You'd have to experiment with it I'm sure.
But first, go in Hammer, make a dynamic prop with one of the classes, and try to trigger a face animation (vertex animation) via inputs. You have to be able to get that working first, and I don't know how you would do that part.
(Edit: Someone correct me if I'm wrong, I'm only part way through my morning coffee)