Usually those kinds of shadows are caused when the side of a brush is also displaced, rather than being culled. You should only select the top face to be dispalced otherwise the whole brush is rendered in a displacement mesh, when you only need/want the top face.
The shadows are caused because displacement faces are automatically put into lighting groups, unlike brush geometry. So shadows often bleed around corners. You can see this for yourself by turning walls into displacements, the shadow will blend around the corner, rather than produce the crisp shadow/lit you would normally find at corners.
What you see here is the sides under the surface are in total shadow and that shadow is bleeding to the surface, but the important thing is that those faces should be deleted regardless and the shadows will disappear inadvertantly.