Displacement is one face. If you select brush and make it displacement, you actually get 6 of them (every side of the brush becomes displacement). You should select the texture tool first and select only the faces you want to turn into displacements.