Yes, toggle Allow swatches only on in the question's configuration.

This will only allow respondents to select the swatches you have included as part of the question.