Yes, toggle Allow swatches only on in the question's configuration.
This will only allow respondents to select the swatches you have included as part of the question.