Can I restrict which colors a submitter can pick with the color picker?

Yes, toggle "Allow swatches only" on in the question's configuration panel.

Allow swatches only toggled on

This will only allow respondents to select the swatches you have included as part of the question.

Related Articles

How does the color picker work and what does it look like?
On a blank line, type "/color" and select "Color Picker" from the popup menu, or click "Add questions" and select "Color Picker" from the question dropdown.
How can I specify certain color swatches for the Color Picker field?
You can configure a Color Picker question to add or remove swatches from the preset list of options.
What are Custom 1, 2, 3, and 4 in the Palette section under Theme & Appearance?
Custom colors are a palette of colors you can choose and then use across your form without having to remember the colors you set initially.