There will come a day (and I’m fighting for it) where color swatches in applications are properly gamma-adjusted. Some applications can, others not, and sometimes it’s not completely consistent. Some day… alas… I dream… 
What to do?
Well, basically, you will have to “unconvert” the Swatch. Myself (personally) have begun to do this “in my head”, I don’t see color swatches as any “direct” representation of the final color, more a principial display of the reflectance, so to me, paradoxically, your image looks fine 
Also, keep in mind that you are getting colored light and reflections from the sun&sky, so you can never actually match exactly the color, unelss you turn off the sun/sky saturation completely… I mean, an object under a blueish/yellowish lightmix does look different than something under precicely white light.
A rule-of-thumb could be to “square” your color values, if you want to do it “by hand”.
If your color is 1, 0.8, 0.5, set it to
1 * 1 = 1
0.8 * 0.8 = 0.64
0.5 * 0.5 = 0.25
It’s an “approximation” (using a gamma of 2) but it can be a useful rule-of-thumb.
/Z