At this time, you have to create your own CSS and target those span tags, which is going to be very difficult. Would you mind taking a screen shot of where you add these color swatches in the Template Manager and posting it? We'll try to find a better solution for you.
