@ian181I can remember reading this a long time ago (may not have been on SOL). Unfortunately I can't remember the title or author but I can add in some additional plot points (if it's the same story).
Various colors of stones / crystals. Typical (IMO) color / power assignments. Red - fire, Blue - water, Brown - earth, black (purple?) - necromancy. I also remember a healing color and possibly air as well.
MC finds a 'meta' crystal capable of duplucating all colors. Meta crystal was also sentient and I believe presented itself as a brown crystal at the beginning.
MC was quite low in rank / social standing (or as OP stated, possibly a slave) of his (town, guild, etc.).
Plot progresses to final battle against another sentient meta crystal (black, iirc) from the same (comet, asteroid, meteorite).
*edit*
I've found the story i was thinking of. Is this your lost story?
Elemental by Etherealism