I think that for whoever votes below five or any number for that matter and that they should give a reason why the vote in either way and if it turns out that that reason is not true that that vote gets taken off what do you think?
I think it is worth breaking down your idea. So:
Since actively moderating every vote is too time consuming to be considered, let's just do away with all votes less than 6.
Doing that leaves the voter a choice from 6 to 10, which is effectively 1 to 5.
Presumably you would then object to receiving a 1 vote or 2, possibly a three.
To fix that perhaps you would prefer a simple good/bad vote choice..??
Would you then object if readers voted your story 'bad' ??
Then again, as has been suggested by others, readers might be voting for you, not your story, and base their vote on your previous behaviour. If that is the case, well, Karma is a bitch...
I suggest you stop complaining and think of it as paying the price for past sins. The only way to gain respect is to earn it, if you feel you are being treated badly, then you have gained a little insight as to how those whose stories you plagiarised felt as a consequence of your actions.
Then again, as has also been pointed out, most readers don't ever use the forum, so will be oblivious to your history, those readers may still award you a 1, it could be due to spelling/grammar, it could be they dislike the subject matter, it could be they had a shit day and are taking out their frustrations on whichever story they read. Conversely you might get a 10 from someone who just loves the name of your main character, regardless of how good or bad the story actually is.
Readers are granted the freedom to vote however they like, for whatever reason, on a scale of 1 to 10. It isn't a free vote if you restrict or moderate those votes*. If you can't live with your readers having that freedom, then you have the freedom to turn voting off.
* Yes I'm aware of the algorithm, I'm just ignoring it.