I've had a long-standing group of readers, actually about five, so they do impact the votes. They actively read each chapter, as the "1" votes all shift to 3s for the exciting chapters, but they registered their complaints initially (it's political, based upon a single minor character in a single story, but they've continued registering their 'protest' by voting down each of my stories ever since (except the latest, where they haven't yet, probably only because they don't visit FS).
There's not much you can do, and like mine, it's probably either philosophical or political. Clearly they're fans, but have a single issue they're trying to punish you for. Personally, if you're curious, I'd enter a blog post about it. They might respond if they think they have a receptive audience. But then, there are quite a few Amazon readers who score hundreds of well-respected books with "1"s, but who've never rating anything higher, meaning they're attempting to take down everyone, rather than trying to highlight anything. It's also a know issue where other authors 1-bomb similar stories to make their own stories look better.
Since their loyal fans, I wouldn't make a big issue out of it--it's an odd 'freedom of expression' issue, but if you're honestly curious, I'd try reaching out to them to see what their issue with your stories is. Otherwise, I'd let it slide.