In answering the inquiries I limited myself to saying more or less exactly that. That's not giving away much.
That might be part of the problem there, if you can't explain the situation until later in the story, many will be lost.
How could you go into detail about your thinking without destroying whatever magic, for lack of a better word, the story has? I don't think you can, but is there ever a reason to do that?
If the readers are complaining about 'not getting it', I doubt they're worried about your motivations for writing it. Instead, you've failed in how you relate the story somehow. Anytime I get responses like that, I like to back away from the story, and consider what's not working in it.
I had a major existential writing crisis recently after I asked readers to contrast the writing style in two books in the same series (written some time apart). The responses weren't good. Essentially, they said that if that was my new style, that they wouldn't bother reading any more stories.
However, that flew in the face of the statistical evidence. They asserted that my efforts to minimize the backstory hurt the character development, and the latter focus on short chapters and action, left them feeling uninterested in the shallow characters. However, the ratings for the early chapters were uniformly low, and the scores mounter over time (once I finished with the back story) and peaked with each action scene.
It wasn't until I got another complaint from a long-time contributor who provided more detail, that I understood the issue. He pointed out that he couldn't understand the motivation of the main character's sister, and instead of developing her further, she was simply dropped later in the story.
That parallels what you're facing. Essentially, you're dealing with general complaints (i.e. either 'I don't understand the story' or 'I don't like the characters'). In those situations, the author has little clue about what is specifically wrong with the story.
After getting the follow-up with the more specific problem, I understood. I'd initially quit the story as being largely unworkable, as it read like three different stories. 1) The intro featuring the MC and his sister. 2) The main character running away with a new character, and 3) the later section, featuring the main character and several people from the original book.
In short, there was no consistency throughout the book, and everyone the readers felt involved with, got shuffled off, while the later characters were never got fully developed. It wasn't that I was writing shorter chapters, or trying to improve the flow of the book, it was the book itself.
I'd realized the issue initially, but when my beta-readers urged me to publish it anyway, I was so focused on fixing secondary issues (minimizing the back story), that I completely forgot about the main issues with the story.
Since I hadn't found any way to successfully combine those separate aspects of the story, I decided the story couldn't be saved. But in other cases, once I figured out what the problem was (say a particular character was extremely unsympathetic), I'd go back, read through the bothersome section, and flesh out the character, explaining their motivations and why they behaved like that.
That approach usually fixed the problem. The readers still didn't appreciate the character, but they at least understood them and could put their actions into context.
I'm afraid that you've got to go back and get some more specific answers. That's what beta-readers are for, but they often never spot the very issues which trip up most readers.
Good luck with the issues, though, as I know how tough it can be second-guessing incomplete complaints.