I too most like more "realistic" settings where there's no specific external *necessary* explanation for why there must be a harem relationship, but it just happens *naturally* for internal dynamic reasons between those specific characters.
However, the first order of business is to break the traditional expectations of a typical couple, there's must be either reason why it wouldn't work, why alternative is preferred or even possible. That's where the "lazy" solutions -- like mind control and similar influence or drastic society shift and collapse, in one way or another allowing our perpetually horny, weak but likeable asshole male to force the *unrealistic* fantasy into (alternative) reality -- come often handy indeed, but with a bit of creativity those aren't necessary.
With "harem" understood broadly as single man (more or less) openly cohabiting (or at least intensely dating) with more than one woman at the same time we of course need said male to be at least not hostile to the very concept, but since that's our implied goal we can pretty much assume that's the case even if he initially denies that himself. Other than that we can ignore him quite freely, because *as everyone knows* all "realistic" (modern) sexual relationships are in fact defined by women.
Even if our horndog is already chasing two or more girls in parallel, it doesn't make it "harem" until at least two of them at least acknowledge such arrangement as at least acceptable. At very least a fiancee and a concubine have not only to discover each other, but also independently conclude it is in their best mutual interest to support their situation as at least temporary status quo, possibly including teaming up for fun.
This does cover open relationships that aren't "true harem" but a "harem" can't be formed *naturally* without at least some phase of opening up the relationship -- unless existing harem is inherited or otherwise acquired as a whole.
So, the most important question we have to answer is, why the First Girlfriend decides sharing her man is a great idea?
The "First Girlfriend" in this context is obviously the first who accepts or even comes up with the very idea to build a harem, not necessarily the historical first of current, even less all girls the man ever dated, but as the trope goes, it's often indeed is the holder of his virginity who *insists* he must have more lovers than just her.
- She may see it as inevitable, that monogamy for this guy isn't an option (or at least an option still involving her), for example:
-- she is a willing and happy slave and believes her master must have more slaves for his happiness (often but necessary in a result of mind control of some kind);
-- devoted to his deityfied partner she sees her goal in life as making his every whim happen even before he has voiced them;
-- she believes his man is of such status, talent or position that other women seeking to exploit him is inevitable, so she engage in "preventive training" actively seeking prospective lovers for him;
-- she believe she has no real choice in the matter, because of external circumstances, such as
--- she is the concubine of a man forced into formal marriage;
--- she is the formal wife of a man known to keep concubine(s);
--- she is King's concubine or otherwise engaged with a third party that cannot be refused but can be worked around with enough caution;
--- she for whatever reason believes she can't fully satisfy her lover's needs fully (reality doesn't need to intrude, but perhaps she is a transgender, born as defunct boy not yet fully finished her transition, and a self declared lesbian, of course except unexplainable attraction to our main male hero);
--- she is going to be partially or fully absent for significant periods of time (this works in reverse too, First Husband planning for extended absence may appoint his official replacement, even not necessarily consulting his wife; it's said to be popular case of polyandry historically);
-- she is his sister (or other relative; in society with active taboo for incest), notwithstanding the additional difficulty of incestuous seduction it can go from there easily, like:
"I never thought that you thought of me as your ideal girl. I'm really flattered that you think of me that highly."
He looked up to see his sister was serious. He was relieved that she wasn't mad at him for what he did. "I know I can't have you, sis. But I hope I can find somebody as great as you."
"We will remember our first times as being really good, not like the girls at school talk about. Too bad I can't tell them what a great lover you are." The last was said with a twinkle in her eyes. "You would be swarmed with virgins wanting you to break them in."
[Of course, she proceeds to make sure that's exactly what happens next.]
- she is herself bisexual and encourages him to bring in lovers for her and/or is teaming up with him in seduction attempts;
- she begs him to help a "friend in need" or otherwise bad situation, from simple horniness to *rescue* of a potential suicide in abusive relationship;
- she is herself taking another lover and would feel better if he did too, her side relationship break fast, but his one last;
Next is, why the Second Girlfriend is coming along?
Obviously, she can follow any of the reasoning above, as there's often little differences in order or preference between the first two as equals, or be the target of those. In addition to that she may be:
- equally attracted to both of the original couple, or even indeed *to the couple* not so much it's members;
- believe she have nowhere else to go, or be otherwise in dependent situation to either or both of the original couple;
- feel greatly indebted to the First Girlfriend (again, it's important what she believes not what truth is) while strongly attracted to the man;
- be a logical complement of a necessarily domestic or social function the orginal couple is lacking, while similarly benefiting from the relationship herself.
I see addition of the Third Girlfriend to be permanent fixture and not just occasional date in open relationship as just as important and difficult to satisfactory justify as going from a couple to a permanent threesome, especially if the Second Girlfriend somehow penetrated otherwise closed seemingly monogamous couple. If the threesome was open to being with, it's much easier. Regardless, she must be bringing something unique to the group. In addition to all previous:
- she may be more on the gay end of the bisexual spectrum and be more attracted to one of the girls and just accepting relationship with the guy as entry point;
- "nothing's more permanent than temporary" an arrangement that may be more or less a forced hand by one or more of the parties may be accepted by some or all as short term, but stick around and develop -- this is typical for "friend in need" and stray puppies of any kind.
After the third, fourth through sixth may follow a pattern. I personally see collections above four to be increasingly unrealistic, unless the structure of the relationship becomes (much) more complex than simple single level star with the man in the center. Unless the man is indeed some kind of a demigod, and even then above six a second center of attraction is pretty much necessary, such as designated butch in a mostly lesbian group.
For an interesting example, in one story series I follow (Transmogrification of Jacob Hopkins by aroslav) a polyamorous "harem" of one guy and eleven or twelve women (one is currently on associate status not a full member, at least he still believes so) was built and function in surprisingly natural and logical way with only few shortcuts, all while he still haven't had full vaginal sex with the secretly transexual friend nor with the youngest member and he have had and still occasionally have (fully accepted) lovers on the side, without being superhuman nor mind controller.