In Battle Cry of Freedom, McPherson tells us that the Union army in the Civil War lost 2 times as many people to illness as to combat deaths. That sounds bad, but he says that the British record in Crimea was 4 to 1 and in the Napoleonic wars, 8 to 1. There are 3 reasons for the high rate of illness death:
1) Weapons (and transportation which enabled rapid-fire weapons to be supplied) improved over this period. (2) Bringing troops together spread disease. (Measles was a major illness in the Union Army.) (3) Medicine was so bad then that everybody got sick more often and died earlier.
If you write anything historical, you have to keep that third factor in mind.
Our modern (first world) experience is that death from illness of people younger than 60 or so is comparatively rare. That was not the case in previous periods.
First, medical care improved, and then things like sewage treatment improved. Experience changed over time, and then -- after a significant delay -- perceptions changed.
If you take your current picture of families and use it for a historical period when -- in actuality -- the infant-mortality rate was 50% or greater, then you are picturing something unrealistic.
And, of course, contraception use is relatively modern. Even when condoms were available, they were not used by married couples. (A century ago, simply advocating this was punishable under pornography statutes.)
So, if you have a group of married couples in the Victorian or earlier period something between half and a third of the wives were pregnant. That means quite large families in the latest portion of that time; it means having lost many children in the earlier times.