I read somewhere that the average male in the U.S. is 5-foot-9 and the average female is 5-foot-3. I have no idea whether either of those is accurate.
For the sake of argument, assume that they are correct. How much taller than the average would a male character need to be for you to invoke the description "tall"? Would you call a 5-foot-10 male "tall"? How about a 5-foot-4 female?