Because for every almost-gifted kid who wasn't challenged, I can point to two more who were believed to be of much lower intelligence and aptitude than they actually were.

EXACTLY! Kids whose parents didn't read to them, who don't have books at home, who don't have that higher level of educational prior knowledge are so often assumed to be less able than they are just because they don't know things we think they "should know." Or if it takes them longer to get a concept, we assume that they're slower, but it's often really that the kids whose parents are more educationally supportive came in farther along in the learning process.