I see nothing ridiculous about this at all. Lee, one of the most well known personages of the ACW, rode a specific horse through most of the war. That horse was a dapple grey. There is a great variety of breeds as well color/pattern variation amongst greys - which can even change as a horse ages.
It seems there is no definitive answer on just how dappled, or grey, or white Traveler actually was, with much of our impression of this animal coming from artist license. - In the mean time we've been able to enjoy viewing some historical photographs as well as some great artworks. What's so ridiculous about that.