'The light from those millions of stars you see is probably many thousands of years old' is a rare example of laypeople substantially OVERestimating astronomical numbers.
Cueball makes the common observation that many of the visible stars in the sky are so distant that it takes thousands of years for light from that star to reach Earth. However, the brightest star Sirius is one of the nearest at a mere 8.6 light-years distance. In other words, the light that was arriving from Sirius in March 2014, when the comic was posted, was emitted some time around August 2005. The previous US president, George W. Bush, was in office from 2001 to 2009 and Megan notes that this isn't a terribly impressive observation. At certain times (mostly when the previous US president served only a single term, but also currently for a short time after the inauguration of any incoming president), looking to events of 8.6 years ago would stretch beyond just the prior administration, but even two (rarely being three, or possibly even more) predecessors ago would not be considered impressive in the circumstances.
The title text references the fact that most people have a hard time imagining the large scale of astronomical numbers. For example, the distance between astronomical bodies or the size of the Sun are hard to imagine; they typically underestimate them by many orders of magnitude and think they are much smaller than they actually are. See Sci-Fi Writers Have No Sense of Scale
In this case, however, people instead overestimate both the number of visible stars and their distance by quite a bit. This could be a result of underestimating the speed of light, which is rather fast. Even though stars are incredibly far away (Sirius's 8.6ly translates to over 80 trillion kilometers), light is so fast that it can crosses those distances as fast as anything can (a relative blink in the eye, for the nearest of these vast distances). This underestimate of light results in an overestimate of the time it takes for light to reach us.
It's frequently cited that about 5,000 to 10,000 stars are visible in the sky by the naked eye. The Bright Star Catalogue is a star catalogue that lists all stars of apparent magnitude 6.5 or brighter, which is roughly every star visible to the naked eye from Earth. The catalog contains 9,110 objects, of which 9,096 are stars, ten are novae or supernovae, and four objects outside of our Milky Way (two globular clusters and two open clusters). To see most of these you need good eyes and a very dark night, and at any point you will only be able to see fewer than half of these as the rest are blocked by the Earth.
This list shows the 93 brightest stars. Of these 60 are more than 100 light years away and only 7 are more than 1,000 light years away. The farthest on this list, Deneb, is "only" 2,600 light years away. Our entire Milky Way contains up to 400 billion (400x10⁹) stars and has a diameter of 100,000 light years.
There are visible objects much farther away, like the Andromeda Galaxy which is 2.5 million light years away and made up of billions of stars. And a gamma ray burst GRB 080319B would have been briefly visible to the naked eye, despite being 7.5 billion light years distant.
See also 1212: Interstellar Memes, 1644: Stargazing and 1440: Geese.