Refraction in Celestial Navigation–still an issue, after all these years

Here is an article written in 2012 that somehow did not get posted.  It is now, because we just got a question about this that I thought we had answered, but could not find it, then found this draft, which a bit late now, we post. Thanks to Nathaniel Fairfield for finding this issue mentioned below in our Emergency Nav book and taking the time to tell us.

Refraction of light as it enters the atmosphere is still one of the largest uncertainties in celestial navigation. When starlight leaves the vacuum of outer space and enters the atmosphere it bends down a small amount depending on how high the star is in the sky.  This happens because the light is slowed down in the gas of the atmosphere. See Refraction in a Sink.

Thus with our sextants we always measure a star height that is slightly too high.  A starlight from stars some 30º above the horizon will bend only some 1' to 2' (if uncorrected yielding an error of some 1 to 2 nmi), but at lower angles the correction can be twice that or more.  At higher angles it is less, being only half a mile or so at 60º and 0 by definition for stars overhead.

For perspective, keep in mind that if we do everything right, we can hope to achieve a cel nav position fix accuracy of about ±0.5 nmi.  Not much better, especially if the boat is moving, and if we are not careful, the uncertainty will be notably larger.  So this refraction issue (on the order of tenths of an arc minute) is right in the same order of magnitude as sextant sight accuracy and nautical almanac accuracy, the other two factors that limit the ultimate accuracy of our fix.... assuming of course that we do everything else right, and if plotting, do so on a large enough scale that small errors don't matter.

There should be no doubt that there is uncertainly associated with refraction. You would not, for example, ever ask that question if you have ever seen a good mirage, which is just abnormal refraction on steroids.

But we do not need to be so qualitative.  When we first made our book called Emergency Navigation, throughout the early and mid 80's, the US-UK Nautical Almanac used a value of -34.5' of correction for the lowest stars, those just over the horizon. In other words, this low starlight is bending more than half a degree.  But the correction goes down fast.  One way we remember it is to note at 5º high the correction is -10', and at 10º high the correction is -5'.  We have formulas and jingles in the book to help remember this if you get stuck without an almanac.

But now, in 2012, we take a look at this correction in the official US-UK Nautical Almanac and we see -33.8' of correction instead of 34.5'.  Granted, there are things about the immovable, permanent stars that have in fact moved.  The Polaris correction used to find latitude from the height of the North Star above the horizon was 49' in 1982, whereas today it is 41'. But this is an entirely different effect and result. This Polaris correction is well understood on the basis of star and earth motions over the years–small as they are, they do add up.

Nothing at all similar has happened to the atmosphere over these years that changes how the starlight bends. Yes, there is global warming and measurable changes to the atmosphere, but these do not change the refraction we are discussing.

This does not mean that navigators in the 80s who knew where they were then, we now know were wrong! In fact, this large shift (0.7 nmi) in the maximum refraction at the horizon has little effect on normal celestial navigation. For one thing, we know there is enhanced refraction uncertainty at low angles, so good texts teach that we should take sights whenever possible above 15º and below 75º.  Above 15º the correction and the uncertainty in the correction go down rapidly. (The goal of taking sights below 75º is for an entirely different reason having to do with the math approximations used in basic procedure. These in fact can be overcome with special procedures.)

Below some 15º, the refraction becomes more sensitive to the properties of the atmosphere used, particularly the pressure, temperature, humidity, temperature changes with altitude (lapse rate), and the wavelength (color) of the light ray.

A thorough description of the values used in the Nautical Almanc is given by C.Y. Hohenkerk (Director of HM Nautical Almanac Office) and A.T. Sinclair in "The Computation of Angular Atmospheric Refraction at Large Zenith Angles," Technical Note 63, 1985.  This paper explains the origin and assumptions made in obtaining the value of -34.5'. They proposed that as a base reference value that can be used to apply corrections to for various conditions of temperature and pressure, and presumably to be used as a standard that could be improved upon if newer data or analysis becomes available. That value was used then and up until about 2004. After that, almanacs (US-UK, French, Russian) switched to the value of -33.8'.

My question has been, what changed that led to the new value and the above paper sheds much light on that question.

Besides explaining the process, they also compute the sensitivities of the result to the input parameters.  First they show that above 60º the result is essentially independent of this input, but at 0º, on the horizon, the effects are large. Summarizing and changing units, they show:

      Change in input                         Change in Refraction on the horizon
Temperature change of 4.5º F                         - 0.6'
Pressure change of 10 mb                                +2.6'

These values have then been incorporated into the Table A4 of the Nautical Almanac for  adjustments to be made for deviations from the standard conditions they used,  namely 50º F and 1010 mb. They also assumed dry air, but showed that changing to saturated air only changed the result by -0.08'.

A difficult challenge of this analysis, however, is not so much the pressure or the temperature effects, but how the temperature changes with altitude, called the lapse rate. This is much more difficult to know in practice and to account for.  The earlier almanac computations used a value called the average lapse rate of 3.5º F/1000 ft and showed that a lapse rate change of 1º F causes a horizon refraction change of -0.3'. Actual air mass lapse rates vary from some 2.5 to 5.5, so we are seeing that there are easily factors floating around that could change the value from 34.5 to 33.8 on improved analysis.

In fact, it is rather more complex than even that. Light rays from bodies on the horizon, must skim along the surface of the ocean to get from outer space to our eye and spend some amount of time within the first meter of air above the surface. This first meter above the ocean is a complex region. Water is continually evaporating into it and condensing back to the ocean, there is salt spray there, and its temperature is affected by the sea temperature as well as local air mass temperature.  The nature of this air is also dependent on the local wind, which mixes the air, and it depends on the sea state, which is more or less active in interacting with it, and the time of day.

See this article Sunset Science. IV. Low-Altitude Refraction by Andrew Young for more details. He is the expert on this and related subjects.