Since we first proposed the Fit-slope method many years ago, and have used it continually since then, I have just realized that the base reason the procedure works is not as obvious as we might have implied in various descriptions of it. The procedure does often lead to improved sight data; it is mentioned in the latest edition of Bowditch: American Practical Navigator.
A look at basic least squares fit analysis reveals the subtlety that makes it such an important aid to celestial sight analysis. But first, a bit of background.
In celestial navigation, we take a series of sextant sights of a specific body whose height is changing with time, so we end up with a list of times (x) and sextant heights (y). Over a short period of time, the celestial body height changes in a linear way. But our measurements always include some random errors, along with possible systematic errors, as well as periodic blunders. The data points then scatter about a straight line that represents the rise or fall of the height with time as perceived from our moving vessel.
Random errors come from varying judgements on best alignment of the body and the horizon as well as random errors in reading and recording the time of the alignment. Systematic errors might be due to a fading horizon during an evening twilight session. Blunders happen by reading the micrometer in correctly, or reading 12 on the sextant dial or watch, and then writing down 21 .
Such sights might be for an LOP from any celestial body, or they could be a series of lunar distance measurements. The process and initial analysis we discuss here is the same.
We take a series of sights of any one body with the goal of getting just one value, a single time and height that best represents the series of measurements, hopefully averaging out at least the random errors, and catching and removing the blunders. An important procedure we follow to improve on this is to try to take at least 4 sights of three bodies, as near 120ยบ apart as possible, and we do so by alternating the sights, rather than take all of one then all of the next. This method also ensures we get a fix as soon as possible in case we are somehow interrupted.
We end up with 4 or 5 sights of each body, and we do this for each. We plot the Hs (y) vs. time (x) and then draw a straight line through the data, which is intended to represent what the values would be if we did them all precisely. Then we can choose any point on this line to represent the full set of measurements of that body.
Doing this by hand, we lay a clear plastic ruler on the data so as to go through the most points possible, adjusting the line to leave as many points above as below the line. The difference between any data point and the line is called the residual, with plus above and minus below. The goal is have the sum of these residuals near zero. Points notably off the line could be ignored or weighed less when choosing the line's position. Note we can move the line up and down and we can rotate it to get the best alignment.
This process is essentially a manual least squares fit to a straight line. Using a spreadsheet program like Excel, we would plot the data and then add a linear trend line, which gives us the equation of the best fit line, y = a + bx, where b is the slope and a is the intercept, meaning value of y when x = 0.
Knowing this mathematically correct best-fit line (we get from Excel), we can compare it with what we get doing in by eye, no math involved. And it turns out that with a bit of practice, the human eye can do a remarkably good job in choosing the best fit, and thus we do not really add much to the analysis with Excel — assuming that is all we wanted out of Excel: the best fit line so we could choose a single sight that we could consider an average of the set for that body.
I think of the above as standard procedure in cel nav, without introducing the Fit-slope Method. Now we look at that process.
The principle behind the Fit-slope Method is simple: we can compute the slope of the Hs vs time line, so when fitting the line, with Excel or by eye with a parallel plotter, we can keep the slope constant as we look for the best fit. That restricts the choice of best fit line position and hence our choice of best representative sight.
We compute the slope by computing Hc1 at a time at the start of the sight session from the DR at that time (t1) and then again from the DR at the end of the sight session (Hc2 at t2). The slope of the line is then (H2-Hc1)/(t2-t1). Plotting by hand we do not need this numeric value, we just plot Hc2 and Hc1 on the same plot as the actual Hs sights and draw a line between them. These Hc values will need a new scale, because they differ from the Hs values by the sum of the altitude corrections. Once on the same plot, we just align a roller plotter with the computed slope and then move it up and down, without rotation, to find the best fit.
That is pretty much the extent of the explanation we have given on how to use the method, followed usually by a couple examples. The Fit-slope Method almost always leads to an improved fix from a set of sights.
____________
But if we look at the details of least squares fitting, we can see that the above discussion has not focused in on why the method actually gets us better averages. It is implied, but not stated, and now I want to clarify how this works
I will state the key points, then at the end show the related math derivations.
As mentioned above, a linear least squares fit to a table of x vs y values gives the equation of the best line as y = a + bx. That best-fit line will always pass through the mid point of each axis (x-bar, y-bar), which is called the centroid point of the data set. x-bar is the average of all the sight times and y-bar is the average of all the Hs values.
Thus if the whole goal of the process is to find a single point on the best fit to all of the sights, then you do not have to plot anything at all. Just choose Hs (best) = the average of all Hs values, and assume this is the value at time T = the average of all the sight times. That collapses the entire analysis to just averaging all of the sight data you have.
A less obvious result to many—through mathematically even simpler—is, if you force the least squares fit to have a fixed, specific slope (ie, constant b) then the best-fit line will also go through the same centroid point of the data set, and you could again choose the midpoints as best representative of the set. The Hs values at any other time than the centroid will be different, but if the goal is just find a single point on the best fit line, the centroid point will still do the job.
According to that analysis, there is no difference between fit-slope and no fit-slope on choosing a single sight to represent the best average of the set!
The subtlety underlying why the fit-sope gets us a better average in light of the above mathematical facts is we have assumed we are averaging all of the sights in both cases.
In most cases, however, we are not using the same set of data in both cases. When using the proper slope we can better tell which sights are likely wrong or outliers, and remove them from the computation of the average. When we change the sights in the average, the average changes, so there is not in practice a common sight solution to the fit-slope vs. no-fit slope solution. Removing only one sight in the set can have a notable affect of the fitted line and centroid value.
In the end, the fit-slope method is not so much a mathematical result. It is simply a guide to helping the human navigator choose which sights belong in the full set to be averaged. It is easy to find sets of real data that show that using the right slope changes the sights we consider too high or too low.
In good conditions, we are looking for small corrections to what we might consider a good set of sights to begin with. In poor conditions, with hit and miss sights, of say a sun popping in and out of the clouds, or sights in big seas where we have to wait for a good view of the real horizon, then we might have a large scatter in the data points and the fit slope method is a hope to pull an improved fix out of poor data.
Note that fitting with fixed slope (b) in Excel is not as easy as just "Add trend line" that fits both a and b, but as noted we can do this all manually just as fast or faster than with Excel.
Another note is for the fit-slope method, we must indeed compute the Hc at the beginning and end using some app or formula to get the right slope. Pub 229 does not work for this as we need values for specific DR positions and those corrections in Pub 229 are tedious, and Pub 249 is not accurate enough.
___________
Here is Claude's proof that the best fit line goes through the centroid without a fixed slope and also with a fixed slope.
Below is then the same proof for a fixed slope:
