The issue at hand is evaluating a fix based on the dimensions of the plot itself. In other words, we look at the lengths of all the lines involved. There are two kinds of lines on the plot of a cel nav fix: the lines of positions (LOPs) themselves, and the lengths of the azimuth lines that run from the assumed positions to the LOPs (the a-values). A sample is shown in Figure 1.
Figure 1. Crucial lengths defined. If any of these are approaching 60 nmi long, we should use the fix for a new DR and redo the sight reduction.
The crucial lengths are the a-values, a1 and a2, as well as the lengths along the LOPs from the fix to the azimuth line (L1 and L2). Chances of long lines is enhanced when the DR is about halfway between two whole latitudes, because we must choose one of these for the assumed latitude.
We make the assumption in standard manual plotting procedure that the circle of equal altitude near our position can be approximated as a straight line (the LOP), and that the azimuth lines, which are segments of a great circle, can be approximated by a straight line as well. Both of these approximations can break down if any of these lines gets too long.
Below is a new addition to textbook Section 11.24, followed by examples and background of this concern.
Evaluate Assumed Positions in Manual Sight Reductions
Before we start to evaluate a full set of sights for optimum accuracy, we should pause at the end of the sight reduction to check that our basic assumptions are in order for the sights at hand. How we do this depends on how we are doing the sight reduction. If we are using a computer or nav app for sight reduction and position fixing, this step can be skipped completely, because the location of the assumed position (AP) does not matter when computing a fix. With calculator solutions, we generally use our DR at the fix time as the basis of the computations, and even if the DR is way off, the programs will work around this either by solving for intersecting circles of position, or iterating lines of position (LOPs) automatically, as explained in the manual approach below. In any event the choice of AP is not a concern for computed solutions, but we must consider this when doing manual solutions using plotting.
When doing sight reduction by hand using tables and manual plotting, we must check that none of the lines leading to our plotted fix are too long. This can happen with manual sight reduction as we must choose an AP based on the minutes part of the GHA and our DR Lat and Lon. We can always choose the AP to be within 30' of the DR, but even then with various configurations of body bearings and APs, we can end up with large a-values. If the DR position was wrong by a lot, meaning the distance between the DR position and the fix found using the APs chosen properly for each sight, then the a-values can get even much larger. In this case the distance between the fix and the azimuth line measured along the LOP can get large as well. The lines that can get too long are illustrated in Figure 1, above.
We might consider that any crucial line in our plotted fix that is over (or approaching) 60 nmi should be considered too long for best results, meaning they may violate our basic assumptions in the plotting. Underlying the manual plotting solution is the assumption that the LOP itself is a valid straight-line approximation to a segment of a circle of equal altitude (see textbook Section 10.6), and we assume that the azimuth line is a segment of a great circle, even though we are plotting it as a straight rhumb line. These approximations can break down when the lines get too long. The consequences of this also depends on the direction they are oriented, but a generic filter on the lengths should catch all cases.
The solution to long lines on the plot, is to plot the fix in the normal way, then read that Lat and Lon and call that the new DR for these sights. Then do the sight reduction again using this DR, which will call for new APs, and then the lines will all be shorter, and the fix you get will be more accurate.
In most cases of routine cel nav—see Hawaii by Sextant for examples—we can proceed as normal, and will not find any excessively long lines, but if we do, we can fix it. On the other hand, if we suspect ahead of time that our DR could be wrong by over 40 miles or so, then we might do a quick 2-LOP fix to check the lines and find a new best DR to use before any further analysis.
The instructions to Pub 229 include a Table of Offsets for curving the LOPs to help with this correction. That table shows corrections of several miles for lines L1 or L2 of just 45 miles. Errors due to long a-values are more subtle and depend on the azimuths; they occur when a straight line approximation to the azimuth line diverges from the curved great circle tract between AP and GP.
In summary, any of the crucial lines in a fix plot approaching 60 nmi long should call for getting a new DR from the fix and redoing the sight reduction. Errors of several miles can occur without this precaution. Such errors are larger for high sights, above 70º or so, and line lengths can get enhanced when DR Lat is about half way between two parallels.
More Background on this Topic
This section goes into more detail to look at how these errors actually come about and the sizes of them in a few examples. At this point, you can say "I know the rule now, and I will use it as needed," and then skip this section!
Below is a sample of the Pub 229 Table of Offsets and how it is used to make the corrections for curved LOPs.
Figure 2. Table of Offsets from Pub 229 and their example of how they are used.
We see from the Table that for a sight that has Ho about 73º, an LOP line length (L1 or L2 in Figure 1) of 45 nmi (45') would cause an error of 1 nmi (1') on the LOP position. At 60 nmi, the error would be much larger. All in all, it is likely faster to just get a new DR leading to shorter lines than to correct the individual LOPs this way, and with that in mind, we generally do not cover the use of the Offsets in our course.
Here is a way to approximate this correction if you might care to.
Figure 2a. We see the 0.9' at 72º in the Pub 229 table, and then can estimate that at 100 nmi, the correction would be about 4.6'. ( To be redrawn when we can. )
Errors due to large a-values are a bit more subtle and there is no easy table for the correction. The situation is illustrated in the graphics below, starting with cel nav data from the USNO.
Figure 3. Cel Nav data from starpath.com/usno.
The Mars and Markab sights are plotted below in OpenCPN, using a neat feature of that program that lets us plot a line segment as rhumb line or great circle, or both.
Figure 4. Geographical positions (GP) of two bodies showing the great circle track to them from the assumed position (AP).
Here we see an interesting example of spherical geometry. From this AP at this time, we would actually be looking just north of west to see a star that is a long way south of us! The direction we look to see a star is called its azimuth, 279.8 T to Markab in this example. This azimuth generated in the sight reduction process is numerically the same as the initial heading of a great circle (GC) track from the AP to the GP. Below we expand the section near the AP.
Figure 5. Deviation of a rhumb line along the initial heading of a great circle track.
In the case of Markab, we would plot the azimuth line in direction 279.8, and then plot the LOP perpendicular to that line. However if the a-value is ~60 miles or so long, this line is some 4 nmi off the GC track as computed in OpenCPN. This, in itself, would not matter, but the offset will effectively rotate the LOP by some small amount. In this simple picture the rotation would be arctan (4/60) which is about 4º—but this example overestimates the effect. We get a better feel for the effect by breaking the GC track down into smaller steps, which is easy to do with a program like StarPilot.
Figure 6. Start of the great circle track to Mars from the AP in 0.25º of Lon steps (~ 12 nmi).
In this example we see that an a-value of 85 nmi would yield an LOP that is rotated by 1º off the true orientation. This could lead to a relatively large fix error if the lines L1 or L2 were also large. On the other hand, a quick redo of the sight reduction using new DR removes this error.
A real exampleAs a practical example that includes a bit of each of these long-line errors, we look at Problem 8 from the real sight data included in Hawaii by Sextant (HBS).
Figure 7. Sample sights from HBS.
In this voyage and the documentation of it, we did a running fix between two sun lines taken at 1102 and 1334, which led to the fix labeled FIX 1334. For now, we do not care about running fixes and just treat these two sights as if we were dead in the water. That would lead to the fix marked with a red circle, where no line was advanced. That is the fix we are studying in light of the fact that L2 from the 1334 sun line is indeed over 60 nmi long.
In the next picture we used the 1334 fix to get a new DR and then did the sight reduction again, using Pub 229 (without any offsets). That is shown below with the new APs.
Figure 8. Redo of Problem 8 sight reductions with new DR position.
The red data are the original plotting using original APs shown in Figure 7. The green plotting is the new fix using the previous fix position as a new DR. You can view both work forms to see where all the numbers come from.
This correction moved the fix by 3.5 nmi, which is a significant amount if we are striving to get the best possible cel nav fix.
To follow up on the point that these long-line problems do not arise in a computed solution, we look at a computed solutions using the raw data from the initial sights, including the inaccurate DR.
Here is the raw sight data of the two sights and the computed LOPs
using 1802 DR 36º 31' N, 133º 28' W. Times in UTC
July 10, 1982, HE = 9 ft, IC=0,
18:02:02, Hs 49º 24.5', a = 21.5' T 098.0
20:34:46, Hs 74º 42.0', a = 10.7' T 158.0
For a fix at 36º 27.8' N, 133º 01.6' W.... which is what we got by plotting, but only after one iteration of the AP.
In other words, again, if you compute the fix with a standard navigation program you do not have to worry about this factor. The crucial feature of a proper cel nav program is it will iterate the results automatically that we are illustrating here manually, meaning it gets a fix from the user input DR, then replaces the DR with the fix, then computes the fix again to see if it changes, and it if did, it repeats again till there is no change. Alternatively, a program can intersect the two circles of position directly and not use any DR at all.
As a closing note on this topic, in the informational section at the back of the Nautical Almanac, called Sight Reduction Procedures: Methods and Formulae for Direct Computation, in Section 11, Position from intercept and azimuth by computation, they address the topic we covered here.
They propose that the distance to be monitored (equivalent to our fix to AP) should be less than 20 nmi and if not, change AP to fix and recompute. They can afford to make the distance smaller because they are computing the solution, as we noted that all programs do. When doing it by hand, we can stick with the "anything approaching 60 nmi" for our trigger.