Great Circle Distance — The Three Options

The great circle (GC) route is the shortest distance between two points on the globe, so we must always keep it in mind when planning an ocean crossing, even if we do not end up following that route. 

The GC route is defined by cutting the earth with a plane that goes through the departure (A), the destination (B),  and the center of the earth (C). That plane cuts the earth in half, and the points A and B lie along a circle (a great circle) whose circumference is the circumference of the earth, and the track along that line from A to B is called the great circle route.  If the plane does not go through the center of the earth, you also get a circle where it intersects the earth, but its circumference will be smaller than that of a great circle.

Distance along a great circle is measured  in nautical miles, which is a unit that was invented for just this purpose. Namely, the full great circle spans 360º, and each degree is 60', so a nautical mile (nmi) is defined as the length of 1 arc minute (1') along the circumference of a great circle of the earth. 

This is very convenient for navigation if we consider the great circle between the north pole, earth center, and south pole, which is a meridian of longitude. Arc minutes along this great circle are minutes of latitude.  Thus a navigator knows immediately if they are to sail from Cape Flattery, WA at about Lat 48 N to San Francisco at about Lat 38 N, they must go 10º of Lat or 600 nmi. Every 1' of Lat = 1 nmi.

There are other implications of this definition that are integrally related to the topic at hand.  For one, this assumes the earth is a sphere... which is not too radical an idea, having been known — or believed to be true — by every educated person on earth except Christopher Columbus for over a thousand years. 

As it turns out, the earth is not a perfect sphere, it is squashed a bit at the poles, as we might slightly compress a beach ball into more of a doorknob shape. Consequently a nautical mile cannot be simply defined as 1' of Lat, because the length of 1' of Lat changes slightly with latitude on this non-spherical shape. That simple definition is reserved for the less precise term sea mile, which is defined as 1' of Lat at a constant Lon. But nautical mile is the official international unit of global navigation so it has to have a definition, and that was given to it 1929: 1 nmi = 1852 meters, exactly.

That definition then tells us what we mean by spherical earth, based on the geometry of a circle. Namely, the circumference (c) of a circle = 2 𝜋 x radius (r) of the circle. Thus we have for spherical earth, c = 2 𝜋 r = 360 x 60 x 1.852 km, or solving for r:

r (spherical earth) = 360 x 60 x 1.852 /(2 x 3.141) = 6,367.9 km.

Thus we are at the first of three types of great circle distance computation, which is assume the earth is spherical with a radius of 6,367.9 km, which makes 1' on the circle = 1 nmi and we can use spherical trigonometry to compute the great circle distance (d) between point 1 and point 2, namely:

Cos(d) = Sin(Lat1) x Sin(Lat2) + Cos(Lat1) x Cos(Lat2) x Cos(Lon2 – Lon1).

This formula can be solved with an inexpensive trig calculator, and indeed this is the solution we would see in many calculators or apps, especially those that are largely celestial navigation oriented, because cel nav assumes the earth is a sphere as defined above.

If we use this method to compute the GC distance between San Francisco (37.8N, 122.8W) and Tokyo (34.8N, 139.8E) we would get 4,473.61 nmi.

But it is not just cel nav apps that use this equation. The Bowditch computations also assume this same 1' = 1 nmi spherical earth, and present the same value.

Besides cel nav focused apps, some chart navigation apps, officially referred to as electronic charting systems (ECS), also use this spherical earth solution, such as Rose Point's Coastal Explorer. We might call this traditional radius, the cel nav radius (6,367.9 km).

But if we open another popular ECS like qtVlm, and ask for the GC distance between these two points we get a different answer, namely 4,476.62 nmi. 

We see essentially the same answer in OpenCPN.

It is not just qtVlm and OpenCPN (two popular free ECS),  other computer or mobile nav apps might show this answer for these two points.

...that is, unless we are looking at a GPS chart plotter app or a handheld GPS unit with routing options, such as the Garmin GPSmap 78 shown below. 

In this case, we get a still different value of this same "great circle distance," namely 4,486.7 nmi. 

We also see this value in the ECS TimeZero.

In short, we have three values for the "great circle" distance between SF and TKY, and the one we get depends on how or who we ask. The differences in these example spans 13.1 nmi — and this, in an age where we pride ourselves with a GPS that gives our position accuracy to about a boat length or two (± 0.01 nmi).

Navigator's do not like inconsistent information, and will usually stop to figure out the source of the discrepancy. This note is intended to help with that.

The three values we noted were presented in increasing accuracy, which is tied to the shape of the earth that was used to compute the value. In most cases, these differences do not have a practical affect on navigation, but it is good to know if something is working right or not, and to understand what we see.

Type 1.  SF to TKY = 4,473.61 nmi. Spherical earth with 1' = 1 nmi. This solution is used in cel nav and other apps, as noted. Earth radius used is 6,367.9 km. The cel nav radius.

Type 2. SF to TKY = 4,476.62 nmi. This is what we would see in selected ECS that want to improve on the accuracy by using an improved earth radius. 

An improved earth shape is more of an oblate ellipsoid (doorknob), which can be approximated with a new spherical earth, but now using the average of the polar and equatorial radii, as shown. This improved method still computes the distance as a spherical earth, but uses this slightly smaller average radius of 6,371.0 km. This can be called the WGS84 average radius.

WGS84 earth dimensions. Keep in mind the scale. The equatorial bulge (7 km)  is just 0.1% of the radius; the depression of the poles (15 km), just 0.2%. The earth is actually pretty spherical.

Type 3.  SF to TKY = 4,486.7 nmi. Is in principle the most accurate solution as it uses not assume a spherical earth shape, but computes the distance along the surface of an oblate ellipsoid, the size and shape of which we get from the geodedic datum we have selected, such as WGS84. We will get this (Type 3) solution in most apps or hardware that lets us choose the horizontal datum, such as any GPS unit, hand-held or console chart plotter. This choice is actually an important thing to check in your GPS to be sure it matches your nautical charts; most should default to WGS84.

We also get this geodetic or ellipsoidal solution for "great circle" distances in several popular computer based ECS, such as TimeZero.

Google Earth will also give this value, but for other locations you may get different results as they may use different datums for different locations, which we do not seem to have control over. (The same is true, by the way, for the elevation data set or model it uses for different parts of the world. It is likely the best we can conveniently come by, but we will not know the details.)

Numerical values of these distances can be checked online with the Jack Williams calculators.

These values can be used to determine what type of computation your device is doing. Use Departure = (37.8, -122.8); Destination =  34.8, 139.8). 

Then check for the GC distance between them.

4473.6 means spherical earth using the cel nav radius (6,367.9 km)

4476.6 means spherical earth using the WGS84 average radius (6,371.0 km)

4486.7 means a WGS84 ellipsoidal computation

A consequence of a true ellipsoidal computation means a nominal, long-distance great circle estimated position depends on which way you are headed. Consider starting from the equator at 130 W and traveling 50º N versus 50º E. Sailing along the surface of a spherical earth, the distance you travel would be the same in both directions, namely 3,000 nmi. But sailing on the surface of an oblate ellipsoid, this is not the case. You have a smaller radius going toward the pole than you do going along the equator. Going north you sail 2,991.8 nmi but sailing east you go 3,005.4 nmi.

For completeness, let me add a 4th solution!  One that goes in the other direction: not striving for high precision, but looking for a solution that can be done with a plastic device that still works if soaking wet, after falling off the nav station and getting stomped on by numerous crew members' wet boots.

Great Circle Solutions with the 2102-D Star Finder