The great circle (GC) route is the shortest distance between two points on the globe, so we must always keep it in mind when planning an ocean crossing, even if we do not end up following that route.
The GC route is defined by cutting the earth with a plane that goes through the departure (A), the destination (B), and the center of the earth (C). That plane cuts the earth in half, and the points A and B lie along a circle (a great circle) whose circumference is the circumference of the earth, and the track along that line from A to B is called the great circle route. If the plane does not go through the center of the earth, you also get a circle where it intersects the earth, but its circumference will be smaller than that of a great circle.
Distance along a great circle is measured in nautical miles, which is a unit that was invented for just this purpose. Namely, the full great circle spans 360º, and each degree is 60', so a nautical mile (nmi) is defined as the length of 1 arc minute (1') along the circumference of a great circle of the earth.
This is very convenient for navigation if we consider the great circle between the north pole, earth center, and south pole, which is a meridian of longitude. Arc minutes along this great circle are minutes of latitude. Thus a navigator knows immediately if they are to sail from Cape Flattery, WA at about Lat 48 N to San Francisco at about Lat 38 N, they must go 10º of Lat or 600 nmi. Every 1' of Lat = 1 nmi.
There are other implications of this definition that are integrally related to the topic at hand. For one, this assumes the earth is a sphere... which is not too radical an idea, having been known — or believed to be true — by every educated person on earth except Christopher Columbus for over a thousand years.
As it turns out, the earth is not a perfect sphere, it is squashed a bit at the poles, as we might slightly compress a beach ball into more of a doorknob shape. Consequently a nautical mile cannot be simply defined as 1' of Lat, because the length of 1' of Lat changes slightly with latitude on this non-spherical shape. That simple definition is reserved for the less precise term sea mile, which is defined as 1' of Lat at a constant Lon. But nautical mile is the official international unit of global navigation so it has to have a definition, and that was given to it 1929: 1 nmi = 1852 meters, exactly.
That definition then tells us what we mean by spherical earth, based on the geometry of a circle. Namely, the circumference (c) of a circle = 2 𝜋 x radius (r) of the circle. Thus we have for spherical earth, c = 2 𝜋 r = 360 x 60 x 1.852 km, or solving for r:
r (spherical earth) = 360 x 60 x 1.852 /(2 x 3.141) = 6,367.9 km.
Thus we are at the first of three types of great circle distance computation, which is assume the earth is spherical with a radius of 6,367.9 km, which makes 1' on the circle = 1 nmi and we can use spherical trigonometry to compute the great circle distance (d) between point 1 and point 2, namely:
Cos(d) = Sin(Lat1) x Sin(Lat2) + Cos(Lat1) x Cos(Lat2) x Cos(Lon2 – Lon1).
This formula can be solved with an inexpensive trig calculator, and indeed this is the solution we would see in many calculators or apps, especially those that are largely celestial navigation oriented, because cel nav assumes the earth is a sphere as defined above.
If we use this method to compute the GC distance between San Francisco (37.8N, 122.8W) and Tokyo (34.8N, 139.8E) we would get 4,473.61 nmi.
But it is not just cel nav apps that use this equation. The Bowditch computations also assume this same 1' = 1 nmi spherical earth, and present the same value.
Besides cel nav focused apps, some chart navigation apps, officially referred to as electronic charting systems (ECS), also use this spherical earth solution, such as Rose Point's Coastal Explorer. We might call this traditional radius, the cel nav radius (6,367.9 km).
But if we open another popular ECS like qtVlm, and ask for the GC distance between these two points we get a different answer, namely 4,476.62 nmi.
We see essentially the same answer in OpenCPN.
It is not just qtVlm and OpenCPN (two popular free ECS), other computer or mobile nav apps might show this answer for these two points.
...that is, unless we are looking at a GPS chart plotter app or a handheld GPS unit with routing options, such as the Garmin GPSmap 78 shown below.
In this case, we get a still different value of this same "great circle distance," namely 4,486.7 nmi.
The three values we noted were presented in increasing accuracy, which is tied to the shape of the earth that was used to compute the value. In most cases, these differences do not have a practical affect on navigation, but it is good to know if something is working right or not, and to understand what we see.
Type 1. SF to TKY = 4,473.61 nmi. Spherical earth with 1' = 1 nmi. This solution is used in cel nav and other apps, as noted. Earth radius used is 6,367.9 km. The cel nav radius.
Type 2. SF to TKY = 4,476.62 nmi. This is what we would see in selected ECS that want to improve on the accuracy by using an improved earth radius.
An improved earth shape is more of an oblate ellipsoid (doorknob), which can be approximated with a new spherical earth, but now using the average of the polar and equatorial radii, as shown. This improved method still computes the distance as a spherical earth, but uses this slightly smaller average radius of 6,371.0 km. This can be called the WGS84 average radius.
Type 3. SF to TKY = 4,486.7 nmi. Is in principle the most accurate solution as it uses not assume a spherical earth shape, but computes the distance along the surface of an oblate ellipsoid, the size and shape of which we get from the geodedic datum we have selected, such as WGS84. We will get this (Type 3) solution in most apps or hardware that lets us choose the horizontal datum, such as any GPS unit, hand-held or console chart plotter. This choice is actually an important thing to check in your GPS to be sure it matches your nautical charts; most should default to WGS84.
We also get this geodetic or ellipsoidal solution for "great circle" distances in several popular computer based ECS, such as TimeZero.
Numerical values of these distances can be checked online with the Jack Williams calculators.