Saturday, January 06, 2007

Excel formula to calculate distance between 2 latitude, longitude (lat/lon) points (GPS positions)

I've been looking for this one for ages, and the one I created based on an equation I found a little while ago didn't work too well :(

Investigation

Calculate distance, bearing and more between two Latitude/Longitude points shows lots of equations in math, and implemented in JavaScript. Chris Veness's page also contains an Excel formula of the ‘Haversine’ equation (actually, using the "spherical law of cosines") for distances between points in kilometres:

=ACOS(SIN(Lat1)*SIN(Lat2) +COS(Lat1)*COS(Lat2)*COS(Lon2-Lon1)) *6371

But it wasn't working, because my GPS points (Lat/Lon) were in decimal degrees, not radians. So I used the formula I found at Latitude And Longitude In Excel (under the "Great Circle Distances" section) by Chip Pearson:

=RadiusEarth*ACOS(COS(RADIANS(90-Lat1)) *COS(RADIANS(90-Lat2)) +SIN(RADIANS(90-Lat1)) *SIN(RADIANS(90-Lat2)) *COS(RADIANS(Long1-Long2)))

where Chip gives RadiusEarth as 6378.135 kilometres. This didn't match up with distances I had when calculated from other systems. But Chris's page has:

R = earth’s radius (mean radius = 6,371km)

Digging deeper, I found Chris's Vincenty formula for distance between two Latitude/Longitude points page which includes a table on different datum models (treating Earth as an ellipsoid), it shows WGS-84 & GRS-80 having the greatest radius on an ellipsoid as 6378.135km & the smallest as 6356.752km.

So Chip was using the maximum radius of the Earth, not the mean radius like Chris (I'm not sure where Chris gets the mean radius from). Substituting Chris's R for RadiusEarth in Chip's formula gives:

Solution

=ACOS(COS(RADIANS(90-Lat1)) *COS(RADIANS(90-Lat2)) +SIN(RADIANS(90-Lat1)) *SIN(RADIANS(90-Lat2)) *COS(RADIANS(Long1-Long2))) *6371

and my final version that uses Excel cell references is:

=ACOS(COS(RADIANS(90-A2)) *COS(RADIANS(90-A3)) +SIN(RADIANS(90-A2)) *SIN(RADIANS(90-A3)) *COS(RADIANS(B2-B3))) *6371

PS. To calculate distances in miles, substitute R (6371) with 3958.756 (and for nautical miles, use 3440.065).