Read more

Graticule computes NULL distance for some records (and how to fix that)

Arne Hartherz
November 30, 2011Software engineer at makandra GmbH

The SQL code generated by Graticule's spherical distance computation is insufficient and can lead to NULL distances in edge cases.

Illustration online protection

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

  • Prevents you from data breaches and liability risks
  • Upgrade at your own pace
  • Works with modern Rubies
Read more Show archive.org snapshot

The reason for this is that MySQL is performing several sine and cosine computations which can result in slight rounding errors -- which is usually okay. Rarely, though, for the exact center of the given circle, it is then asked to compute the arc cosine of a result like 1.0000000001.

Since this is undefined, MySQL correctly returns NULL (instead of the expected distance of 0 for the circle's center).

The solution to this problem is to discard rounding errors by forcing any results into bounds [-1; 1] before computing the arc cosine on it. Put the following into an initializer like config/initializers/graticule_spherical_to_sql_without_nulls.rb which patches Graticule's to_sql Show archive.org snapshot method:

Graticule::Distance::Spherical.class_eval do
  def self.to_sql(options)
    options = {
      :units => :miles,
      :latitude_column => 'latitude',
      :longitude_column => 'longitude'
    }.merge(options)
    %{
      (ACOS(
        GREATEST(-1,
          LEAST(1,
            SIN(RADIANS(#{options[:latitude]})) *
            SIN(RADIANS(#{options[:latitude_column]})) +
            COS(RADIANS(#{options[:latitude]})) *
            COS(RADIANS(#{options[:latitude_column]})) *
            COS(RADIANS(#{options[:longitude_column]}) - RADIANS(#{options[:longitude]}))
          )
        )
      ) * #{Graticule::Distance::EARTH_RADIUS[options[:units].to_sym]})
    }.gsub("\n", '').squeeze(" ")
  end
end
Posted by Arne Hartherz to makandra dev (2011-11-30 14:52)