Read more

Using Rationals to avoid rounding errors in calculations

Martin Schaflitzl
November 20, 2023Software engineer at makandra GmbH

Ruby has the class Rational Show archive.org snapshot which allows you to store exact fractions. Any calculation on these variables will now use fractional calculations internally, until you convert the result to another data type or do a calculation which requires an implicit conversion.

Example use case:

Illustration money motivation

Opscomplete powered by makandra brand

Save money by migrating from AWS to our fully managed hosting in Germany.

  • Trusted by over 100 customers
  • Ready to use with Ruby, Node.js, PHP
  • Proactive management by operations experts
Read more Show archive.org snapshot

Lets say you want to store the conversion factor from MJ to kWh in a variable, which is 1/3.6. Using BigDecimals for this seems like a good idea, it usually helps with rounding errors over a float, but there are cases where you still have to worry about rounding errors.

conversion_factor = BigDecimal('1') / BigDecimal('3.6')
# => 0.277777777777777777777777777777777778e0

Here the conversion factor got rounded after some decimal places. If you now use this value now to convert a Value from MJ to kWh it can lead to ugly rounding errors.

BigDecimal('7.2') * conversion_factor
# => 0.20000000000000000000000000000000000016e1

This might be OK for you as the value is still very accurate. When you later round the value, e.g. for displaying it might not be an issue, but can be annoying to deal with during development and in tests.

To avoid this use Rationals

conversion_factor = Rational('1') / Rational('3.6')
# => (5/18)
result = Rational('7.2') * conversion_factor
# => (2/1)
result.to_d(0)
# => 0.2e1

Note

The 0 in result.to_d(0) is the required precision parameter Show archive.org snapshot , which is applied to the BigDecimal return value.

Using Rationals has some pitfalls you should be aware of:

  • Multiplying a Rational with a BigDecimal still produces rounding errors.

    BigDecimal('7.2') * (Rational('1') / Rational('3.6'))
    # => 0.20000000000000000016e1
    
  • Rationals don't like floats very much, when initializing a Rational with a float you don't get accurate values.

    float_rational = Rational(0.1)
    # => (3602879701896397/36028797018963968)
    float_rational.to_d(0)
    # => 0.100000000000000005551115123125782702e0
    
Posted by Martin Schaflitzl to makandra dev (2023-11-20 16:11)