Read more

Ruby: Generating and parsing JSON, or: understanding JSON::ParserError "unexpected token"

Dominik Schöler
March 02, 2021Software engineer at makandra GmbH

The described issue and suggestion only apply to the JSON gem in versions < 2. From JSON 2.0.0, a newer specification is implemented that is more liberal and will parse '"a"' to "a" without complaining.

json is part of the standard library of Ruby and deals with JSON Show archive.org snapshot , obviously. As you know, JSON is the string format that represents simple data structures. Ruby data structures that resemble Javascript objects can be serialized to JSON with #to_json. These can be restored from a JSON string with JSON.parse().

Illustration online protection

Rails professionals since 2007

Our laser focus on a single technology has made us a leader in this space. Need help?

  • We build a solid first version of your product
  • We train your development team
  • We rescue your project in trouble
Read more Show archive.org snapshot

So what could go wrong here?

JSON.parse("a".to_json)

It will raise JSON::ParserError (784: unexpected token at '"a"'). But why?

Generating JSON vs serializing objects

JSON is built on two data structures, "object" and "array". In Ruby these are Hash and Array. However, #to_json is defined on more classes than just Hash and Array. It will serialize Ruby objects to JSON values (see the "value" section on json.org Show archive.org snapshot ).

In order to parse a JSON value generated by #to_json, use JSON.load(). So the solution to the failing code example above is JSON.load("a".to_json). Since it creates Ruby objects from its input, you must not use JSON.load() with untrusted input.

What about JSON.parse()? Should we forget about it? Of course not. JSON.parse() should be used to parse a valid JSON string. How can one be generated? Use JSON.generate(). In contrast to #to_json, it will only generate valid JSON strings and fail early if it can't.

Summary

Handling valid JSON strings:

De-/Serializing Ruby objects to JSON values:

Also see How to fix: "unexpected token" error for JSON.parse.

Posted by Dominik Schöler to makandra dev (2021-03-02 15:41)