Ruby: Generating and parsing JSON, or: understanding JSON::ParserError "unexpected token"

Posted . Visible to the public. Deprecated.

The described issue and suggestion only apply to the JSON gem in versions < 2. From JSON 2.0.0, a newer specification is implemented that is more liberal and will parse '"a"' to "a" without complaining.

json is part of the standard library of Ruby and deals with JSON Show archive.org snapshot , obviously. As you know, JSON is the string format that represents simple data structures. Ruby data structures that resemble Javascript objects can be serialized to JSON with #to_json. These can be restored from a JSON string with JSON.parse().

So what could go wrong here?

JSON.parse("a".to_json)

It will raise JSON::ParserError (784: unexpected token at '"a"'). But why?

Generating JSON vs serializing objects

JSON is built on two data structures, "object" and "array". In Ruby these are Hash and Array. However, #to_json is defined on more classes than just Hash and Array. It will serialize Ruby objects to JSON values (see the "value" section on json.org Show archive.org snapshot ).

In order to parse a JSON value generated by #to_json, use JSON.load(). So the solution to the failing code example above is JSON.load("a".to_json). Since it creates Ruby objects from its input, you must not use JSON.load() with untrusted input.

What about JSON.parse()? Should we forget about it? Of course not. JSON.parse() should be used to parse a valid JSON string. How can one be generated? Use JSON.generate(). In contrast to #to_json, it will only generate valid JSON strings and fail early if it can't.

Summary

Handling valid JSON strings:

De-/Serializing Ruby objects to JSON values:

Also see How to fix: "unexpected token" error for JSON.parse.

Dominik Schöler
Last edit
Dominik Schöler
License
Source code in this card is licensed under the MIT License.
Posted by Dominik Schöler to makandra dev (2021-03-02 14:41)