Month: February 2022

jpt 1.0 now has the ability to detect and help truncation in strings, arrays and objects. The -D option can detect this in newline delimited and concatenated JSON and -H will help the JSON by closing up strings, arrays, and objects. For example, here’s two JSON text concatenated both with truncation:

jpt -i0 -DH <<< $'{"a":[1,2 {"b":[3,4,'
{"a":[1,2]}
{"b":[3,4,null]}

#heredoc to more clearly show the input data
% jpt -i0 -DH <<EOF
heredoc> {"a":[1,2
heredoc> {"b":[3,4,
heredoc> EOF
{"a":[1,2]}
{"b":[3,4,null]}

If there is a trailing comma, a null will be put there to signify the fact there was an additional value before truncation occured, in all cases one should assume some data has been lost. The -i0 option is for zero indents, single line JSON however it is not NDJSON. When multiple JSON texts are input, as above, JSON Sequence (RFC 7464) are the default output. JSON Sequences are simply a record separator (RS) character at the beginning of JSON text. RS characters do not appear in the Terminal. If you pipe the output to bbedit or redirect to a file you’ll see them. To get some commentary on the repairs, use the -c option:

jpt -i0 -cDH <<< $'{"a":[1,2 {"b":[3,4,'         
{"a":[1,2]}
{"b":[3,4,null]}
--> Original error: SyntaxError: JSON Parse error: Expected ']'
--> Byte 10: Truncation Detected
--> Truncation help for JSON Text (1), bytes: 1-10
--> Truncation help after byte 10, array end: ]
--> Truncation help after byte 10, object end: }
--> Truncation help for JSON Text (2), bytes: 11-21
--> Truncation help after byte 21, missing value and array end: null]
--> Truncation help after byte 21, object end: }
--> Bytes 1-10: JSON Text (1)
--> Bytes 11-21: JSON Text (2)

echo $?                                 
1

As you can see -c can get pretty verbose depending on the issues. If there is any output with -c the exit code of jpt will be 1, if there is nothing to fix it will exit 0. This can be useful if you need to know if the source JSON needed any help or not.

jpt 1.0 is there when your JSON needs some help, not just when it’s perfectly formatted. If that sounds cool give it a try, download the Mac .pkg at github.com/brunerd/jpt/releases

JSON5 is not JSON nor is an IETF standard but jpt 1.0 can rehabilitate it back to standards compliant JSON. Let’s take the example from its web page.

{
  // comments
  unquoted: 'and you can quote me on that',
  singleQuotes: 'I can use "double quotes" here',
  lineBreaks: "Look, Mom! \
No \\n's!",
  hexadecimal: 0xdecaf,
  leadingDecimalPoint: .8675309, andTrailing: 8675309.,
  positiveSign: +1,
  trailingComma: 'in objects', andIn: ['arrays',],
  "backwardsCompatible": "with JSON",
}

Now let’s send it through jpt

% jpt json5_example.txt        
{
  "unquoted": "and you can quote me on that",
  "singleQuotes": "I can use \"double quotes\" here",
  "lineBreaks": "Look, Mom! No \\n's!",
  "hexadecimal": 912559,
  "leadingDecimalPoint": 0.8675309,
  "andTrailing": 8675309,
  "positiveSign": 1,
  "trailingComma": "in objects",
  "andIn": [
    "arrays"
  ],
  "backwardsCompatible": "with JSON"
}

That last line is a bit misleading. JSON5 texts are totally not backward compatible with JSON parsers. What they mean is that JSON5 parsers are backward compatible with JSON. FYI – Every single JSON5 addition is not compatible with JSON, I should know I had to write parsing rules and behaviors to deal with it! Let’s use the -c flag to comment on all the work that went into un-junking the above example

% jpt -c ./json5_example.txt 
{
  "unquoted": "and you can quote me on that",
  "singleQuotes": "I can use \"double quotes\" here",
  "lineBreaks": "Look, Mom! No \\n's!",
  "hexadecimal": 912559,
  "leadingDecimalPoint": 0.8675309,
  "andTrailing": 8675309,
  "positiveSign": 1,
  "trailingComma": "in objects",
  "andIn": [
    "arrays"
  ],
  "backwardsCompatible": "with JSON"
}
--> Original error: SyntaxError: JSON Parse error: Unrecognized token '/'
--> Byte 5: // line comment (JSON5)
--> Byte 19: unquoted object key name (JSON5)
--> Byte 29: single quoted string (JSON5)
--> Byte 63: unquoted object key name (JSON5)
--> Byte 77: single quoted string (JSON5)
--> Byte 113: unquoted object key name (JSON5)
--> Byte 138: \ escaped newline in string (JSON5)
--> Byte 153: unquoted object key name (JSON5)
--> Byte 164: 0x hex value (JSON5)
--> Byte 177: unquoted object key name (JSON5)
--> Byte 198: decimal point wihtout preceding digit (JSON5)
--> Byte 208: unquoted object key name (JSON5)
--> Byte 228: trailing decimal lacking mantissa (JSON5)
--> Byte 233: unquoted object key name (JSON5)
--> Byte 247: explicit plus + for value (JSON5)
--> Byte 253: unquoted object key name (JSON5)
--> Byte 268: single quoted string (JSON5)
--> Byte 282: unquoted object key name (JSON5)
--> Byte 290: single quoted string (JSON5)
--> Byte 296: trailing comma (JSON5)
--> Byte 336: trailing comma (JSON5)

Yeah there’s so much wrong with JSON5 vs JSON its not funny. It’s baffling that Apple is acknowledging it by adding JSON5 parsing to their frameworks! At least they don’t generate it! Despite not being funny, let’s try anyway, here’s some more JSON5 example to show some of the other ways it can lead you astray.

{
	"jpt_lineBreakHandling": "Extended parsing to ignore tabs \
	after escaped newlines.\nIf JSON5 is supposed to be human readable \
	shouldn't tabs be included in the escaping?",
	"leadingDecimalPoint": .8675309, 
	"andTrailingDecimal": 8675309.,
	"negativeLeadingDecimalPoint": -.1234, 

	//Infinity converts to null, use -I for string
	"+Infinity": +Infinity,
	"-Infinity": -Infinity,
	"Infinity": Infinity,
	//null is not a number now is it? done.
	"NaN": NaN,
}

Running this through jpt we’ll get pure JSON output

% jpt ./json5_examples_brunerd.txt 
{
  "jpt_lineBreakHandling": "Extended parsing to ignore tabs after escaped newlines.\nIf JSON5 is supposed to be human readable shouldn't tabs be included in the escaping?",
  "leadingDecimalPoint": 0.8675309,
  "andTrailingDecimal": 8675309,
  "negativeLeadingDecimalPoint": -0.1234,
  "+Infinity": null,
  "-Infinity": null,
  "Infinity": null,
  "NaN": null
}


#-I to convert to signed Infinity strings instead of null
#-8 to convert NaN to string "NaN" (-N was taken, like NaN 8 is like Infinity but not quite :)
% jpt -I8 ./json5_examples_brunerd.txt 
{
  "jpt_lineBreakHandling": "Extended parsing to ignore tabs after escaped newlines.\nIf JSON5 is supposed to be human readable shouldn't tabs be included in the escaping?",
  "leadingDecimalPoint": 0.8675309,
  "andTrailingDecimal": 8675309,
  "negativeLeadingDecimalPoint": -0.1234,
  "+Infinity": "+Infinity",
  "-Infinity": "-Infinity",
  "Infinity": "Infinity",
  "NaN": "NaN"
}

The fact that they added the data type of +/- Infinity and NaN really complicates the conversion since JSON doesn’t have equivalents. By default Infinity converts to null but -I will convert it to the string "Infinity" with signedness if specified. NaN is also converted to null by default but the -8 flag will convert it to the string "NaN". Why 8? Because it’s sort like infinity but it’s not, just like NaN.

If you have need to rehab JSON5 back to JSON then jpt might be able to help you out. It can run on Mac all the way back to OS X Tiger (10.4) all the way up to the newest macOS Monterey (12) and on other *nixes with jsc installed. Check the jpt Releases page