A funny thing happened on the way to writing a new blog entry. Instead of finding an example JSON file that ships with macOS, I stumbled upon a slow creep of malformed JSON files that started in OS X El Capitan and continues into Big Sur. While I can’t do much about deviant JSON files in macOS, I can control how jpt handles them. So like any good Teachable Moment™ and life lesson, I’ve turned lemons into lemonade and out of challenge comes innovation: jpt v0.9.6 is now nonplussed by these uncouth JSON files, with their comments and trailing commas. Like water off ducks back, jpt keeps on keepin’ on, unphased.
Finding the Misfits
To find these misfits I wrote made a script jsonParsingReportCSV-macOS. This will attempt to parse the JSON it finds using the json_pp binary that ships from Apple, the results are output as CSV and you can then easily sort it to spot troublemakers.
I’ve uploaded a couple annotated version of those searches along with the kind of deviation they exhibit: Malformed-macOS-10.15.csv and Malformed-macOS-11.1.csv
What’s their damage, man?
The two most prevalent kinds of JSON malformation I found in macOS were: trailing commas and comments. Trailing commas meaning the last item in an array or object should not have a comma after it. While Javascript has always allowed trailing commas in array literals and later in object literals (ES5), then in function parameters (ES2017), JSON has never allowed trailing commas nor comments. Neither are in the JSON spec (RFC8259).
The biggest group of offenders were the USD generated JSONs within USDKit (a private framework) and Model I/O. USD is interesting because of it’s lineage and purpose: It’s Pixar’s open source software for combining multiple 3D elements within a common scene. Or as Pixar’s page put it: “Universal Scene Description (USD) is the first publicly available software that addresses the need to robustly and scalably interchange and augment arbitrary 3D scenes that may be composed from many elemental assets.” Keep an eye on this framework 🕶
JSON Lines (Don’t Do It)
An outlier in the mix is defaultConfig.json within CoreAnalytics.framework, it is what some call “JSON Lines” or “concatenated JSON”, where each line is its own a JSON value or object. While doing some research on JSON Lines I came upon an intriguing comment in this post by rurban:
“The delimiters don’t get in the way, they protect you from MITM and other attacks. It’s a security feature, not a bug.”
Reini Urban on JSON vs. “concatenated JSON”
Taking a look at Reini’s GitHub and the level of his work, I’m going to give his statement some credence. To that end, I really didn’t want to add another output mode nor change query behavior to accommodate JSON Lines. My current compromise is to simply to attempt conversion of a JSON Lines file into an array: tack on commas and ensconce in square brackets. It Just Works™ if every line is truly a self contained JSON object. If you want to be made aware of this, use the -c option so jpt it will “complain” on stderr and exit with code 1.
The Rabbit Hole of CoreAnalytics
Within the CoreAnalytics private framework is defaultConfig.json, it is a collection of 4019 objects currently that describe the events and actions Apple is interested in collecting usage information on. They want to know if you’re actually using all those new features in CarPlay, Reminders, Photos, eGPU App usage, etc… If you want to see all the key names try this: jpt $..addTransform.name /System/Library/PrivateFrameworks/CoreAnalytics.framework/Versions/A/Resources/defaultConfig.json
A Comment on JSON Comments
Comments in JSON has inspired debate far and wide, this post by getify sums it up very nicely:
A JSON encoder MUST NOT output comments. A JSON decoder MAY accept and ignore comments.
Douglas Crockford in some old Yahoo group that even Wayback can’t find anymore
So that’s just what jpt does now, it will never encode comments but it will strip out any and all comments it recognizes automatically, no switches needed. Which style comments? All of them (or most of them):
| Comment | kind | lineage |
# comment | line | shell, Perl, PHP, Python, R, Ruby, MySQL |
// comment | line | C(99), Javascript, Java, Swift |
/* comment */ | block | C, Java, Javascript, Swift |
; comment | line | assembly language (ASM) |
-- comment | line | Ada, Applescript, Haskell, Lua, SQL |
<!-- comment --> | block | XML |
*** comment *** | line | macOS’s ionodecache.json |
If you need a test file you can use my worstJSONEver.json. A most horrible JSON file if there ever was one! jpt soldiers on, use -c to see what it does to clean the input (-i0 for no level indents). It shows the initial parse error and each attempt at recovery.
Fixing the Misfits
There is no fixing, there is only acceptance. Since most of these files are in SIP protected areas of macOS so it will take an act of God, a well filed bug report, for Apple to correct these in a future update. But they usually only act if there’s a demonstrable bug, in this case it’s more of an observation about the condition of the JSON files than a clear issue. If someone is really bothered, they could disable SIP and go in and fix them, however future macOS updates will likely revert these changes. Since I haven’t tried this, who knows, macOS may get ornery and trip OS integrity protections, rendering your system un-bootable, YMMV 🤷♂️. The real question is if they impact how macOS works and I’m guessing they don’t. The apps and frameworks which use these off-spec JSON file must be doing something to correct the issues before parsing. Rest assured though, if you attempt to parse one of these files with jpt there will be no issues, it Just Works™. However if you would like to know it did something, use -c to “complain” about comments, commas, JSON Lines, and hard wrapping. It will send those notices to stderr and exit with a status of 1.
jpt: malformed JSON? Bring it on.
So after all this I don’t have a good example page for jpt’s multitude of features and usage (yet) but what I do have is a JSON Power Tool with more “dummy-proof” features than it did before. I’ll take that as a win! Check out the latest release of jpt at my Github.
jpt (JSON Power Tool) is a non-compiled tool that can manipulate and query JSON in a variety of ways, it runs on macOS (10.4 – 11.0) PPC/Intel/Apple Silicon; Linux with jsc; Windows with Linux subsystem for Windows and jsc. It can be used standalone or embedded within your shell scripts. I hope you find it useful. Thanks for reading!