JSON command-line toolbox (jq, gron, jc, etc)

Posted on 2023-12-12, by Racum. Tags: JSON Tools

Table of Contents

The JSON format is the backbone of current Internet protocols and APIs, and this article contains a list of tools to make a better use of it from the command-line, without the need of writing complex programs to read, search, parse, transform and transmit JSON data.

For the sake of example, every time you see test.json here, it refers to a file with the following content:

{
  "countries": [
    {
      "code": "sm",
      "name": "San Marino",
      "population": 33881
    },
    {
      "code": "mc",
      "name": "Monaco",
      "population": 39050
    }
  ]
}

Querying with "jq"

This is one of the sharpest tools: jq; it applies a query language over a JSON content: you define the “path” of your data inside the JSON structure, and you get the object(s) referenced back.

For example: “get the name of the first country”:

$ cat test.json | jq '.countries[0].name'
"San Marino"

One of the most common use-cases of jq if iterating over a list, for that, just ommit the index of an array to match all items inside it.

Example: “get the name of all countries”:

$ cat test.json | jq '.countries[].name'
"San Marino"
"Monaco"

Please refer of the jq Manual, you’ll see that the tool is very advanced, with support to recursion, math/string operations, conditionals, regular expressions, variables, etc. Or, if you think the manual is too long for your taste, try The Ultimate Interactive JQ Guide: this is the best entry point if you want a “crash course” about jq!

Before moving forward to the next tool, here is a quick tip: use jq without parameters to prettify a JSON file, or jq -c to minimize it:

$ cat test.json | jq -c
{"countries":[{"code":"sm","name":"San Marino","population":33881},{"code":"mc","name":"Monaco","population":39050}]}

Searching with "gron"

The tagline of gron is "Make JSON greppable!", and yes! it does exactly that! To be more precise: it flattens a hierarchical JSON file into a simple key-value list of all leaf elements, making it perfect to be searched (or “grepped”).

For example...

$ cat test.json | gron
json = {};
json.countries = [];
json.countries[0] = {};
json.countries[0].code = "sm";
json.countries[0].name = "San Marino";
json.countries[0].population = 33881;
json.countries[1] = {};
json.countries[1].code = "mc";
json.countries[1].name = "Monaco";
json.countries[1].population = 39050;

...could be grepped by any key or value, like this:

$ cat test.json | gron | grep code
json.countries[0].code = "sm";
json.countries[1].code = "mc";

And the coolest part is that this operation is reversible, you can “ungron” gronned data:

$ cat test.json | gron | grep code | gron -u
{
  "countries": [
    {
      "code": "sm"
    },
    {
      "code": "mc"
    }
  ]
}

Pro tip: the first thing you should do after install gron is to alias “ungron” on your shell, like this: alias ungron="gron -u", and so you can easily get in the flow to gron ➤ grep ➤ ungron.

Crafting with "jo"

Raw JSON is not very terminal-friendly, it forces you to alternate single and double quotes, making the process very error-prone; for situations like that, try to synthesize JSON values with jo: it is a simple tool that transforms command-line parameters into JSON objects or list (with -a):

$ jo a=1 b=2
{"a":1,"b":2}

$ jo -a 1 2 3
[1,2,3]

Here is a more complete example (the -p stands for "pretty"):

$ jo -p code=ad name=Andorra population=83523
{
   "code": "ad",
   "name": "Andorra",
   "population": 83523
}

A jo call can be nested inside another jo call, reflecting the structure of JSON you want to craft:

$ jo -p countries=$(jo -a $(jo code=ad name=Andorra population=83523))
{
   "countries": [
      {
         "code": "ad",
         "name": "Andorra",
         "population": 83523
      }
   ]
}

Merging with "spruce"

Spruce is actually an YAML tool, but it understands JSON input. This is a merging tool, it combines two structures (objects get added, lists get appended, etc).

For example, consider the combination of our test.json with the new fiji.json file:

$ cat fiji.json
{"countries": [{"code": "fj", "name": "Fiji", "population": 893468}]}

$ spruce merge test.json fiji.json
countries:
- code: sm
  name: San Marino
  population: 33881
- code: mc
  name: Monaco
  population: 39050
- code: fj
  name: Fiji
  population: 893468

And, of course, you can use Spruce itself to transform that combined YAML back into JSON (I’m only adding a jq here to make it pretty, since Spruce JSON output is minimized):

$ spruce merge test.json fiji.json | spruce json | jq
{
  "countries": [
    {
      "code": "sm",
      "name": "San Marino",
      "population": 33881
    },
    {
      "code": "mc",
      "name": "Monaco",
      "population": 39050
    },
    {
      "code": "fj",
      "name": "Fiji",
      "population": 893468
    }
  ]
}

Connecting with "curl"

This is an old friend of all developers: curl, the command-line “swiss army knife” for all network protocols! If you are messing with JSON, you’ll be piping from/to curl sooner or later!

But, before we get into curl’s JSON features, please meet a very nice debug service: httpbin.org, this reflects back your request (URL, header, body, etc) in JSON format, so you can understand better what is being transferred. I'll be using the /anything endpoint in my examples below.

The simplest use of curl is calling it just with the URL as a parameter, this triggers a GET request, like so:

$ curl https://httpbin.org/anything
{
  "args": {},
  "data": "",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.5.0",
  },
  "json": null,
  "method": "GET",
  "origin": "123.123.123.123",
  "url": "https://httpbin.org/anything"
}

If you pass data with the --json parameter, it triggers a set of changes:

$ curl https://httpbin.org/anything --json '{"hello": "world"}'
{
  "args": {},
  "data": "{\"hello\": \"world\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json",
    "Content-Length": "18",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.5.0",
  },
  "json": {
    "hello": "world"
  },
  "method": "POST",
  "origin": "123.123.123.123",
  "url": "https://httpbin.org/anything"
}

Notice that:

The method became automatically a POST.
The "Content-Type" and "Accept" headers were set to application/json.
The content was sent inside the request body.

The --json shortcut replaces the pattern curl -X POST … -H … -d … with a single convenient parameter. And it also supports different ways to load data:

Load from file: curl https://httpbin.org/post --json @test.json
Pipe from another command: jo hello=world | curl https://httpbin.org/post --json @-

Importing everything with "jc"

This tool is just overkill! jc can convert basically everything you throw to it into JSON.

From XML

Before the popularity of JSON, XML was the preferred format in the late 90’s and early 2000’s. It is rarely picked as a format for new modern APIs, but protocols created at that time are still used today; for example most websites have a RSS feed, or a sitemap, those are just XML files.

Consider the following test.xml example file...

<countries>
    <country code="sm">
        <name>San Marino</name>
        <population>33881</population>
    </country>
    <country code="mc">
        <name>Monaco</name>
        <population>39050</population>
    </country>
</countries>

...gets parsed into:

$ cat test.xml | jc --xml | jq
{
  "countries": {
    "country": [
      {
        "@code": "sm",
        "name": "San Marino",
        "population": "33881"
      },
      {
        "@code": "mc",
        "name": "Monaco",
        "population": "39050"
      }
    ]
  }
}

Notice that inner-tag parameters are supported, with a @ prefix.

From everything else

It supports:

Data formats: YAML, CSV, TOML, INI and PLIST.
Most of the common terminal commands, like crontab, df, dig, env, file, find, free, ls, ping, ps, uname, etc.
Addresses: URLs, emails and IPs (v4 and v6).
JWTs (JSON Web Tokens).
Dates: ISO-8601 and timestamp.
A lot more! ...please refer to the list of parsers.

Fun combinations

First (last?) entries of a blog

$ curl https://racum.blog/rss.xml | jc --xml | jq -r '.rss.channel.item[].title' | head -5
Minimal Docker for GeoDjango + PostGIS
Back to Blogging
Sherlock & Co
Paquito D'Riveria - Mozart's Adagio
Festival Interceltique de Lorient 2023

Access public APIs

Check out this list of public APIs and have fun with it!

For example, here is one to guess your country based on your IP:

$ curl https://api.country.is | jq -r '.country'
DE

And another one to tell times for sunrise and sunset:

$ curl "https://api.sunrise-sunset.org/json?tzId=Europe/Berlin&lat=52.522&lng=13.411" \
  | gron | grep sun | gron -u | jq '.results'
{
  "sunrise": "8:01:10 AM",
  "sunset": "3:55:02 PM"
}

Or, iterate over many API pages to get the names of all Star Wars characters:

$ curl 'https://swapi.dev/api/people/?page=[1-9]' | jq -r '.results[].name'
Luke Skywalker
C-3PO
R2-D2
Darth Vader
Leia Organa
...

Conclusion

I hope this article expanded your toolbox! JSON is a very important format for all kind of developers and data scientists, and knowing what is possible from the command-line can help a lot during debug and automation.