How to Use jq Command to Process JSON in Linux

How to Use jq Command to Process JSON in Linux

Jq is a effective and highly convenient parser prospectus that can stream and filter JSON documents out of records and UNIX hoses. This article will teach you the requisites of jq, current code examples, as nicely as some with one voice with one voice-natural makes that you can mount today.

What Is jq Offered For?

The most typical utilise for jq is for handling and manipulating JSON responses from Software application-as-a-Solution (SaaS) APIs. For instance, you can utilise jq along through cURL to faucet into Digitalocean’s API endpoints to avail your account niceties.

Aside from that, jq is in a indistinguishable way a effective energy for governing big JSON records. Some of the most newfangled-fashioned database regimen today such as MongoDB, PostgreSQL, and MySQL stabilize JSON as a way to storefront documents. As such, situating out jq provides you an side in specialization how those database contraptions job.

Amazing to comprehend: locate out some of the pristine contraptions to edit JSON records inside Chrome.

Installing and Making gain service of of jq

To overture through jq, mount its binary package to your mechanism:

sudo apt install jq

Position an responsive API endpoint that you can check-up jq on. In my flay, I’m visiting utilise the ipinfo.io’s IP mosaic API.

The most natural filter for jq is the dot (.) filter. This will pretty print the JSON answer as jq retrieved it from its perfunctory input:

curl https://ipinfo.io/ | jq '.'

Another natural filter is the pipeline (|) icon. This is a systematic filter that enacts the outcome of one filter as the input of one more:

curl https://ipinfo.io/ | jq '. | .ip'

The top crispness after the pipeline vehicle driver is the “Object Identifier-Index.” This scours your JSON input for any kind of variable that suits its text and prints its top crispness on the incurable. In this flay, I’m glancing for the top crispness of the “ip:” way.

Doning the requisites used and dusted, the complying through sectors will substantiate you some of the secrets that you can implement making utilise of jq.

Amazing to comprehend: locate out how you can readjust text streams making utilise of sed in Linux.

1. Concocting a Fundamental Feed Traveller through jq

Most vibrant websites today administer responsive API endpoints for analysis documents inside their contraptions. For instance, every Github repository possesses its super own API URL for you to retrieve the latest dedicates and troublemakers for that project.

You can utilise an API endpoint like this through jq to invent your super own humble “RSS-like” feed. To overture, utilise cURL to check-up if the endpoint is working properly:

curl https://api.github.com/repos/bitcoin/bitcoin/issues

Run the complying through to print the initially access in your feed:

curl https://api.github.com/repos/bitcoin/bitcoin/issues | jq '.[0]'

This will substantiate the miscellaneous arenas that the Github API sends to jq. You can utilise these to invent your super own custom JSON points by piping the input to the curly braces ({}) filter:

curl https://api.github.com/repos/bitcoin/bitcoin/issues | jq '.[0] | { title: .title }'

Adding the comma (,) filter inside the curly braces makes it viable for you to encompass multiple arenas to your custom points:

curl https://api.github.com/repos/bitcoin/bitcoin/issues | jq '.[0] | {title: .title, url: .html_url, author: .user.login}'

Retrieving rid of the “0” inside the square braces will apply your jq filter to the entirety feed:

curl https://api.github.com/repos/bitcoin/bitcoin/issues | jq '.[] | {title: .title, url: .html_url, author: .user.login}'

You can in a indistinguishable way invent a tiny Fun script to brandish the latest troublemakers from your favorite Github project. Paste the complying through block of code inside an unfurnished capsule script document:

#!/bin/bash

# usage: ./script.sh [0 ... 29]

REPO="https://api.github.com/repos/bitcoin/bitcoin/issues"

curl $REPO | jq ".[$1] | {title: .title, url: .html_url, author: .user.login}"

Conserve your document, after that sprinted the complying through command to render it executable:

chmod u+x ./script.sh

Checkup your newfangled feed viewers by checklist the latest question in your favorite Github repo:

./script.sh 0

FYI: locate out how to invent custom RSS feeds making utilise of RSS-Bridge in Linux.

2. Estimation and Scouring through a JSON Documents source

Aside from analysis documents off of APIs, you can in a indistinguishable way utilise jq to steal care of JSON records in your municipal machine. Prelude by inventing a humble JSON database document making utilise of your favorite text editor:

nano ./database.json

Paste the complying through block of documents inside your document, after that conserve it:

[<br>    {"id": 1, "name": "Ramces", "balance": 20},<br>    {"id": 2, "name": "Alice", "balance": 30},<br>    {"id": 3, "name": "Bob", "balance": 10},<br>    {"id": 4, "name": "Charlie", "balance": 20},<br>    {"id": 5, "name": "Maria", "balance": 50}<br>]

Checkup whether jq investigates your JSON document properly by printing the initially points in your database accord:

jq '.[0]' database.json

Administer a inquiry on your JSON database making utilise of the “Object Identifier-Index” filter. In my flay, I am scouring for the top crispness of the “.tag” way on every access in my database:

jq '.[] | .name' database.json

You can in a indistinguishable way utilise some of jq’s constructed-in chores to filter your questions based on precise affectations. For instance, you can pursuit for and print with one voice the JSON objects that have a “.tag” top crispness through added than six individualities:

jq '.[] | select((.name|length)>6)' database.json

Operating on JSON Documents sources through jq

In enhancement to that, jq can sprinted on JSON documents sources the super same to a natural spreadsheet. For instance, the complying through command prints the sum unexpurgated of the “.stability” way for every points in the database:

jq '[.[] | .balance] | add' database.json

You can even prolong this by including a conditional news to your inquiry. The complying through will single encompass the “.stability” if the “.tag” top crispness of the 2nd points is “Alice”:

jq 'if .[1].name == "Alice" then [ .[] | .balance ] | add else "Second name is not Alice" end' database.json

It’s you can probably picture to momentarily recover rid of variables from your JSON database. This can be justifiable if you’re purging your filter and you yearn to render sure that it can still process your dataset:

jq 'del(.[1].name) | .[]' database.json

You can in a indistinguishable way insert newfangled variables to your database making utilise of the “+” vehicle driver. For instance, the complying through nonsense has the variable “spirited: true” to the initially points in the database:

jq '.[0] + {active: true}' database.json

Tab: You can render your matches incurable by piping the outcome of your jq command to your original database document: jq '.[0] + {active: true}' database.json > database.json.

3. Revolutionizing Non-JSON Information in jq

Another vivid center of jq is that it can accept and job through non-JSON documents. To attain that, the prospectus gains gain service of of an with one voice with one voice-natural “slurp position” in which it converts any kind of void and newline delimited documents into a JSON accord.

You can enable this center by piping documents into jq through an -s flag:

echo '1 2' | jq -s .

One merit of reinventing your raw documents into an accord is that you can address them making utilise of accord index digits. The complying through command has 2 payoffs by referring to their revolutionized accord void:

echo '1 2' | jq -s '.[0] + .[1]'

You can steal this accord void even more and construct newfangled JSON code roughly it. For instance, this code converts the text from the echo command to a JSON points through the curly braces filter:

echo '6 "Mallory" 10' | jq -s '{"id": .[0], "name": .[1], "balance": .[2]}'

Indifferent from seizing in raw documents, jq can in a indistinguishable way antiphon non-JSON documents as its outcome. This is justifiable if you’re making utilise of jq as part of a bigger capsule script and you single call for the outcome from its filters.

To implement that, sprinted jq adhered to by the -r flag. For instance, the complying through command investigates with one voice the names from my database document and reverts it as raw text documents:

jq -r '.[] | .name' database.json

All with one voice-natural JSON Parsers to jq

Since the code for jq is responsive source, assorted founders have unleashed their super own designs of the JSON parser. Each of these possesses its super own unprecedented selling time that either improves on or matches a core part of jq.

1. jaq

Jaq is a effective JSON parser that provides a unprejudiced homogenous center kit to jq.

Concocted in Rust, one of the greatest selling elements of Jaq is that it can sprinted the jq language upwards to 30 times sooner than the original parser while still retaining in overturn compatibility. This alone performs it justifiable once you’re running big jq filters and you yearn to maximize the suitability out of your machine.

That claimed, one disadvantage of jaq is that it’s not currently accessible on Debian, Ubuntu, and Fedora repositories. The single way to attain it is to either download Homebrew or compile it from source.

2. gojq

Gojq is an with one voice with one voice-natural JSON parser that’s written solitarily in Go. It provides an accessible and simplistic-to-utilise difference of jq that you can mount on practically any kind of system.

The original jq prospectus can be unbelievably terse in its misstep posts. As a outcome, debugging jq scripts is specifically hard for a newfangled jq audience. Gojq solves this question by authenticating you in which the misstep is in your script as nicely as offering extensive posts on the kind of misstep that arised.

Another selling time of gojq is that it can read and process both JSON and YAML records. This can be specifically advantageous if you’re a Docker and Docker Devise audience and you yearn to automate your deployment workflow.

Gojq’s greatest question is that it remote some of the features that come by default on the original jq parser. For instance, selections such as --ascii-output, --seq, and --sort-keys doesn’t exist on gojq.

FYI: locate out how to streamline your JSON code by making utilise of some of the pristine JSON beautifiers today.

3. fq

Polymorphous jaq and gojq, fq is a diligent software training course toolkit that can review both text and binary documents. It can job through a hodgepodge of newfangled-fashioned layouts such as JSON, YAML, HTML, and even FLAC.

The greatest center of fq is that it has a constructed-in hex viewers for records. This performs it insignificant to stare at a document’s internal structure to discern how it’s earned and whether there’s anything erroneous through it. Aside from that, fq in a indistinguishable way gains gain service of of the super same syntax for jq once solving text which performs it simplistic to locate out for any individual already acquainted through jq.

One disadvantage of this enthusiastic urge is that fq is still under heavy delusion. As such, some of the prospectus’s features and actions are still subject to sweeping matches.

Touring jq, how it works, and what performs it systematic is simply the initially interfere situating out how to render regimen on your computer. Nab a deep dive into the magical universe of coding by analysis the requisites of capsule shows.

Image credit report: Ferenc Almasi through Unsplash. All alterations and screenshots by Ramces Red.