Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.

All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.

You can make a much simpler version of your jq program:

jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)'

which outputs:

URL='example.com'
AUTHOR='John Doe'
CREATED='10/22/2017'

There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.

All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.

This string is evalable if you completely trust the input data and has the effect you want:

eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)"
echo "$AUTHOR"

As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.

If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:

jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...'

You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.


It's also possible to build up an associative array instead where both keys and values are quoted:

eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))"

After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.

Answer from Michael Homer on Stack Exchange
Top answer
1 of 4
56

Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.

All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.

You can make a much simpler version of your jq program:

jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)'

which outputs:

URL='example.com'
AUTHOR='John Doe'
CREATED='10/22/2017'

There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.

All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.

This string is evalable if you completely trust the input data and has the effect you want:

eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)"
echo "$AUTHOR"

As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.

If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:

jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...'

You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.


It's also possible to build up an associative array instead where both keys and values are quoted:

eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))"

After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.

2 of 4
20

Building on @Michael Homer's answer, you can avoid a potentially-unsafe eval entirely by reading the data into an associative array.

For example, if your JSON data is in a file called file.json:

#!/bin/bash

typeset -A myarray

while IFS== read -r key value; do
    myarray["value"
done < <(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + .value ' file.json)

# show the array definition
typeset -p myarray

# make use of the array variables
echo "URL = '${myarray[URL]}'"
echo "CREATED = '${myarray[CREATED]}'"
echo "AUTHOR = '${myarray[URL]}'"

Output:

$ ./read-into-array.sh 
declare -A myarray=([CREATED]="10/22/2017" [AUTHOR]="John Doe" [URL]="example.com" )
URL = 'example.com'
CREATED = '10/22/2017'
AUTHOR = 'example.com'
Discussions

Writing bash code for array with multiple values
I wouldn't recommend using bash to parse JSON into a RDBMS. That's not really in its wheelhouse. For one thing, bash arrays can't be nested, so it doesn't really have its own way of storing values decoded from JSON. What I usually do when I have some JSON I need multiple things from in a bash script is just store the JSON itself in a variable and run jq on it as needed to extract bits, e.g. json=$( More on reddit.com
🌐 r/bash
12
7
October 24, 2022
linux - save json jq output as seperate variables or as an array - Stack Overflow
I want to be able to have both ... in a array, that way it's not dependent on me having 2 workspaces exclusively, as for the format i would like no quotation marks just the variable example 7:Entertainment or 1:Browsing ... Assuming the values in .name have no spaces or certain other characters, you would be able to get away with something along the lines of the following: X=($(i3-msg -t get_workspaces | jq -r '.[] | ... More on stackoverflow.com
🌐 stackoverflow.com
Assigning an Array Parsed With jq to Bash Script Array - Stack Overflow
I parsed a json file with jq like this : # cat test.json | jq '.logs' | jq '.[]' | jq '._id' | jq -s It returns an array like this : [34,235,436,546,.....] Using bash script i described an array... More on stackoverflow.com
🌐 stackoverflow.com
bash - jq: output array of json objects - Stack Overflow
44 How to convert a JSON object stream into an array with jq More on stackoverflow.com
🌐 stackoverflow.com
🌐
jq
jqlang.org › manual
jq 1.8 Manual
This option passes a JSON-encoded value to the jq program as a predefined variable. If you run jq with --argjson foo 123, then $foo is available in the program and has the value 123.
🌐
Julius Gamanyi
til.juliusgamanyi.com › posts › jq-bash-read-multiple-vars
jq with bash - read multiple values and save in variable | Julius Gamanyi
April 18, 2023 - Sometimes I need to read multiple string values from json, and save them into their own variables as a pre-processing step. Assigning multiple variables in Bash using process substitution, i.e., <(command) and using join with an empty string saved the day. Let the variable jsonData hold example json (from another jq TIL) using the here-doc
🌐
Reddit
reddit.com › r/bash › writing bash code for array with multiple values
r/bash on Reddit: Writing bash code for array with multiple values
October 24, 2022 -

I have a json file which i'm using bash to extract.

sample.json

{"extract": { "data": [ {"name": "John Smith", "id": 8752, "address": "1 Anywhere Street", "tel": 1234567890, "email": "john.smith@gmail.com" }, { "name": "Jane Smith", "id": 4568, "address": "719 Anywhere Street", "tel": 0987654321, "email": "janesmith@hotmail.com" } ] } }

and store the value within an array

id=($(cat sample.json | jq -r '.extract.data[] .name'))

so in the case of ${id[0]} will output John Smith and ${id[1]} will output Jane Smith.

I am intending to store the values in a database (this will be my first attempt) which will be in a similar to that of the json, each object needs to be relative to how it is in the json so it might be better to go with:

data1=($(cat sample.json | jq -r '.extract.data[0] | .[]))

Lets say i have 1000 names to save to my database along with their id's. I'm some advice whether if there a more sensible (more effective) approach on how:

- Pull the data from Json? will I need to write this 1000 times?e.g

data1=($(cat sample.json | jq -r '.extract.data[0] | .[]))
data2=($(cat sample.json | jq -r '.extract.data[1] | .[]))
data3=($(cat sample.json | jq -r '.extract.data[2] | .[]))
..
data1=($(cat sample.json | jq -r '.extract.data[1000] | .[]))

-Put the data into the DB from the first array? will the code need to reference the array as:

${data1[0]}
${data1[1]} 
${data1[2]}

Would be grateful for a steer in the right direction? - thanks.

🌐
Phpfog
phpfog.com › home › blog › using variables in jq (command line json parser)
Using Variables in JQ (Command Line JSON Parser) - PHPFog.com
January 29, 2021 - To use a simple variable in jq use the --arg option. Using --arg var value will set the variable $var to value.
Find elsewhere
Top answer
1 of 5
33

We can solve this problem by two ways. They are:

Input string:

// test.json
{
    "keys": ["key1","key2","key3"]
}

Approach 1:

1) Use jq -r (output raw strings, not JSON texts) .

KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]

2) Use @sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.

KEYS=$(<test.json jq -r '.keys | @sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'

3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.

KEYS=$((<test.json jq -r '.keys | @sh')| tr -d \') 
echo $KEYS
# Output: key1 key2 key3

4) We can convert the comma-separated string to the array by placing our string output in a round bracket(). It also called compound Assignment, where we declare the array with a bunch of values.

ARRAYNAME=(value1 value2  .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | @sh') | tr -d \'\"))

echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}

# Output: 
# Array size:  3
# Array elements: key1 key2 key3

Approach 2:

1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.

#!/bin/bash
KEYS=$(jq -r '.keys' test.json  | tr -d '[],"')
echo $KEYS

# Output: key1 key2 key3

2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().

#!/bin/bash
KEYS=($(jq -r '.keys' test.json  | tr -d '[]," '))

echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}

# Output:
# Array size:  3
# Array elements: key1 key2 key3
2 of 5
16

To correctly parse values that may have newlines (and any other arbitrary (non-NUL) characters) use jq's @sh filter to generate space-separated quoted strings, and Bash's declare -a to parse the quoted strings as array elements. (No pre-processing required)

foo.json:

{"data": ["$0", " \t\n", "*", "\"", ""]}
str=$(jq -r '.data | @sh' foo.json)
declare -a arr="($str)"   # must be quoted like this
declare -p arr
# declare -a arr=([0]="\$0" [1]=$' \t\n' [2]="*" [3]="\"" [4]="")

Update: jq 1.7 (2023-09)

As of version 1.7, jq has a --raw-output0 option, enabling it to output null-terminated strings which can be read into an array as usual:

mapfile -d '' arr < <(jq --raw-output0 '.data[]' foo.json)
wait "$!"  # use in bash-4.4+ to get exit status of the process substitution

Note on NUL characters in JSON strings

JSON strings may contain NUL characters while shell variables cannot. If your JSON input may contain NUL's, you may need to add some special handling.

  • When using the @sh filter, NUL characters from JSON strings will be silently replaced with the sequence \0. Note that this makes the JSON strings "\\0" and "\u0000" indistinguishable.

  • When using the --raw-output0 option, NUL characters will trigger an error and jq will terminate with an exit status of 5.

Reading multiple/nested arrays

The @sh filter can be combined with --raw-output0 to reliably read multiple arrays at once (or a single nested array) as it will produce a NUL-separated list of space-separated quoted strings.

json='[[1,2],[3,4]]' i=0
while read -r -d ''; do
    declare -a "arr$((i++))=($REPLY)"
done < <(jq --raw-output0 '.[]|@sh' <<<$json)
for ((n=0; n<i; n++)); { declare -p "arr$n"; }
# declare -a arr0=([0]="1" [1]="2")
# declare -a arr1=([0]="3" [1]="4")
🌐
Exercism
exercism.org › tracks › jq › concepts › variables
Variables in jq on Exercism
$ jq -c -n '42 | (. * 2) as $double | [., $double]' [42,84] The scope of the variable is "lexical" -- the variable will be in scope for the rest of the pipeline. Variables defined in functions (we'll get to functions later) are "local" to the function.
🌐
Programming Historian
programminghistorian.org › en › lessons › json-and-jq
Reshaping JSON with jq | Programming Historian
May 24, 2016 - [.tag, .count] | Create simple arrays with just the tag name and count ... These final challenges will help you test your understanding of how to pipe together jq commands on your own. What function do we need to add to the hashtag-counting filter to only count hashtags when their tweet has been retweeted at least 200 times? Hint: the retweet count is saved under the key retweet_count.
🌐
GitHub
gist.github.com › joar › 776b7d176196592ed5d8
Add a field to an object with JQ · GitHub
$ echo '{"hello": "world"}' | jq --arg foo bar '. + {foo: ("not" + $foo)}' { "hello": "world", "foo": "notbar" } ... I have json which is an object containing a property that is an array, and I want to create another property with the same value as an existing property, without changing anything else.
Top answer
1 of 4
66

You can use separate variables with read :

read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' | 
     jq -r '.id, .name, .full_name'))

echo "id        : $var1"
echo "name      : $var2"
echo "full_name : $var3"

Using array :

read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' | 
     jq -r '.id, .name, .full_name'))

echo "id        : ${arr[0]}"
echo "name      : ${arr[1]}"
echo "full_name : ${arr[2]}"

Also you can split jq output with some character :

IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data | 
    map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] | 
    join("|")) | join("\n")')

Or use an array like :

set -f; IFS='|' data=($(curl '......' | jq -r '.data | 
    map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] | 
    join("|")) | join("\n")')); set +f

absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.

For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")

If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:

IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data | 
    map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] | 
    join("|")) | join("\n")')
2 of 4
8

jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:

.data[] | ( .episodeName, .id )

For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.

For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:

Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?

🌐
jq
jqlang.org › manual › v1.6
jq 1.6 Manual
This option passes a JSON-encoded value to the jq program as a predefined variable. If you run jq with --argjson foo 123, then $foo is available in the program and has the value 123.
Top answer
1 of 6
36

Using jq :

readarray arr < <(jq '.[].item2' json)
printf '%s\n' "${arr[@]}"

If you need a more hardened way:

readarray -td '' arr

for inputs with newlines or other special characters, avoiding word splitting.

Output:

value2
value2_2

Check:

Process Substitution >(command ...) or <(...) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection: cmd1 2> >(cmd2). See http://mywiki.wooledge.org/ProcessSubstitution http://mywiki.wooledge.org/BashFAQ/024

2 of 6
9

The following is actually buggy:

# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )

If you have bash 4.4 or newer, a best-of-all-worlds option is available:

# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
  set -o pipefail
  curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)

...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:

# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )

...or bash 3.x compatibility and failure detection, but without newline support:

# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
  set -o pipefail
  curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)

...or bash 3.x compatibility and newline support, but without failure detection:

# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
  arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')
🌐
GitHub
gist.github.com › olih › f7437fb6962fb3ee9fe95bda8d2c8fa4
jq Cheet Sheet · GitHub
Save olih/f7437fb6962fb3ee9fe95bda8d2c8fa4 to your computer and use it in GitHub Desktop. ... Not available as yum install on our current AMI. It should be on the latest AMI though: https://aws.amazon.com/amazon-linux-ami/2015.09-release-notes/ Installing from the source proved to be tricky. When running jq, the following arguments may become handy: