Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.
All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.
You can make a much simpler version of your jq program:
jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)'
which outputs:
URL='example.com'
AUTHOR='John Doe'
CREATED='10/22/2017'
There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.
All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.
This string is evalable if you completely trust the input data and has the effect you want:
eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)"
echo "$AUTHOR"
As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.
If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:
jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...'
You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.
It's also possible to build up an associative array instead where both keys and values are quoted:
eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))"
After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.
Your original version isn't going to be evalable because the author name has spaces in it - it would be interpreted as running a command Doe with the environment variable AUTHOR set to John. There's also virtually never a need to pipe jq to itself - the internal piping & dataflow can connect different filters together.
All of this is only sensible if you completely trust the input data (e.g. it's generated by a tool you control). There are several possible problems otherwise detailed below, but let's assume the data itself is certain to be in the format you expect for the moment.
You can make a much simpler version of your jq program:
jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)'
which outputs:
URL='example.com'
AUTHOR='John Doe'
CREATED='10/22/2017'
There's no need for a map: .[] deals with taking each object in the array through the rest of the pipeline as a separate item, so everything after the last | is applied to each one separately. At the end, we just assemble a valid shell assignment string with ordinary + concatenation, including appropriate quotes & escaping around the value with @sh.
All the pipes matter here - without them you get fairly unhelpful error messages, where parts of the program are evaluated in subtly different contexts.
This string is evalable if you completely trust the input data and has the effect you want:
eval "$(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + (.value | @sh)' < data.json)"
echo "$AUTHOR"
As ever when using eval, be careful that you trust the data you're getting, since if it's malicious or just in an unexpected format things could go very wrong. In particular, if the key contains shell metacharacters like $ or whitespace, this could create a running command. It could also overwrite, for example, the PATH environment variable unexpectedly.
If you don't trust the data, either don't do this at all or filter the object to contain just the keys you want first:
jq '.SITE_DATA | { AUTHOR, URL, CREATED } | ...'
You could also have a problem in the case that the value is an array, so .value | tostring | @sh will be better - but this list of caveats may be a good reason not to do any of this in the first place.
It's also possible to build up an associative array instead where both keys and values are quoted:
eval "declare -A data=($(jq -r '.SITE_DATA | to_entries | .[] | @sh "[\(.key)]=\(.value)"' < test.json))"
After this, ${data[CREATED]} contains the creation date, and so on, regardless of what the content of the keys or values are. This is the safest option, but doesn't result in top-level variables that could be exported. It may still produce a Bash syntax error when a value is an array, or a jq error if it is an object, but won't execute code or overwrite anything.
Building on @Michael Homer's answer, you can avoid a potentially-unsafe eval entirely by reading the data into an associative array.
For example, if your JSON data is in a file called file.json:
#!/bin/bash
typeset -A myarray
while IFS== read -r key value; do
myarray["
value"
done < <(jq -r '.SITE_DATA | to_entries | .[] | .key + "=" + .value ' file.json)
# show the array definition
typeset -p myarray
# make use of the array variables
echo "URL = '${myarray[URL]}'"
echo "CREATED = '${myarray[CREATED]}'"
echo "AUTHOR = '${myarray[URL]}'"
Output:
$ ./read-into-array.sh
declare -A myarray=([CREATED]="10/22/2017" [AUTHOR]="John Doe" [URL]="example.com" )
URL = 'example.com'
CREATED = '10/22/2017'
AUTHOR = 'example.com'
In jq 1.3 and up you can use the --arg VARIABLE VALUE command-line option:
jq -n --arg v "$VAR" '{"foo": $v}'
I.e., --arg sets a variable to the given value so you can then use $varname in your jq program, and now you don't have to use shell variable interpolation into your jq program.
EDIT: From jq 1.5 and up, you can use --argjson to pass in an array directly, e.g.
jq -n --argjson v '[1,2,3]' '{"foo": $v}'
Once you have your variable loaded, you should use the split filter to split that string into an array.
$ jq -n --arg inarr "${ARR}" '{ arr: $inarr | split("\n") }'
Writing bash code for array with multiple values
linux - save json jq output as seperate variables or as an array - Stack Overflow
Assigning an Array Parsed With jq to Bash Script Array - Stack Overflow
bash - jq: output array of json objects - Stack Overflow
I have a json file which i'm using bash to extract.
sample.json
{"extract": { "data": [ {"name": "John Smith", "id": 8752, "address": "1 Anywhere Street", "tel": 1234567890, "email": "john.smith@gmail.com" }, { "name": "Jane Smith", "id": 4568, "address": "719 Anywhere Street", "tel": 0987654321, "email": "janesmith@hotmail.com" } ] } }and store the value within an array
id=($(cat sample.json | jq -r '.extract.data[] .name'))
so in the case of ${id[0]} will output John Smith and ${id[1]} will output Jane Smith.
I am intending to store the values in a database (this will be my first attempt) which will be in a similar to that of the json, each object needs to be relative to how it is in the json so it might be better to go with:
data1=($(cat sample.json | jq -r '.extract.data[0] | .[]))
Lets say i have 1000 names to save to my database along with their id's. I'm some advice whether if there a more sensible (more effective) approach on how:
- Pull the data from Json? will I need to write this 1000 times?e.g
data1=($(cat sample.json | jq -r '.extract.data[0] | .[])) data2=($(cat sample.json | jq -r '.extract.data[1] | .[])) data3=($(cat sample.json | jq -r '.extract.data[2] | .[])) .. data1=($(cat sample.json | jq -r '.extract.data[1000] | .[]))
-Put the data into the DB from the first array? will the code need to reference the array as:
${data1[0]}
${data1[1]}
${data1[2]}Would be grateful for a steer in the right direction? - thanks.
Assuming the values in .name have no spaces or certain other characters, you would be able to get away with something along the lines of the following:
X=($(i3-msg -t get_workspaces | jq -r '.[] | select(.visible == true).name' ))
for var in "${X[@]}"
do
echo "$var"
done
Output:
9
1:Browsing
More robustly, if your shell has readarray:
readarray -t X < <(... | jq -r '.[] | select(.visible == true).name ')
Otherwise, you could use the idiom:
vars=(); while IFS= read -r line; do vars+=("$line"); done < .....
the ideal solution would be to have it stored in a array
Assuming you mean a bash array, you can use declare to declare it, and use jq's @sh to properly escape the contents.
For an indexed array, use declare -a:
$ declare -a cw="($(i3-msg … | jq -r '.[] | select(.visible).name | @sh'))"
$ declare -p cw # print for testing
declare -a cw=([0]="9" [1]="1:Browsing")
For an associative array, use declare -A, and output each keys before its value:
$ declare -A cw="($(i3-msg … | jq -r '.[] | select(.visible) | .output, .name | @sh'))"
$ declare -p cw # print for testing
declare -A cw=([DisplayPort-2]="9" [DisplayPort-1]="1:Browsing" )
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use @sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | @sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | @sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket(). It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | @sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that may have newlines (and any other arbitrary (non-NUL) characters) use jq's @sh filter to generate space-separated quoted strings, and Bash's declare -a to parse the quoted strings as array elements. (No pre-processing required)
foo.json:
{"data": ["$0", " \t\n", "*", "\"", ""]}
str=$(jq -r '.data | @sh' foo.json)
declare -a arr="($str)" # must be quoted like this
declare -p arr
# declare -a arr=([0]="\$0" [1]=$' \t\n' [2]="*" [3]="\"" [4]="")
Update: jq 1.7 (2023-09)
As of version 1.7, jq has a --raw-output0 option, enabling it to output null-terminated strings which can be read into an array as usual:
mapfile -d '' arr < <(jq --raw-output0 '.data[]' foo.json)
wait "$!" # use in bash-4.4+ to get exit status of the process substitution
Note on NUL characters in JSON strings
JSON strings may contain NUL characters while shell variables cannot. If your JSON input may contain NUL's, you may need to add some special handling.
When using the
@shfilter, NUL characters from JSON strings will be silently replaced with the sequence\0. Note that this makes the JSON strings"\\0"and"\u0000"indistinguishable.When using the
--raw-output0option, NUL characters will trigger an error andjqwill terminate with an exit status of 5.
Reading multiple/nested arrays
The @sh filter can be combined with --raw-output0 to reliably read multiple arrays at once (or a single nested array) as it will produce a NUL-separated list of space-separated quoted strings.
json='[[1,2],[3,4]]' i=0
while read -r -d ''; do
declare -a "arr$((i++))=($REPLY)"
done < <(jq --raw-output0 '.[]|@sh' <<<$json)
for ((n=0; n<i; n++)); { declare -p "arr$n"; }
# declare -a arr0=([0]="1" [1]="2")
# declare -a arr1=([0]="3" [1]="4")
You can use separate variables with read :
read var1 var2 var3 < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : $var1"
echo "name : $var2"
echo "full_name : $var3"
Using array :
read -a arr < <(echo $(curl -s 'https://api.github.com/repos/torvalds/linux' |
jq -r '.id, .name, .full_name'))
echo "id : ${arr[0]}"
echo "name : ${arr[1]}"
echo "full_name : ${arr[2]}"
Also you can split jq output with some character :
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')
Or use an array like :
set -f; IFS='|' data=($(curl '......' | jq -r '.data |
map([.absoluteNumber, .airedEpisodeNumber, .episodeName, .overview] |
join("|")) | join("\n")')); set +f
absoluteNumber, airedEpisodeNumber, episodeName & overview are respectively ${data[0]}, ${data[1]}, ${data[2]}, ${data[3]}. set -f and set +f are used to respectively disable & enable globbing.
For the jq part, all your required fields are mapped and delimited with a '|' character with join("|")
If your are using jq < 1.5, you'll have to convert Number to String with tostring for each Number fields eg:
IFS='|' read var1 var2 var3 var4 < <(curl '......' | jq -r '.data |
map([.absoluteNumber|tostring, .airedEpisodeNumber|tostring, .episodeName, .overview] |
join("|")) | join("\n")')
jq always produces a stream of zero or more values. For example, to produce the two values corresponding to "episodeName" and "id"' you could write:
.data[] | ( .episodeName, .id )
For your purposes, it might be helpful to use the -c command-line option, to ensure each JSON output value is presented on a single line. You might also want to use the -r command-line option, which removes the outermost quotation marks from each output value that is a JSON string.
For further variations, please see the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, e.g. the question:
Q: How can a stream of JSON texts produced by jq be converted into a bash array of corresponding values?
Using jq :
readarray arr < <(jq '.[].item2' json)
printf '%s\n' "${arr[@]}"
If you need a more hardened way:
readarray -td '' arr
for inputs with newlines or other special characters, avoiding word splitting.
Output:
value2
value2_2
Check:
Process Substitution >(command ...) or <(...) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection: cmd1 2> >(cmd2).
See http://mywiki.wooledge.org/ProcessSubstitution http://mywiki.wooledge.org/BashFAQ/024
The following is actually buggy:
# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )
If you have bash 4.4 or newer, a best-of-all-worlds option is available:
# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
set -o pipefail
curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)
...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:
# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )
...or bash 3.x compatibility and failure detection, but without newline support:
# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
set -o pipefail
curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)
...or bash 3.x compatibility and newline support, but without failure detection:
# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')