The problem is that jq is still just outputting lines of text; you can't necessarily preserve each array element as a single unit. That said, as long as a newline is not a valid character in any object, you can still output each object on a separate line.
get_json_array | jq -c '.[]' | while read object; do
api_call "$object"
done
Under that assumption, you could use the readarray command in bash 4 to build an array:
readarray -t conversations < <(get_json_array | jq -c '.[]')
for conversation in "${conversations[@]}"; do
api_call "$conversation"
done
Answer from chepner on Stack OverflowThe problem is that jq is still just outputting lines of text; you can't necessarily preserve each array element as a single unit. That said, as long as a newline is not a valid character in any object, you can still output each object on a separate line.
get_json_array | jq -c '.[]' | while read object; do
api_call "$object"
done
Under that assumption, you could use the readarray command in bash 4 to build an array:
readarray -t conversations < <(get_json_array | jq -c '.[]')
for conversation in "${conversations[@]}"; do
api_call "$conversation"
done
Here is a solution without loops:
json=$(jq -c ".my_key[]" ./my-file.json)
json_without_quotes=$(echo ${json//\"/""})
declare -a all_apps_array=($(echo $json_without_quotes | tr "\n" " "))
More fun with jq, getting results into a usable array
bash - jq: output array of json objects - Stack Overflow
Assigning an Array Parsed With jq to Bash Script Array - Stack Overflow
ubuntu - parse one field from an JSON array into bash array - Unix & Linux Stack Exchange
What does the jq command do in converting JSON arrays to Bash arrays?
How to convert a JSON array into a Bash array?
What is the jq command in Linux?
I'm using this in a cURL to get the data from result[]:
foo=$(curl --request GET \ --silent \ --url https://example.com \ --header 'Content-Type: application/json' | jq -r '.result[]')
When I print $foo, this is what I have:
[key]
default
firewall_custom
zone
34
[
{
"id": "example",
"version": "6",
"action": "block",
"expression": "lorem",
"description": "ipsum",
"last_updated": "2024-08-15T19:10:24.913784Z",
"ref": "example",
"enabled": true
},
{
"id": "example2",
"version": "7",
"action": "block",
"expression": "this",
"description": "that",
"last_updated": "2024-08-15T19:10:24.913784Z",
"ref": "example2",
"enabled": true
}
]What I need from this is to create a loop where, in a series of addtional cURLs, I can insert action, expression, and description.
I'm imagining that I would push these to 3 separate arrays (action, expression, and description), so that ${action[0]} would coincide with ${expression[0]} and ${description[0]}, and so on.
Something along the lines of:
# assuming that I have somehow created the following arrays:
# action=("block" "block")
# expression=("lorem" "this")
# description=("ipsum" "that")
for x in ${action[@]}; do
bar=$(curl --request GET \
--silent \
--url https://example.com \
--data '{
"action": ${action[$x]},
"expression": ${expression[$x]},
"description": ${description[$x]}
}' | jq '.success')
if [[ $bar == true ]]
then
printf "$x succeeded\n"
else
printf "$x failed\n"
fi
# reset bar
bar=''
doneThe question is, how to create action, expression, and description arrays from the results of $foo (that original cURL)?
We can solve this problem by two ways. They are:
Input string:
// test.json
{
"keys": ["key1","key2","key3"]
}
Approach 1:
1) Use jq -r (output raw strings, not JSON texts) .
KEYS=$(jq -r '.keys' test.json)
echo $KEYS
# Output: [ "key1", "key2", "key3" ]
2) Use @sh (Converts input string to a series of space-separated strings). It removes square brackets[], comma(,) from the string.
KEYS=$(<test.json jq -r '.keys | @sh')
echo $KEYS
# Output: 'key1' 'key2' 'key3'
3) Using tr to remove single quotes from the string output. To delete specific characters use the -d option in tr.
KEYS=$((<test.json jq -r '.keys | @sh')| tr -d \')
echo $KEYS
# Output: key1 key2 key3
4) We can convert the comma-separated string to the array by placing our string output in a round bracket(). It also called compound Assignment, where we declare the array with a bunch of values.
ARRAYNAME=(value1 value2 .... valueN)
#!/bin/bash
KEYS=($((<test.json jq -r '.keys | @sh') | tr -d \'\"))
echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
Approach 2:
1) Use jq -r to get the string output, then use tr to delete characters like square brackets, double quotes and comma.
#!/bin/bash
KEYS=$(jq -r '.keys' test.json | tr -d '[],"')
echo $KEYS
# Output: key1 key2 key3
2) Then we can convert the comma-separated string to the array by placing our string output in a round bracket().
#!/bin/bash
KEYS=($(jq -r '.keys' test.json | tr -d '[]," '))
echo "Array size: " ${#KEYS[@]}
echo "Array elements: "${KEYS[@]}
# Output:
# Array size: 3
# Array elements: key1 key2 key3
To correctly parse values that may have newlines (and any other arbitrary (non-NUL) characters) use jq's @sh filter to generate space-separated quoted strings, and Bash's declare -a to parse the quoted strings as array elements. (No pre-processing required)
foo.json:
{"data": ["$0", " \t\n", "*", "\"", ""]}
str=$(jq -r '.data | @sh' foo.json)
declare -a arr="($str)" # must be quoted like this
declare -p arr
# declare -a arr=([0]="\$0" [1]=$' \t\n' [2]="*" [3]="\"" [4]="")
Update: jq 1.7 (2023-09)
As of version 1.7, jq has a --raw-output0 option, enabling it to output null-terminated strings which can be read into an array as usual:
mapfile -d '' arr < <(jq --raw-output0 '.data[]' foo.json)
wait "$!" # use in bash-4.4+ to get exit status of the process substitution
Note on NUL characters in JSON strings
JSON strings may contain NUL characters while shell variables cannot. If your JSON input may contain NUL's, you may need to add some special handling.
When using the
@shfilter, NUL characters from JSON strings will be silently replaced with the sequence\0. Note that this makes the JSON strings"\\0"and"\u0000"indistinguishable.When using the
--raw-output0option, NUL characters will trigger an error andjqwill terminate with an exit status of 5.
Reading multiple/nested arrays
The @sh filter can be combined with --raw-output0 to reliably read multiple arrays at once (or a single nested array) as it will produce a NUL-separated list of space-separated quoted strings.
json='[[1,2],[3,4]]' i=0
while read -r -d ''; do
declare -a "arr
REPLY)"
done < <(jq --raw-output0 '.[]|@sh' <<<$json)
for ((n=0; n<i; n++)); { declare -p "arr$n"; }
# declare -a arr0=([0]="1" [1]="2")
# declare -a arr1=([0]="3" [1]="4")
Using jq :
readarray arr < <(jq '.[].item2' json)
printf '%s\n' "${arr[@]}"
If you need a more hardened way:
readarray -td '' arr
for inputs with newlines or other special characters, avoiding word splitting.
Output:
value2
value2_2
Check:
Process Substitution >(command ...) or <(...) is replaced by a temporary filename. Writing or reading that file causes bytes to get piped to the command inside. Often used in combination with file redirection: cmd1 2> >(cmd2).
See http://mywiki.wooledge.org/ProcessSubstitution http://mywiki.wooledge.org/BashFAQ/024
The following is actually buggy:
# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )
If you have bash 4.4 or newer, a best-of-all-worlds option is available:
# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
set -o pipefail
curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)
...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:
# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )
...or bash 3.x compatibility and failure detection, but without newline support:
# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
set -o pipefail
curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)
...or bash 3.x compatibility and newline support, but without failure detection:
# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')
To convert your JSON to a bash array, with help of jq:
$ readarray -t arr < <(jq '.value' file)
$ printf '%s\n' "${arr[@]}"
"1"
"3"
"4"
To fix your expanded example (the exact command), just don't use object construction {value: .Value}, but instead only .Value:
$ readarray -t arr < <(aws ec2 describe-instances --region=us-east-1 --filters --filters "Name=tag:NodeType,Values=worker" --query "Reservations[].Instances[].Tags[]" | jq -r '.[] | select(.Key == "NodeNumber") | .Value')
$ printf '%s\n' "${arr[@]}"
1
3
4
Notice the lack of double quotes, since the -r option now prints only raw string values, not raw JSON Objects.
After you get arr populated with values like this, you can easily iterate over it and perform tests, just as you described in your question.
First, Store the Data
Given your raw data stored as a string in a json variable, perhaps with a here-document:
json=$(
cat <<- EOF
{
"value": "1"
}
{
"value": "3"
}
{
"value": "4"
}
EOF
)
Bash itself will do a reasonable job of prettifying it:
$ echo $json
{ "value": "1" } { "value": "3" } { "value": "4" }
Parsing the Data Into a Bash Array
There's more than one way to do this. Two of the more obvious ways are to use use jq or grep to extract the values into a Bash array with the shell's simple array notation:
values=( `echo $json | jq .value` )
echo "${values[@]}"
"1" "3" "4"
unset values
values=$(egrep -o '[[:digit:]]+' <<< "$json")
echo "${values[@]}"
1
3
4
There are certainly other ways to accomplish this task, but this seems like the low-hanging fruit. Your mileage may vary.
Caveat
The main thing to remember about Bash arrays is that they need to use expansions such as "${foo[@]}" when quoted, or ${bar[*]} when unquoted, if you want to use them for loops rather than indexing into them directly. Once they're in an array, though, you can access the entire array easily as long as you understand the different expansions and quoting rules.
Hi guys,
I am a linux noob and am trying to write a script to extract info from a mkv file using mkvmerge but am not able to convert the target json script to a bash array. I have tried a number of solutions from stack overflow but with no success.
here are some of my attempts
dir="/mnt/Anime/Series/KonoSuba/Season 2/[Nep_Blanc] KonoSuba II 10 .mkv"
*********************************************************************************
ARRAY_SIZE=$(mkvmerge -J "$dir" | jq '.tracks | length')
count=0
arr=()
while [ $count -lt $ARRAY_SIZE ];
do
arr+=($(mkvmerge -J "$dir" | jq '.tracks'[$count]))
((count++))
done
*********************************************************************************
readarray -t test_array < <(mkvmerge -J "$dir" | jq '.tracks')
for element in "${test_array[@]}";
do
echo "$element"
done
*********************************************************************************
array=($(mkvmerge -J "$dir" | jq '.tracks' | sed -e 's/^\[/(/' -e 's/\]$/)/'))but the echo prints out lines instead of the specific objects.
Though now it is helpling me with my python, originally the project was to help me learn bash scripting. I would really like to have a bash implementation so any help overcoming this roadblock would be appreciated.
This trick with the jq 1.5 inputs streaming filter seems to do it
... | jq -n '.items |= [inputs]'
Ex.
$ find ~/ -maxdepth 1 -name "D*" |
while read line; do
jq -n --arg name "$(basename "$line")" \
--arg path "$line" \
'{name: $name, path: $path}'
done | jq -n '.items |= [inputs]'
{
"items": [
{
"name": "Downloads",
"path": "/home/steeldriver/Downloads"
},
{
"name": "Desktop",
"path": "/home/steeldriver/Desktop"
},
{
"name": "Documents",
"path": "/home/steeldriver/Documents"
}
]
}
Calling jq directly from find, and then collecting the resulting data with jq to construct the final output, without any shell loops:
find ~ -maxdepth 1 -name '[[:upper:]]*' \
-exec jq -n --arg path {} '{ name: ($path|sub(".*/"; "")), path: $path }' \; |
jq -n -s '{ items: inputs }'
The jq that is being executed via -exec creates a JSON object per found pathname. It strips off everything in the pathname up to the last slash for the name value, and uses the pathname as is for the path value.
The final jq reads the data from find into an array with -s, and simply inserts it as the items array in a new JSON object. The final jq invocation could also be written jq -n '{ items: [inputs] }.
Example result (note that I was using [[:upper:]* in place of D* for the -name pattern with find):
{
"items": [
{
"name": "Documents",
"path": "/home/myself/Documents"
},
{
"name": "Mail",
"path": "/home/myself/Mail"
},
{
"name": "Work",
"path": "/home/myself/Work"
}
]
}
I have a json file which i'm using bash to extract.
sample.json
{"extract": { "data": [ {"name": "John Smith", "id": 8752, "address": "1 Anywhere Street", "tel": 1234567890, "email": "john.smith@gmail.com" }, { "name": "Jane Smith", "id": 4568, "address": "719 Anywhere Street", "tel": 0987654321, "email": "janesmith@hotmail.com" } ] } }and store the value within an array
id=($(cat sample.json | jq -r '.extract.data[] .name'))
so in the case of ${id[0]} will output John Smith and ${id[1]} will output Jane Smith.
I am intending to store the values in a database (this will be my first attempt) which will be in a similar to that of the json, each object needs to be relative to how it is in the json so it might be better to go with:
data1=($(cat sample.json | jq -r '.extract.data[0] | .[]))
Lets say i have 1000 names to save to my database along with their id's. I'm some advice whether if there a more sensible (more effective) approach on how:
- Pull the data from Json? will I need to write this 1000 times?e.g
data1=($(cat sample.json | jq -r '.extract.data[0] | .[])) data2=($(cat sample.json | jq -r '.extract.data[1] | .[])) data3=($(cat sample.json | jq -r '.extract.data[2] | .[])) .. data1=($(cat sample.json | jq -r '.extract.data[1000] | .[]))
-Put the data into the DB from the first array? will the code need to reference the array as:
${data1[0]}
${data1[1]}
${data1[2]}Would be grateful for a steer in the right direction? - thanks.