First, obtain an array containing all the different object property names in your object array input. Those will be the columns of your CSV:
(map(keys) | add | unique) as $cols
Then, for each object in the object array input, map the column names you obtained to the corresponding properties in the object. Those will be the rows of your CSV.
map(. as
cols | map(
rows
Finally, put the column names before the rows, as a header for the CSV, and pass the resulting row stream to the @csv filter.
$cols, $rows[] | @csv
All together now. Remember to use the -r flag to get the result as a raw string:
jq -r '(map(keys) | add | unique) as $cols | map(. as
cols | map(
rows | $cols, $rows[] | @csv'
Answer from user3899165 on Stack OverflowFirst, obtain an array containing all the different object property names in your object array input. Those will be the columns of your CSV:
(map(keys) | add | unique) as $cols
Then, for each object in the object array input, map the column names you obtained to the corresponding properties in the object. Those will be the rows of your CSV.
map(. as
cols | map(
rows
Finally, put the column names before the rows, as a header for the CSV, and pass the resulting row stream to the @csv filter.
$cols, $rows[] | @csv
All together now. Remember to use the -r flag to get the result as a raw string:
jq -r '(map(keys) | add | unique) as $cols | map(. as
cols | map(
rows | $cols, $rows[] | @csv'
The Skinny
jq -r '(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[] | @csv'
or:
jq -r '(.[0] | keys_unsorted) as $keys | ([$keys] + map([.[ $keys[] ]])) [] | @csv'
The Details
Aside
Describing the details is tricky because jq is stream-oriented, meaning it operates on a sequence of JSON data, rather than a single value. The input JSON stream gets converted to some internal type which is passed through the filters, then encoded in an output stream at program's end. The internal type isn't modeled by JSON, and doesn't exist as a named type. It's most easily demonstrated by examining the output of a bare index (.[]) or the comma operator (examining it directly could be done with a debugger, but that would be in terms of jq's internal data types, rather than the conceptual data types behind JSON).
jq -cn '"a", "b"' "a" "b"
Note that the output isn't an array (which would be ["a", "b"]). Compact output (the -c option) shows that each array element (or argument to the , filter) becomes a separate object in the output (each is on a separate line).
A stream is like a JSON-seq, but uses newlines rather than RS as an output separator when encoded. Consequently, this internal type is referred to by the generic term "sequence" in this answer, with "stream" being reserved for the encoded input and output.
Constructing the Filter
The first object's keys can be extracted with:
.[0] | keys_unsorted
Keys will generally be kept in their original order, but preserving the exact order isn't guaranteed. Consequently, they will need to be used to index the objects to get the values in the same order. This will also prevent values being in the wrong columns if some objects have a different key order.
To both output the keys as the first row and make them available for indexing, they're stored in a variable. The next stage of the pipeline then references this variable and uses the comma operator to prepend the header to the output stream.
(.[0] | keys_unsorted) as $keys | $keys, ...
The expression after the comma is a little involved. The index operator on an object can take a sequence of strings (e.g. "name", "value"), returning a sequence of property values for those strings. $keys is an array, not a sequence, so [] is applied to convert it to a sequence,
$keys[]
which can then be passed to .[]
.[ $keys[] ]
This, too, produces a sequence, so the array constructor is used to convert it to an array.
[.[ $keys[] ]]
This expression is to be applied to a single object. map() is used to apply it to all objects in the outer array:
map([.[ $keys[] ]])
Lastly for this stage, this is converted to a sequence so each item becomes a separate row in the output.
map([.[ $keys[] ]])[]
Why bundle the sequence into an array within the map only to unbundle it outside? map produces an array; .[ $keys[] ] produces a sequence. Applying map to the sequence from .[ $keys[] ] would produce an array of sequences of values, but since sequences aren't a JSON type, so you instead get a flattened array containing all the values.
["NSW","AU","state","New South Wales","AB","CA","province","Alberta","ABD","GB","council area","Aberdeenshire","AK","US","state","Alaska"]
The values from each object need to be kept separate, so that they become separate rows in the final output.
Finally, the sequence is passed through @csv formatter.
Alternate
The items can be separated late, rather than early. Instead of using the comma operator to get a sequence (passing a sequence as the right operand), the header sequence ($keys) can be wrapped in an array, and + used to append the array of values. This still needs to be converted to a sequence before being passed to @csv.
Using jq to extract values from column-oriented JSON and format in CSV - Unix & Linux Stack Exchange
text processing - Convert JSON of arrays to CSV with headers using JQ - Unix & Linux Stack Exchange
Convert json to csv with jq tool
Need help with using Array value as input for `jq`
Videos
You may pick the headers as an array of strings from the keys of the first element in the list, then extract all elements' values as separate arrays. Applying the @csv output operator to each element of the resulting list will CSV-quote the data (jq quotes all strings, but not booleans or numbers):
$ jq -r '[first|keys_unsorted] + map([.[]]) | .[] | @csv' file
"bytes","checked"
276697,false
276697,false
Or,
$ jq -r '(first|keys_unsorted), (.[]|[.[]]) | @csv' file
"bytes","checked"
276697,false
276697,false
Or,
$ jq -r '(first|keys_unsorted), map(map(.))[] | @csv' file
"bytes","checked"
276697,false
276697,false
Or any other way to extract the values into separate arrays.
Note that this relies on the keys to occur in the same order all throughout the input data.
However, it's even easier with Miller (mlr):
$ mlr --j2c cat file
bytes,checked
276697,false
276697,false
This simply passes the data through Miller's cat command (which, when used like this, does not modify the data), while converting from JSON to CSV using the --j2c option (short for --ijson --ocsv). Note that since Miller is properly CSV-aware, it only quotes the fields that actually need quoting.
You may also get a nicely formatted table by choosing the pretty-printed output format together with --barred:
$ mlr --j2p --barred cat file
+--------+---------+
| bytes | checked |
+--------+---------+
| 276697 | false |
| 276697 | false |
+--------+---------+
(--j2p is short for --ijson --opprint.)
Or without --barred:
$ mlr --j2p cat file
bytes checked
276697 false
276697 false
I got it:
cat file.json | jq '(.[0] | keys), .[] | join(",")'
looks like you can surround any part in parentheses to prevent it from "consuming" the stream (I don't know if that's what it's called or even if I got it right here, because I couldn't find anything on jq's documentation and had to piece everything together with bits and bobs scattered throughout various blogs and stackoverflows, so if there's a "proper" way to do it please let me know).
By the way if you have a shell for loop like I did use -s option to combine separate json objects:
$ for i in {3,4,5,8}
do rclone rc core/stats --rc-user USER --rc-pass PASS --rc-addr :557$i
done | jq -rs '(.[0] | keys), .[] | join(",")' | column -ts,
bytes checks deletedDirs deletes elapsedTime errors eta fatalError renames retryError speed totalBytes totalChecks totalTransfers transferTime transfers
1660182 0 0 0 258038.009782457 0 0 false 0 false 1664.9322505627426 1660182 0 6 0 6
407752609 0 0 0 258038.054874325 0 0 false 0 false 10615.04533495996 407752609 0 86 0 86
7403585 0 0 0 258038.103563555 0 0 false 0 false 20892.381593377457 7403585 0 2 0 2
0 0 0 0 258038.156466825 0 false 0 false 0 0 0 0 0 0
jq has a filter, @csv, for converting an array to a CSV string. This filter takes into account most of the complexities associated with the CSV format, beginning with commas embedded in fields. (jq 1.5 has a similar filter, @tsv, for generating tab-separated-value files.)
Of course, if the headers and values are all guaranteed to be free of commas and double quotation marks, then there may be no need to use the @csv filter. Otherwise, it would probably be better to use it.
For example, if the 'Company Name' were 'Smith, Smith and Smith', and if the other values were as shown below, invoking jq with the "-r" option would produce valid CSV:
$ jq -r '.data | map(.displayName), map(.value) | @csv' so.json2csv.json
"First Name","Last Name","Position","Company Name","Country"
"John (""Johnnie"")","Doe","Director, Planning and Posterity","Smith, Smith and Smith","Transylvania"
I prefer to make each record a row in my CSV.
jq -r '.data | map([.displayName, .rank, .value] | join(", ")) | join("\n")'
Given the data in the question, this outputs
First Name, 1, VALUE
Last Name, 2, VALUE
Position, 3, VALUE
Company Name, 4, VALUE
Country, 5, VALUE
Answering my own question after playing with given answers, generic solution for json arrays
jq --raw-output 'to_entries|map(.key),(map(.value)|transpose[])|@csv'
A solution using the transpose function:
<file jq -r '(to_entries|map(.key)),([.ObjectID,.Name,.Color,.Acidity]|transpose|.[])|@csv'
The object name is extracted using to_entries|map(.key).
The object content is put inside an array and transposed in order to get arrays with element of each object.
The result is then going through the @csv operator that converts the array into a comma separated string.