With jo, which makes it easy to generate JSON on the command line:

$ jo -p key1="$value1" key2="$value2"
{
   "key1": "foo",
   "key2": "bar"
}

or, depending on what you want the end result to be,

$ jo -a -p "$(jo key1="$value1")" "$(jo key2="$value2")"
[
   {
      "key1": "foo"
   },
   {
      "key2": "bar"
   }
]

Note that jo will also properly encode the values in the strings $value1 and $value2.

You can do this:

X=("hello world" "goodnight moon")
printf '%s\n' "${X[@]}" | jq -R . | jq -s .

output

[
  "hello world",
  "goodnight moon"
]

There are two main ways to create a valid JSON array from arbitrary lists of strings. One is using jq with its --args option:

jq -c -n '$ARGS.positional' --args "${mylist[@]}"

or, if the elements are not to be encoded as strings (they may be plain numbers, for example, or previously prepared JSON strings or whole JSON objects),

jq -c -n '$ARGS.positional' --jsonargs "${mylist[@]}"

Note that --args and --jsonargs and its arguments need to be the last thing on the command line of jq.

I'm using -n to disable reading any external input and -c to create JSON in the "compact" output format, as shown in the question.

The other way is using jo, a tool specifically made to make it easy to generate JSON on the command line:

jo -a "${mylist[@]}"

The -a option to jo generates an array from the given arguments.

The jo tool infers the type of the object itself, so a list like

mylist=(1 2 3 4 5 b)

would be turned into the JSON array

[1,2,3,4,5,"b"]

One suggestion is to use --args with jq to create the two arrays and then collect these in the correct location in the main document. Note that --args is required to be the last option on the command line and that all the remaining command line arguments will become elements of the $ARGS.positional array.

{
    jq -n --arg key APP-Service1-Admin '{(ARGS.positional}' --args a b
    jq -n --arg key APP-Service1-View  '{(ARGS.positional}' --args c d
} |
jq -s --arg key 'AD Accounts' '{($key): add}' |
jq --arg Service service1-name --arg 'AWS account' service1-dev '$ARGS.named + .'

The first two jq invocations create a set of two JSON objects:

{
  "APP-Service1-Admin": [
    "a",
    "b"
  ]
}
{
  "APP-Service1-View": [
    "c",
    "d"
  ]
}

The third jq invocation uses -s to read that set into an array, which becomes a merged object when passed through add. The merged object is assigned to our top-level key:

{
  "AD Accounts": {
    "APP-Service1-Admin": [
      "a",
      "b"
    ],
    "APP-Service1-View": [
      "c",
      "d"
    ]
  }
}

The last jq adds the remaining top-level keys and their values to the object:

{
  "Service": "service1-name",
  "AWS account": "service1-dev",
  "AD Accounts": {
    "APP-Service1-Admin": [
      "a",
      "b"
    ],
    "APP-Service1-View": [
      "c",
      "d"
    ]
  }
}

With jo:

jo -d . \
    Service=service1-name \
    'AWS account'=service1-dev  \
    'AD Accounts.APP-Service1-Admin'="$(jo -a a b)" \
    'AD Accounts.APP-Service1-View'="$(jo -a c d)"

The "internal" object is created using .-notation (enabled with -d .), and a couple of command substitutions for creating the arrays.

Or you can drop the -d . and use a form of array notation:

jo  Service=service1-name \
    'AWS account'=service1-dev \
    'AD Account[APP-Service1-Admin]'="$(jo -a a b)" \
    'AD Account[APP-Service1-View]'="$(jo -a c d)"

One way to do this is to provide a jq function that generates your repeated structure, given the specific inputs you want to modify. Consider the following:

# generate this however you want to -- hardcoded, built by a loop, whatever.
source_dest_pairs=(
  sourcebucket1:destinationbucket1
  sourcebucket2:destinationbucket2
  sourcebucket3:destinationbucket3
)

# -R accepts plain text, not JSON, as input; -n doesn't read any input automatically
# ...but instead lets "inputs" or "input" be used later in your jq code.
jq -Rn '
  def instructionsForPair($source; $dest): {
    "Name":"S3DistCp step",
    "HadoopJarStep": {
      "Args":[
        "s3-dist-cp",
        "--s3Endpoint=s3.amazonaws.com",
        "--src=\($source)",
        "--dest=\($dest)"
      ],
      "Jar":"command-runner.jar"
    }
  };

  [ inputs 
  | capture("^(?<source>[^:]+):(?<dest>.*)$"; "")
  | select(.)
  | instructionsForPair(.source; .dest) ]
' < <(printf '%s\n' "${source_dest_pairs[@]}")

...correctly emits as output:

[
  {
    "Name": "S3DistCp step",
    "HadoopJarStep": {
      "Args": [
        "s3-dist-cp",
        "--s3Endpoint=s3.amazonaws.com",
        "--src=sourcebucket1",
        "--dest=destinationbucket1"
      ],
      "Jar": "command-runner.jar"
    }
  },
  {
    "Name": "S3DistCp step",
    "HadoopJarStep": {
      "Args": [
        "s3-dist-cp",
        "--s3Endpoint=s3.amazonaws.com",
        "--src=sourcebucket2",
        "--dest=destinationbucket2"
      ],
      "Jar": "command-runner.jar"
    }
  },
  {
    "Name": "S3DistCp step",
    "HadoopJarStep": {
      "Args": [
        "s3-dist-cp",
        "--s3Endpoint=s3.amazonaws.com",
        "--src=sourcebucket3",
        "--dest=destinationbucket3"
      ],
      "Jar": "command-runner.jar"
    }
  }
]

What about running though the generated string again?

arr=($(echo "['1', '2', '3', '4']" | grep -oP "'[^']*'" | awk '{print substr($0, 2, length($0) - 2)}'))

This extracts everything between single quotes, then strips the quotes on the sides, and gives what you want.

ibug@linux:~ $ echo "${arr[@]}"
1 2 3 4
ibug@linux:~ $ for i in "${arr[@]}"; do echo "$i"; done
1
2
3
4
ibug@linux:~ $

Simply use printf to format the output into JSON

First, you have a very blatant typo in this part of your code right here:

echo "${array[3]:$var-3:4}

Note there is no closing straight quote: ". Fixed it in the rewrite I did below:

But more to the point, doing something like this (using printf) as suggested in this StackOverflow answer. Tested and works in CentOS 7.

#!/bin/bash

readarray -t array <<< "$(df -h)";
var=$(echo "${array[1]}"| grep -aob '%' | grep -oE '[0-9]+');
df_output="${array[3]:$var-3:4}";

manufacturer=$(cat /sys/class/dmi/id/chassis_vendor);
product_name=$(cat /sys/class/dmi/id/product_name);
version=$(cat /sys/class/dmi/id/bios_version);
serial_number=$(cat /sys/class/dmi/id/product_serial);
hostname=$(hostname);
operating_system=$(hostnamectl | grep "Operating System" | cut -d ' ' -f5-);
architecture=$(arch);
processor_name=$(awk -F':' '/^model name/ {print $2}' /proc/cpuinfo | uniq | sed -e 's/^[ \t]*//');
memory$(dmidecode -t 17 | grep "Size.*MB" | awk '{s+=$2} END {print s / 1024"GB"}');
hdd_model=$(cat /sys/block/sda/device/model);
system_main_ip=$(hostname -I);

printf '{"manufacturer":"%s","product_name":"%s","version":"%s","serial_number":"%s","hostname":"%s","operating_system":"%s","architecture":"%s","processor_name":"%s","memory":"%s","hdd_model":"%s","system_main_ip":"%s"}' "$manufacturer" "$product_name" "$version" "$serial_number" "$hostname" "$operating_system" "$architecture" "$processor_name" "$memory" "$hdd_model" "$system_main_ip"

The output I get is this:

{"manufacturer":"Oracle Corporation","product_name":"VirtualBox","version":"VirtualBox","serial_number":"","hostname":"sandbox-centos-7","operating_system":"CentOS Linux 7 (Core)","architecture":"x86_64","processor_name":"Intel(R) Core(TM) i5-1030NG7 CPU @ 1.10GHz","memory":"","hdd_model":"VBOX HARDDISK   ","system_main_ip":"10.0.2.15 192.168.56.20 "}

And if you have jq installed, you can pipe the output of the shell script to jq to “pretty print” the output into some human readable format. Like let’s say your script is named my_script.sh, just pipe it to jq like this:

./my_script.sh | jq

And the output would look like this:

{
  "manufacturer": "Oracle Corporation",
  "product_name": "VirtualBox",
  "version": "VirtualBox",
  "serial_number": "",
  "hostname": "sandbox-centos-7",
  "operating_system": "CentOS Linux 7 (Core)",
  "architecture": "x86_64",
  "processor_name": "Intel(R) Core(TM) i5-1030NG7 CPU @ 1.10GHz",
  "memory": "",
  "hdd_model": "VBOX HARDDISK   ",
  "system_main_ip": "10.0.2.15 192.168.56.20 "
}

One possible solution to this:

declare -A aliases
aliases[Index]=components/Index/Exports
aliases[Shared]=components/Shared/Exports
aliases[Icons]=components/Icons/Exports

jq -n --argjson n "${#aliases[@]}" '
        { compileroption: {
                baseurl: ".",
                paths:
                (
                        reduce range($n) as $i ({};
                                .[$ARGS.positional[$i]] = [$ARGS.positional[$i+$n]]
                        )
                )
        } }' --args "${!aliases[@]}" "${aliases[@]}"

Does not use jo and instead pass the keys and values of the associative array aliases into jq as positional parameters with --args at the end of the command (--args must always be the last option, if it's used at all). The jq utility receives the keys and values as a single array, $ARGS.positional. This means the first half of the array contains the keys, and the second half of the array contains the corresponding values.

The body of the jq expression creates the output object and uses a reduce operation over a range of $n integers from zero up, where $n is the number of elements in the aliases array. The reduce operation builds the paths object by adding the positional arguments, one by one, using $i:th argument as the key and the $i+$n:th argument as the element in the corresponding array value.

A slightly different approach using jo to create leaf objects of each key-value pair of the associative array:

declare -A aliases
aliases[Index]=components/Index/Exports
aliases[Shared]=components/Shared/Exports
aliases[Icons]=components/Icons/Exports

for key in "${!aliases[@]}"; do
        jo "$key[]=${aliases[$key]}"
done

This would output the three objects

{"Icons":["components/Icons/Exports"]}
{"Index":["components/Index/Exports"]}
{"Shared":["components/Shared/Exports"]}

Since we're using jo like this, we impose some obvious restrictions on the keys of the array (may not contain =, [] etc.)

We could use jq in place of jo like so:

for key in "${!aliases[@]}"; do
        jq -n --arg key "$key" --arg value "${aliases[$key]}" '.[$key] = [$value]'
done

We may then read these and add them in the correct place in the object we're creating in jq:

declare -A aliases
aliases[Index]=components/Index/Exports
aliases[Shared]=components/Shared/Exports
aliases[Icons]=components/Icons/Exports

for key in "${!aliases[@]}"; do
        jo "$key[]=${aliases[$key]}"
done |
jq -n '{ compileroptions: {
        baseURL: ".",
        paths: (reduce inputs as $item ({}; . += $item)) } }'

The main difference here is that we don't pass stuff into jq as command line options, but rather as a stream of JSON objects.

There are two main issues in your data and code:

1. You have an input file in DOS or Windows text file format.
2. Your code creates multiple single-element arrays rather than a single array with multiple elements.

Your input file, lol, appears to be a text file in DOS/Windows format. This means that when a utility that expects a Unix text file as input reads the file, each line will have an additional carriage-return character (\r) at the end.

You should convert the file to Unix text file format. This can be done with e.g. dos2unix.

As for your code, you can avoid the shell loop and let jq read the whole file in one go. This allows you to create a single result array rather than a set of arrays, each with a single object, which your code does.

The following assumes that the only thing that varies between the elements of the top-level array in the result is the source value (there is nothing in the question that explains how the max and min values of the source and destination ports should be picked):

jq -n -R '
        [inputs] |
        map( {
                source: .,
                protocol: "17",
                isStateless: true,
                udpOptions: {
                        sourcePortRange: { min: 521, max: 65535 },
                        destinationPortRange: { min: 1, max: 65535 }
                }
        } )' cidr.txt

or in the same compact one-line form as in your question:

jq -n -R '[inputs]|map({source:.,protocol:"17",isStateless:true,udpOptions:{sourcePortRange:{min:521,max:65535},destinationPortRange:{min:1,max:65535}}})' cidr.txt

Using inputs, jq reads the remaining inputs. Together with -R, it will read each line of cidr.txt as a single string. Putting this in an array with [inputs] we create an array of strings. The map() call takes each string from this array and transforms it into the source value for a larger, otherwise static object.

Add -c to the invocation of jq to get "compact" output.

If you don't want to, or are unable to, convert the input data from DOS to Unix text form, you can remove the carriage-return characters from within the jq expression instead.

To do this, replace the . after source: with (.|rtrimstr("\r")), including the outer parentheses. This trims the carriage-return from the end of each string read from the file.

First, your data is not valid json, there is a comma too much:

{
  "TestNames": [
    {
      "Name": "test1",
      "CreateDate": "2016-08-30T10:52:52Z",
      "Id": "testId1", <--- Remove that!
    },
    {
      "Name": "test2",
      "CreateDate": "2016-08-30T10:52:13Z",
      "Id": "testId2"
    }
  ]
}

Once you've fixed that you can use jq for parsing json on the command line:

echo "$x" | jq -r '.TestNames[]|"\(.Name) , \(.Id)"'

if you need to keep the output values.

declare -A map1

while read name id ; do
    echo "$name"
    echo "$id"
    map1[$name]=$id

done < <(echo "$x" | jq -r '.TestNames[]|"\(.Name) \(.Id)"')

echo "count : ${#map1[@]}"
echo "in loop: ${map1[$name]}"

Further to Jeff's answer, please note that the transformation can be accomplished with a single invocation of jq. If your jq has the inputs filter:

jq -Rn '[inputs] | {cassandra:{nodes:map({ip_address:.,type:"seed"})}}' 

Otherwise:

jq -Rs 'split("\n") | {cassandra:{nodes:map({ip_address:.,type:"seed"})}}' ips.txt

If you really cannot use a proper JSON parser such as jq^[1] , try an awk-based solution:

Bash 4.x:

Copyreadarray -t values < <(awk -F\" 'NF>=3 {print $4}' myfile.json)

Bash 3.x:

CopyIFS=$'\n' read -d '' -ra values < <(awk -F\" 'NF>=3 {print $4}' myfile.json)

This stores all property values in Bash array ${values[@]}, which you can inspect with
declare -p values.

These solutions have limitations:

• each property must be on its own line,
• all values must be double-quoted,
• embedded escaped double quotes are not supported.

All these limitations reinforce the recommendation to use a proper JSON parser.

^{Note: The following alternative solutions use the Bash 4.x+ readarray -t values command, but they also work with the Bash 3.x alternative, IFS=$'\n' read -d '' -ra values.}

grep + cut combination: A single grep command won't do (unless you use GNU grep - see below), but adding cut helps:

Copyreadarray -t values < <(grep '"' myfile.json | cut -d '"' -f4)

GNU grep: Using -P to support PCREs, which support \K to drop everything matched so far (a more flexible alternative to a look-behind assertion) as well as look-ahead assertions ((?=...)):

Copyreadarray -t values < <(grep -Po ':\s*"\K.+(?="\s*,?\s*$)' myfile.json)

Finally, here's a pure Bash (3.x+) solution:

^{What makes this a viable alternative in terms of performance is that no external utilities are called in each loop iteration; however, for larger input files, a solution based on external utilities will be much faster.}

Copy#!/usr/bin/env bash

declare -a values # declare the array                                                                                                                                                                  

# Read each line and use regex parsing (with Bash's `=~` operator)
# to extract the value.
while read -r line; do
  # Extract the value from between the double quotes
  # and add it to the array.
  [[ $line =~ :[[:blank:]]+\"(.*)\" ]] && values+=( "${BASH_REMATCH[1]}" )
done < myfile.json                                                                                                                                          

declare -p values # print the array

^{[1] Here's what a robust jq-based solution would look like (Bash 4.x):
readarray -t values < <(jq -r '.[]' myfile.json)}