[THIS IS NOT WORKING. READ BELOW.]

There are many ways to do this.

Assuming elements of the array are supposed to be unique, which your use case strongly implies, you can just pass the resulting array after the addition through the unique filter.

$ cat foo.json | jq '.widgets[] | select(.name=="foo").properties |= (.+ ["cat"] | unique)'

There are a few problems here.

One is that the resulting output is partial as it is missing the container object.

Another one is that the edited array looses the commas separating the objects thus becoming illegal JSON.

The actual result from the above command is:

{
  "name": "foo",
  "properties": [
    "baz",
    "cat"
  ]
}
{
  "name": "bar"
}

You can use with_entries() function on the parent element to convert the sub elements into a pair keyed by key and value and add the string to value array, if it doesn't have it already

jq '.contact_groups |= ( with_entries( if ( .value | index("/contact_group/109") | not ) then .value += [ "/contact_group/109" ] else . end ) )'

The with_entries(..) builtin is a shorthand for doing to_entries | map | from_entries. The index("/contact_group/109") | not part ensures to add the entry if the string is not already present in the .value array.

^{See demo on jqplay}

Here is a safe if pedestrian solution:

.elements 
|=  (map(.name) | index("foo")) as $ix
    | if $ix 
      then .[$ix]["value"] = "BAR" 
      else . + [{name: "foo", value: "BAR"}]
      end

You might want to abstract away the "foo" and "BAR" bits:

upsert

# Input is assumed to be an array of {name:_, value:_} objects
def upsert($foo; $bar):
  (map(.name) | index($foo)) as $ix
  | if $ix then .[$ix]["value"] = $bar else . + [{name: $foo, value: $bar}] end;

Usage:

.elements |= upsert("foo"; "BAR")

jq has a flag for feeding actual JSON contents with its --argjson flag. What you need to do is, store the content of the first JSON file in a variable in jq's context and update it in the second JSON

jq --argjson groupInfo "$(<input.json)" '.[].groups += [$groupInfo]' orig.json

The part "$(<input.json)" is shell re-direction construct to output the contents of the file given and with the argument to --argjson it is stored in the variable groupInfo. Now you add it to the groups array in the actual filter part.

Putting it in another way, the above solution is equivalent of doing this

jq --argjson groupInfo '{"id": 9,"version": 0,"lastUpdTs": 1532371267968,"name": "Training" }' \
   '.[].groups += [$groupInfo]' orig.json

"The question"

In answer to The question: Yes. jq 1.5 has keys_unsorted, so you can use the following def of walk/1, which is now standard in the “master” version of jq:

Copy# Apply f to composite entities recursively, and to atoms
def walk(f):
  . as $in
  | if type == "object" then
      reduce keys_unsorted[] as $key
        ( {}; . + { ($key):  ($in[$key] | walk(f)) } ) | f
  elif type == "array" then map( walk(f) ) | f
  else f
  end;

For further details and examples, see the “development” version of the jq manual, the jq FAQ https://github.com/stedolan/jq/wiki/FAQ, etc.

"No duplicates are allowed within the elements of an array"

This is readily accomplished using index/1; you might like to use a helper function such as:

Copydef ensure_has($x): if index([$x]) then . else . + [$x] end;

"If any of the parents of .name do not exist, should be able to add them on the fly"

If I understand this requirement correctly, it will useful for you to know that jq will create objects based on assignments, e.g.

{} | .a.b.c = 1

yields

Copy{"a":{"b":{"c":1}}}

Thus, using your example, you will probably want to include something like this in your walk:

Copyif type == "object" and has("spec")
   then (.spec.template.spec.containers? // null) as $existing
   | if $existing then .spec.template.spec.containers |= ... 
     else .spec.template.spec.containers = ...
     end
else .
end

The |= .+ part in the filter adds a new element to the existing array. You can use jq with filter like:

jq '.data.messages[3] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

To avoid using the hardcoded length value 3 and dynamically add a new element, use . | length which returns the length, which can be used as the next array index, i.e.,

jq '.data.messages[.data.messages| length] |= . + {
      "date": "2010-01-07T19:55:99.999Z", 
      "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
      "status": "OKKK", 
      "message": "metadata loaded into iRODS successfullyyyyy"
}' inputJson

(or) as per peak's suggestion in the comments, using the += operator alone

jq '.data.messages += [{
     "date": "2010-01-07T19:55:99.999Z",
     "xml": "xml_samplesheet_2017_01_07_run_09.xml", 
     "status": "OKKK", 
     "message": "metadata loaded into iRODS successfullyyyyy"
}]'

which produces the output you need:

{
  "report": "1.0",
  "data": {
    "date": "2010-01-07",
    "messages": [
      {
        "date": "2010-01-07T19:58:42.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OK",
        "message": "metadata loaded into iRODS successfully"
      },
      {
        "date": "2010-01-07T20:22:46.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata duplicated into iRODS"
      },
      {
        "date": "2010-01-07T22:11:55.949Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "NOK",
        "message": "metadata was not validated by XSD schema"
      },
      {
        "date": "2010-01-07T19:55:99.999Z",
        "xml": "xml_samplesheet_2017_01_07_run_09.xml",
        "status": "OKKK",
        "message": "metadata loaded into iRODS successfullyyyyy"
      }
    ]
  }
}

Use jq-play to dry-run your jq-filter and optimize any way you want.

Reassign the .Statement array with the new one containing the former entries with the target object updated:

jq --arg root "arn:aws:iam::$ACCOUNT_ID:root" \
'.Statement |= map(select(.Sid=="Allow use of the key").Principal.AWS |= . + [$root])' <<<$policy

Do not try to parse JSON text with standard text processing tools like awk, sed or without JSON modules in perl, as they are non JSON syntax aware. jq is a lightweight JSON processor that allows you manipulate content on the fly.

For e.g. your requirement could be simply written as

jq --arg new "arn:aws:iam::xxxxxxx:role/3" \
  '.Statement[].Principal.AWS? += [$new]' json

If there is more than one element to be appended, create the arg as a JSON blob

jq --argjson new \
  '["arn:aws:iam::xxxxxxx:role/3", "arn:aws:iam::xxxxxxx:role/4"]' \
  '.Statement[].Principal.AWS? += $new'

Instructions for downloading and using the tool are at Download jq and usage tutorial at jq Manual

If the contents are in a file and you want to modify this file permanently, use a in-place edit tool like sponge available from GNU moreutils package which you can do

jq --arg new "arn:aws:iam::xxxxxxx:role/3" \
  '.Statement[].Principal.AWS? += [$new]' json | sponge json

or use a POSIX tool like mktemp

temp="$(mktemp)"
jq --arg new "arn:aws:iam::xxxxxxx:role/3" \
  '.Statement[].Principal.AWS? += [$new]' json > "$temp" && mv "$temp" json