There are a number of tools specifically designed for the purpose of manipulating JSON from the command line, and will be a lot easier and more reliable than doing it with Awk, such as jq:

curl -s 'https://api.github.com/users/lambda' | jq -r '.name'

You can also do this with tools that are likely already installed on your system, like Python using the json module, and so avoid any extra dependencies, while still having the benefit of a proper JSON parser. The following assume you want to use UTF-8, which the original JSON should be encoded in and is what most modern terminals use as well:

Python 3:

curl -s 'https://api.github.com/users/lambda' | \
    python3 -c "import sys, json; print(json.load(sys.stdin)['name'])"

Python 2:

export PYTHONIOENCODING=utf8
curl -s 'https://api.github.com/users/lambda' | \
    python2 -c "import sys, json; print json.load(sys.stdin)['name']"

Frequently Asked Questions

Why not a pure shell solution?

The standard POSIX/Single Unix Specification shell is a very limited language which doesn't contain facilities for representing sequences (list or arrays) or associative arrays (also known as hash tables, maps, dicts, or objects in some other languages). This makes representing the result of parsing JSON somewhat tricky in portable shell scripts. There are somewhat hacky ways to do it, but many of them can break if keys or values contain certain special characters.

Bash 4 and later, zsh, and ksh have support for arrays and associative arrays, but these shells are not universally available (macOS stopped updating Bash at Bash 3, due to a change from GPLv2 to GPLv3, while many Linux systems don't have zsh installed out of the box). It's possible that you could write a script that would work in either Bash 4 or zsh, one of which is available on most macOS, Linux, and BSD systems these days, but it would be tough to write a shebang line that worked for such a polyglot script.

Finally, writing a full fledged JSON parser in shell would be a significant enough dependency that you might as well just use an existing dependency like jq or Python instead. It's not going to be a one-liner, or even small five-line snippet, to do a good implementation.

Why not use awk, sed, or grep?

It is possible to use these tools to do some quick extraction from JSON with a known shape and formatted in a known way, such as one key per line. There are several examples of suggestions for this in other answers.

However, these tools are designed for line based or record based formats; they are not designed for recursive parsing of matched delimiters with possible escape characters.

So these quick and dirty solutions using awk/sed/grep are likely to be fragile, and break if some aspect of the input format changes, such as collapsing whitespace, or adding additional levels of nesting to the JSON objects, or an escaped quote within a string. A solution that is robust enough to handle all JSON input without breaking will also be fairly large and complex, and so not too much different than adding another dependency on jq or Python.

I have had to deal with large amounts of customer data being deleted due to poor input parsing in a shell script before, so I never recommend quick and dirty methods that may be fragile in this way. If you're doing some one-off processing, see the other answers for suggestions, but I still highly recommend just using an existing tested JSON parser.

Historical notes

This answer originally recommended jsawk, which should still work, but is a little more cumbersome to use than jq, and depends on a standalone JavaScript interpreter being installed which is less common than a Python interpreter, so the above answers are probably preferable:

curl -s 'https://api.github.com/users/lambda' | jsawk -a 'return this.name'

This answer also originally used the Twitter API from the question, but that API no longer works, making it hard to copy the examples to test out, and the new Twitter API requires API keys, so I've switched to using the GitHub API which can be used easily without API keys. The first answer for the original question would be:

curl 'http://twitter.com/users/username.json' | jq -r '.text'

The availability of parsers in nearly every programming language is one of the advantages of JSON as a data-interchange format.

Rather than trying to implement a JSON parser, you are likely better off using either a tool built for JSON parsing such as jq or a general purpose script language that has a JSON library.

For example, using jq, you could pull out the ImageID from the first item of the Instances array as follows:

jq '.Instances[0].ImageId' test.json

Alternatively, to get the same information using Ruby's JSON library:

ruby -rjson -e 'j = JSON.parse(File.read("test.json")); puts j["Instances"][0]["ImageId"]'

I won't answer all of your revised questions and comments but the following is hopefully enough to get you started.

Suppose that you had a Ruby script that could read a from STDIN and output the second line in your example output[0]. That script might look something like:

#!/usr/bin/env ruby
require 'json'

data = JSON.parse(ARGF.read)
instance_id = data["Instances"][0]["InstanceId"]
name = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Name" }["Value"]
owner = data["Instances"][0]["Tags"].find {|t| t["Key"] == "Owner" }["Value"]
cost_center = data["Instances"][0]["SubnetId"].split("-")[1][0..3]
puts "#{instance_id}\t#{name}\t#{cost_center}\t#{owner}"

How could you use such a script to accomplish your whole goal? Well, suppose you already had the following:

• a command to list all your instances
• a command to get the json above for any instance on your list and output it to STDOU

One way would be to use your shell to combine these tools:

echo -e "Instance id\tName\tcost centre\tOwner"
for instance in $(list-instances); do
    get-json-for-instance $instance | ./ugly-ruby-scriptrb
done

Now, maybe you have a single command that give you one json blob for all instances with more items in that "Instances" array. Well, if that is the case, you'll just need to modify the script a bit to iterate through the array rather than simply using the first item.

In the end, the way to solve this problem, is the way to solve many problems in Unix. Break it down into easier problems. Find or write tools to solve the easier problem. Combine those tools with your shell or other operating system features.

[0] Note that I have no idea where you get cost-center from, so I just made it up.

There is jq for parsing json on the command line:

 jq '.Body'

Visit this for jq: https://stedolan.github.io/jq/

First, consider whether you really need JSON. In your example code, you create a JSON string and then immediately decode it into a Python dict. Would it be simpler to just use a dict directly?

The problem with your current string is that you're not escaping your quotation marks properly. To avoid confusing multi-level escaping, use a triple-quoted string to represent the shell script and json.dumps to convert a dict into a JSON string:

import json
jsonstr = json.dumps({"script": """\
#!/bin/bash
set -e

readonly PRIMARY=/tech01/primary
readonly SECONDARY=/tech02/secondary
readonly LOCATION=(machineA machineB)
readonly MAPPED_LOCATION=/bat/data/snapshot
HOSTNAME=$hostname

dir1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

length1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} "ls '$dir1' | wc -l")
length2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} "ls '$dir2' | wc -l")

echo $length1
echo $length2

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    rm -rf $PRIMARY/*
    rm -rf $SECONDARY/*
    for el in $primary_partition
    do
        scp david@${LOCATION[0]}:$dir1/weekly_8880_"$el"_5.data $PRIMARY/. || scp david@${LOCATION[1]}:$dir2/weekly_8880_"$el"_5.data $PRIMARY/.
    done
fi"""})

Alternatively, you could put the shell script in its own file and open the file to read the string from it.

One suggestion is to use --args with jq to create the two arrays and then collect these in the correct location in the main document. Note that --args is required to be the last option on the command line and that all the remaining command line arguments will become elements of the $ARGS.positional array.

{
    jq -n --arg key APP-Service1-Admin '{($key): $ARGS.positional}' --args a b
    jq -n --arg key APP-Service1-View  '{($key): $ARGS.positional}' --args c d
} |
jq -s --arg key 'AD Accounts' '{($key): add}' |
jq --arg Service service1-name --arg 'AWS account' service1-dev '$ARGS.named + .'

The first two jq invocations create a set of two JSON objects:

{
  "APP-Service1-Admin": [
    "a",
    "b"
  ]
}
{
  "APP-Service1-View": [
    "c",
    "d"
  ]
}

The third jq invocation uses -s to read that set into an array, which becomes a merged object when passed through add. The merged object is assigned to our top-level key:

{
  "AD Accounts": {
    "APP-Service1-Admin": [
      "a",
      "b"
    ],
    "APP-Service1-View": [
      "c",
      "d"
    ]
  }
}

The last jq adds the remaining top-level keys and their values to the object:

{
  "Service": "service1-name",
  "AWS account": "service1-dev",
  "AD Accounts": {
    "APP-Service1-Admin": [
      "a",
      "b"
    ],
    "APP-Service1-View": [
      "c",
      "d"
    ]
  }
}

With jo:

jo -d . \
    Service=service1-name \
    'AWS account'=service1-dev  \
    'AD Accounts.APP-Service1-Admin'="$(jo -a a b)" \
    'AD Accounts.APP-Service1-View'="$(jo -a c d)"

The "internal" object is created using .-notation (enabled with -d .), and a couple of command substitutions for creating the arrays.

Or you can drop the -d . and use a form of array notation:

jo  Service=service1-name \
    'AWS account'=service1-dev \
    'AD Account[APP-Service1-Admin]'="$(jo -a a b)" \
    'AD Account[APP-Service1-View]'="$(jo -a c d)"

Since you control the content of that JSON file, and since you need to process it with shell on a low-memory device, don't use JSON: output shell syntax and you can source it directly

mac="8d:ff:86:7d:9c"
adblock="off"
parental="off"
vpn="newyork1.ovpn"
vpnstatus="off"
status="active" 

Then, after you be downloaded it to, say, status.sh

. ./status.sh
do_something_with "$status"
do_something_else_with "$vpn" "$vpnstatus"