So I do cloud devops and have managed to create a lot of automations using BASH. For example, using AWS cli tools with the default output format of json, I have written many scripts using the AWS cli where I pipe the output to jq and get the results I am looking for. Combined with tools like jqplay, I have accomplished a lot. But there is a limit to BASH's usefulness when you are doing more complex operations. For that reason I have tried to lean in and do more stuff in python. I have gotten pretty good at modifying existing code and have written some pretty useful smaller python scripts.

But several times over the last few months, I keep trying and failing to really comprehend pythons handling of json. Such that I have given up and gone back to bash to complete a project.

So I am asking for help with two things from r/learnpython.

1. Solving the particular problem I am having right now.

2. Finally understanding how to parse any json with python.

ONE - - - - - - My current problem.

So using the code below, I have learned how to just get my data using boto3, convert it to json using json.dumps() and pretty print the json. (By the way, I need to use the standard json library here)

import boto3, json
from sys import argv

account = argv[1]

##THE FUNCTION BELOW WORKS FINE AND IS NOT REALLY RELEVANT TO MY QUESTION
def get_app_vpc_name(account): 
    if 'sbx' in account:
        return 'sbx-app-' + account
    elif 'dev' in account:
        return 'dev-app-' + account
    elif 'tst' in account:
        return 'tst-app-' + account
    elif 'prd' in account:
        return 'prd-app-' + account

## SETUP boto3 FOR AWS API 
boto3.setup_default_session(profile_name=account)
ec2 = boto3.client('ec2')

## GET A PARTICULAR VPC OUTPUT
def get_app_vpc_cidr_block(account):
    app_vpc_cidr_blk_name = '-'.join([account, 'app-vpc-cidr-block'])
    vpc = ec2.describe_vpcs(
    Filters=[
        {
            'Name': 'tag:Name',
            'Values': [
                get_app_vpc_name(account)
            ]
        }
    ]
    )
## CONVERT PYTHON DICTIONARY TO JSON USING json.dumps
    vpc_json = json.dumps(vpc, indent=6)

## PRETTY PRINT THE JSON
    print(vpc_json)

---- output FROM ABOVE CODE

{
      "Vpcs": [
            {
                  "CidrBlock": "10.215.188.0/22",
                  "DhcpOptionsId": "dopt-a370e999",
                  "State": "available",
                  "VpcId": "vpc-046b1f660f8337999",
                  "OwnerId": "999092819999",
                  "InstanceTenancy": "default",
                  "CidrBlockAssociationSet": [
                        {
                              "AssociationId": "vpc-cidr-assoc-027d7fe136117b999",
                              "CidrBlock": "10.215.188.0/22",
                              "CidrBlockState": {
                                    "State": "associated"
                              }
                        }
                  ],
                  "IsDefault": false,
                  "Tags": [
                        {
                              "Key": "network_tier",
                              "Value": "app"
                        },
                        {
                              "Key": "network_name",
                              "Value": "llh-devapp"
                        },
                        {
                              "Key": "ingress_support",
                              "Value": "false"
                        },
                        {
                              "Key": "usage",
                              "Value": "central-network"
                        },
                        {
                              "Key": "network_environment",
                              "Value": "dev"
                        },
                        {
                              "Key": "Name",
                              "Value": "dev-app-llh-devapp"
                        },
                        {
                              "Key": "account_name",
                              "Value": "N/A"
                        }
                  ]
            }
      ],
      "ResponseMetadata": {
            "RequestId": "9619218c-be04-4e36-bc4c-e8c8411a7999",
            "HTTPStatusCode": 200,
            "HTTPHeaders": {
                  "x-amzn-requestid": "9619218c-be04-4e36-bc4c-e8c8411a7999",
                  "cache-control": "no-cache, no-store",
                  "strict-transport-security": "max-age=31536000; includeSubDomains",
                  "content-type": "text/xml;charset=UTF-8",
                  "content-length": "1966",
                  "date": "Fri, 30 Sep 2022 19:40:09 GMT",
                  "server": "AmazonEC2"
            },
            "RetryAttempts": 0
      }
}

So I have made a lot of progress, but where I have struggled for weeks is parsing the json (or even the dictionary before I convert it to json) to get the exact data I need. Over and over and over, I keep running into type and other errors, but I have not succeeded in just parsing and get just the data I need. from the json/dictionary.

In this particular case, all I want is to get the CidrBlock from the data. I have had partial success by working with the code below, appended to the above script. (I will just show the function with the extra code)

def get_app_vpc_cidr_block(account):
    app_vpc_cidr_blk_name = '-'.join([account, 'app-vpc-cidr-block'])
    vpc = ec2.describe_vpcs(
    Filters=[
        {
            'Name': 'tag:Name',
            'Values': [
                get_app_vpc_name(account)
            ]
        }
    ]
    )
    vpc_json = json.dumps(vpc, indent=6)
    print(vpc_json)
    ## EXTRA CODE. USING DICTIONARY ITEMS. JSON CODE ABOVE IS IRELEVANT
    for key, value in vpc.items():
        results = value
        print(results[0]['CidrBlock'])

I would like to add that I have tried a LOT of differnet things to simply retrieve the CidrBlock data. Depending on whether or not I dumped it to json, I have gotten so many errors. (Often type errors, but no real success.

Here is what the code above returns after the json print. (NOTE that it does return the CidrBlock before the error).

10.215.188.0/22
Traceback (most recent call last):
  File "./caller.py", line 6, in <module>
     get_app_vpc_cidr_block(aws_acct)
  File "/var/lib/jenkins/testscripts/getAppVpcCidrBlock.py", line 35, in get_app_vpc_cidr_block
print(results[0]['CidrBlock'])
KeyError: 0

Regarding this problem in particular, my only question is this.

What is the correct pythonic way to retrieve the damned CidrBlock value from the above dictionary/json and assign the value to a variable?

2----MORE GENERAL JSON QUESTIONS

• Do I even need to convert output like above from a dictionary to json? (After retrieving the data using boto3, it is of the dictionary type.)

• Once you have an object (json or dictionary), what is the proper way to parse it and get particular data from it. In my case I will almost always want to retrieve certain values from the data and assign variables to those values.

• To illustrate the above question, suppose I wanted to retrive the following values from the json and assign each value to a variable

  • CidrBLock

  • VpcId

  • AssociationId (from CidrBlockAssociationSet)

  • Name Value (from tags)

• Is there a python tool similar to jqplay that I can use to play with json to find the right python to get the data I want?

Thanks and I appreciate any help y'all can give me on this.