Hi all,

Long time programmer here, but new to Python. This will be a long and, I think, complicated issue, so appreciate anyone who reads through it all and has any suggestions. I've looked up different ways to pull this data and don't seem to be making any progress. I'm sure there's a much better way.

I'm writing a program that will connect to our library to pull a list of everything we have checked out and I want to output a sorted list by due date and whether it has holds or not. I've got the code working to log in and pull a json data structure, but I cannot get it to export the data in the correct order. The json data is(to me) hideously complex with some data(due date) in one section and other data in another section. I'm able to pull the fields I want, but keeping them together is proving challenging.

For example, the title and subtitle are in the 'bibs/briefinfo' section with a key value of 'title' or 'subtitle'. Due Date is also in the 'checkouts' section with a key value of 'dueDate'. When I loop through them, though, the Titles are in one order, the due dates are in another order and the subtitles another.

I used BeautifulSoup because it's a webpage with json in it, so used BS to read the webpage.

I'm wanting to pull the following fields for each book so I can display the info for each book:

title, subtitle, contentType from briefinfo section

duedate from checkouts section

heldcopies and availablecopies from the availability section

Here's the pertinent section of my code:

soup = BeautifulSoup(index_page.text, 'html.parser')
            all_scripts = soup.find_all('script', {"type":"application/json"})

            for script in all_scripts:
                jsondata = json.loads(script.text)
                print(jsondata)
                
                output = []
                for i in item_generator(jsondata, "bibTitle"):
                    ans = {i}
                    print(i)
                    output.append(ans)

                for i in item_generator(jsondata, "dueDate"):
                    ans = {i}
                    output.append(ans)

                print("Subtitle----------------------")
                for i in item_generator(jsondata, "subtitle"):
                    ans = {i}
                    print(i)
                    output.append(ans)

print(output)

Here's the json output from my print statement so I can see what I'm working with. I tried to format it so it's easier to read. I removed a lot of other elements to keep the size down. Hopefully I didn't break any of the brackets.

{

'app':

{

'coreCssFingerprint': '123123123',

'coreAssets':

{

'cdnHost': 'https://xyz.com',

'cssPath': '/dynamic_stylesheet',

'defaultStylesheet': 'xyz.css'

},

},

'entities':

{

'listItems': {},

'cards': {},

'accounts':

{

'88888888':

  {

  'barcode': '999999999',

  'expiryDate': None, 

  'id': 88888888, 

  }

},

'shelves':

  {

  '88888888': 

  	{

  	'1111222222': 

{

'id': 1111222222,

'metadataId': 'S00A1122334',

'shelf': 'for_later',

'privateItem': True,

'dateAdded': '2023-12-30',

},

  	}

  }, 

'users':

  {

  '88888888': 

  	{ 

  	'accounts': \[88888888\], 

  	'status': 'A', 

  	'showGroupingDebug': False, 

  	'avatarUrl': '', 

  	'id': 88888888, 

  	}

  }, 

  'eventPrograms': {}, 

  'checkouts': 

  	{

  	'112233445566778899': 

{

'checkoutId': '112233445566778899',

'materialType': 'PHYSICAL',

'dueDate': '2024-08-26',

'metadataId': 'S99Z000000',

'bibTitle': "The Lord of the Rings"

},

  	'998877665544332211': 

{

'checkoutId': 998877665544332211',

'materialType': 'PHYSICAL',

'dueDate': '2024-08-26',

'metadataId': 'S88Y00000',

'bibTitle': 'The Lord of the Rings'

},

  	}, 

  'eventSeries': {}, 

  'catalogBibs': {},

  'bibs': 

  	{

  	'S88Y00000': 

{

'id': 'S88Y00000',

'briefInfo':

{

'superFormats': ['BOOKS', 'MODERN_FORMATS'],

'genreForm': [],

'callNumber': '123.456',

'authors': ['Tolkien, J.R.R.'],

'metadataId': 'S88Y00000',

'jacket':

{

'type': 'hardcover',

'local_url': None

},

'contentType': 'FICTION',

'format': 'BK',

'subtitle': 'The Two Towers',

'title': 'The Lord of the Rings',

'id': 'S88Y00000',

},

'availability':

{

'heldCopies': 0,

'singleBranch': False,

'metadataId': 'S88Y00000',

'statusType': 'AVAILABLE',

'totalCopies': 3,

'availableCopies': 2

}

},

'S77X12345':

{

'id': 'S77X12345',

'briefInfo':

{

'superFormats': ['BOOKS', 'MODERN_FORMATS'],

'genreForm': [],

'callNumber': '123.457',

'authors': ['Tolkien, J.R.R.'],

'metadataId': 'S77X12345',

'jacket':

{

'type': 'hardcover',

'local_url': None

},

'contentType': 'FICTION',

'format': 'BK',

'subtitle': 'The Fellowship of the Ring',

'title': 'The Lord of the Rings',

'id': 'S77X12345',

},

'availability':

{

'heldCopies': 0,

'singleBranch': False,

'metadataId': 'S77X12345',

'statusType': 'AVAILABLE',

'totalCopies': 2,

'availableCopies': 1

}

}

Anyone know of a better way to parse this data? Thanks!

I was giving the task to flatten a JSON file which basically means to take every arrays and turn it table (not a actual table, all I need to do is give them a spreadsheet with the mapping). The only thing that I need to make sure is that there are a key between those 2 tables. There are many layered / nested arrays in the JSON.

Is there any automatic way of achieving this? There isn't any logic, just making sure that all layers talk to each other.