I tried MySQL, imported the XML data set files into the database, then exported them to CSV format and processed 82.2GB files in just 3 hours.

The XML Utilities library is worth a try, assuming valid & flat XML structure - it even comes with a command line xml2csv utility.

It specifically states:

xmlutils.py is a set of Python utilities for processing xml files serially for converting them to various formats (SQL, CSV, JSON). The scripts use ElementTree.iterparse() to iterate through nodes in an XML document, thus not needing to load the entire DOM into memory. The scripts can be used to churn through large XML files (albeit taking long :P) without memory hiccups.

I recommend just using libraries because they're usually very optimised. I'll talk about that later. For now, here's a way that utilises the xml.dom.minidom module, which is a part of the Python standard library, so no additional libraries are required.

Edit: rewrote the last part using the standard CSV library instead of manually writing the file, as suggested by a comment. That makes for 2 Python built-in modules, not 1. The original code for the CSV writing will be at the end of the reply, if you're interested.

from xml.dom import minidom
from csv import DictWriter

# Step 1: Read and parse the XML file
# Write it as a string, or open the file and read it
xml_file = open('employees.xml', 'r')
xml_data = xml_file.read()

dom = minidom.parseString(xml_data)
employees = dom.getElementsByTagName('employee')

xml_file.close()

# Step 2: Extract the required information
data = []
for employee in employees:
    emp_data = {}
    for child in employee.childNodes:
        if child.nodeType == minidom.Node.ELEMENT_NODE:
            emp_data[child.tagName] = child.firstChild.data
    data.append(emp_data)

# Step 3: Write the extracted information to a CSV file
with open('output.csv', 'w', newline = '') as csv_file:
    fieldnames = ['id', 'name', 'age', 'salary', 'division']
    writer = DictWriter(csv_file, fieldnames = fieldnames)

    writer.writeheader()
    for emp_data in data:
        writer.writerow(emp_data)

Don't reinvent the wheel, just realign it.

— Anthony J. D'Angelo, I think

I recommend NOT using this code. You should really just use lxml. It's extremely simple and easy to use and can handle complex XML structures with nested elements and attributes. Let me know how everything goes!

Original CSV write code without CSV library

# Step 3: Write the extracted information to a CSV file
with open('output.csv', 'w') as f:
    f.write('id,name,age,salary,division\n')
    for emp_data in data:
        f.write(f"{emp_data['id']},{emp_data['name']},{emp_data['age']},{emp_data['salary']},{emp_data['division']}\n")