https://lxml.de/

If parsing speed is a key factor for you, consider using cElementTree or lxml.

There are definitely more options out there, see these threads:

• What is the fastest way to parse large XML docs in Python?
• How do I parse XML in Python?
• High-performance XML parsing in Python with lxml

Here's good answer about xml.etree.ElementTree.iterparse practice on huge XML files. lxml has the method as well. The key to stream parsing with iterparse is manual clearing and removing already processed nodes, because otherwise you will end up running out of memory.

Another option is using xml.sax. The official manual is too formal to me, and lacks examples so it needs clarification along with the question. Default parser module, xml.sax.expatreader, implement incremental parsing interface xml.sax.xmlreader.IncrementalParser. That is to say xml.sax.make_parser() provides suitable stream parser.

For instance, given a XML stream like:

<?xml version="1.0" encoding="utf-8"?>
<root>
  <entry><a>value 0</a><b foo='bar' /></entry>
  <entry><a>value 1</a><b foo='baz' /></entry>
  <entry><a>value 2</a><b foo='quz' /></entry>
  ...
</root>

Can be handled in the following way.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import xml.sax


class StreamHandler(xml.sax.handler.ContentHandler):

  lastEntry = None
  lastName  = None


  def startElement(self, name, attrs):
    self.lastName = name
    if name == 'entry':
      self.lastEntry = {}
    elif name != 'root':
      self.lastEntry[name] = {'attrs': attrs, 'content': ''}

  def endElement(self, name):
    if name == 'entry':
      print({
        'a' : self.lastEntry['a']['content'],
        'b' : self.lastEntry['b']['attrs'].getValue('foo')
      })
      self.lastEntry = None
    elif name == 'root':
      raise StopIteration

  def characters(self, content):
    if self.lastEntry:
      self.lastEntry[self.lastName]['content'] += content


if __name__ == '__main__':
  # use default ``xml.sax.expatreader``
  parser = xml.sax.make_parser()
  parser.setContentHandler(StreamHandler())
  # feed the parser with small chunks to simulate
  with open('data.xml') as f:
    while True:
      buffer = f.read(16)
      if buffer:
        try:
          parser.feed(buffer)
        except StopIteration:
          break
  # if you can provide a file-like object it's as simple as
  with open('data.xml') as f:
    parser.parse(f)

Scrapy is used for scraping web pages (extracting data from web pages) hence the name.

Beautiful Soup is library for parsing/pulling data from XML and HTML files.

xml.elementtree provides object representation of the XML file and it is a XML processing module of Python XML package. It is neat to use for parsing and manipulating data in XML format.

lxml is as they claim compatible yet superior to elementtree of the Python XML module but essentially does the same however, I never used it for parsing of HTML files.

In my experience I used Scrapy for fetching data from various user panels that did not have any kind of API for pulling the data. However, parsing of HTML files I mostly did with Beautiful Soup as it is really neat and easy to use. Regarding XML parsing I mostly used Python XML package however, I never had any complicated XML parsing to perform so Python XML package covered everything I need.

The right tool really depends on your requirements. If you need library to parse XML and HTML files both I would go with Beautiful Soup as it is really easy to use and you have vast documentation online.