It's may be not an easy decision

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("https://edwvb.blogspot.ru/2018/03/3-tipa-povedeniya-kotorye-opredelyayut-uspeshnyh-prodavcov.html")
bsObj = BeautifulSoup(html, "html.parser")

nameList = bsObj.findAll("div", {"dir":"ltr", "style":"text-align: left;", "trbidi":"on"})
nameList = [i.text for i in nameList]

After that we need first convert nameList[1] to pd.Series and then to DataFrame

S = pd.Series(nameList[1])
S.to_frame()

Try:

import requests
import pandas as pd
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
}

user_input = "Solarpanels"
site = f"https://news.google.com/rss/search?q={user_input}+when:14d&hl=en-GB&gl=DE&ceid=GB:en"


soup = BeautifulSoup(requests.get(site, headers=headers).content, "xml")

all_data = []
for item in soup.select("item"):
    all_data.append(
        {
            "title": item.title.text,
            "link": item.link.text,
            "pubDate": item.pubDate.text,
            "description": BeautifulSoup(
                item.description.text, "html.parser"
            ).get_text(strip=True), # or .get_text(strip=True, separator=" ")
            "source": item.source.text,
            "source_url": item.source["url"],
        }
    )

df = pd.DataFrame(all_data)
print(df.head().to_markdown(index=False))

Prints:

I believe that the below is what you are looking for. The idea is to look at each entry in xml_list as xml document, parse it and get the attrributes.

import xml.etree.ElementTree as ET

xml_list = ['<TAG attr_1="A" attr_2="1" attr_3="01"/>',
              '<TAG attr_1="B" attr_3="02"/>',
              '<TAG attr_1="C" attr_2="3" attr_3="03"/>']

result = []
for xml in xml_list:
  root = ET.fromstring(xml)
  entry = {}
  for i in range(1,4):
    entry[f'attr_{i}'] = root.attrib.get(f'attr_{i}',None)
  result.append(entry)
    
print(result)

output

[{'attr_1': 'A', 'attr_2': '1', 'attr_3': '01'}, {'attr_1': 'B', 'attr_2': None, 'attr_3': '02'}, {'attr_1': 'C', 'attr_2': '3', 'attr_3': '03'}]

If the pattern is always identical and no other deviations occur, the following procedure can be followed:

Copypd.DataFrame([e.text.split('-') for e in forcast])

Note: For reliable results, more detailed information is needed in the questionnaire.

Example

Copyfrom bs4 import BeautifulSoup
import pandas as pd

html = '''<div class="cell "><span>1.2</span><span class="m-unit"></span> - <span>2.0</span><span class="m-unit"></span></div>
<div class="cell "><span>1.5</span><span class="m-unit"></span> - <span>2.6</span><span class="m-unit"></span></div>'''

soup = BeautifulSoup(html)

forcast = soup.select('div')

pd.DataFrame([e.text.split('-') for e in forcast])

Output

Here is the list of things I would think about to improve:

• you are doubling on calls to handle_bs4_element() here:

  data.append([handle_bs4_element(rc) for rc in row_cells if handle_bs4_element(rc)])
  

  Instead, you can either allow "falsy" values for the row cells and filter them afterwards, or expand the loop:

  result = []
  for rc in row_cells:
      cell_text = handle_bs4_element(rc)
      if cell_text:
          result.append(cell_text)
  data.append(result)
  

• the DRY principle. There are several repeated blocks of code, like:

  if len(_res) == 1:
      return _res[0]
  else:
      return _res
  

• using list comprehensions is not only more Pythonic, but actually faster. E.g. you can replace:

  _res = []
  for td_content in element.contents:
      _res.append(handle_bs4_element(td_content))
  

  with:

  _res = [handle_bs4_element(td_content) for td_content in element.contents]
  

• you can use the short if/else one-liner, replacing:

  if len(_res) == 1:
      return _res[0]
  else:
      return _res
  

  with:

  return _res[0] if len(_res) == 1 else _res
  

• variable naming. _res should not be started with an underscore. You are confusing private class or instance attributes with regular variables. _res should probably be called result, or may be cell_data?

• if you will have more of this kind of tag-specific processing logic, continuing to put it as an another elif would hurt readability and does not scale well. Consider using the "Extract Method" refactoring method and defining a separate functions for each of the cases.

• instead of using the .contents list directly, look into using .get_text(), which completes an element's text including the children texts recursively. Not sure if applicable for your problem.

• or, instead of .contents list, you can use the .children generator

As a side note, there is also a simpler way to parse HTML tables - pandas.read_html() which would load an HTML table into a DataFrame, you can then easily dump the dataframe into a list or into CSV, or into an Excel file directly. For example, the following code:

from pprint import pprint

import pandas as pd


df = pd.read_html('table_sample.html')[0]  # get the first parsed dataframe
pprint(df.values.tolist())

Would automagically produce:

[[nan, 'Description', 'Col 1', 'Col 2', 'Col 3'],
 [1.0, 'Some paragraph text', 'x', '5', '2'],
 [2.0, 'HEADER 1', nan, nan, nan],
 [3.0, 'Some text: (1) Check out this Figure 1.0.', 'x', '2', '1'],
 [4.0, '(2) Some more text', 'x', '2', '1'],
 [5.0, '(3) Additional text', 'x', '2', '1'],
 [6.0, '(4) A bit more text', 'x', '2', '1'],
 [7.0, '(5) A span Figure 1.0 for  edited text. At this point the span starts again', 'x', '2', '1'],
 [8.0, 'HEADER 2', nan, nan, nan],
 [9.0, 'Weird formatting, because Confluence', 'x', '4', '2'],
 [10.0, 'HEADER 3', nan, nan, nan],
 [11.0, 'A paragraph about header 3.  This is just silly. Strong indeed.', 'x', '3', '3'],
 [12.0, 'Something about things or what not. Why is this in a span?', 'x', '2', '2'],
 [13.0, 'HEADER 4', nan, nan, nan],
 [14.0, 'Section 4 baby! Or header.  Confluence formatting fun.', 'x', '2', '3'],
 [15.0, 'Pretty boring span of text', 'x', '2', '2'],
 [16.0, 'HEADER 5', nan, nan, nan],
 [17.0, 'A big paragraph describing more stuff. Super exciting.', 'x', '4', '2']]

Where you write .get_text(), you could use .get_text().strip() to strip off whitespace.

You are storing several columns, which may work well enough, you can combine them with zip(x, y) if need be. But you might find it more convenient to ask BeautifulSoup to find the table, and then find_all('tr') within the table, that is, iterate over the rows.

Consider representing (part of) a table row like this:

row = dict(opponent='vs. Northfield Mt. Hermon',
           advantage='Home',
           score='1-1')

If you have a tr object, a table row, you could easily find those values.

With that in hand, you could represent the whole table as a list of rows, with each row being a dict.

Then output the rows to a spreadsheet as you've been doing. Or $ pip install pandas and you can do:

rows = read_html_table_rows()
df = pandas.Dataframe(rows)
df.to_excel('results.xls')

Document:

pandas.read_html(url, attrs={'class': 'table_grey_border'})

Try this:

Replace:

df = pd.DataFrame(data, columns=['Job title', 'URL'])

With:

df = pd.DataFrame({"Job title": list_job_titles, "URL": list_job_URLs})

The table variable contains a list. You would need to call find_all on its members (even though you know it's a list with only one member), not on the entire thing.

>>> type(table)
<class 'bs4.element.ResultSet'>
>>> type(table[0])
<class 'bs4.element.Tag'>
>>> len(table[0].find_all('tr'))
6
>>>