The following will read everything into a dictionary keyed by player name. The value associated with each player is itself a dictionary acting as a record with named fields associated with the items converted to a format suitable for further processing.
info = {}
with open('scoring_info.txt') as input_file:
for line in input_file:
player, stats, outcome, date = (
item.strip() for item in line.split('-', 3))
stats = dict(zip(('kills', 'deaths', 'assists'),
map(int, stats.split('/'))))
date = tuple(map(int, date.split('-')))
info[player] = dict(zip(('stats', 'outcome', 'date'),
(stats, outcome, date)))
print('info:')
for player, record in info.items():
print(' player %r:' % player)
for field, value in record.items():
print(' %s: %s' % (field, value))
# sample usage
player = 'Fizz'
print('\n%s had %s kills in the game' % (player, info[player]['stats']['kills']))
Output:
info:
player 'Shyvana':
date: (2012, 11, 22)
outcome: Loss
stats: {'assists': 5, 'kills': 12, 'deaths': 4}
player 'Miss Fortune':
date: (2012, 11, 22)
outcome: Win
stats: {'assists': 3, 'kills': 12, 'deaths': 4}
player 'Fizz':
date: (2012, 11, 22)
outcome: Win
stats: {'assists': 5, 'kills': 12, 'deaths': 4}
Fizz had 12 kills in the game
Alternatively, rather than holding most of the data in dictionaries, which can make nested-field access a little awkward — info[player]['stats']['kills'] — you could instead use a little more advanced "generic" class to hold them, which will let you write info2[player].stats.kills instead.
To illustrate, here's almost the same thing using a class I've named Struct because it's somewhat like the C language's struct data type:
class Struct(object):
""" Generic container object """
def __init__(self, **kwds): # keyword args define attribute names and values
self.__dict__.update(**kwds)
info2 = {}
with open('scoring_info.txt') as input_file:
for line in input_file:
player, stats, outcome, date = (
item.strip() for item in line.split('-', 3))
stats = dict(zip(('kills', 'deaths', 'assists'),
map(int, stats.split('/'))))
victory = (outcome.lower() == 'win') # change to boolean T/F
date = dict(zip(('year','month','day'), map(int, date.split('-'))))
info2[player] = Struct(champ_name=player, stats=Struct(**stats),
victory=victory, date=Struct(**date))
print('info2:')
for rec in info2.values():
print(' player %r:' % rec.champ_name)
print(' stats: kills=%s, deaths=%s, assists=%s' % (
rec.stats.kills, rec.stats.deaths, rec.stats.assists))
print(' victorious: %s' % rec.victory)
print(' date: %d-%02d-%02d' % (rec.date.year, rec.date.month, rec.date.day))
# sample usage
player = 'Fizz'
print('\n%s had %s kills in the game' % (player, info2[player].stats.kills))
Output:
info2:
player 'Shyvana':
stats: kills=12, deaths=4, assists=5
victorious: False
date: 2012-11-22
player 'Miss Fortune':
stats: kills=12, deaths=4, assists=3
victorious: True
date: 2012-11-22
player 'Fizz':
stats: kills=12, deaths=4, assists=5
victorious: True
date: 2012-11-22
Fizz had 12 kills in the game
Answer from martineau on Stack OverflowThe following will read everything into a dictionary keyed by player name. The value associated with each player is itself a dictionary acting as a record with named fields associated with the items converted to a format suitable for further processing.
info = {}
with open('scoring_info.txt') as input_file:
for line in input_file:
player, stats, outcome, date = (
item.strip() for item in line.split('-', 3))
stats = dict(zip(('kills', 'deaths', 'assists'),
map(int, stats.split('/'))))
date = tuple(map(int, date.split('-')))
info[player] = dict(zip(('stats', 'outcome', 'date'),
(stats, outcome, date)))
print('info:')
for player, record in info.items():
print(' player %r:' % player)
for field, value in record.items():
print(' %s: %s' % (field, value))
# sample usage
player = 'Fizz'
print('\n%s had %s kills in the game' % (player, info[player]['stats']['kills']))
Output:
info:
player 'Shyvana':
date: (2012, 11, 22)
outcome: Loss
stats: {'assists': 5, 'kills': 12, 'deaths': 4}
player 'Miss Fortune':
date: (2012, 11, 22)
outcome: Win
stats: {'assists': 3, 'kills': 12, 'deaths': 4}
player 'Fizz':
date: (2012, 11, 22)
outcome: Win
stats: {'assists': 5, 'kills': 12, 'deaths': 4}
Fizz had 12 kills in the game
Alternatively, rather than holding most of the data in dictionaries, which can make nested-field access a little awkward — info[player]['stats']['kills'] — you could instead use a little more advanced "generic" class to hold them, which will let you write info2[player].stats.kills instead.
To illustrate, here's almost the same thing using a class I've named Struct because it's somewhat like the C language's struct data type:
class Struct(object):
""" Generic container object """
def __init__(self, **kwds): # keyword args define attribute names and values
self.__dict__.update(**kwds)
info2 = {}
with open('scoring_info.txt') as input_file:
for line in input_file:
player, stats, outcome, date = (
item.strip() for item in line.split('-', 3))
stats = dict(zip(('kills', 'deaths', 'assists'),
map(int, stats.split('/'))))
victory = (outcome.lower() == 'win') # change to boolean T/F
date = dict(zip(('year','month','day'), map(int, date.split('-'))))
info2[player] = Struct(champ_name=player, stats=Struct(**stats),
victory=victory, date=Struct(**date))
print('info2:')
for rec in info2.values():
print(' player %r:' % rec.champ_name)
print(' stats: kills=%s, deaths=%s, assists=%s' % (
rec.stats.kills, rec.stats.deaths, rec.stats.assists))
print(' victorious: %s' % rec.victory)
print(' date: %d-%02d-%02d' % (rec.date.year, rec.date.month, rec.date.day))
# sample usage
player = 'Fizz'
print('\n%s had %s kills in the game' % (player, info2[player].stats.kills))
Output:
info2:
player 'Shyvana':
stats: kills=12, deaths=4, assists=5
victorious: False
date: 2012-11-22
player 'Miss Fortune':
stats: kills=12, deaths=4, assists=3
victorious: True
date: 2012-11-22
player 'Fizz':
stats: kills=12, deaths=4, assists=5
victorious: True
date: 2012-11-22
Fizz had 12 kills in the game
You want to use split (' - ') to get the parts, then perhaps again to get the numbers:
for line in yourfile.readlines ():
data = line.split (' - ')
nums = [int (x) for x in data[1].split ('/')]
Should get you all the stuff you need in data[] and nums[]. Alternatively, you can use the re module and write a regular expression for it. This doesn't seem complex enough for that, though.
Extract data from text file using Python (or any language) - Stack Overflow
I need help using python to extract specific data from a text file and put it into an excel file
python - Extract information from sentence - Data Science Stack Exchange
extracting specific data from a text file in python - Stack Overflow
first of all you'll need to open the text file in read mode. I'd suggest using a context manager like so:
with open('path/to/your/file.txt', 'r') as file:
for line in file.readlines():
# do something with the line (it is a string)
as for managing the info you could build some intermediate structure, for example a dictionary or a list of dictionaries, and then translate that into a CSV file with the csv module.
you could for example split the file whenever there is a blank line, maybe like this:
with open('Downloads/test.txt', 'r') as f:
my_list = list() # this will be the final list
entry = dict() # this contains each user info as a dict
for line in f.readlines():
if line.strip() == "": # if line is empty start a new dict
my_list.append(entry) # and append the old one to the list
entry = dict()
else: # otherwise split the line and create new dict
line_items = line.split(r' ')
print(line_items)
entry[line_items[0]] = line_items[1]
print(my_list)
this code won't work because your text is not formatted in a consistent way: you need to find a way to make the split between "title" and "content" (like "first name" and "bob") in a consistent way. I suggest maybe looking at regex and fixing the txt file by making spacing more consistent.
assuming the data resides in a:
a="""
First Name Bob
Last name Smith
Phone 555-555-5555
Email [email protected]
Date of Birth 11/02/1986
Preferred Method of Contact Text Message
Desired Appointment Date 04/29
Desired Appointment Time 10am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
First Name john
Last name Smith
Phone 555-555-4444
Email [email protected]
Date of Birth 03/02/1955
Preferred Method of Contact Text Message
Desired Appointment Date 05/22
Desired Appointment Time 9am
City Pittsburgh
Location State
IP Address x.x.x.x
User-Agent (Browser/OS) Apple Safari 14.0.3 / OS X
Referrer http://www.example.com
"""
line_sep = "\n" # CHANGE ME ACCORDING TO DATA
fields = ["First Name", "Last name", "Phone",
"Email", "Date of Birth", "Preferred Method of Contact",
"Desired Appointment Date", "Desired Appointment Time",
"City", "Location", "IP Address", "User-Agent","Referrer"]
records = a.split(line_sep * 2)
all_records = []
for record in records:
splitted_record = record.split(line_sep)
one_record = {}
csv_record = []
for f in fields:
found = False
for one_field in splitted_record:
if one_field.startswith(f):
data = one_field[len(f):].strip()
one_record[f] = data
csv_record.append(data)
found = True
if not found:
csv_record.append("")
all_records.append(";".join(csv_record))
one_record will have the record as dictionary and csv_record will have it as a list of fields (ordered as fields variable)
I have been tasked with creating a spreadsheet that shows every a mainframe job accesses a specific application. The application is being retired, and each of the jobs that use it will have to be recoded. I get a daily log file that lists all of the activity in that application, but I need specific items from each of those text blocks exported into a spreadsheet.
Here is a sample block of text from the output:
RECOVERY SUMMARY FOR CDDZD : JOB01513 READER TIME: 0:28:26 READER DATE: 12/29/2020 SMFID: HMP1 0 PGM NAME: SORT DDNAME: SORTOF1 VOLSER: PRD004 DSORG: PS DATA SET: xxx.xxx.xxx.xxx *** TIME OF DAY: 0:30:26 CPU TIME SAVED: 0:00:09 ELAPSE TIME SAVED: 0:01:58 NUMBER OF EXTENTS: 2 *** TYPE OF ATTEMPT: SPACSECI *** SYSLOG MESSAGE: SVM4874I INCREASED SPACE FROM 10 CYLS TO 20
Each daily file has 500-1000 of these blocks.
What I need is a spreadsheet with columns for: PGM NAME, DDNAME, VOLSER, DATA SET, and NUMBER OF EXTENTS. The rest of the data is useless for this project.
The basics of the script I know; opening and reading the file, creating the xls doc, and closing them, etc... But I don't know how to get the info after each data point. I can get output that tells me how many times "PGM NAME" appears in the file, but not the actual program name its listing.
You can possibly use a combination of Named Entity Recognition and Syntactical Analysis - while the word Edwin is certainly propping up, imagine a situation where the name is Edward Philip Martel. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. John & Ramsey dined at Winterfell).
This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP can help upto a certain extent only).
One way of leveraging on syntax analysis / parsing and NER is in the following examples -
1. User: Edwin is my name.
2. User: I am Edwin.
3. User: My name is Edwin.
In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem. This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.
You should use Named Entity Recognition, for example from NLTK. You can find a usage example here. It would work for your described case quite well.
split should work just fine so long you ignore the header in your file
processes = []
with open("file.txt", "r") as f:
lines = f.readlines()
# Loop through all lines, ignoring header.
# Add last element to list (i.e. the process name)
for l in lines[1:]:
processes.append(l.split()[-1])
print processes
Result:
['dns.exe', 'lsass.exe', 'svchost.exe', 'lsass.exe', 'System']
You can use pandas DataFrames to do this without getting into the hassle of split:
parsed_file = pandas.read_csv("filename", header = 0)
will automatically read this into a DataFrame for you. You can then filter by those rows containing dns.exe, etc. You may need to define your own header
Here is a more general replacement for read_csv if you want more control. I've assumed your columns are all tab separated, but you can feel free to change the splitting character however you like:
with open('filename','r') as logs:
logs.readline() # skip header so you can can define your own.
columns = ["Proto","Local Address","Foreign Address","State","PID", "Process"]
formatted_logs = pd.DataFrame([dict(zip(columns,line.split('\t'))) for line in logs])
Then you can just filter the rows by
formatted_logs = formatted_logs[formatted_logs['Process'].isin(['dns.exe','lsass.exe', ...])]
If you want just the process names, it is even simpler. Just do
processes = formatted_logs['Process'] # returns a Series object than can be iterated through