There are some oddities with the code above, notably that "w" option will overwrite the csv file. from my test this would actually append to an already existing file.
import csv
with open(r'loop.csv','a') as f1: # need "a" and not w to append to a file, if not will overwrite
writer=csv.writer(f1, delimiter='\t',lineterminator='\n',)
# two options here, either:
for i in range(0,10):
row = [i]
writer.writerow(row)
#OR
writer.writerows([i for i in range(10)]) #note that range(0,10) and range(10) are the same thing
Answer from Flying Turtle on Stack OverflowThere are some oddities with the code above, notably that "w" option will overwrite the csv file. from my test this would actually append to an already existing file.
import csv
with open(r'loop.csv','a') as f1: # need "a" and not w to append to a file, if not will overwrite
writer=csv.writer(f1, delimiter='\t',lineterminator='\n',)
# two options here, either:
for i in range(0,10):
row = [i]
writer.writerow(row)
#OR
writer.writerows([i for i in range(10)]) #note that range(0,10) and range(10) are the same thing
import csv
with open('loop.csv','w') as f1:
writer=csv.writer(f1, delimiter='\t',lineterminator='\n',)
for i in range(0,10):
row = [i]
writer.writerow(row)
python - Append data to CSV using a nested loop - Stack Overflow
Appending output of a for loop for Python to a csv file - Stack Overflow
python 3.x - append entries in csv in a for loop by using dataframe.to_csv - Stack Overflow
python 3.x - Appending data to Pandas DataFrame with for loop - Stack Overflow
There are two each_dict objects in json_response. They have 5 and 13 tweets, respectively (each_dict['data']).
In addition, there are 5 and 13 elements in each_dict['includes']['users'], respectively.
You got 194 elements because in the first iteration of for each_dict in json_response: you save data 5x5=25 times (loop 2 is executed 5 times for every tweet in loop 1). While in the second iteration you save data 13x13=169 times (loop 2 is executed 13 times for every tweet in loop 1).
You should append data to your csv outside loop 2. That is,
for each_dict in json_response:
for tweet in each_dict['data']:
# ...
for dic in each_dict['includes']['users']:
# ...
res = [author_id, created_at, tweet_id, text, bio, image_url]
csvWriter.writerow(res)
In addition, I recommend using a pandas dataframe to store the info you need and save to csv. It makes the code more readable and you do not have to worry about opening a buffer. See my recommendation below, including renaming:
import pandas as pd
df = pd.DataFrame()
for each_dict in json_response:
for tweet in each_dict['data']:
row = {}
row["author_id"] = tweet['author_id']
row["created_at"] = dateutil.parser.parse(tweet['created_at'])
row["tweet_id"] = tweet['id']
row["text"] = tweet['text']
for user in each_dict['includes']['users']:
if user["id"] == row["author_id"]:
row["bio"] = user['description']#.encode('utf-16','surrogatepass').decode('utf-16') # uncomment this if you get UnicodeError
for media in each_dict['includes']['media']:
row['image_url'] = media.get('url', ' ')
df = df.append(row, ignore_index=True)
# Note, since the dataframe is initially empty with no columns, appending a dictionary (i.e, row) will automatically generate the header based on the dictionary's keys.
df.to_csv('path/to/file.csv')
Output
tweet_id author_id created_at ...
0 1375057688355336195 2877379617 2021-03-25T12:11:14.000Z ...
1 1374085719472361474 1265018154444562440 2021-03-22T19:48:59.000Z ...
...
17 1360693490880032770 926909484 2021-02-13T20:53:03.000Z ...
Looks like the else branch of if 'description' in dic: is never executed. If your code is indented correctly, then also the csvWriter.writerow part is never executed because of this.
That yields that no contents are written to your file.
A comment on code style:
- use
with open(file) as file_variable:instead of manually using open and close. That can save you some trouble, e.g. the trouble you would get when the else branch would indeed be executed and the file would be closed multiple times :)
Hello. This is a follow up question regarding the project I originally made involving this post: https://www.reddit.com/r/learnpython/comments/61hb2k/looking_to_scrape_historical_weather_data_new_to/
My current issue right now is that I can't store more than a couple of years data in a single list in Python, because it runs out of memory. When I try to append continuously onto one dataframe (instead of a list), I receive this warning and my final CSV file is completely blank:
'<' not supported between instances of 'str' and 'int', sort order is undefined for incomparable objects, result = result.union(other)
Another idea I had was to create one CSV file at the beginning through pandas, and then append that CSV file as my code keeps looping, but I'm unsure of how to do that. Any suggestions would be greatly appreciated!
This is the link to my code: https://pastebin.com/A31JNWp9