【443】Tweets Analysis Q&A

 

 

【Question 01】

　　When converting Tweets info to csv file, commas in the middle of data (i.e. location: Sydney, NSW) can make a mistake of the csv file (creaing more columns).

　　The solution is to add double quotation marks on both sides of the content, like this:

fo.write("\"" + str(tweet["user"]["location"]) + "\"")

【Question 02】

　　When open csv file with Excel, sometimes it will show messy code, but it can show well with Notepad.

　　ref：csv 文件打开乱码，有哪些方法可以解决？

　　One solution is opening this file with notepad++.

　　Another solution is adding codes at the beginning of the writing file, like this:

fo = open(r"D:\Twitter Data\Data\test\tweets.csv", "w")
fo.write("\ufeff")

【Question 03】

　　Text contents contain carriage return, double quotation marks, single quotation marks. Those info will make mistakes when creating csv file.

　　So we should replace those characters with space or nothing, like this:

text = str(tweet["text"])
text = text.replace("\n", " ")
text = text.replace("\"", "")
text = text.replace("\'", "")
fo.write("\"" + text + "\"")

　　Including tweet["user"]["location"] and tweet["text"], for these two attributes, user can write whatever they want, so it's easy to make mistakes.

【Question 04】

　　After converting Tweets to csv file, but I can't open this file by pandas.read_csv(). The reason is there must be some problems in those data. Since there are about more than 100000+ rows of this csv file, how can I locate the error line?

　　Solution is coverting the first 10000 rows, if there are not errors, and then converting the next 10000 rows. If error occurs, trying to narrow the range of numbers, like error occurs between 20000 to 30000, we can change the range of numbers with 20000 to 25000. Using this method several times, we can locate the error line and find the real problems. For this spicific case, most problems are about contents include carriage return, double quotation marks, etc.

　　Codes like this:

...

count = 0
or line in tweets_file:
    try:
        count += 1
        if (count < 10000):
            continue
        ...

        if (count > 20000):
            break
    except:
        continue
...