[Machine Learning with Python] How to get your data?
Using Pandas Library
The simplest way is to read data from .csv
files and store it as a data frame object:
import pandas as pd
df = pd.read_csv('olympics.csv', index_col=0, skiprows=1)
You can also read .xsl
files and directly select the rows and columns you are interested in by setting parameters skiprows
, usecols
. Also, you can indicate index column by parameter index_col
.
energy=pd.read_excel('Energy Indicators.xls', sheet_name='Energy',skiprows=8,usecols='E,G', index_col=None, na_values=['NA'])
For .txt
files, you can also use read_csv
function by defining the separation symbol:
university_towns=pd.read_csv('university_towns.txt',sep='\n',header=None)
See more about pandas io operations in http://pandas.pydata.org/pandas-docs/stable/io.html
Using os Module
Read .csv
files:
import os
import csv
for file in os.listdir("objective_folder"):
with open('objective_folder/'+file, newline='') as csvfile:
rows = csv.reader(csvfile) # read csc file
for row in rows: # print each line in the file
print(row)
Read .xsl
files:
import os
import xlrd
for file in os.listdir("objective_folder/"):
data = xlrd.open_workbook('objective_folder/'+file)
table = sheel_1 = data.sheet_by_index(0)#the first sheet in Excel
nrows = table.nrows #row number
for i in range(nrows):
if i == 0: # skip the first row if it defines variable names
continue
row_values = table.row_values(i) #read each row value
print(row_values)
Download from Website Automatically
We can also try to read data directly from url link. This time, the .csv
file is compressed as housing.tgz
. We need to download the file and then decompress it. So you can write a small function as below to realize it. It is a worthy effort because you can get the most recent data every time you run the function.
import os
import tarfile
from six.moves import urllib
DOWNLOAD_ROOT = "https://raw.githubusercontent.com/ageron/handson-ml/master/"
HOUSING_PATH = "datasets/housing"
HOUSING_URL = DOWNLOAD_ROOT + HOUSING_PATH + "/housing.tgz"
def fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH):
if not os.path.isdir(housing_path):
os.makedirs(housing_path)
tgz_path = os.path.join(housing_path, "housing.tgz")
urllib.request.urlretrieve(housing_url, tgz_path)
housing_tgz = tarfile.open(tgz_path)
housing_tgz.extractall(path=housing_path)
housing_tgz.close()
when you call fetch_housing_data()
, it creates a datasets/housing
directory in your workspace, downloads the housing.tgz
file, and extracts the housing.csv
from it in this directory.
Now let’s load the data using Pandas. Once again you should write a small function to load the data:
import pandas as pd
def load_housing_data(housing_path=HOUSING_PATH):
csv_path = os.path.join(housing_path, "housing.csv")
return pd.read_csv(csv_path)
What’s more?
These methods are what I have met so far. In typical environments your data would be available in a relational database (or some other common datastore) and spread across multiple tables/documents/files. To access it, you would first need to get your credentials and access authorizations, and familiarize yourself with the data schema. I will supplement more methods if I encounter in the future.
[Machine Learning with Python] How to get your data?的更多相关文章
- 【Machine Learning】Python开发工具:Anaconda+Sublime
Python开发工具:Anaconda+Sublime 作者:白宁超 2016年12月23日21:24:51 摘要:随着机器学习和深度学习的热潮,各种图书层出不穷.然而多数是基础理论知识介绍,缺乏实现 ...
- Python (1) - 7 Steps to Mastering Machine Learning With Python
Step 1: Basic Python Skills install Anacondaincluding numpy, scikit-learn, and matplotlib Step 2: Fo ...
- Getting started with machine learning in Python
Getting started with machine learning in Python Machine learning is a field that uses algorithms to ...
- 《Learning scikit-learn Machine Learning in Python》chapter1
前言 由于实验原因,准备入坑 python 机器学习,而 python 机器学习常用的包就是 scikit-learn ,准备先了解一下这个工具.在这里搜了有 scikit-learn 关键字的书,找 ...
- Machine Learning的Python环境设置
Machine Learning目前经常使用的语言有Python.R和MATLAB.如果采用Python,需要安装大量的数学相关和Machine Learning的包.一般安装Anaconda,可以把 ...
- [Machine Learning with Python] Familiar with Your Data
Here I list some useful functions in Python to get familiar with your data. As an example, we load a ...
- [Machine Learning with Python] My First Data Preprocessing Pipeline with Titanic Dataset
The Dataset was acquired from https://www.kaggle.com/c/titanic For data preprocessing, I firstly def ...
- [Machine Learning with Python] Data Preparation through Transformation Pipeline
In the former article "Data Preparation by Pandas and Scikit-Learn", we discussed about a ...
- [Machine Learning with Python] Data Preparation by Pandas and Scikit-Learn
In this article, we dicuss some main steps in data preparation. Drop Labels Firstly, we drop labels ...
随机推荐
- idea 关于高亮显示与选中字符串相同的内容踩过的坑
由 Eclipse 切换到 idea 遇到了很多不熟的地方和踩过的坑,下面记录 idea 关于高亮显示与选中字符串相同的内容踩过的坑. IDEA 2017.2.1 版本修改 Identifier un ...
- 【nginx】nginx.sh nginx 安装脚本
#! /bin/shcd /usr/local/srcwget http://nginx.org/download/nginx-1.10.1.tar.gzecho 'download success' ...
- Python基础(二)——常用内置函数
1. 常用内置函数 (1)isinstance(object, classinfo) 用于判断一个对象是否为某一类型. object 是实例对象 classinfo 是基本类型如 int, floa ...
- matplotlib学习记录 三
# 绘制自己和朋友在各个年龄的女友数量的折线图 from matplotlib import pyplot as plt # 让matplotlib能够显示中文 plt.rcParams['font. ...
- Python头脑风暴4
IT是全国平均薪资最高的行业,2017年全国最高,人均13点4万每年. 但技术固然好,创业拼的还是世界观下的创意. 蘑菇街,并夕夕,TikTok,头条,哪个不是创意用IT技术的现实化?? 未来,大平台 ...
- leetcode-4-basic
解题思路:这道题比较简单,代码不贴了.需要注意的是: 数字与字符串之间的转换, char str[100]; sprintf(str, "%d", num); 解题思路: 这道题是 ...
- 数学基础:HUD1406-完数
完数 Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 65536/32768 K (Java/Others) Problem Descript ...
- 二叉排序树:POJ2418-Hardwood Species(外加字符串处理)
Hardwood Species Time Limit: 10000MS Memory Limit: 65536K Description Hardwoods are the botanical gr ...
- UVa 10655 Contemplation! Algebra 矩阵快速幂
题意: 给出\(p=a+b\)和\(q=ab\),求\(a^n+b^n\). 分析: 这种题目关键还是在于构造矩阵: \(\begin{bmatrix} 0 & 1 \\ -(a+b) &am ...
- SQL中使用关键词创建表或字段
有时候我们给表或者字段命名时,会无意中选择了一个SQL中的关键字进行命名,然后就报错了: ERROR: syntax error at or near "limit" MySQL ...