Using Apache Spark and MySQL for Data Analysis

What is Spark

Apache Spark is a cluster computing framework, similar to Apache Hadoop. Wikipedia has a great description of it:

Apache Spark is an open source cluster computing framework originally developed in the AMPLab at University of California, Berkeley but was later donated to the Apache Software Foundation where it remains today. In contrast to Hadoop’s two-stage disk-based MapReduce paradigm, Spark’s multi-stage in-memory primitives provides performance up to 100 times faster for certain applications. By allowing user programs to load data into a cluster’s memory and query it repeatedly, Spark is well-suited to machine learning algorithms.

In contrast to popular belief, Spark does not require all data to fit into memory but will use caching to speed up the operations (just like MySQL). Spark can also run in standalone mode and does not require Hadoop; it can also be run on a single server (or even laptop or desktop) and utilize all your CPU cores.

Starting it in a distributed mode is really easy. Start the “master” first. You can run the “slave” on the same node:

Shell

root@thor:~/spark# ./sbin/start-master.sh

less ../logs/spark-root-org.apache.spark.deploy.master.Master-1-thor.out

15/08/25 11:21:21 INFO Master: Starting Spark master at spark://thor:7077

15/08/25 11:21:21 INFO Master: Running Spark version 1.4.1

15/08/25 11:21:21 INFO Utils: Successfully started service 'MasterUI' on port 8080.

15/08/25 11:21:21 INFO MasterWebUI: Started MasterWebUI at http://10.60.23.188:8080

root@thor:~/spark# ./sbin/start-slave.sh spark://thor:7077

Then run Spark Worker on any additional nodes (make sure to add the hostname to /etc/hosts or use DNS):

Shell

1	root@d31:~/spark# ./sbin/start-slave.sh spark://thor:7077

Why Spark and Not MySQL?

There are a number of tasks where MySQL (out-of-the-box) does not show great performance. One of the MySQL limitations is: 1 query = 1 cpu core. It means that even if you have 48 fast cores and a large dataset to process (i.e. group by, sort, etc) it will not utilize the full computing power. Spark, on the contrary, will be able to utilize all your CPU cores.

Another difference between MySQL and Spark:

MySQL uses so called “schema on write” – it will need the data to be converted into MySQL. If our data is not inside MySQL you can’t use “sql” to query it.
Spark (and Hadoop/Hive as well) uses “schema on read” – it can apply a table structure on top of a compressed text file, for example, (or any other supported input format) and see it as a table; then we can use SQL to query this “table.”

In other words, MySQL is storage+processing while Spark’s job is processing only, and it can pipe data directly from/toexternal datasets, i.e., Hadoop, Amazon S3, local files, JDBC (MySQL/other databases). Spark supports text files (compressed), SequenceFiles, and any other Hadoop InputFormat as well as Parquet Columnar storage. Spark is more flexible in this regard compared to Hadoop: Spark can read data directly from MySQL, for example.

The typical pipeline to load external data to MySQL is:

Uncompress (typically the external data is in compressed text files)
Load it into MySQL’s staging table with “LOAD DATA INFILE”
Only then we can filter/group by and save the result in another table

That can cause additional overhead. In many cases we do not need the “raw” data but we still have to load it into MySQL.

Why Spark Together With MySQL

On the contrary, the result of our analysis (i.e. aggregated data) should be in MySQL. It does not have to be, but it is much more convenient to store the result of your analysis in MySQL. Let’s say you want to analyze a big dataset (i.e. year to year sales comparison) and you will need to present it in the form of a table or graph. The result set will be significantly smaller as it will be aggregated and it will be much easier to store it in MySQL as many standard applications will work with that.

Real World Test Case

One of interesting free datasets is Wikipedia Page Counts. (>1TB compressed, available since 2008). This data can be downloaded (as gzipped space delimited text files) and is also available (limited dataset) on AWS. The data is aggregated by the hour and has the following fields:

project (i.e. “en”, “fr”, etc, which is usually a language)
title of the page (uri), urlencoded
number of requests
size of the content returned

(the date field is encoded inside the filename, 1 file per hour)

Our goal will be to find the top 10 pages by the number of requests per day in English Wikipedia, but also to support searching for an arbitrary word so we can show how, for example, the number of requests for the wikipedia article about “myspace” will compare to the article about “facebook” (2008 to 2015).

To do that in MySQL we will have to load it as is into MySQL. The files are distributed with the date part encoded. The uncompressed size of all files is > 10TB. Here are the possible steps (as per our typical MySQL pipeline):

Uncompress the file and run “LOAD DATA INFILE” into a staging (temporary) table:

MySQL

load data local infile '$file'

into table wikistats.wikistats_full CHARACTER SET latin1

FIELDS TERMINATED BY ' '

(project_name, title, num_requests, content_size)

set request_date = STR_TO_DATE('$datestr', '%Y%m%d %H%i%S');

Aggregate with “insert into” a final table

MySQL

1

2

3

4

insert into wikistats.wikistats_by_day

select date(request_date) as request_day, title, count(*), sum(num_requests)

from wikistats.wikistats_full

group by request_day, title;
Somehow url decode the title (may be using UDF).

This is a big overhead. We will uncompress and transform the data into MySQL just to discard most if it.

According to my calculations it should take > 1 month to do the whole pipeline for 6 years of data (this time does not include the uncompress time and does not include the load time depreciation as the table get bigger and bigger and indexes need to be updated). There are a lots of things we can do here to speed it up of course, i.e., load into different MySQL instances, load into MEMORY table first, then group by into InnoDB, etc.

But one of the easiest ways here will be using Apache Spark and Python script (pyspark). Pyspark can read the original gziped text files, query those text files with SQL, apply any filters, functions, i.e. urldecode, group by day and save the resultset into MySQL.

Here is the Python script to perform those actions:

Python

from pyspark import SparkContext

sc=SparkContext()

# sc is an existing SparkContext.

from pyspark.sql import SQLContext, Row

sqlContext = SQLContext(sc)

import urllib

from datetime import timedelta, date

def load_day(filename, mydate):

# Load a text file and convert each line to a Row.

lines = sc.textFile(filename)

parts = lines.map(lambda l: l.split(" ")).filter(lambda line: line[0]=="en").filter(lambda line: len(line)>3).cache()

wiki = parts.map(lambda p: Row(project=p[0], url=urllib.unquote(p[1]).lower(), num_requests=int(p[2]), content_size=int(p[3])))

#wiki.count()

# Infer the schema, and register the DataFrame as a table.

schemaWiki = sqlContext.createDataFrame(wiki)

schemaWiki.registerTempTable("wikistats")

group_res = sqlContext.sql("SELECT '"+ mydate + "' as mydate, url, count(*) as cnt, sum(num_requests) as tot_visits FROM wikistats group by url")

# Save to MySQL

mysql_url="jdbc:mysql://thor?user=wikistats&password=wikistats"

group_res.write.jdbc(url=mysql_url, table="wikistats.wikistats_by_day_spark", mode="append")

# Write to parquet file - if needed

group_res.saveAsParquetFile("/ssd/wikistats_parquet_bydate/mydate=" + mydate)

mount = "/data/wikistats/"

d= date(2008, 1, 1)

end_date = date(2008, 2, 1)

delta = timedelta(days=1)

while d < end_date:

print d.strftime("%Y-%m-%d")

filename=mount + "wikistats//dumps.wikimedia.org/other/pagecounts-raw/2008/2008-01/pagecounts-200801" + d.strftime("%d") + "-*.gz"

print(filename)

load_day(filename, d.strftime("%Y-%m-%d"))

d += delta

In the script I used Spark to read the original gzip files (1 day at a time). We can use directory as “input” or a list of files. I will then use Resilient Data Set (RDD) transformations; python has lambda functions: map and filter which will allow us to split the “input files” and filter them.

The next step will be to apply the schema (declare fields); here we can also apply any other functions; i.e., I use urllib.unquote to decode the title (urldecode). Finally, we can register the temp table and then use familiar SQL to do the group by.

The script will normally utilize all cpu cores. In addition it is very easy to run it in distributed mode even without Hadoop: just copy the files to all machines in a Spark cluster or use NFS/external storage.

The script took about an hour on 3 boxes to process 1 month of data and load the aggregated data to MySQL (single instance). We can estimate that to load all 6 years (aggregated) to MySQL is ~3 days.

You may now ask, why is it significantly faster (and we still have the result loaded to the same MySQL instance)? The answer is, it is a different, more efficient pipeline. In our original MySQL pipeline (which will probably take months) we load the raw data to MySQL. Here we filter and group on read, and write only what we need to MySQL.

One question may also come up here: do we actually need this whole “pipeline?” Can we simply run our analytical queries on top of the “raw” data? Well, that is possible, but will probably require 1000 nodes Spark Cluster to do it efficiently as it will need to scan through 5TB of data (see “more reading” below).

Multi-treaded Performance for MySQL Inserts

When using group_res.write.jdbc(url=mysql_url, table=”wikistats.wikistats_by_day_spark”, mode=”append”) Spark will use multiple threads to insert into MySQL.

MySQL

+------+-----------+------------+-----------+---------+------+--------+--------------------------------------------------------------------------------------------------------+-----------+---------------+

25 rows in set (0.00 sec)

Monitoring your jobs

Spark provides you with a web interface to monitor and manage your job. Here is the example: I’m running the wikistats.py application:

Result: Using Parquet Columnar Format vs MySQL InnoDB table

Spark supports Apache Parquet Columnar format, so we can save RDD as a parquet file (it can be saved to a directory to HDFS):

Python

1	group_res.saveAsParquetFile("/ssd/wikistats_parquet_bydate/mydate=" + mydate)

Here we save the result of our pipeline (aggregated data) into Spark. I also utilize partitioning by day (“mydate=20080101”) and Spark can auto discover partitions in this format.

When I have my results, I want to query it. Let’s say I want to find the top 10 most frequently queried wiki pages in January 2018. I can do this query with MySQL (I will need to filter out main page and search pages):

MySQL

mysql> SELECT lower(url) as lurl, sum(tot_visits) as max_visits , count(*) FROM wikistats_by_day_spark where lower(url) not like '%special%' and lower(url) not like '%page%' and lower(url) not like '%test%' and lower(url) not like '%wiki%' group by lower(url) order by max_visits desc limit 10;

+--------------------------------------------------------+------------+----------+

| lurl | max_visits | count(*) |

+--------------------------------------------------------+------------+----------+

| heath_ledger | 4247338 | 131 |

| cloverfield | 3846404 | 131 |

| barack_obama | 2238406 | 153 |

| 1925_in_baseball#negro_league_baseball_final_standings | 1791341 | 11 |

| the_dark_knight_(film) | 1417186 | 64 |

| martin_luther_king,_jr. | 1394934 | 136 |

| deaths_in_2008 | 1372510 | 67 |

| united_states | 1357253 | 167 |

| scientology | 1349654 | 108 |

| portal:current_events | 1261538 | 125 |

+--------------------------------------------------------+------------+----------+

10 rows in set (1 hour 22 min 10.02 sec)

Please note, here we are using our already aggregated (summary by data) table, not the “raw” data.

As we can see, the query took 1 hour 22 mins. I have also saved the same results to Parquet (see the script), so now I can use it with Spark-SQL:

Shell

1	./bin/spark-sql --master local

This will use a local version of spark-sql, using 1 host only.

Shell

spark-sql> CREATE TEMPORARY TABLE wikistats_parquet

USING org.apache.spark.sql.parquet

OPTIONS (

path "/ssd/wikistats_parquet_bydate"

);

Time taken: 3.466 seconds

spark-sql> select count(*) from wikistats_parquet;

select count(*) from wikistats_parquet;

227846039

Time taken: 82.185 seconds, Fetched 1 row(s)

spark-sql> SELECT lower(url) as lurl, sum(tot_visits) as max_visits , count(*) FROM wikistats_parquet where lower(url) not like '%special%' and lower(url) not like '%page%' and lower(url) not like '%test%' and lower(url) not like '%wiki%' group by lower(url) order by max_visits desc limit 10;

heath_ledger 4247335 42

cloverfield 3846400 42

barack_obama 2238402 53

1925_in_baseball#negro_league_baseball_final_standings 1791341 11

the_dark_knight_(film) 1417183 36

martin_luther_king,_jr. 1394934 46

deaths_in_2008 1372510 38

united_states 1357251 55

scientology 1349650 44

portal:current_events 1261305 44

Time taken: 1239.014 seconds, Fetched 10 row(s)

That took ~20 minutes, which is much faster.

Conclusion

Apache Spark provides a great and easy way to analyze and aggregate data. What I love about Spark vs other big data and analytical frameworks:

Open-source and actively developed
No dependency on tools, i.e., the input data and output data does not have to be in Hadoop
Standalone mode for quick start, easy to deploy
Massively parallel, easy to add nodes
Support of variety of input and output format; i.e., it can read/write to MySQL (vs JDBC driver) and Parquet Columnar format

However, there are a number of drawbacks:

It is still new so you can expect some bugs and undocumented behavior. Many of the errors are hard to explain.
It requires Java; Spark 1.5 only supports Java 7 and higher. That also means it will require additional memory, which is reasonable nowadays.
You will need to run jobs through “spark-submit”

I believe Apache Spark is a great tool and can complement MySQL for data analytical and BI purposes.

来源： https://www.percona.com/blog/2015/10/07/using-apache-spark-mysql-data-analysis/

来自为知笔记(Wiz)

Using Apache Spark and MySQL for Data Analysis的更多相关文章

使用Apache Spark 对 mysql 调优查询速度提升10倍以上
在这篇文章中我们将讨论如何利用 Apache Spark 来提升 MySQL 的查询性能. 介绍在我的前一篇文章Apache Spark with MySQL 中介绍了如何利用 Apache Spa ...
Introducing DataFrames in Apache Spark for Large Scale Data Science（中英双语）
文章标题 Introducing DataFrames in Apache Spark for Large Scale Data Science 一个用于大规模数据科学的API——DataFrame ...
【译】Using .NET for Apache Spark to Analyze Log Data
.NET for Spark可用于处理成批数据.实时流.机器学习和ad-hoc查询.在这篇博客文章中,我们将探讨如何使用.NET for Spark执行一个非常流行的大数据任务,即日志分析. 1 什么 ...
使用 Apache Spark 让 MySQL 查询速度提升 10 倍以上
转: https://coyee.com/article/11012-how-apache-spark-makes-your-slow-mysql-queries-10x-faster-or-more ...
Apache Spark 2.2.0 中文文档
Apache Spark 2.2.0 中文文档 - 快速入门 | ApacheCN Geekhoo 关注 2017.09.20 13:55* 字数 2062 阅读 13评论 0喜欢 1 快速入门使用 ...
Apache Spark 2.2.0 中文文档 - Spark 编程指南 | ApacheCN
Spark 编程指南概述 Spark 依赖初始化 Spark 使用 Shell 弹性分布式数据集 (RDDs) 并行集合外部 Datasets(数据集) RDD 操作基础传递 Functio ...
Learning Spark: Lightning-Fast Big Data Analysis 中文翻译
Learning Spark: Lightning-Fast Big Data Analysis 中文翻译行为纯属个人对于Spark的兴趣,仅供学习. 如果我的翻译行为侵犯您的版权,请您告知,我将停止 ...
Why Apache Spark is a Crossover Hit for Data Scientists [FWD]
Spark is a compelling multi-purpose platform for use cases that span investigative, as well as opera ...
CS100.1x Introduction to Big Data with Apache Spark
CS100.1x简介这门课主要讲数据科学,也就是data science以及怎么用Apache Spark去分析大数据. Course Software Setup 这门课主要介绍如何编写和调试Py ...

随机推荐

[原创] PHP 使用Redis实现锁
目录锁实现的注意点加锁 connect 与 pconnect 解锁 Redis 中使用 Lua 脚本的注意点 Redis集群分布式锁 RedLock 算法锁实现的注意点互斥: 任意时刻, 只能 ...
git 远程库和url
我们使用 git remote add origin <url> 来关联远程主机,这个origin就是关联的远程主机名,如果我们想同时关联两个远程主机,我们可以用 git remote a ...
linux下实践导入导出MySQL数据库
一.导出: 用mysqldump命令行命令格式 mysqldump -u 用户名 -p 数据库名 > 数据库名.sql 范例: mysqldump -u root -p abc > ab ...
Compile android source and kernel for emulator in Debian
1.download the android source code Reference from http://source.android.com/source/downloading.html ...
CentOS 6.4下安装 Mono 3.2 和Jexus 5.4
1.安装Mono源码安装需要的库 yum -y install gcc gcc-c++ bison pkgconfig glib2-devel gettext make libpng-devel li ...
深度学习（二）BP求解过程和梯度下降
一.原理重点:明白偏导数含义,是该函数在该点的切线,就是变化率,一定要理解变化率. 1)什么是梯度梯度本意是一个向量(矢量),当某一函数在某点处沿着该方向的方向导数取得该点处的最大值,即函数在该点 ...
大数乘法的C代码实现
在C语言中,宽度最大的无符号整数类型是unsigned long long, 占8个字节.那么,如果整数超过8个字节,如何进行大数乘法呢? 例如: $ python Python 2.7.6 (def ...
java.lang.NoClassDefFoundError: org/springframework/ui/jasperreports/JasperReportsUtils原因
在springMVC结合jasperReporter中发现的问题 java.lang.NoClassDefFoundError: org/springframework/ui/jasperreport ...
08 Java 集合的线程安全问题
1 Java中的集合 Java中的集合分为同步的集合(线程安全)和线程不安全的集合例如 : ArrayList和Vector的区别: 一.同步性:Vector是线程安全的,也就是说是同步的,而Arr ...
ibatis内置类型
别名 java类型事务管理器类型 JDBC com.ibatis.sqlmap.engine.transaction.jdbc.JdbcTransactionConfig JTA com.iba ...

Using Apache Spark and MySQL for Data Analysis

Using Apache Spark and MySQL for Data Analysis的更多相关文章

随机推荐

热门专题