GeoSpark入门-可视化
GeoSpark是一种用于大规模空间数据处理的集群计算。 GeoSpark通过一组out-of-the-box空间弹性分布式数据集( SRDDs ) 扩展 Apache Spark,它可以跨机器高效地加载。处理、分析、展示大规模空间数据。
准备工作
- Windows 和 spark
- IDEA
- GeoSpark支持Java、Scala两种,本次开发语言选择Java。
GeoSpark
参考https://github.com/jiayuasu/GeoSparkTemplateProject,下载项目到本地。
GeoSpark-Viz Java项目构建
cd ./geospark-viz/java
mvn clean install
由于项目中的数据生成图片不太满意,将map.shp数据解析成polygon.csv,修改下java代码
ConfFile= new FileInputStream(resourcePath+"babylon.polygon2.properties");
通过buildChoroplethMap统计面内得点数生成分级统计图,修改buildScatterPlot和
buildHeatMap输入数据为点数据生成散点图和热力图。
完整代码:
package example;
import com.vividsolutions.jts.geom.Envelope;
import com.vividsolutions.jts.geom.Polygon;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.serializer.KryoSerializer;
import org.apache.spark.storage.StorageLevel;
import org.datasyslab.geospark.enums.FileDataSplitter;
import org.datasyslab.geospark.enums.GridType;
import org.datasyslab.geospark.enums.IndexType;
import org.datasyslab.geospark.formatMapper.EarthdataHDFPointMapper;
import org.datasyslab.geospark.spatialOperator.JoinQuery;
import org.datasyslab.geospark.spatialRDD.PointRDD;
import org.datasyslab.geospark.spatialRDD.PolygonRDD;
import org.datasyslab.geospark.spatialRDD.RectangleRDD;
import org.datasyslab.geosparkviz.core.ImageGenerator;
import org.datasyslab.geosparkviz.core.ImageStitcher;
import org.datasyslab.geosparkviz.core.RasterOverlayOperator;
import org.datasyslab.geosparkviz.core.Serde.GeoSparkVizKryoRegistrator;
import org.datasyslab.geosparkviz.extension.visualizationEffect.ChoroplethMap;
import org.datasyslab.geosparkviz.extension.visualizationEffect.HeatMap;
import org.datasyslab.geosparkviz.extension.visualizationEffect.ScatterPlot;
import org.datasyslab.geosparkviz.utils.ColorizeOption;
import org.datasyslab.geosparkviz.utils.ImageType;
import java.awt.*;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties;
// TODO: Auto-generated Javadoc
/**
* The Class Example.
*/
public class Example2019 {
/** The spark context. */
static JavaSparkContext sparkContext;
/** The prop. */
static Properties prop;
/** The Point input location. */
static String PointInputLocation;
/** The Point offset. */
static Integer PointOffset;
/** The Point splitter. */
static FileDataSplitter PointSplitter;
/** The Point num partitions. */
static Integer PointNumPartitions;
/** The Rectangle input location. */
static String RectangleInputLocation;
/** The Rectangle offset. */
static Integer RectangleOffset;
/** The Rectangle splitter. */
static FileDataSplitter RectangleSplitter;
/** The Rectangle num partitions. */
static Integer RectangleNumPartitions;
/** The Polygon input location. */
static String PolygonInputLocation;
/** The Polygon offset. */
static Integer PolygonOffset;
/** The Polygon splitter. */
static FileDataSplitter PolygonSplitter;
/** The Polygon num partitions. */
static Integer PolygonNumPartitions;
/** The Line string input location. */
static String LineStringInputLocation;
/** The Line string offset. */
static Integer LineStringOffset;
/** The Line string splitter. */
static FileDataSplitter LineStringSplitter;
/** The Line string num partitions. */
static Integer LineStringNumPartitions;
/** The US main land boundary. */
static Envelope USMainLandBoundary;
/** The earthdata input location. */
static String earthdataInputLocation;
/** The earthdata num partitions. */
static Integer earthdataNumPartitions;
/** The HDF increment. */
static int HDFIncrement = 5;
/** The HDF offset. */
static int HDFOffset = 2;
/** The HDF root group name. */
static String HDFRootGroupName = "MOD_Swath_LST";
/** The HDF data variable name. */
static String HDFDataVariableName = "LST";
/** The HDF data variable list. */
static String[] HDFDataVariableList = {"LST","QC","Error_LST","Emis_31","Emis_32"};
/** The HD fswitch XY. */
static boolean HDFswitchXY = true;
/** The url prefix. */
static String urlPrefix = "";
/**
* Builds the scatter plot.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildScatterPlot(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
//PolygonRDD spatialRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions, StorageLevel.MEMORY_ONLY());
ScatterPlot visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG);
// visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false,-1,-1,false,true);
// visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
// visualizationOperator.Visualize(sparkContext, spatialRDD);
// imageGenerator = new ImageGenerator();
// imageGenerator.SaveVectorImageAsLocalFile(visualizationOperator.vectorImage, outputPath,ImageType.SVG);
//
// visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false,-1,-1,true,true);
// visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
// visualizationOperator.Visualize(sparkContext, spatialRDD);
// imageGenerator = new ImageGenerator();
// imageGenerator.SaveVectorImageAsLocalFile(visualizationOperator.distributedVectorImage, outputPath+"-distributed",ImageType.SVG);
//
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
}
/**
* Builds the heat map.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildHeatMap(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,5);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath,ImageType.PNG);
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
}
/**
* Builds the choropleth map.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildChoroplethMap(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
PolygonRDD queryRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions, StorageLevel.MEMORY_ONLY());
spatialRDD.spatialPartitioning(GridType.RTREE);
queryRDD.spatialPartitioning(spatialRDD.grids);
spatialRDD.buildIndex(IndexType.RTREE,true);
JavaPairRDD<Polygon,Long> joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD,queryRDD,true,false);
long start = System.currentTimeMillis();
ChoroplethMap visualizationOperator = new ChoroplethMap(1000,600,USMainLandBoundary,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.RED, true);
visualizationOperator.Visualize(sparkContext, joinResult);
ScatterPlot frontImage = new ScatterPlot(1000,600,USMainLandBoundary,false);
frontImage.CustomizeColor(0, 0, 0, 255, Color.GREEN, true);
frontImage.Visualize(sparkContext, queryRDD);
RasterOverlayOperator overlayOperator = new RasterOverlayOperator(visualizationOperator.rasterImage);
overlayOperator.JoinImage(frontImage.rasterImage);
ImageGenerator imageGenerator = new ImageGenerator();
//imageGenerator.SaveRasterImageAsLocalFile(frontImage.rasterImage, outputPath,ImageType.PNG);
imageGenerator.SaveRasterImageAsLocalFile(overlayOperator.backRasterImage, outputPath,ImageType.PNG);
//imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
//ImageStitcher.stitchImagePartitionsFromLocalFile(outputPath, 1000,600,0,4, 4);
System.out.println("散点图生成完成,共耗时" + (System.currentTimeMillis() - start) + "ms");
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
}
/**
* Parallel filter render no stitch.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean parallelFilterRenderNoStitch(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,2,4,4,true,true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
}
/**
* Parallel filter render stitch.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean parallelFilterRenderStitch(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,2,4,4,true,true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
ImageStitcher.stitchImagePartitionsFromLocalFile(outputPath, 1000,600,0,4, 4);
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
}
/**
* Earthdata visualization.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean earthdataVisualization(String outputPath)
{
try {
EarthdataHDFPointMapper earthdataHDFPoint = new EarthdataHDFPointMapper(HDFIncrement,HDFOffset,HDFRootGroupName,
HDFDataVariableList,HDFDataVariableName,HDFswitchXY,urlPrefix);
PointRDD spatialRDD = new PointRDD(sparkContext, earthdataInputLocation, earthdataNumPartitions, earthdataHDFPoint,StorageLevel.MEMORY_ONLY());
ScatterPlot visualizationOperator = new ScatterPlot(1000,600,spatialRDD.boundaryEnvelope,ColorizeOption.EARTHOBSERVATION,false,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.BLUE, true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG);
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
}
/**
* The main method.
*
* @param args the arguments
* @throws IOException Signals that an I/O exception has occurred.
*/
public static void main(String[] args) throws IOException {
long start = System.currentTimeMillis();
Logger.getLogger("org").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.WARN);
SparkConf sparkConf = new SparkConf().setAppName("GeoSparkVizDemo").setMaster("local[*]").set("spark.serializer", KryoSerializer.class.getName())
.set("spark.kryo.registrator", GeoSparkVizKryoRegistrator.class.getName());
sparkContext = new JavaSparkContext(sparkConf);
prop = new Properties();
String resourcePath = "src/test/resources/";
String demoOutputPath = "target/demo";
FileInputStream ConfFile= new FileInputStream(resourcePath+"babylon.point.properties");
prop.load(ConfFile);
String scatterPlotOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath + "/scatterplot";
String heatMapOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/heatmap";
String choroplethMapOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/choroplethmap";
String parallelFilterRenderStitchOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/parallelfilterrenderstitchheatmap";
String earthdataScatterPlotOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/earthdatascatterplot";
PointInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
PointOffset = Integer.parseInt(prop.getProperty("offset"));;
PointSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
PointNumPartitions = Integer.parseInt(prop.getProperty("numPartitions"));
ConfFile= new FileInputStream(resourcePath+"babylon.rectangle.properties");
prop.load(ConfFile);
RectangleInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
RectangleOffset = Integer.parseInt(prop.getProperty("offset"));
RectangleSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
RectangleNumPartitions = Integer.parseInt(prop.getProperty("numPartitions"));
ConfFile= new FileInputStream(resourcePath+"babylon.polygon2.properties");
prop.load(ConfFile);
PolygonInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
PolygonOffset = Integer.parseInt(prop.getProperty("offset"));
PolygonSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
PolygonNumPartitions = Integer.parseInt(prop.getProperty("numPartitions"));
ConfFile= new FileInputStream(resourcePath+"babylon.linestring.properties");
prop.load(ConfFile);
LineStringInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
LineStringOffset = Integer.parseInt(prop.getProperty("offset"));
LineStringSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
LineStringNumPartitions = Integer.parseInt(prop.getProperty("numPartitions"));
USMainLandBoundary = new Envelope(-126.790180,-64.630926,24.863836,50.000);
earthdataInputLocation = System.getProperty("user.dir")+"/src/test/resources/modis/modis.csv";
earthdataNumPartitions = 5;
HDFIncrement=5;
HDFOffset=2;
HDFRootGroupName = "MOD_Swath_LST";
HDFDataVariableName = "LST";
HDFswitchXY = true;
urlPrefix = System.getProperty("user.dir")+"/src/test/resources/modis/";
if(buildScatterPlot(scatterPlotOutputPath)&&buildHeatMap(heatMapOutputPath)
&&buildChoroplethMap(choroplethMapOutputPath)&¶llelFilterRenderStitch(parallelFilterRenderStitchOutputPath+"-stitched")
&¶llelFilterRenderNoStitch(parallelFilterRenderStitchOutputPath)&&earthdataVisualization(earthdataScatterPlotOutputPath))
{
System.out.println("散点图生成完成,共耗时" + (System.currentTimeMillis() - start) + "ms");
System.out.println("All GeoSparkViz Demos have passed.");
}
else
{
System.out.println("GeoSparkViz Demos failed.");
}
sparkContext.stop();
}
}
项目可视化一览
参考资料:
https://www.jianshu.com/p/1a531de087df
https://www.helplib.com/GitHub/article_127813
GeoSpark入门-可视化的更多相关文章
- (一)Superset 1.3图表篇——Table
本系列文章基于Superset 1.3.0版本.1.3.0版本目前支持分布,趋势,地理等等类型共59张图表.本次1.3版本的更新图表有了一些新的变化,而之前也一直没有做过非常细致的图表教程. 而且目前 ...
- 一起学微软Power BI系列-官方文档-入门指南(4)Power BI的可视化
在前面的系列文章中,我们介绍了官方有关获取数据,以及建模的原始文档和基本介绍.今天继续给大家介绍官方文档中,有关可视化的内容.实际上获获取数据和建模更注重业务关系的处理,而可视化则关注对数据的解读.这 ...
- d3.js:数据可视化利器之快速入门
hello,data! 在进入d3.js之前,我们先用一个小例子回顾一下将数据可视化的基本流程. 任务 用横向柱状图来直观显示以下数据: var data = [10,15,23,78,57,29,3 ...
- Roslyn 入门:使用 Visual Studio 的语法可视化窗格查看和了解代码的语法树
使用 Visual Studio 提供的 Syntax Visualizer,我们可以实时看到一个代码文件中的语法树.这对我们基于 Roslyn 编写静态分析和修改工具非常有帮助.本文将介绍如何安装它 ...
- 数据可视化入门之show me the numbers
数据的可视化一直是自己瞎玩着学,近来想系统的学数据可视化的东西,于是搜索资料时看到有人推荐<show me the numbers>作为入门. 由于搜不到具体的书籍内容,只能 ...
- 数据可视化-svg入门基础(二)
接上一篇:数据可视化-svg入门基础(一),基础一主要是介绍了svg概念,元素样式设置等. svg是(scalable vector graphic)伸缩矢量图像. 一.目录 (1)图形元素 (2)文 ...
- Seaborn数据可视化入门
在本节学习中,我们使用Seaborn作为数据可视化的入门工具 Seaborn的官方网址如下:http://seaborn.pydata.org 一:definition Seaborn is a Py ...
- Matplotlib数据可视化(1):入门介绍
1 matplot入门指南¶ matplotlib是Python科学计算中使用最多的一个可视化库,功能丰富,提供了非常多的可视化方案,基本能够满足各种场景下的数据可视化需求.但功能丰富从另一方面来 ...
- Keras入门(六)模型训练实时可视化
在北京做某个项目的时候,客户要求能够对数据进行训练.预测,同时能导出模型,还有在页面上显示训练的进度.前面的几个要求都不难实现,但在页面上显示训练进度当时笔者并没有实现. 本文将会分享如何在K ...
随机推荐
- 【2018寒假集训 Day2】【动态规划】钱币兑换(exchange)(自己翻译的2333)
钱币兑换(exchange) 问题描述: Dave偶然获得了未来几天的美元(dollars)与马克(marks)之间的兑换率.例如Dave开始有100marks,请编写个程序帮助Dave找出最好的买卖 ...
- 个人收藏--未整理—C# http/https 上传下载文件
c# HTTP/HTTPS 文件上传. 分类: .net 2015-02-03 08:36 541人阅读 评论(0) 收藏 举报 方法主体 [csharp] view plaincopy public ...
- centos 7 MysSQL 5.6.39 源码安装
MySQL 5.6.39 二进制安装 CentOS 7 将默认数据库MySQL替换成了Mariadb. 这里会从系统的环境准备开始一步一步安装. 环境准备 系统版本 内核版本 IP地址 Centos ...
- Windows的定时任务(Schedule Task)设置
一.设置 1 点击“开始” 2 点击“控制面板” 3 双击“任务计划” 4 双击“添加任务计划” 5 到了“任务计划向导”界面,点击“下一步” 6 点击“浏览”选择需要定时运行的程序(exe文件,ba ...
- PTA 1139 1138 1137 1136
PAT 1139 1138 1137 1136 一个月不写题,有点生疏..脑子跟不上手速,还可以啦,反正今天很开心. PAT 1139 First Contact 18/30 找个时间再修bug 23 ...
- 机器学习笔记(六) ---- 支持向量机(SVM)
支持向量机(SVM)可以说是一个完全由数学理论和公式进行应用的一种机器学习算法,在小批量数据分类上准确度高.性能好,在二分类问题上有广泛的应用. 同样是二分类算法,支持向量机和逻辑回归有很多相似性,都 ...
- 深入比特币原理(四)——锁定脚本(locking script)与解锁脚本(unlocking script)
通常比特币都是以虚拟货币的概念出现在大众眼前,实际上比特币是第一个真正的区块链"平台",利用它去中心化.不可篡改.可追溯等特点不光可以实现多种交易模式(如点对点交易.多重签名交易等 ...
- shell 循环读取文件及字符串转为数组
文件/etc/hdocker_config内容如下: 30.72.63.94 30.72.63.95 30.72.63.96 30.72.63.97 /tmp/lasclocker.tar maste ...
- php上传下载文件
之前做一个上传下载的项目,发现网上的和自己需求不是很一样,翻阅了下书籍和整理了下网上的一些代码.做了一个上传下载的demo,上传通过php本身的uploadfile函数,并返回以时间戳命名的文件名后, ...
- luogu P2507 [SCOI2008]配对
题目描述 你有 n 个整数Ai和n 个整数Bi.你需要把它们配对,即每个Ai恰好对应一个Bp[i].要求所有配对的整数差的绝对值之和尽量小,但不允许两个相同的数配对.例如A={5,6,8},B={5, ...