GeoSpark是一种用于大规模空间数据处理的集群计算。 GeoSpark通过一组out-of-the-box空间弹性分布式数据集( SRDDs ) 扩展 Apache Spark,它可以跨机器高效地加载。处理、分析、展示大规模空间数据。

准备工作

  1. Windows 和 spark
  2. IDEA
  3. GeoSpark支持Java、Scala两种,本次开发语言选择Java。

GeoSpark

参考https://github.com/jiayuasu/GeoSparkTemplateProject,下载项目到本地。

GeoSpark-Viz Java项目构建

cd ./geospark-viz/java
mvn clean install

由于项目中的数据生成图片不太满意,将map.shp数据解析成polygon.csv,修改下java代码

ConfFile= new FileInputStream(resourcePath+"babylon.polygon2.properties");



通过buildChoroplethMap统计面内得点数生成分级统计图,修改buildScatterPlot和

buildHeatMap输入数据为点数据生成散点图和热力图。





完整代码:

package example;

import com.vividsolutions.jts.geom.Envelope;
import com.vividsolutions.jts.geom.Polygon;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.serializer.KryoSerializer;
import org.apache.spark.storage.StorageLevel;
import org.datasyslab.geospark.enums.FileDataSplitter;
import org.datasyslab.geospark.enums.GridType;
import org.datasyslab.geospark.enums.IndexType;
import org.datasyslab.geospark.formatMapper.EarthdataHDFPointMapper;
import org.datasyslab.geospark.spatialOperator.JoinQuery;
import org.datasyslab.geospark.spatialRDD.PointRDD;
import org.datasyslab.geospark.spatialRDD.PolygonRDD;
import org.datasyslab.geospark.spatialRDD.RectangleRDD;
import org.datasyslab.geosparkviz.core.ImageGenerator;
import org.datasyslab.geosparkviz.core.ImageStitcher;
import org.datasyslab.geosparkviz.core.RasterOverlayOperator;
import org.datasyslab.geosparkviz.core.Serde.GeoSparkVizKryoRegistrator;
import org.datasyslab.geosparkviz.extension.visualizationEffect.ChoroplethMap;
import org.datasyslab.geosparkviz.extension.visualizationEffect.HeatMap;
import org.datasyslab.geosparkviz.extension.visualizationEffect.ScatterPlot;
import org.datasyslab.geosparkviz.utils.ColorizeOption;
import org.datasyslab.geosparkviz.utils.ImageType; import java.awt.*;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.Properties; // TODO: Auto-generated Javadoc /**
* The Class Example.
*/
public class Example2019 { /** The spark context. */
static JavaSparkContext sparkContext; /** The prop. */
static Properties prop; /** The Point input location. */
static String PointInputLocation; /** The Point offset. */
static Integer PointOffset; /** The Point splitter. */
static FileDataSplitter PointSplitter; /** The Point num partitions. */
static Integer PointNumPartitions; /** The Rectangle input location. */
static String RectangleInputLocation; /** The Rectangle offset. */
static Integer RectangleOffset; /** The Rectangle splitter. */
static FileDataSplitter RectangleSplitter; /** The Rectangle num partitions. */
static Integer RectangleNumPartitions; /** The Polygon input location. */
static String PolygonInputLocation; /** The Polygon offset. */
static Integer PolygonOffset; /** The Polygon splitter. */
static FileDataSplitter PolygonSplitter; /** The Polygon num partitions. */
static Integer PolygonNumPartitions; /** The Line string input location. */
static String LineStringInputLocation; /** The Line string offset. */
static Integer LineStringOffset; /** The Line string splitter. */
static FileDataSplitter LineStringSplitter; /** The Line string num partitions. */
static Integer LineStringNumPartitions; /** The US main land boundary. */
static Envelope USMainLandBoundary; /** The earthdata input location. */
static String earthdataInputLocation; /** The earthdata num partitions. */
static Integer earthdataNumPartitions; /** The HDF increment. */
static int HDFIncrement = 5; /** The HDF offset. */
static int HDFOffset = 2; /** The HDF root group name. */
static String HDFRootGroupName = "MOD_Swath_LST"; /** The HDF data variable name. */
static String HDFDataVariableName = "LST"; /** The HDF data variable list. */
static String[] HDFDataVariableList = {"LST","QC","Error_LST","Emis_31","Emis_32"}; /** The HD fswitch XY. */
static boolean HDFswitchXY = true; /** The url prefix. */
static String urlPrefix = ""; /**
* Builds the scatter plot.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildScatterPlot(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
//PolygonRDD spatialRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions, StorageLevel.MEMORY_ONLY());
ScatterPlot visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG); // visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false,-1,-1,false,true);
// visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
// visualizationOperator.Visualize(sparkContext, spatialRDD);
// imageGenerator = new ImageGenerator();
// imageGenerator.SaveVectorImageAsLocalFile(visualizationOperator.vectorImage, outputPath,ImageType.SVG);
//
// visualizationOperator = new ScatterPlot(1000,600,USMainLandBoundary,false,-1,-1,true,true);
// visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.GREEN, true);
// visualizationOperator.Visualize(sparkContext, spatialRDD);
// imageGenerator = new ImageGenerator();
// imageGenerator.SaveVectorImageAsLocalFile(visualizationOperator.distributedVectorImage, outputPath+"-distributed",ImageType.SVG);
//
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true; } /**
* Builds the heat map.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildHeatMap(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,5);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath,ImageType.PNG);
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
} /**
* Builds the choropleth map.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean buildChoroplethMap(String outputPath)
{
try{ PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
PolygonRDD queryRDD = new PolygonRDD(sparkContext, PolygonInputLocation, PolygonSplitter, false, PolygonNumPartitions, StorageLevel.MEMORY_ONLY());
spatialRDD.spatialPartitioning(GridType.RTREE);
queryRDD.spatialPartitioning(spatialRDD.grids);
spatialRDD.buildIndex(IndexType.RTREE,true);
JavaPairRDD<Polygon,Long> joinResult = JoinQuery.SpatialJoinQueryCountByKey(spatialRDD,queryRDD,true,false); long start = System.currentTimeMillis();
ChoroplethMap visualizationOperator = new ChoroplethMap(1000,600,USMainLandBoundary,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.RED, true);
visualizationOperator.Visualize(sparkContext, joinResult); ScatterPlot frontImage = new ScatterPlot(1000,600,USMainLandBoundary,false);
frontImage.CustomizeColor(0, 0, 0, 255, Color.GREEN, true);
frontImage.Visualize(sparkContext, queryRDD); RasterOverlayOperator overlayOperator = new RasterOverlayOperator(visualizationOperator.rasterImage);
overlayOperator.JoinImage(frontImage.rasterImage); ImageGenerator imageGenerator = new ImageGenerator();
//imageGenerator.SaveRasterImageAsLocalFile(frontImage.rasterImage, outputPath,ImageType.PNG);
imageGenerator.SaveRasterImageAsLocalFile(overlayOperator.backRasterImage, outputPath,ImageType.PNG);
//imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
//ImageStitcher.stitchImagePartitionsFromLocalFile(outputPath, 1000,600,0,4, 4); System.out.println("散点图生成完成,共耗时" + (System.currentTimeMillis() - start) + "ms");
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
} /**
* Parallel filter render no stitch.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean parallelFilterRenderNoStitch(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,2,4,4,true,true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
}
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
} /**
* Parallel filter render stitch.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean parallelFilterRenderStitch(String outputPath)
{
try{
PointRDD spatialRDD = new PointRDD(sparkContext, PointInputLocation, PointOffset, PointSplitter, false, PointNumPartitions, StorageLevel.MEMORY_ONLY());
HeatMap visualizationOperator = new HeatMap(1000,600,USMainLandBoundary,false,2,4,4,true,true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.distributedRasterImage, outputPath,ImageType.PNG);
ImageStitcher.stitchImagePartitionsFromLocalFile(outputPath, 1000,600,0,4, 4); }
catch(Exception e)
{
e.printStackTrace();
return false;
}
return true;
} /**
* Earthdata visualization.
*
* @param outputPath the output path
* @return true, if successful
*/
public static boolean earthdataVisualization(String outputPath)
{ try {
EarthdataHDFPointMapper earthdataHDFPoint = new EarthdataHDFPointMapper(HDFIncrement,HDFOffset,HDFRootGroupName,
HDFDataVariableList,HDFDataVariableName,HDFswitchXY,urlPrefix);
PointRDD spatialRDD = new PointRDD(sparkContext, earthdataInputLocation, earthdataNumPartitions, earthdataHDFPoint,StorageLevel.MEMORY_ONLY());
ScatterPlot visualizationOperator = new ScatterPlot(1000,600,spatialRDD.boundaryEnvelope,ColorizeOption.EARTHOBSERVATION,false,false);
visualizationOperator.CustomizeColor(255, 255, 255, 255, Color.BLUE, true);
visualizationOperator.Visualize(sparkContext, spatialRDD);
ImageGenerator imageGenerator = new ImageGenerator();
imageGenerator.SaveRasterImageAsLocalFile(visualizationOperator.rasterImage, outputPath, ImageType.PNG);
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
} /**
* The main method.
*
* @param args the arguments
* @throws IOException Signals that an I/O exception has occurred.
*/
public static void main(String[] args) throws IOException {
long start = System.currentTimeMillis();
Logger.getLogger("org").setLevel(Level.WARN);
Logger.getLogger("akka").setLevel(Level.WARN); SparkConf sparkConf = new SparkConf().setAppName("GeoSparkVizDemo").setMaster("local[*]").set("spark.serializer", KryoSerializer.class.getName())
.set("spark.kryo.registrator", GeoSparkVizKryoRegistrator.class.getName());
sparkContext = new JavaSparkContext(sparkConf); prop = new Properties(); String resourcePath = "src/test/resources/";
String demoOutputPath = "target/demo";
FileInputStream ConfFile= new FileInputStream(resourcePath+"babylon.point.properties");
prop.load(ConfFile); String scatterPlotOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath + "/scatterplot";
String heatMapOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/heatmap";
String choroplethMapOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/choroplethmap";
String parallelFilterRenderStitchOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/parallelfilterrenderstitchheatmap";
String earthdataScatterPlotOutputPath = System.getProperty("user.dir")+"/"+demoOutputPath+"/earthdatascatterplot"; PointInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
PointOffset = Integer.parseInt(prop.getProperty("offset"));;
PointSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
PointNumPartitions = Integer.parseInt(prop.getProperty("numPartitions")); ConfFile= new FileInputStream(resourcePath+"babylon.rectangle.properties");
prop.load(ConfFile);
RectangleInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
RectangleOffset = Integer.parseInt(prop.getProperty("offset"));
RectangleSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
RectangleNumPartitions = Integer.parseInt(prop.getProperty("numPartitions")); ConfFile= new FileInputStream(resourcePath+"babylon.polygon2.properties");
prop.load(ConfFile);
PolygonInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
PolygonOffset = Integer.parseInt(prop.getProperty("offset"));
PolygonSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
PolygonNumPartitions = Integer.parseInt(prop.getProperty("numPartitions")); ConfFile= new FileInputStream(resourcePath+"babylon.linestring.properties");
prop.load(ConfFile);
LineStringInputLocation = System.getProperty("user.dir")+"/"+resourcePath+prop.getProperty("inputLocation");
LineStringOffset = Integer.parseInt(prop.getProperty("offset"));
LineStringSplitter = FileDataSplitter.getFileDataSplitter(prop.getProperty("splitter"));
LineStringNumPartitions = Integer.parseInt(prop.getProperty("numPartitions")); USMainLandBoundary = new Envelope(-126.790180,-64.630926,24.863836,50.000); earthdataInputLocation = System.getProperty("user.dir")+"/src/test/resources/modis/modis.csv";
earthdataNumPartitions = 5;
HDFIncrement=5;
HDFOffset=2;
HDFRootGroupName = "MOD_Swath_LST";
HDFDataVariableName = "LST";
HDFswitchXY = true;
urlPrefix = System.getProperty("user.dir")+"/src/test/resources/modis/"; if(buildScatterPlot(scatterPlotOutputPath)&&buildHeatMap(heatMapOutputPath)
&&buildChoroplethMap(choroplethMapOutputPath)&&parallelFilterRenderStitch(parallelFilterRenderStitchOutputPath+"-stitched")
&&parallelFilterRenderNoStitch(parallelFilterRenderStitchOutputPath)&&earthdataVisualization(earthdataScatterPlotOutputPath))
{
System.out.println("散点图生成完成,共耗时" + (System.currentTimeMillis() - start) + "ms");
System.out.println("All GeoSparkViz Demos have passed.");
}
else
{
System.out.println("GeoSparkViz Demos failed."); }
sparkContext.stop();
} }

项目可视化一览



参考资料:

https://www.jianshu.com/p/1a531de087df

https://www.helplib.com/GitHub/article_127813

GeoSpark入门-可视化的更多相关文章

  1. (一)Superset 1.3图表篇——Table

    本系列文章基于Superset 1.3.0版本.1.3.0版本目前支持分布,趋势,地理等等类型共59张图表.本次1.3版本的更新图表有了一些新的变化,而之前也一直没有做过非常细致的图表教程. 而且目前 ...

  2. 一起学微软Power BI系列-官方文档-入门指南(4)Power BI的可视化

    在前面的系列文章中,我们介绍了官方有关获取数据,以及建模的原始文档和基本介绍.今天继续给大家介绍官方文档中,有关可视化的内容.实际上获获取数据和建模更注重业务关系的处理,而可视化则关注对数据的解读.这 ...

  3. d3.js:数据可视化利器之快速入门

    hello,data! 在进入d3.js之前,我们先用一个小例子回顾一下将数据可视化的基本流程. 任务 用横向柱状图来直观显示以下数据: var data = [10,15,23,78,57,29,3 ...

  4. Roslyn 入门:使用 Visual Studio 的语法可视化窗格查看和了解代码的语法树

    使用 Visual Studio 提供的 Syntax Visualizer,我们可以实时看到一个代码文件中的语法树.这对我们基于 Roslyn 编写静态分析和修改工具非常有帮助.本文将介绍如何安装它 ...

  5. 数据可视化入门之show me the numbers

           数据的可视化一直是自己瞎玩着学,近来想系统的学数据可视化的东西,于是搜索资料时看到有人推荐<show me the numbers>作为入门. 由于搜不到具体的书籍内容,只能 ...

  6. 数据可视化-svg入门基础(二)

    接上一篇:数据可视化-svg入门基础(一),基础一主要是介绍了svg概念,元素样式设置等. svg是(scalable vector graphic)伸缩矢量图像. 一.目录 (1)图形元素 (2)文 ...

  7. Seaborn数据可视化入门

    在本节学习中,我们使用Seaborn作为数据可视化的入门工具 Seaborn的官方网址如下:http://seaborn.pydata.org 一:definition Seaborn is a Py ...

  8. Matplotlib数据可视化(1):入门介绍

      1 matplot入门指南¶ matplotlib是Python科学计算中使用最多的一个可视化库,功能丰富,提供了非常多的可视化方案,基本能够满足各种场景下的数据可视化需求.但功能丰富从另一方面来 ...

  9. Keras入门(六)模型训练实时可视化

      在北京做某个项目的时候,客户要求能够对数据进行训练.预测,同时能导出模型,还有在页面上显示训练的进度.前面的几个要求都不难实现,但在页面上显示训练进度当时笔者并没有实现.   本文将会分享如何在K ...

随机推荐

  1. 使用scrapy-redis搭建分布式爬虫环境

    scrapy-redis简介 scrapy-redis是scrapy框架基于redis数据库的组件,用于scrapy项目的分布式开发和部署. 有如下特征:  分布式爬取 您可以启动多个spider工 ...

  2. 5分钟上手自动化测试——Airtest+Poco快速上手

    版权声明:该文章为AirtestProject原创文章:允许转载,但转载必须注明“转载”并保留原链接 前言 本文档将演示使用`AirtestProject`专用的编辑器AirtestIDE,编写`Ai ...

  3. ASP.NET Core 2 preview 1中Program.cs,Startup.cs和CreateDefaultBuilder的探索

    Exploring Program.cs, Startup.cs and CreateDefaultBuilder in ASP.NET Core 2 preview 1 ASP.NET Core 2 ...

  4. 使用 SecureRandom 产生随机数采坑记录

    公众号「码海」欢迎关注 背景 我们的项目工程里经常在每个函数需要用到 Random 的地方定义一下 Random 变量(如下) public void doSomethingCommon() { Ra ...

  5. CSS自定义默认样式

    html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, ...

  6. Springboot整合webservice

    Springboot整合webservice 2019-12-10 16:34:42 星期二 WebService是什么 WebService是一种跨编程语言和跨操作系统平台的远程调用技术,服务之间的 ...

  7. 写一个umi插件 自动生成代码 解放cv的双手

    引言 最近在写一个中台项目,使用的react的umi框架. 各种增删改查.基本是列表页 新建页 详情页这种页面 为了避免不必要的简单重复(主要是想偷懒) 于是想去实现自己的一个代码生成器 探索 首先, ...

  8. redis--linux环境搭建

    1.redis诞生的背景 在这要从08年开始说起,一个意大利的小伙子创建一个访问网站信息的LLOOGG.COM网站,用来记录网站的访问记录,查看最近一万条访问信息,每次访问都会将数据存入mysql当中 ...

  9. 洛谷 题解 P1842 【奶牛玩杂技】

    本蒟蒻又双叒叕被爆踩辣! Solution: 我们先看数据,50000,那么O(n)或者O(n log(n))是可以过的,非严格O(n * sqrt(n))要卡卡常,说不定也可以过. 那么什么算法可以 ...

  10. 压缩感知重构算法之IHT算法python实现

    压缩感知重构算法之OMP算法python实现 压缩感知重构算法之CoSaMP算法python实现 压缩感知重构算法之SP算法python实现 压缩感知重构算法之IHT算法python实现 压缩感知重构 ...