println("--------------------"+data.rdd.getNumPartitions) // 获取DF中partition的数目

     val partitions = data.rdd.glom().collect() // 获取所有data下所有的partition，返回一个partition的集合

     for(part <- partitions){

       println(part.getClass.getName + "::::::::" + part.length) // 每个partition中的数据量

     }

结果：

--------------------100

[Lorg.apache.spark.sql.Row;::::::::61516

[Lorg.apache.spark.sql.Row;::::::::61656

[Lorg.apache.spark.sql.Row;::::::::61991

[Lorg.apache.spark.sql.Row;::::::::61269

[Lorg.apache.spark.sql.Row;::::::::61654

[Lorg.apache.spark.sql.Row;::::::::61780

[Lorg.apache.spark.sql.Row;::::::::62059

[Lorg.apache.spark.sql.Row;::::::::61675

[Lorg.apache.spark.sql.Row;::::::::61339

[Lorg.apache.spark.sql.Row;::::::::61783

[Lorg.apache.spark.sql.Row;::::::::61620

[Lorg.apache.spark.sql.Row;::::::::61883

[Lorg.apache.spark.sql.Row;::::::::61631

[Lorg.apache.spark.sql.Row;::::::::61930

[Lorg.apache.spark.sql.Row;::::::::61451

[Lorg.apache.spark.sql.Row;::::::::61797

[Lorg.apache.spark.sql.Row;::::::::61367

[Lorg.apache.spark.sql.Row;::::::::61647

[Lorg.apache.spark.sql.Row;::::::::61488

[Lorg.apache.spark.sql.Row;::::::::61584

[Lorg.apache.spark.sql.Row;::::::::61733

[Lorg.apache.spark.sql.Row;::::::::61491

[Lorg.apache.spark.sql.Row;::::::::61809

[Lorg.apache.spark.sql.Row;::::::::61062

[Lorg.apache.spark.sql.Row;::::::::61658

[Lorg.apache.spark.sql.Row;::::::::61599

[Lorg.apache.spark.sql.Row;::::::::61911

[Lorg.apache.spark.sql.Row;::::::::61602

[Lorg.apache.spark.sql.Row;::::::::61348

[Lorg.apache.spark.sql.Row;::::::::61677

[Lorg.apache.spark.sql.Row;::::::::61722

[Lorg.apache.spark.sql.Row;::::::::61482

[Lorg.apache.spark.sql.Row;::::::::61714

[Lorg.apache.spark.sql.Row;::::::::61241

[Lorg.apache.spark.sql.Row;::::::::61737

[Lorg.apache.spark.sql.Row;::::::::62015

[Lorg.apache.spark.sql.Row;::::::::62062

[Lorg.apache.spark.sql.Row;::::::::61557

[Lorg.apache.spark.sql.Row;::::::::61607

[Lorg.apache.spark.sql.Row;::::::::61175

[Lorg.apache.spark.sql.Row;::::::::61653

[Lorg.apache.spark.sql.Row;::::::::61460

[Lorg.apache.spark.sql.Row;::::::::61705

[Lorg.apache.spark.sql.Row;::::::::61492

[Lorg.apache.spark.sql.Row;::::::::61340

[Lorg.apache.spark.sql.Row;::::::::61767

[Lorg.apache.spark.sql.Row;::::::::61756

[Lorg.apache.spark.sql.Row;::::::::61793

[Lorg.apache.spark.sql.Row;::::::::61417

[Lorg.apache.spark.sql.Row;::::::::61376

[Lorg.apache.spark.sql.Row;::::::::62039

[Lorg.apache.spark.sql.Row;::::::::61571

[Lorg.apache.spark.sql.Row;::::::::61849

[Lorg.apache.spark.sql.Row;::::::::61553

[Lorg.apache.spark.sql.Row;::::::::61612

[Lorg.apache.spark.sql.Row;::::::::61980

[Lorg.apache.spark.sql.Row;::::::::61714

[Lorg.apache.spark.sql.Row;::::::::62376

[Lorg.apache.spark.sql.Row;::::::::61884

[Lorg.apache.spark.sql.Row;::::::::61273

[Lorg.apache.spark.sql.Row;::::::::61669

[Lorg.apache.spark.sql.Row;::::::::61695

[Lorg.apache.spark.sql.Row;::::::::61515

[Lorg.apache.spark.sql.Row;::::::::61247

[Lorg.apache.spark.sql.Row;::::::::61909

[Lorg.apache.spark.sql.Row;::::::::61879

[Lorg.apache.spark.sql.Row;::::::::61913

[Lorg.apache.spark.sql.Row;::::::::61199

[Lorg.apache.spark.sql.Row;::::::::61678

[Lorg.apache.spark.sql.Row;::::::::61619

[Lorg.apache.spark.sql.Row;::::::::61909

[Lorg.apache.spark.sql.Row;::::::::61406

[Lorg.apache.spark.sql.Row;::::::::61775

[Lorg.apache.spark.sql.Row;::::::::61559

[Lorg.apache.spark.sql.Row;::::::::61773

[Lorg.apache.spark.sql.Row;::::::::61888

[Lorg.apache.spark.sql.Row;::::::::61634

[Lorg.apache.spark.sql.Row;::::::::61786

[Lorg.apache.spark.sql.Row;::::::::61666

[Lorg.apache.spark.sql.Row;::::::::61519

[Lorg.apache.spark.sql.Row;::::::::61563

[Lorg.apache.spark.sql.Row;::::::::61481

[Lorg.apache.spark.sql.Row;::::::::61295

[Lorg.apache.spark.sql.Row;::::::::61343

[Lorg.apache.spark.sql.Row;::::::::61750

[Lorg.apache.spark.sql.Row;::::::::61328

[Lorg.apache.spark.sql.Row;::::::::61650

[Lorg.apache.spark.sql.Row;::::::::61541

[Lorg.apache.spark.sql.Row;::::::::61397

[Lorg.apache.spark.sql.Row;::::::::61505

[Lorg.apache.spark.sql.Row;::::::::61761

[Lorg.apache.spark.sql.Row;::::::::61795

[Lorg.apache.spark.sql.Row;::::::::62291

[Lorg.apache.spark.sql.Row;::::::::61566

[Lorg.apache.spark.sql.Row;::::::::61213

[Lorg.apache.spark.sql.Row;::::::::62028

[Lorg.apache.spark.sql.Row;::::::::62634

[Lorg.apache.spark.sql.Row;::::::::61838

[Lorg.apache.spark.sql.Row;::::::::61243

[Lorg.apache.spark.sql.Row;::::::::61585

样例：

--------------------100
[Lorg.apache.spark.sql.Row;::::::::61516

[Lorg.apache.spark.sql.Row;::::::::61656

[Lorg.apache.spark.sql.Row;::::::::61991

[Lorg.apache.spark.sql.Row;::::::::61269

[Lorg.apache.spark.sql.Row;::::::::61654

[Lorg.apache.spark.sql.Row;::::::::61780

spark查看DF的partition数目及每个partition中的数据量【集群模式】的更多相关文章

  1. Spark集群模式&Spark程序提交

    Spark集群模式&Spark程序提交 1. 集群管理器 Spark当前支持三种集群管理方式 Standalone-Spark自带的一种集群管理方式,易于构建集群. Apache Mesos- ...

  2. 【待补充】Spark 集群模式 && Spark Job 部署模式

    0. 说明 Spark 集群模式 && Spark Job 部署模式 1. Spark 集群模式 [ Local ] 使用一个 JVM 模拟 Spark 集群 [ Standalone ...

  3. Spark Tachyon编译部署（含单机和集群模式安装）

    Tachyon编译部署 编译Tachyon 单机部署Tachyon 集群模式部署Tachyon 1.Tachyon编译部署 Tachyon目前的最新发布版为0.7.1,其官方网址为http://tac ...

  4. Spark Streaming揭秘 Day31 集群模式下SparkStreaming日志分析（续）

    Spark Streaming揭秘 Day31 集群模式下SparkStreaming日志分析(续) 今天延续昨天的内容,主要对为什么一个处理会分解成多个Job执行进行解析. 让我们跟踪下Job调用过 ...

  5. Spark Streaming揭秘 Day30 集群模式下SparkStreaming日志分析

    Spark Streaming揭秘 Day30 集群模式下SparkStreaming日志分析 今天通过集群运行模式观察.研究和透彻的刨析SparkStreaming的日志和web监控台. Day28 ...

  6. Spark集群模式概述

    作者:foreyou出处:http://www.foreyou.net/2015/06/22/spark-cluster-mode-overview/声明:本文采用以下协议进行授权: 署名-非商用|C ...

  7. Apache Spark 2.2.0 中文文档 - 集群模式概述 | ApacheCN

    集群模式概述 该文档给出了 Spark 如何在集群上运行.使之更容易来理解所涉及到的组件的简短概述.通过阅读 应用提交指南 来学习关于在集群上启动应用. 组件 Spark 应用在集群上作为独立的进程组 ...

  8. Spark 官方文档（2）——集群模式

    Spark版本:1.6.2 简介:本文档简短的介绍了spark如何在集群中运行,便于理解spark相关组件.可以通过阅读应用提交文档了解如何在集群中提交应用. 组件 spark应用程序通过主程序的Sp ...

  9. Spark学习笔记3（IDEA编写scala代码并打包上传集群运行）

    Spark学习笔记3 IDEA编写scala代码并打包上传集群运行 我们在IDEA上的maven项目已经搭建完成了,现在可以写一个简单的spark代码并且打成jar包 上传至集群,来检验一下我们的sp ...

随机推荐

  1. Android--UI之ViewStub

    前言 按照最近博客的路线,继续讲Android的UI开发.今天讲解一下ViewStub控件,惰性装载控件.在本篇博客中,将了解到ViewStub的常用属性.方法,以及注意事项.最后将以一个简单的Dem ...

  2. JavaSSM框架整合

    SSM整合 ssm框架 框架整合  在博客的前面介绍了mybatis,spring,springmvc的使用,那么这篇博客将介绍将mybatis和spring,springmvc的整合. 整合之前,我 ...

  3. Dubbo 源码分析系列之三 —— 架构原理

    1 核心功能 首先要了解Dubbo提供的三大核心功能: Remoting:远程通讯 提供对多种NIO框架抽象封装,包括"同步转异步"和"请求-响应"模式的信息交 ...

  4. mybatis-generator插件执行报错：Cannot resolve classpath entry

    记录一个小问题 使用了mybatis-generator插件自动生成实体类,DAO,Mapper,在执行时报错.报错信息如下 Failed to execute goal org.mybatis.ge ...

  5. YARN集群的mapreduce测试（一）

    hadoop集群搭建中配置了mapreduce的别名是yarn [hadoop@master01 hadoop]$ mv mapred-site.xml.template mapred-site.xm ...

  6. spark之JDBC开发（实战）

    一.概述 Spark Core.Spark-SQL与Spark-Streaming都是相同的,编写好之后打成jar包使用spark-submit命令提交到集群运行应用$SPARK_HOME/bin#. ...

  7. scala程序开发入门

    scala程序开发入门,快速步入scala的门槛: 1.Scala的特性: A.纯粹面向对象(没有基本类型,只有对象类型).Scala的安装与JDK相同,只需要解压之后配置环境变量即可:B.Scala ...

  8. 如何定义一个有效的OWIN Startup Class

    命名约定 Katana在程序集内的程序集名称空间下查找一个叫做Startup的类, 通过属性指定 [assembly: OwinStartup(typeof(OwinConsoleApp.Startu ...

  9. asp.net-缓存技术-20180409

    asp.net缓存技术有三种 1.页面输出缓存 2.页面部分缓存 3.页面数据缓存 ---------------------------------------------------------- ...

  10. FastDFS搭建

    a.上传安装时需要的文件 ①:FastDFS_v5.05.tar.gz ②:fastdfs-nginx-module_v1.16.tar.gz ③:libfastcommon-1.0.7.tar.gz ...