PolyBase is a technology that accesses and combines(整合) both non-relational and relational data, all from within SQL Server. It allows you to run queries on external data in Hadoop or Azure blob storage. The queries are optimized(优化) to push computation to Hadoop

目录:

  • feature
  • Performance
  • cale-out groups
  • use cases
  • 参考资料

feature:


  • By simply using Transact-SQL (T-SQL) statements, you an import and export data back and forth(反复、来回) between relational tables in SQL Server and non-relational data stored in Hadoop or Azure Blob Storage. You can also query the external data from within a T-SQL query and join it with relational data
  • Query data stored in Hadoop: Users are storing data in cost-effective distributed and scalable systems(可伸缩系统), such as Hadoop. PolyBase makes it easy to query the data by using T-SQL
  • Query data stored in Azure blob storage: Azure blob storage is a convenient(方便) place to store data for use by Azure services. PolyBase makes it easy to access the data by using T-SQL.
  • Import data from Hadoop or Azure blob storage: Leverage the speed of Microsoft SQL's columnstore technology and analysis capabilities by importing data from Hadoop or Azure blob storage into relational tables. There is no need for a separate ETL or import tool
  • Export data to Hadoop or Azure blob storage: Archive data to Hadoop or Azure blob storage to achieve cost-effective storage and keep it online for easy access
  • Integrate with BI tools:Use PolyBase with Microsoft’s business intelligence and analysis stack, or use any third party tools that is compatible with SQL Server

Performance:


  • Push computation to Hadoop:The query optimizer (查询优化器)makes a cost-based decision to push computation to Hadoop when doing so will improve query performance. It uses statistics on external tables to make the cost-based decision. Pushing computation creates MapReduce jobs and leverages Hadoop's distributed computational resources.
  • Scale compute resources:To improve query performance, you can use SQL Server PolyBase scale-out groups. This enables parallel data transfer between SQL Server instances and Hadoop nodes, and it adds compute resources for operating on the external data

cale-out groups:


  • polybase 使用单一的sqlserver 实例来处理基于hadoop 或 Azure blobl Storage 的大量数据集时,可能会出现性能瓶颈, group feature允许用户创建 sqlserver instance 集群来处理扩展的大数据集
  • headnode:  The head node contains the SQL Server instance to which PolyBase queries are submitted. Each PolyBase group can have only one head node. A head node is a logical group of SQL Database Engine, PolyBase Engine and PolyBase Data Movement Service on the SQL Server instance
  • Compute node:A compute node contains the SQL Server instance that assists with(帮助) scale-out query processing on external data. A compute node is a logical group of SQL Server and the PolyBase data movement service on the SQL Server instance. A PolyBase group can have multiple compute nodes
  • Distributed query processing:
    1. PolyBase queries are submitted to the SQL Server on the head node. The part of the query that refers to external tables is handed-off (移交)to the PolyBase engine
    2. The PolyBase engine is the key component behind PolyBase queries. It parses the query on external data, generates the query plan and distributes the work to the data movement service on the compute nodes for execution. After completion of the work, it receives the results from the compute nodes and submits them to SQL Server for processing and returning to the client
    3. The PolyBase data movement service receives instructions(指令) from the PolyBase engine and transfers data between HDFS and SQL Server, and between SQL Server instances on the head and compute nodes
  • Editions availability:
    1. After setup of SQL Server, the instance can be designated(指定) as either a head node or a compute node.
    2. The choice depends on which version of SQL Server PolyBase is running on.
    3. On an Enterprise edition installation, the instance can be designated either as head node or a compute node.
    4. On a Standard edition, the instance can only be designated as a compute node

use cases


  • polybase primary use cases 如下图:
  • (a) query submitted to PDW requires “unstructured” data from Hadoop for its execution. This might be as simple as a scan whose input is an HDFS file or a join between a file in HDFS and a table in PDW. The output in this case flows back to the user or application program that submitted the query
  • (b)  is similar except that the output of the query is materialized as an output file in HDFS, where it might be consumed by either a subsequent PDW query or by a MapReduce job. Polybase, when appropriate,will translate operations on HDFS-resident data into MapReduce jobs and push those jobs to Hadoop for execution in order to minimize the data imported from HDFS into PDW and maximize the use of Hadoop cluster resources. With Hadoop 2.0 we envision supporting a variety of techniques for processing joins that involve HDFS and PDW resident tables, including, for example, the use of semi-join techniques.

pushdown:


  • 和linked servers一样,PolyBase会设法将尽量多的处理工作转移到源数据库。也就是说,当查询Hadoop或Azure blob存储时,会生成恰当的map/reduce操作。这就是所谓的“下推(pushdown)”,开发人员需要了解的下推限制:
    1. 用于数值、日期、时间值的二元比较操作符(<、>、=、!=、<>、>=、<=)
    2. 算术运算符( +、-、*、/、%)
    3. 逻辑运算符(AND、OR)
    4. 一元运算符(NOT、IS NULL、IS NOT NULL)
    5. BETWEEN、NOT、IN和LIKE操作符可能也可以下推。这取决于查询优化器如何将它们改写为一系列使用基本关系运算符的语句
    6. 下推可以通过OPTION (FORCE EXTERNALPUSHDOWN)显式启用,或通过OPTION (DISABLE EXTERNALPUSHDOWN)显式禁用

参考资料:


  • jdk: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
  • 注意:英文版的window2012 OS 不支持安装 sqlserver2016中文版

DW(三):polybase基本理论的更多相关文章

  1. [转] DDD领域驱动设计(三) 之 理论知识收集汇总

    最近一直在学习领域驱动设计(DDD)的理论知识,从网上搜集了一些个人认为比较有价值的东西,贴出来和大家分享一下: 我一直觉得不要盲目相信权威,比如不能一谈起领域驱动设计,就一定认为国外的那个Eric ...

  2. DW(六):polybase访问Azure Blob Storage

    目录: 连接hadoop配置语法 配置hadoop连接 Pushdown配置 Create external tables for Azure blob storage 连接hadoop配置语法: g ...

  3. DW(五):polybase集群安装

    目录: Prerequisites 集群配置规划 polybase install firewall config 集群配置 删除计算节点 install Prerequisites Microsof ...

  4. 【Hive三】Hive理论

    1. Hive基础 1. Hive基础 Hive基本概念 引入原因: Hive是什么 Hive数据管理 四种数据模型 Hive内部表和外部表 Hive数据类型 Hive的优化 Map的优化: Redu ...

  5. 马凯军201771010116《面向对象程序设计(java)》第三周学习总结

    第一部分  理论知识学习与复习部分 1.在第一章里主要对Java中常见的误解这部分进行了细读,也对Java的“白皮书”术语认真的看了一遍,对Java术语有了更深的理解. 2.在第二章中对Java程序的 ...

  6. 杨其菊201771010134《面向对象程序设计(Java)》第三周学习总结

    <面向对象程序设计(Java)>第三周学习总结 第一部分:理论知识 这周课程没有新进度,由于感觉对基础语法的不熟悉,复习了一遍前三章的细碎知识,学到一些之前不知道的原理: 1.计算机高级语 ...

  7. 深入浅出Git(偏向理论)

    目录 一.理论概述 1. 什么是Git 版本控制系统分类 2. GitLab和GitHub是什么 3.Git功能 二.结合具体命令了解其工作 1.环境 2.部署 Git仓库的使用 简单命令解释 Git ...

  8. day38 并发编程(理论)

    目录 一.操作系统发展史 二.多道技术 1 单核实现并发的效果 2 多道技术图解 3 多道技术重点 三.进程理论 1 必备知识点 2 进程调度 3 进程的三状态 4 两对重要概念 四.开启进程的两种方 ...

  9. redis教程(整理中)

    一.redis简介 1.Redis:键值对类型的内存数据库:应用于高并发和实时请求的场景: 2.Redis常用数据类型: (1) string(基本数据类型)     (2)hash 注:hash中的 ...

随机推荐

  1. 【题解】【DP】【Leetcode】Climbing Stairs

    You are climbing a stair case. It takes n steps to reach to the top. Each time you can either climb ...

  2. Python TF-IDF计算100份文档关键词权重

    上一篇博文中,我们使用结巴分词对文档进行分词处理,但分词所得结果并不是每个词语都是有意义的(即该词对文档的内容贡献少),那么如何来判断词语对文档的重要度呢,这里介绍一种方法:TF-IDF. 一,TF- ...

  3. 让Grub 2来拯救你的 bootloader

    没有什么事情比 bootloader 坏掉更气人的了,充分发挥 Grub 2 的作用,让 bootloader 安分工作吧.为什么这么说? Grub 2 是最受欢迎的 bootloader ,几乎用在 ...

  4. 比较字符串,equals防空指针问题

    1,比较两个字符串内容的话,用a.equals(b)比较,其中a,b是两个字符串,用a==b的话比较的是a和b的内存地址.2,如果一个字符串是变量,另一个字符串是常量的话,一定要把常量写在前面,变量写 ...

  5. HOG特征提取分析(转)

    背景引言 方向梯度直方图(Histogram of Oriented Gradient,HOG)是用于在计算机视觉和图像处理领域,目标检测的特征描述子.该项技术是用来计算图像局部出现的方向梯度次数或信 ...

  6. hdu1712 ACboy needs your help 分组背包

    最基础的分组背包~ #include <iostream> #include <cstdio> #include <cstdlib> #include <cs ...

  7. 第二篇T语言代码编写技巧

    控件事件 控件 控件是对数据和方法的封装.控件可以有自己的属性和方法.属性是控件数据的简单访问者.方法则是控件 的一些简单而可见的功能. 概述 1.控件应用 使用现成的控件来开发应用程序时,控件工作在 ...

  8. 《Java程序设计》第8周学习总结

    学号20145220 <Java程序设计>第8周学习总结 教材学习内容总结 15.1.1日志API简介 java.util.logging包提供了日志功能相关类与接口,不必额外配置日志组件 ...

  9. lightoj1074 最短路

    题意:有n个城市,每个城市有拥挤值,有一些单向道路,从某个城市到另一个城市的花费是拥挤值差的三次方,当然可能是负的值.问从1点到某点最少的花费,若小于3或不能到达输出“?” 建图的边权是拥挤值差的三次 ...

  10. 【NOIP2010】【P1317】乌龟棋

    似乎很像搜索的DP(应该也可以用搜索写) 原题: 小明过生日的时候,爸爸送给他一副乌龟棋当作礼物.乌龟棋的棋盘是一行N 个格子,每个格子上一个分数(非负整数).棋盘第1格是唯一的起点,第N 格是终点, ...