DW(六):polybase访问Azure Blob Storage
目录:
- 连接hadoop配置语法
- 配置hadoop连接
- Pushdown配置
- Create external tables for Azure blob storage
连接hadoop配置语法:
global configuration settings for PolyBase Hadoop and Azure blob storage connectivity, Syntax:
- sp_configure: List all of the configuration options
- 配置与hadoop连接语法:
sp_configure [ @configname = ] 'hadoop connectivity',
[ @configvalue = ] { 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 }
[;] RECONFIGURE
[;]
- Option 0: Disable Hadoop connectivity
- Option 1: Hortonworks HDP 1.3 on Windows Server
- Option 1: Azure blob storage (WASB[S])
- Option 2: Hortonworks HDP 1.3 on Linux
- Option 3: Cloudera CDH 4.3 on Linux
- Option 4: Hortonworks HDP 2.0 on Windows Server
- Option 4: Azure blob storage (WASB[S])
- Option 5: Hortonworks HDP 2.0 on Linux
- Option 6: Cloudera 5.1, 5.2, 5.3, 5.4, and 5.5 on Linux
- Option 7: Hortonworks 2.1, 2.2, and 2.3 on Linux
- Option 7: Hortonworks 2.1, 2.2, and 2.3 on Windows Server
- Option 7: Azure blob storage (WASB[S])
- 示例代码:
- 显示所有可用的配置列表: EXEC sp_configure;
- 显示某个配置项的配置信息: EXEC sp_configure @configname='hadoop connectivity';
- 设置配置项示例语法如下:
sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
GO RECONFIGURE
GO
配置hadoop连接
- 在本节示例中,连接HDInsight3.2集,对应关系如下:
- 配置polybase连接配置项信息信息如下:
sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
GO RECONFIGURE
GO- 命令行: services.msc, 重启下面的服务
- SQL Server PolyBase Data Movement Service
- SQL Server PolyBase Engine
Pushdown配置:
- To improve query performance, enable pushdown computation to a Hadoop cluster
- Find the file yarn-site.xml in the installation path of SQL Server. Typically, the path is: C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf
- On the Hadoop machine, find the analogous(类似) file in the Hadoop configuration directory. In the file, find and copy the value of the configuration key yarn.application.classpath
- On the SQL Server machine, in the yarn-site.xml file, find the yarn.application.classpath property. Paste the value from the Hadoop machine into the value element.
$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*
Create external tables for Azure blob storage
- The Elastic(弹性) Database query feature relies on(依靠) the these four DDL statements. Typically, these DDL statements are used once or rarely when the schema of your application changes
- [CREATE MASTER KEY] (https://msdn.microsoft.com/library/ms174382.aspx)
- [CREATE CREDENTIAL] (https://msdn.microsoft.com/library/ms189522.aspx)
- [CREATE DATABASE SCOPED CREDENTIAL] (https://msdn.microsoft.com/library/mt270260.aspx)
- [CREATE/DROP EXTERNAL DATA SOURCE] (https://msdn.microsoft.com/library/dn935022.aspx)
- [CREATE/DROP EXTERNAL TABLE] (https://msdn.microsoft.com/library/dn935021.aspx)
- You can use the following syntax to drop the master key and credentials
- DROP CREDENTIAL <credential_name> ON DATABASE;
- DROP DATABASE SCOPED CREDENTIAL <credential_name>;
- DROP MASTER KEY;
- 模拟简单文本数据,如下:
- 将文本文件上传到 Azure blob storage, 如下图:
- 远程连接ploybase1, 打开ssms(注:此版本需单独安装),执行脚本如下:
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'Password01!'; CREATE DATABASE SCOPED CREDENTIAL JNAzureCredit
WITH IDENTITY = 'zhushy', Secret = XXXX'; /*存储访问Key*/ CREATE EXTERNAL DATA SOURCE JNAzureStorage with (
TYPE = HADOOP,
LOCATION ='wasb://<blob_container_name>@<azure_storage_account_name>.blob.core.chinacloudapi.cn',
CREDENTIAL = JNAzureCredit
); CREATE EXTERNAL FILE FORMAT TextFileFormat WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (FIELD_TERMINATOR =' ', USE_TYPE_DEFAULT = TRUE)) CREATE EXTERNAL TABLE [dbo].[hospital] (
[ID] varchar(128) NULL,
[name] varchar(128) null
)
WITH (LOCATION='/zhu/data1.txt',
DATA_SOURCE = JNAzureStorage,
FILE_FORMAT = TextFileFormat
);- 验证数据,输入如下SQL: select * from [dbo].[hospital] ,如下图,则OK
DW(六):polybase访问Azure Blob Storage的更多相关文章
- presto访问 Azure blob storage
当集群使用Azure Blog Storage时,prestoDB无法获取返回结果,在此记录下 如下,hive里面的两个表,一个使用的是本地的hdfs,一个是使用 azure blob storage ...
- hadoop(四): 本地 hbase 集群配置 Azure Blob Storage
基于 HDP2.4安装(五):集群及组件安装 创建的hadoop集群,修改默认配置,将hbase 存储配置为 Azure Blob Storage 目录: 简述 配置 验证 FAQ 简述: hadoo ...
- Azure Storage 系列(六)使用Azure Queue Storage
一,引言 在之前介绍到 Azure Storage 第一篇文章中就有介绍到 Azure Storage 是 Azure 上提供的一项存储服务,Azure 存储包括 对象.文件.磁盘.队列和表存储.这里 ...
- Azure Blob Storage 基本用法 -- Azure Storage 之 Blob
Azure Storage 是微软 Azure 云提供的云端存储解决方案,当前支持的存储类型有 Blob.Queue.File 和 Table. 笔者在<Azure Table storage ...
- Azure Blob Storage从入门到精通
今天推荐的是一个系列文章,让读者阅读完成后可以对Azure Blob Storage的开发有一个全面的了解,可谓是从入门到精通. Azure在最初的版本里面就提供了非结构化数据的存储服务,也即Blob ...
- Python 操作 Azure Blob Storage
笔者在<Azure 基础:Blob Storage>一文中介绍了 Azure Blob Storage 的基本概念,并通过 C# 代码展示了如何进行基本的操作.最近笔者需要在 Linux ...
- [AWS vs Azure] 云计算里AWS和Azure的探究(6) - Amazon Simple Storage Service 和 Microsoft Azure Blob Storage
这几天Nasuni公司出了一份报告,分析了各个云厂商的云存储的性能,包括Amazon S3,Azure Blob Storage, Google Drive, HP以及Rackspace.其中性能上A ...
- Azure Functions(二)集成 Azure Blob Storage 存储文件
一,引言 上一篇文章有介绍到什么是 SeverLess ,ServerLess 都有哪些特点,以及多云环境下 ServerLess 都有哪些解决方案.在这众多解决方案中就包括 Function App ...
- [Windows Azure] How to use the Windows Azure Blob Storage Service in .NET
How to use the Windows Azure Blob Storage Service in .NET version 1.7 version 2.0 This guide will de ...
随机推荐
- magento二次开发的基本步骤分享
Magento后台添加新模块的体会 确定命名空间(Namespace)和模块(Modulename)的命名: 在app/etc/modules/ 路径下,创建 Namespace_Modulename ...
- magento如何获取某一产品的订单量代码
<?php $sku = nl2br($_product->getSku()); $_productCollection = Mage::getResourceModel('reports ...
- MySQL 添加列,修改列,删除列
创建后表的修改 alter table 语句用于创建后对表的修改, 基础用法如下: 添加列 基本形式: alter table 表名 add 列名 列数据类型 [after 插入位置]; 示例: 在表 ...
- UI学习笔记---第一天
一.iOS概述 iOS是Apple公司的移动操作系统,主要⽤用于iPhone.iPad.iPad Mini.iPod Touch等移动产品. 借助iOS,我们可以开发视频类.美图类.新闻类.⾳乐类.团 ...
- 输出NSRange类型的数据
如果你想知道字符串内的某处是否包含其他字符串,使用rangeOfString: 方法 -(NSRange)rangeOfString :(NSString *)aString; NSString *n ...
- 转载:LBP代码详细注释
%LBP returns the local binary pattern image or LBP histogram of an image.% J = LBP(I,R,N,MAPPING,MOD ...
- Codeforces Round #368 (Div. 2)A B C 水 图 数学
A. Brain's Photos time limit per test 2 seconds memory limit per test 256 megabytes input standard i ...
- SQL根据现有表新建一张表
SQL根据现有表新建表,新建的这张表结构要跟现有表结构相同,但不要现有表里面的数据! 执行DML语句依据数据库类型而定: SQLITE -----复制表结构及数据到新表 CREATE TABLE TA ...
- ASP.NET Web API与Rest web api:发布到IIS(二)(同发布.NET webservice)
本文档大部分来源于:http://www.cnblogs.com/zqzjs/p/4705994.html 工具VS2010,window环境win7 一:Webservice的创建与方法查看调用 1 ...
- Fresco好案例
http://www.open-open.com/lib/view/open1436094840774.html