HBase(一): c#访问hbase组件开发

HDP2.4安装系列介绍了通过ambari创建hbase集群的过程，但工作中一直采用.net的技术路线，如何去访问基于Java搞的Hbase呢？ Hbase提供基于Java的本地API访问，同时扩展了通过 Thrift、Rest 实现Web访问的API。 so 决定开发基于.net的 sdk，通过其提供的 rest webAPI 来访问 hbase, 其中c#与Java的数据交互过程采用protobuf协议。

目录：

参考资料
基本原理
c#、java数据交互
hbase filter 实体
WebRequester
hbaseClient

参考资料：

hbase官网（http://hbase.apache.org/book.html#hbase_apis）
hbase 源码 (github下载)
Microsoft.HBase.Client 源码 (github下载)

基本原理：

HBase Rest 是建立在HBase java 客户端基础之上提供的web 服务，示意图如下：
可以通过 start /stop 等命令来启动或停止Hbase的 Rest server 服务，如下：

1. 命令：hbase rest start (默认的方式启动rest服务，端口是8080)
2. 命令：hbase rest start 9000 （这种方式以端口9000方式启动）
3. 命令：hbase-daemon.sh start rest -p 9000

当服务启动的时候，系统内嵌的jetty servlet container启动并部署servlet.服务默认的监听端口为8080，可通过修改hbase 配置文件来替换其它端口。
简单概述需求：将下面列表中的访问及传递参数用c#进行封装

1. http://192.168.2.21: 为HBase master 对应的IP地址
2. 8080: 是HBase Rest Server对应的端口
3. yourTable: 操作HBase 数据库的表名
4. schema/regions/scanner: 约定关键字

c#与java通过protobuf数据交互:

Hbase 为java与其它开发语言通过protobuf进行数据交互制定一个特定的数据结构（见hbase官网REST Protobufs Schema 的结构描述），网上有一堆的工具可根据据protobufs schemal 文件生成java、c#源码。意思是双方都遵守这个数据文件格式，来实现夸平台的数据交互与共享。这个就是做了一个平台无关的文件与平台和语言相关的数据对象之间的适配转化工作，如很多xml解析器一样的原理。

协议文件是.proto为后缀的文件，格式如下代码示例

package org.apache.hadoop.hbase.rest.protobuf.generated;

message TableInfo {

  required string name = ;

  message Region {

    required string name = ;

    optional bytes startKey = ;

    optional bytes endKey = ;

    optional int64 id = ;

    optional string location = ;

  }

  repeated Region regions = ;

}

package：在Java里面代表这个文件所在的包名，在c#里面代表该文件的命名空间
message：代表一个类；
required：代表该字段必填；
optional：代表该字段可选，并可以为其设置默认值

从github上下载window版的转换工具，将解压后包中的ProtoGen.exe.config,protoc.exe,ProtoGen.exe及Google.ProtocolBuffers.dll文件放到某个新建的文件夹( 如：c:\zhu)
将hbase 规定的协议文件同时copy至该目录（hbase源码包中 \hbase\hbase-rest\src\main\resources\org\apache\hadoop\hbase\rest\protobuf 下的文件）
以TableInfoMessage.proto 为例进行说明, windows系统下打开命令提示符，切换至 c:\zhu 文件夹下
执行：protoc --descriptor_set_out=TableInfoMessage.protobin --include_imports TableInfoMessage.proto
上述命令之后，c:\zhu 文件夹内生成了一个TableInfoMessage.protobin文件
执行：protogen AddressBook.protobin (目录下会生成名为TableInfoMessage.cs文件，这就是生成的c#源码）
当然你可以写一个批处理命令来执行，完成后生成的9个文件引入到你的Visual studio 工程即可使用。

hbase filter 实体:

在hbase读取数据时设置的过滤参数，参照 (hbase\hbase-client\src\main\java\org\apache\hadoop\hbase\filter)源码，用c#翻译一次
完成后如下图

WebRequester：

封装http请求 WebRequester 类

public class WebRequester

    {

        private string url = string.Empty;

        /// <summary>

        ///

        /// </summary>

        /// <param name="urlString"></param>

        public WebRequester(string urlString)

        {

            this.url = urlString;

        }

        /// <summary>

        /// Issues the web request.

        /// </summary>

        /// <param name="endpoint">The endpoint.</param>

        /// <param name="method">The method.</param>

        /// <param name="input">The input.</param>

        /// <param name="options">request options</param>

        /// <returns></returns>

        public HttpWebResponse IssueWebRequest(string endpoint, string method, Stream input, RequestOptions options)

        {

            return IssueWebRequestAsync(endpoint, method, input,options).Result;

        }

        /// <summary>

        /// Issues the web request asynchronous.

        /// </summary>

        /// <param name="endpoint">The endpoint.</param>

        /// <param name="method">The method.</param>

        /// <param name="input">The input.</param>

        /// <param name="options">request options</param>

        /// <returns></returns>

        public async Task<HttpWebResponse> IssueWebRequestAsync(string endpoint, string method, Stream input, RequestOptions options)

        {

            string uri = string.Format("{0}/{1}", this.url, endpoint);

            HttpWebRequest httpWebRequest = HttpWebRequest.CreateHttp(uri);

            httpWebRequest.Timeout = options.TimeoutMillis;

            httpWebRequest.PreAuthenticate = true;

            httpWebRequest.Method = method;

            httpWebRequest.ContentType = options.ContentType;

            if (options.AdditionalHeaders != null)

            {

                foreach (var kv in options.AdditionalHeaders)

                {

                    httpWebRequest.Headers.Add(kv.Key, kv.Value);

                }

            }

            if (input != null)

            {

                using (Stream req = await httpWebRequest.GetRequestStreamAsync())

                {

                    await input.CopyToAsync(req);

                }

            }

            return (await httpWebRequest.GetResponseAsync()) as HttpWebResponse;

        }

    }

http 操作实体类

public class RequestOptions

    {

        public string AlternativeEndpoint { get; set; }

        public bool KeepAlive { get; set; }

        public int TimeoutMillis { get; set; }

        public int SerializationBufferSize { get; set; }

        public int ReceiveBufferSize { get; set; }

        public bool UseNagle { get; set; }

        public int Port { get; set; }

        public Dictionary<string, string> AdditionalHeaders { get; set; }

        public string AlternativeHost { get; set; }

        public string ContentType { get; set; }

        public static RequestOptions GetDefaultOptions()

        {

            return new RequestOptions()

            {

                KeepAlive = true,

                TimeoutMillis = ,

                ReceiveBufferSize =  *  * ,

                SerializationBufferSize =  *  * ,

                UseNagle = false,

                //AlternativeEndpoint = Constants.RestEndpointBase,

                //Port = 443,

                AlternativeEndpoint = string.Empty,

                Port = ,

                AlternativeHost = null,

                ContentType = "application/x-protobuf"

            };

        }

    }

hbaseClient：

定义hbase 常用操作接口IHbaseClient（包含基于表的操作以及数据的读写），示例如下

public interface IHBaseClient

    {        

        /// <summary>

        ///

        /// </summary>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<org.apache.hadoop.hbase.rest.protobuf.generated.Version> GetVersionAsync(RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="schema"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<bool> CreateTableAsync(TableSchema schema, RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task DeleteTableAsync(string table, RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<TableInfo> GetTableInfoAsync(string table, RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<TableSchema> GetTableSchemaAsync(string table, RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<TableList> ListTablesAsync(RequestOptions options = null);

        /// <summary>

        ///

        /// </summary>

        /// <param name="tableName"></param>

        /// <param name="scannerSettings"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<ScannerInformation> CreateScannerAsync(string tableName, Scanner scannerSettings, RequestOptions options);

        /// <summary>

        ///

        /// </summary>

        /// <param name="scannerInfo"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<CellSet> ScannerGetNextAsync(ScannerInformation scannerInfo, RequestOptions options);

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="cells"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        Task<bool> StoreCellsAsync(string table, CellSet cells, RequestOptions options = null);

    }

实现接口类 HBaseClient

public class HBaseClient : IHBaseClient

    {

        private WebRequester _requester;

        private readonly RequestOptions _globalRequestOptions;

        /// <summary>

        ///

        /// </summary>

        /// <param name="endPoints"></param>

        /// <param name="globalRequestOptions"></param>

        public HBaseClient(string url, RequestOptions globalRequestOptions = null)

        {

            _globalRequestOptions = globalRequestOptions ?? RequestOptions.GetDefaultOptions();

            _requester = new WebRequester(url);

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<org.apache.hadoop.hbase.rest.protobuf.generated.Version> GetVersionAsync(RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            return await GetRequestAndDeserializeAsync<org.apache.hadoop.hbase.rest.protobuf.generated.Version>(EndPointType.Version, optionToUse);

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="schema"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<bool> CreateTableAsync(TableSchema schema, RequestOptions options = null)

        {

            if (string.IsNullOrEmpty(schema.name))

                throw new ArgumentException("schema.name was either null or empty!", "schema");

            var optionToUse = options ?? _globalRequestOptions;

            string endpoint = string.Format("{0}/{1}", schema.name, EndPointType.Schema);

            using (HttpWebResponse webResponse = await PutRequestAsync(endpoint,schema, optionToUse))

            {

                if (webResponse.StatusCode == HttpStatusCode.Created)

                {

                    return true;

                }

                // table already exits

                if (webResponse.StatusCode == HttpStatusCode.OK)

                {

                    return false;

                }

                // throw the exception otherwise

                using (var output = new StreamReader(webResponse.GetResponseStream()))

                {

                    string message = output.ReadToEnd();

                    throw new WebException(

                       string.Format("Couldn't create table {0}! Response code was: {1}, expected either 200 or 201! Response body was: {2}",

                          schema.name,webResponse.StatusCode,message));

                }

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task DeleteTableAsync(string table, RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            string endPoint = string.Format("{0}/{1}", table, EndPointType.Schema);

            using (HttpWebResponse webResponse = await ExecuteMethodAsync<HttpWebResponse>(WebMethod.Delete, endPoint, null, optionToUse))

            {

                if (webResponse.StatusCode != HttpStatusCode.OK)

                {

                    using (var output = new StreamReader(webResponse.GetResponseStream()))

                    {

                        string message = output.ReadToEnd();

                        throw new WebException(

                            string.Format("Couldn't delete table {0}! Response code was: {1}, expected 200! Response body was: {2}",

                                table, webResponse.StatusCode, message));

                    }

                }

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<TableInfo> GetTableInfoAsync(string table, RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            string endPoint = string.Format("{0}/{1}", table, EndPointType.Regions);

            return await GetRequestAndDeserializeAsync<TableInfo>(endPoint, optionToUse);

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<TableSchema> GetTableSchemaAsync(string table, RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            string endPoint = string.Format("{0}/{1}", table, EndPointType.Schema);

            return await GetRequestAndDeserializeAsync<TableSchema>(endPoint, optionToUse);

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<TableList> ListTablesAsync(RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            return await GetRequestAndDeserializeAsync<TableList>("", optionToUse);

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="tableName"></param>

        /// <param name="scannerSettings"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<ScannerInformation> CreateScannerAsync(string tableName, Scanner scannerSettings, RequestOptions options)

        {

            string endPoint = string.Format("{0}/{1}", tableName, EndPointType.Scanner);

            using (HttpWebResponse response = await ExecuteMethodAsync(WebMethod.Post, endPoint, scannerSettings, options))

            {

                if (response.StatusCode != HttpStatusCode.Created)

                {

                    using (var output = new StreamReader(response.GetResponseStream()))

                    {

                        string message = output.ReadToEnd();

                        throw new WebException(

                            string.Format( "Couldn't create a scanner for table {0}! Response code was: {1}, expected 201! Response body was: {2}",

                                tableName, response.StatusCode, message));

                    }

                }

                string location = response.Headers.Get("Location");

                if (location == null)

                {

                    throw new ArgumentException("Couldn't find header 'Location' in the response!");

                }

                return new ScannerInformation(new Uri(location), tableName, response.Headers);

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="scannerInfo"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<CellSet> ScannerGetNextAsync(ScannerInformation scannerInfo, RequestOptions options)

        {

            string endPoint = string.Format("{0}/{1}/{2}", scannerInfo.TableName, EndPointType.Scanner, scannerInfo.ScannerId);

            using (HttpWebResponse webResponse = await GetRequestAsync(endPoint, options))

            {

                if (webResponse.StatusCode == HttpStatusCode.OK)

                {

                    return Serializer.Deserialize<CellSet>(webResponse.GetResponseStream());

                }

                return null;

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="table"></param>

        /// <param name="cells"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        public async Task<bool> StoreCellsAsync(string table, CellSet cells, RequestOptions options = null)

        {

            var optionToUse = options ?? _globalRequestOptions;

            string path = table + "/somefalsekey";

            using (HttpWebResponse webResponse = await PutRequestAsync(path, cells, options))

            {

                if (webResponse.StatusCode == HttpStatusCode.NotModified)

                {

                    return false;

                }

                if (webResponse.StatusCode != HttpStatusCode.OK)

                {

                    using (var output = new StreamReader(webResponse.GetResponseStream()))

                    {

                        string message = output.ReadToEnd();

                        throw new WebException(

                           string.Format("Couldn't insert into table {0}! Response code was: {1}, expected 200! Response body was: {2}",

                              table, webResponse.StatusCode, message));

                    }

                }

            }

            return true;

        }

        /// <summary>

        ///

        /// </summary>

        /// <typeparam name="T"></typeparam>

        /// <param name="endpoint"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        private async Task<T> GetRequestAndDeserializeAsync<T>(string endpoint, RequestOptions options)

        {

            using (WebResponse response = await _requester.IssueWebRequestAsync(endpoint, WebMethod.Get, null, options))

            {

                using (Stream responseStream = response.GetResponseStream())

                {

                    return Serializer.Deserialize<T>(responseStream);

                }

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <typeparam name="TReq"></typeparam>

        /// <param name="endpoint"></param>

        /// <param name="query"></param>

        /// <param name="request"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        private async Task<HttpWebResponse> PutRequestAsync<TReq>(string endpoint, TReq request, RequestOptions options)

          where TReq : class

        {

            return await ExecuteMethodAsync(WebMethod.Post, endpoint, request, options);

        }

        /// <summary>

        ///

        /// </summary>

        /// <typeparam name="TReq"></typeparam>

        /// <param name="method"></param>

        /// <param name="endpoint"></param>

        /// <param name="request"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        private async Task<HttpWebResponse> ExecuteMethodAsync<TReq>(string method,string endpoint,TReq request,RequestOptions options) where TReq : class

        {

            using (var input = new MemoryStream(options.SerializationBufferSize))

            {

                if (request != null)

                {

                    Serializer.Serialize(input, request);

                }

                input.Seek(, SeekOrigin.Begin);

                return await _requester.IssueWebRequestAsync(endpoint,method, input, options);

            }

        }

        /// <summary>

        ///

        /// </summary>

        /// <param name="endpoint"></param>

        /// <param name="query"></param>

        /// <param name="options"></param>

        /// <returns></returns>

        private async Task<HttpWebResponse> GetRequestAsync(string endpoint, RequestOptions options)

        {

            return await _requester.IssueWebRequestAsync(endpoint, WebMethod.Get, null, options);

        }

    }

按步骤完成上面的代码，编译通过即OK，下一篇进入sdk的测试验证之旅

HBase(一): c#访问hbase组件开发的更多相关文章

HBase(二): c#访问HBase之股票行情Demo
上一章完成了c#访问hbase的sdk封装,接下来以一个具体Demo对sdk进行测试验证.场景:每5秒抓取指定股票列表的实时价格波动行情,数据下载后,一方面实时刷新UI界面,另一方面将数据放入到在内存 ...
使用C#通过Thrift访问HBase
前言因为项目需要要为客户程序提供C#.Net的HBase访问接口,而HBase并没有提供原生的.Net客户端接口,可以通过启动HBase的Thrift服务来提供多语言支持. Thrift介绍环境 ...
CDH 6.0.1 版本默认配置下 HUE | happybase 无法访问 Hbase 的问题
第一个问题 HUE 无法直接连接到 HBase 在默认配置下 CDH 6.0.1 版本下的 HBase2.0 使用了默认配置 hbase.regionserver.thrift.compact = T ...
Hbase记录-client访问zookeeper大量断开以及参数调优分析（转载）
1.hbase client配置参数超时时间.重试次数.重试时间间隔的配置也比较重要,因为默认的配置的值都较大,如果出现hbase集群或者RegionServer以及ZK关掉,则对应用程序是灾难性的 ...
使用C#和Thrift来访问Hbase实例
今天试着用C#和Thrift来访问Hbase,主要参考了博客园上的这篇文章.查了Thrift,Hbase的资料,结合博客园的这篇文章,终于搞好了.期间经历了不少弯路,下面我尽量详细的记录下来,免得大家 ...
Pyspark访问Hbase
作者:Syn良子出处:http://www.cnblogs.com/cssdongl/p/7347167.html 转载请注明出处记录自己最近抽空折腾虚拟机环境时用spark2.0的pyspark ...
windows平台下用C#访问HBase
Hadoop中的HBase有多种数据访问方式,ubuntu里可以用hbase shell查看操作hbase数据库,但windows平台下需要用thrift对它进行访问. 例如hadoop安装在/usr ...
JAVA API访问Hbase org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=32
Java使用API访问Hbase报错: 我的hbase主节点是spark1 java代码访问hbase的时候写的是ip 结果运行程序报错不能够识别主机名修改主机名修改主机hosts文 ...
PHP通过thrift2访问HBASE
前一段时间需要在网页上显示HBASE查询的结果,考虑用PHP来实现,在网上搜了一下,普遍都是用thrift作为接口来实现的. 参考博文: http://www.cnblogs.com/scotom ...

随机推荐

转：Repeater嵌套绑定Repeater以及内层调用外层数据
<table border=" style="margin-bottom: 5px" width="100%"> <asp:Repe ...
chromium 安装 pepper flash player
打开终端,输入以下命令即可安装: 1. sudo apt-get update 2. sudo apt-get install chromium-browser 3. sudo apt-get ins ...
Java 使用对话框选择文件并输出到控制台
import java.util.*; import java.io.*; import javax.swing.JFileChooser; public class ReadFileUsingJFi ...
用python处理数学问题
一, 计算对数: >>> import math #导入数学模块>>> math.log(8,2) #计算以2为底 8的对数3.0>&g ...
Nagios：企业级系统监控方案
在大多数情况下Cacti + RRDtool已经实现对系统各种参数的监测.但很多企业可能不满足于仅仅监测系统基本参数的需求,而是需要监测除基本参数之外的各种应用程序的运行状况.很显然在这种情况下对于一 ...
codevs 2216 线段树两种更新方式的冲突
题目描述 Description “神州“载人飞船的发射成功让小可可非常激动,他立志长大后要成为一名宇航员假期一始,他就报名参加了“小小宇航员夏令营”,在这里小可可不仅学到了丰富的宇航知识,还参与解决 ...
小三角图标如何用CSS写
上三角▲ 1 width: 0; 2 height: 0; 3 line-height: 0; 4 font-size: 0; 5 border-width: 10px; 6 border-s ...
init/main.c
/* * linux/init/main.c * * Copyright (C) 1991, 1992 Linus Torvalds */ #include <stdarg.h> #inc ...
Fresco最最最简单使用
http://blog.csdn.net/wa991830558/article/details/46005063
Mysql 下 Insert、Update、Delete、Order By、Group By注入
Insert: 语法:INSERT INTO table_name (列1, 列2,...) VALUES (值1, 值2,....) 报错注入: insert into test(id,name,p ...

HBase(一): c#访问hbase组件开发

HBase(一): c#访问hbase组件开发的更多相关文章

随机推荐

热门专题