Java读写avro例子
一、avro是一个数据序列化框架,可以高效得进行序列化和反序列化,支持C, C++, C#, Java, PHP, Python, 和Ruby语言。现在使用Java来读写。
二、环境搭建
1、下载avro-1.7.7.jar and avro-tools-1.7.7.jar两个jar包,放到指定文件目录。下载地址 http://www.trieuvan.com/apache/avro/avro-1.7.7/java/
我放到了D:\soft\avro 文件夹,在改目录下新建java文件夹,用来存放生成的Java代码
2、该目录下新建user.avsc文件,内容是:
{"namespace": "example.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
3、打开cmd,进入到该目录,执行命令生成User类,注意命令后面有个".",表示生成的代码放在本目录下。
java -jar avro-tools-1.7.7.jar compile schema user.avsc java .
在该文件夹下的Java文件下的../example/avro/目录下就会生成User.java文件。
4.使用eclipse新建maven项目,在pom.xml加入avro的依赖。
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.7.7</version>
</dependency>
三、生成的User.java文件的内容如下。
把生成的User.java类复制到工程中,注意这个User.java里面生成的User类及其内部类的包名默认是user.avsc文件中的namespace的值,
在本例中也就是example.avro。需要全部替换为自己的包名。
最简单的方法就是把User.java中的example.avro全部替换为自己的包名。
/**
* Autogenerated by Avro
*
* DO NOT EDIT DIRECTLY
*/
package example.avro;
@SuppressWarnings("all")
@org.apache.avro.specific.AvroGenerated
public class User extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"example.avro\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"favorite_number\",\"type\":[\"int\",\"null\"]},{\"name\":\"favorite_color\",\"type\":[\"string\",\"null\"]}]}");
public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; }
@Deprecated public java.lang.CharSequence name;
@Deprecated public java.lang.Integer favorite_number;
@Deprecated public java.lang.CharSequence favorite_color; /**
* Default constructor. Note that this does not initialize fields
* to their default values from the schema. If that is desired then
* one should use <code>newBuilder()</code>.
*/
public User() {} /**
* All-args constructor.
*/
public User(java.lang.CharSequence name, java.lang.Integer favorite_number, java.lang.CharSequence favorite_color) {
this.name = name;
this.favorite_number = favorite_number;
this.favorite_color = favorite_color;
} public org.apache.avro.Schema getSchema() { return SCHEMA$; }
// Used by DatumWriter. Applications should not call.
public java.lang.Object get(int field$) {
switch (field$) {
case 0: return name;
case 1: return favorite_number;
case 2: return favorite_color;
default: throw new org.apache.avro.AvroRuntimeException("Bad index");
}
}
// Used by DatumReader. Applications should not call.
@SuppressWarnings(value="unchecked")
public void put(int field$, java.lang.Object value$) {
switch (field$) {
case 0: name = (java.lang.CharSequence)value$; break;
case 1: favorite_number = (java.lang.Integer)value$; break;
case 2: favorite_color = (java.lang.CharSequence)value$; break;
default: throw new org.apache.avro.AvroRuntimeException("Bad index");
}
} /**
* Gets the value of the 'name' field.
*/
public java.lang.CharSequence getName() {
return name;
} /**
* Sets the value of the 'name' field.
* @param value the value to set.
*/
public void setName(java.lang.CharSequence value) {
this.name = value;
} /**
* Gets the value of the 'favorite_number' field.
*/
public java.lang.Integer getFavoriteNumber() {
return favorite_number;
} /**
* Sets the value of the 'favorite_number' field.
* @param value the value to set.
*/
public void setFavoriteNumber(java.lang.Integer value) {
this.favorite_number = value;
} /**
* Gets the value of the 'favorite_color' field.
*/
public java.lang.CharSequence getFavoriteColor() {
return favorite_color;
} /**
* Sets the value of the 'favorite_color' field.
* @param value the value to set.
*/
public void setFavoriteColor(java.lang.CharSequence value) {
this.favorite_color = value;
} /** Creates a new User RecordBuilder */
public static example.avro.User.Builder newBuilder() {
return new example.avro.User.Builder();
} /** Creates a new User RecordBuilder by copying an existing Builder */
public static example.avro.User.Builder newBuilder(example.avro.User.Builder other) {
return new example.avro.User.Builder(other);
} /** Creates a new User RecordBuilder by copying an existing User instance */
public static example.avro.User.Builder newBuilder(example.avro.User other) {
return new example.avro.User.Builder(other);
} /**
* RecordBuilder for User instances.
*/
public static class Builder extends org.apache.avro.specific.SpecificRecordBuilderBase<User>
implements org.apache.avro.data.RecordBuilder<User> { private java.lang.CharSequence name;
private java.lang.Integer favorite_number;
private java.lang.CharSequence favorite_color; /** Creates a new Builder */
private Builder() {
super(example.avro.User.SCHEMA$);
} /** Creates a Builder by copying an existing Builder */
private Builder(example.avro.User.Builder other) {
super(other);
if (isValidValue(fields()[0], other.name)) {
this.name = data().deepCopy(fields()[0].schema(), other.name);
fieldSetFlags()[0] = true;
}
if (isValidValue(fields()[1], other.favorite_number)) {
this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);
fieldSetFlags()[1] = true;
}
if (isValidValue(fields()[2], other.favorite_color)) {
this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);
fieldSetFlags()[2] = true;
}
} /** Creates a Builder by copying an existing User instance */
private Builder(example.avro.User other) {
super(example.avro.User.SCHEMA$);
if (isValidValue(fields()[0], other.name)) {
this.name = data().deepCopy(fields()[0].schema(), other.name);
fieldSetFlags()[0] = true;
}
if (isValidValue(fields()[1], other.favorite_number)) {
this.favorite_number = data().deepCopy(fields()[1].schema(), other.favorite_number);
fieldSetFlags()[1] = true;
}
if (isValidValue(fields()[2], other.favorite_color)) {
this.favorite_color = data().deepCopy(fields()[2].schema(), other.favorite_color);
fieldSetFlags()[2] = true;
}
} /** Gets the value of the 'name' field */
public java.lang.CharSequence getName() {
return name;
} /** Sets the value of the 'name' field */
public example.avro.User.Builder setName(java.lang.CharSequence value) {
validate(fields()[0], value);
this.name = value;
fieldSetFlags()[0] = true;
return this;
} /** Checks whether the 'name' field has been set */
public boolean hasName() {
return fieldSetFlags()[0];
} /** Clears the value of the 'name' field */
public example.avro.User.Builder clearName() {
name = null;
fieldSetFlags()[0] = false;
return this;
} /** Gets the value of the 'favorite_number' field */
public java.lang.Integer getFavoriteNumber() {
return favorite_number;
} /** Sets the value of the 'favorite_number' field */
public example.avro.User.Builder setFavoriteNumber(java.lang.Integer value) {
validate(fields()[1], value);
this.favorite_number = value;
fieldSetFlags()[1] = true;
return this;
} /** Checks whether the 'favorite_number' field has been set */
public boolean hasFavoriteNumber() {
return fieldSetFlags()[1];
} /** Clears the value of the 'favorite_number' field */
public example.avro.User.Builder clearFavoriteNumber() {
favorite_number = null;
fieldSetFlags()[1] = false;
return this;
} /** Gets the value of the 'favorite_color' field */
public java.lang.CharSequence getFavoriteColor() {
return favorite_color;
} /** Sets the value of the 'favorite_color' field */
public example.avro.User.Builder setFavoriteColor(java.lang.CharSequence value) {
validate(fields()[2], value);
this.favorite_color = value;
fieldSetFlags()[2] = true;
return this;
} /** Checks whether the 'favorite_color' field has been set */
public boolean hasFavoriteColor() {
return fieldSetFlags()[2];
} /** Clears the value of the 'favorite_color' field */
public example.avro.User.Builder clearFavoriteColor() {
favorite_color = null;
fieldSetFlags()[2] = false;
return this;
} @Override
public User build() {
try {
User record = new User();
record.name = fieldSetFlags()[0] ? this.name : (java.lang.CharSequence) defaultValue(fields()[0]);
record.favorite_number = fieldSetFlags()[1] ? this.favorite_number : (java.lang.Integer) defaultValue(fields()[1]);
record.favorite_color = fieldSetFlags()[2] ? this.favorite_color : (java.lang.CharSequence) defaultValue(fields()[2]);
return record;
} catch (Exception e) {
throw new org.apache.avro.AvroRuntimeException(e);
}
}
}
}
四、用Java实现序列化,即写avro文件。
新建一个Java类
public static void main(String[] args) throws IOException {
// 声明并初始化User对象
// 方式一
User user1 = new User();
user1.setName("zhangsan");
user1.setFavoriteNumber(21);
user1.setFavoriteColor(null);
// 方式二 使用构造函数
// Alternate constructor
User user2 = new User("Ben", 7, "red");
// 方式三,使用Build方式
// Construct via builder
User user3 = User.newBuilder()
.setName("Charlie")
.setFavoriteColor("blue")
.setFavoriteNumber(null)
.build();
String path = "D:\\tmp\\user.avro"; // avro文件存放目录
DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class);
DataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter);
dataFileWriter.create(user1.getSchema(), new File(path));
// 把生成的user对象写入到avro文件
dataFileWriter.append(user1);
dataFileWriter.append(user2);
dataFileWriter.append(user3);
dataFileWriter.close();
}
run一下代码,查看指定文件目录是否生成了avro文件。
五、Java读取avro文件,即实现avro反序列化。
public static void main(String[] args) throws IOException {
DatumReader<User> reader = new SpecificDatumReader<User>(User.class);
DataFileReader<User> dataFileReader = new DataFileReader<User>(new File("D:\\tmp\\user.avro"), reader);
User user = null;
while (dataFileReader.hasNext()) {
user = dataFileReader.next();
System.out.println(user);
}
}
运行结果:
{"name": "zhangsan", "favorite_number": 21, "favorite_color": null}
{"name": "Ben", "favorite_number": 7, "favorite_color": "red"}
{"name": "Charlie", "favorite_number": null, "favorite_color": "blue"}
至此,例子写完。
Java读写avro例子的更多相关文章
- java 读写文件例子
在linux下可以读写中文 import java.io.*; import java.text.SimpleDateFormat; import java.util.*; public class ...
- java读写文件大全
java读写文件大全 最初java是不支持对文本文件的处理的,为了弥补这个缺憾而引入了Reader和Writer两个类,这两个类都是抽象类,Writer中 write(char[] ch,int o ...
- Java读写文本文件操作
package com.test; import java.io.BufferedReader; import java.io.BufferedWriter; import java.io.File; ...
- java 读写word java 动态写入 模板文件
import java.io.ByteArrayOutputStream; import java.io.File; import java.io.FileInputStream; import ja ...
- Java读写文件方法总结
Java读写文件方法总结 Java的读写文件方法在工作中相信有很多的用处的,本人在之前包括现在都在使用Java的读写文件方法来处理数据方面的输入输出,确实很方便.奈何我的记性实在是叫人着急,很多时候既 ...
- Java读写文件的几种方式
自工作以后好久没有整理Java的基础知识了.趁有时间,整理一下Java文件操作的几种方式.无论哪种编程语言,文件读写操作时避免不了的一件事情,Java也不例外.Java读写文件一般是通过字节.字符和行 ...
- Java正则表达式匹配例子
Java正则表达式匹配例子 package com.ibm.test; import java.util.regex.Matcher; import java.util.regex.Pattern; ...
- Java读写Windows共享文件夹 .
版权声明:本文为博主原创文章,未经博主允许不得转载. 项目常常需要有访问共享文件夹的需求,例如共享文件夹存储照片.文件等.那么如何使用Java读写Windows共享文件夹呢? Java可以使用JCIF ...
- 使用java注解的例子有没有
使用java注解的例子 参考文档:http://www.cnblogs.com/pepcod/archive/2013/02/20/2918719.html http://www.shaoqun.co ...
随机推荐
- hadoop集群部署
1) 安装jdk 下载jdk-6u21-linux-i586.bin 然后修改/etc/profile: export JAVA_HOME=/usr/local/jdk export CLASSPAT ...
- docker容器内外相互拷贝数据
从宿主机上拷贝文件到容器内 注意:一下红色字体为宿主机文件 docker cp 文件 容器名:目录 # docker cp httpd-2.4.34.tar.gz node4:/opt/ 从宿主机上拷 ...
- win10以上系统设定PPTP自动拨号
:bohaorasdial adsl 123 123if not %errorlevel% == 0 goto :bohaoexit rasdial adsl 123 123 rasdial是开始拨号 ...
- gulp 编译es6 探究
1.gulp配置: var gulp = require('gulp') var fs = require("fs") var babelify = require('babeli ...
- concurrent.futures进线程池和协程
concurrent.futures 异步执行进程线程池的模块,一个抽象类,定义submit,map,shutdown方法 from concurrent.futures import Process ...
- Mybatis-Plus3.0入门手册
Mybatis-Plus3.0入门手册 ref: https://blog.csdn.net/moshowgame/article/details/81008485 Mybatis-Plus简介 ...
- java list按照 对象 指定多个字段属性进行排序
ListUtils.Java---功能类 package PjectUtils; import java.lang.reflect.Field; import java.text.NumberForm ...
- jmeter获取token并请求失败Internal authentication failed 400
jmeter访问token报错400 1.请求token地址 2.运行jmeter报错-run 3.400的意思是: 400(错误请求) 服务器不理解请求的语法. 4.报错信息如下 {"er ...
- WPF 交替行背景属性
交替行背景色:RowBackground奇数行,AlternatingRowBackground偶数行 <!--#region 表格--> <DataGrid x:Name=&quo ...
- QSetting 说明和简单使用
今天看到服务端代码有一个QSetting.一开始以为是STL模板中的Set(弄到QT中改了个名字而已).仔细一看吓一跳,不是STL模板.是qt特有的一个类. 用来保存或读取一些配置信息用的.看了后,感 ...