把Flume的Source设置为 Spooling directory source
把Flume的Source设置为 Spooling directory source,在设定的目录下放置需要读取的文件,一些文件在读取过程中会报错。
文件格式和报错如下:
实验一
读取汉子和“;"符号的TXT文档
会报一下错误
- 2015-11-06 22:16:02,386 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /home/hadoop/flumeSpool-2 }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
- java.nio.charset.MalformedInputException: Input length = 1
- at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
- at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:195)
实验二
读取数字和”;“符号的TXT文档
报错
- 2015-11-06 22:22:49,012 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /home/hadoop/flumeSpool-3 }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
- java.nio.charset.MalformedInputException: Input length = 1
- at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
- at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:195)
此报错与前面的报错相同。
实验三
读取数字和”回车符“的TXT文档
报错:
- 2015-11-06 22:28:06,889 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /home/hadoop/flumeSpool-7 }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
- java.nio.charset.MalformedInputException: Input length = 1
- at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
- at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:195)
实验四
读取数字,字母,汉字组成的csv文件
报错:
- 2015-11-06 22:40:34,781 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /home/hadoop/flumeSpool-6 }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
- java.nio.charset.MalformedInputException: Input length = 1
- at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
- at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:195)
- at org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:133)
- at org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:71)
实验五
读取数字和汉字组成的TXT文件
报错:
- 2015-11-06 23:34:29,455 (pool-3-thread-1) [ERROR - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:256)] FATAL: Spool Directory source r1: { spoolDir: /home/hadoop/flumeSpool-10 }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
- java.nio.charset.MalformedInputException: Input length = 1
- <span style="white-space:pre"> </span>at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
- <span style="white-space:pre"> </span>at org.apache.flume.serialization.ResettableFileInputStream.readChar(ResettableFileInputStream.java:195)
- <span style="white-space:pre"> </span>at org.apache.flume.serialization.LineDeserializer.readLine(LineDeserializer.java:133)
- <span style="white-space:pre"> </span>at org.apache.flume.serialization.LineDeserializer.readEvent(LineDeserializer.java:71)
- <span style="white-space:pre"> </span>at org.apache.flume.serialization.LineDeserializer.readEvents(LineDeserializer.java:90)
- <span style="white-space:pre"> </span>at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:252)
- <span style="white-space:pre"> </span>at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
- <span style="white-space:pre"> </span>at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
- <span style="white-space:pre"> </span>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
- <span style="white-space:pre"> </span>at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
成功的实验一
一排数字,什么符号都没有
成功的实验二
如上所示,不缺定是否使用了”回车符“。
把Flume的Source设置为 Spooling directory source的更多相关文章
- Spooling Directory Source使用技巧
1.使用文件原来的名字 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 a1.sources=r1 a1.sinks=k1 a1.sources.r1.ty ...
- Flume-Spooling Directory Source 监控目录下多个新文件
使用 Flume 监听整个目录的文件,并上传至 HDFS. 一.创建配置文件 flume-dir-hdfs.conf https://flume.apache.org/FlumeUserGuide.h ...
- gen already exists but is not a source folder. Convert to a source folder or rename it.
异常提示: gen already exists but is not a source folder. Convert to a source folder or rename it. 错误原因 ...
- 将Samba设置为Active Directory域控制器
一 简介 从版本4.0开始,samba可以作为Active Directory(AD)域控制器(DC)运行,如果在生产环境中安装samba,建议运行两个或者多个DC用于故障转移 本文介绍如何让将一个S ...
- Flume的Avro Sink和Avro Source研究之一: Avro Source
问题 : Avro Source提供了怎么样RPC服务,是怎么提供的? 问题 1.1 Flume Source是如何启动一个Netty Server来提供RPC服务. 由GitHub上avro-rpc ...
- flume 1.4 设置FLUME_CONF_DIR没用
今天安装Apache Flume 1.4的时候,发现设置了FLUME_CONF_DIR环境变量和flume-env.sh后,运行flume-ng时flume-env.sh中的JAVA_HOME没有生效 ...
- Configuration must specify a spooling directory
启动spooling源时报错: 原因:spooling配置文件有误 a1.sources.r1.type = spooldir a1.sources.r1.spooldir = /usr/local/ ...
- WPF Image Source 设置相对路径图片
BitmapImage bt = new BitmapImage(new Uri("Images\\3_u10484.png", UriKind.Relative));this.I ...
- eclipse调试(debug)的时候,出现Source not found,Edit Source Lookup Path,一闪而过
问题描述 使用Eclipse调试代码的时候,打了断点,经常出现Source not found,网上找了半天,大部分提示点击Edit Source Lookup Path,添加被调试的工程,然而往往没 ...
随机推荐
- 转载 在 Linux 虚拟机中手动安装或升级 VMware Tools
http://pubs.vmware.com/workstation-12/index.jsp?lang=zh_CN&topic=/com.vmware.ws.using.doc/GUID-0 ...
- 使用Canvas把照片转换成素描画
原文:http://www.alloyteam.com/2012/07/convert-picture-to-sketch-by-canvas/ 腾讯的alloy team写的一个素描效果,挺不错的. ...
- JedisPoolConfig配置
JedisPoolConfig config = new JedisPoolConfig(); //连接耗尽时是否阻塞, false报异常,ture阻塞直到超时, 默认true config. ...
- C#调用webService的几种方法
转自: WebClient 用法小结 http://www.cnblogs.com/hfliyi/archive/2012/08/21/2649892.html http://www.cnblogs. ...
- 查找DOM
<!doctype html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
- (剑指Offer)面试题30:最小的k个数
题目: 输入n个整数,找出其中最小的K个数.例如输入4,5,1,6,2,7,3,8这8个数字,则最小的4个数字是1,2,3,4,. 思路: 1.排序 把输入的n个整数排序,然后取前k个数: 时间复杂度 ...
- css3 过度效果之物体向上冒出
<!DOCTYPE HTML> <html> <head> <meta charset="utf-8"/> <meta nam ...
- hdu4291之矩阵快速幂
A Short problem Time Limit: 2000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others) ...
- Objective-C运行时编程 - 方法混写 Method Swizzling
摘要: 本文描述方法混写对实例.类.父类.不存在的方法等情况处理,属于Objective-C(oc)运行时(runtime)编程范围. 编程环境:Xcode 6.1.1, Yosemite,iOS 8 ...
- C++的优秀特性3:构造函数和析构函数
(转载请注明原创于潘多拉盒子) 构造函数和析构函数是C++中再熟悉不过的概念了,几乎每个了解一点C++的人都知道这两个概念是什么意思.一个对象的全部生命期中构造函数和析构函数执行的时机如下: 1. 为 ...