我想还有很多人没有听说过ZModem协议,更不知道有rz/sz这样方便的工具。 好东西不敢独享。以下给出我知道的一点皮毛。 下面一段是从SecureCRT的帮助中copy的:

ZModem is a full-duplex file transfer protocol that supports fast data transfer rates and effective error detection. ZModem is very user friendly, allowing either the sending or receiving party to initiate a file transfer. ZModem supports multiple file ("batch") transfers, and allows the use of wildcards when specifying filenames. ZModem also supports resuming most prior ZModem file transfer attempts.

rz,sz是便是Linux/Unix同Windows进行ZModem文件传输的命令行工具 windows端需要支持ZModem的telnet/ssh客户端,SecureCRT就可以用SecureCRT登陆到Unix/Linux主机(telnet或ssh均可) O 运行命令rz,即是接收文件,SecureCRT就会弹出文件选择对话框,选好文件之后关闭对话框,文件就会上传到当前目录 O 运行命令sz file1 file2就是发文件到windows上(保存的目录是可以配置) 比FTP命令方便多了,而且服务器不用再开FTP服务了 PS:Linux上rz/sz这两个小工具安装lrzsz-x.x.xx.rpm即可,Unix可用源码自行 编译,Solaris spac的可以到sunfreeware下载执行码

如果安装的是hadoop-0.20.2,那么eclipse-plugin的具体位置位在:/home/hadoop/hadoop-0.20.2/contrib/eclipse-plugin下面。 
如果安装的是hadoop-0.21.0,那么eclipse-plugin的具体位置位在:/home/hadoop/hadoop-0.21.0/mapred/contrib/eclipse/hadoop-0.21.0-eclipse-plugin.jar下面

将hadoop-0.21.0-eclipse-plugin.jar这个插件保存到eclipse目录下的pluging中,eclipse就能够自动识别。

本机的环境如下:

Eclipse 3.6

Hadoop-0.20.2

Hive-0.5.0-dev

1. 安装hadoop-0.20.2-eclipse-plugin的插件。注意:Hadoop目录中的/hadoop-0.20.2/contrib /eclipse-plugin/hadoop-0.20.2-eclipse-plugin.jar在Eclipse3.6下有问题,无法在 Hadoop Server上运行,可以从http://code.google.com/p/hadoop-eclipse-plugin/下载

2. 选择Map/Reduce视图:window ->  open pers.. ->  other.. ->  map/reduce

3. 增加DFS Locations:点击Map/Reduce Locations—> New Hadoop Loaction,填写对应的host和port

1
2
3
4
5
6
7
8
9
10
  1. Map/Reduce Master:
  2. Host: 10.10.xx.xx
  3. Port: 9001
  4. DFS Master:
  5. Host: 10.10.xx.xx(选中 User M/R Master host即可)
  6. Port: 9000
  7. User name: root
  8. 更改Advance parameters 中的 hadoop.job.ugi, 默认是 DrWho,Tardis, 改成:root,Tardis。如果看不到选项,则使用Eclipse -clean重启Eclipse
  9. 否则,可能会报错org.apache.hadoop.security.AccessControlException

4. 设置本机的Host:

1
2
3
4
5
  1. 10.10.xx.xx zw-hadoop-master. zw-hadoop-master
  2. #注意后面需要还有一个zw-hadoop-master.,否则运行Map/Reduce时会报错:
  3. java.lang.IllegalArgumentException: Wrong FS: hdfs://zw-hadoop-master:9000/user/root/oplog/out/_temporary/_attempt_201008051742_0135_m_000007_0, expected: hdfs://zw-hadoop-master.:9000
  4. at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352)

5. 新建一个Map/Reduce Project,新建Mapper,Reducer,Driver类,注意,自动生成的代码是基于老版本的Hadoop,自己修改:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
  1. <span>package</span> <span>com.sohu.hadoop.test</span><span>;</span>
  2. <span>import</span> <span>java.util.StringTokenizer</span><span>;</span>
  3. <span>import</span> <span>org.apache.hadoop.io.IntWritable</span><span>;</span>
  4. <span>import</span> <span>org.apache.hadoop.io.Text</span><span>;</span>
  5. <span>import</span> <span>org.apache.hadoop.mapreduce.Mapper</span><span>;</span>
  6. <span>public</span> <span>class</span> MapperTest <span>extends</span> Mapper<span><</span>Object, Text, Text, IntWritable<span>></span> <span>{</span>
  7. <span>private</span> <span>final</span> <span>static</span> IntWritable one <span>=</span> <span>new</span> IntWritable<span>(</span><span>1</span><span>)</span><span>;</span>
  8. <span>public</span> <span>void</span> map<span>(</span><span>Object</span> key, Text value, <span>Context</span> context<span>)</span>
  9. <span>throws</span> <span>IOException</span>, <span>InterruptedException</span> <span>{</span>
  10. <span>String</span> userid <span>=</span> value.<span>toString</span><span>(</span><span>)</span>.<span>split</span><span>(</span><span>"[|]"</span><span>)</span><span>[</span><span>2</span><span>]</span><span>;</span>
  11. context.<span>write</span><span>(</span><span>new</span> Text<span>(</span>userid<span>)</span>, <span>new</span> IntWritable<span>(</span><span>1</span><span>)</span><span>)</span><span>;</span>
  12. <span>}</span>
  13. <span>}</span>
  14. <span>package</span> <span>com.sohu.hadoop.test</span><span>;</span>
  15. <span>import</span> <span>java.io.IOException</span><span>;</span>
  16. <span>import</span> <span>org.apache.hadoop.io.IntWritable</span><span>;</span>
  17. <span>import</span> <span>org.apache.hadoop.io.Text</span><span>;</span>
  18. <span>import</span> <span>org.apache.hadoop.mapreduce.Reducer</span><span>;</span>
  19. <span>public</span> <span>class</span> ReducerTest <span>extends</span> Reducer<span><</span>Text, IntWritable, Text, IntWritable<span>></span> <span>{</span>
  20. <span>private</span> IntWritable result <span>=</span> <span>new</span> IntWritable<span>(</span><span>)</span><span>;</span>
  21. <span>public</span> <span>void</span> reduce<span>(</span>Text key, Iterable<span><</span>IntWritable<span>></span> values, <span>Context</span> context<span>)</span>
  22. <span>throws</span> <span>IOException</span>, <span>InterruptedException</span> <span>{</span>
  23. <span>int</span> sum <span>=</span> <span>0</span><span>;</span>
  24. <span>for</span> <span>(</span>IntWritable val <span>:</span> values<span>)</span> <span>{</span>
  25. sum <span>+=</span> val.<span>get</span><span>(</span><span>)</span><span>;</span>
  26. <span>}</span>
  27. result.<span>set</span><span>(</span>sum<span>)</span><span>;</span>
  28. context.<span>write</span><span>(</span>key, result<span>)</span><span>;</span>
  29. <span>}</span>
  30. <span>}</span>
  31. <span>package</span> <span>com.sohu.hadoop.test</span><span>;</span>
  32. <span>import</span> <span>org.apache.hadoop.conf.Configuration</span><span>;</span>
  33. <span>import</span> <span>org.apache.hadoop.fs.Path</span><span>;</span>
  34. <span>import</span> <span>org.apache.hadoop.io.IntWritable</span><span>;</span>
  35. <span>import</span> <span>org.apache.hadoop.io.Text</span><span>;</span>
  36. <span>import</span> <span>org.apache.hadoop.io.compress.CompressionCodec</span><span>;</span>
  37. <span>import</span> <span>org.apache.hadoop.io.compress.GzipCodec</span><span>;</span>
  38. <span>import</span> <span>org.apache.hadoop.mapreduce.Job</span><span>;</span>
  39. <span>import</span> <span>org.apache.hadoop.mapreduce.lib.input.FileInputFormat</span><span>;</span>
  40. <span>import</span> <span>org.apache.hadoop.mapreduce.lib.output.FileOutputFormat</span><span>;</span>
  41. <span>import</span> <span>org.apache.hadoop.util.GenericOptionsParser</span><span>;</span>
  42. <span>public</span> <span>class</span> DriverTest <span>{</span>
  43. <span>public</span> <span>static</span> <span>void</span> main<span>(</span><span>String</span><span>[</span><span>]</span> args<span>)</span> <span>throws</span> <span>Exception</span> <span>{</span>
  44. Configuration conf <span>=</span> <span>new</span> Configuration<span>(</span><span>)</span><span>;</span>
  45. <span>String</span><span>[</span><span>]</span> otherArgs <span>=</span> <span>new</span> GenericOptionsParser<span>(</span>conf, args<span>)</span>
  46. .<span>getRemainingArgs</span><span>(</span><span>)</span><span>;</span>
  47. <span>if</span> <span>(</span>otherArgs.<span>length</span> <span>!=</span> <span>2</span><span>)</span>
  48. <span>{</span>
  49. <span>System</span>.<span>err</span>.<span>println</span><span>(</span><span>"Usage: DriverTest <in> <out>"</span><span>)</span><span>;</span>
  50. <span>System</span>.<span>exit</span><span>(</span><span>2</span><span>)</span><span>;</span>
  51. <span>}</span>
  52. Job job <span>=</span> <span>new</span> Job<span>(</span>conf, <span>"Driver Test"</span><span>)</span><span>;</span>
  53. job.<span>setJarByClass</span><span>(</span>DriverTest.<span>class</span><span>)</span><span>;</span>
  54. job.<span>setMapperClass</span><span>(</span>MapperTest.<span>class</span><span>)</span><span>;</span>
  55. job.<span>setCombinerClass</span><span>(</span>ReducerTest.<span>class</span><span>)</span><span>;</span>
  56. job.<span>setReducerClass</span><span>(</span>ReducerTest.<span>class</span><span>)</span><span>;</span>
  57. job.<span>setOutputKeyClass</span><span>(</span>Text.<span>class</span><span>)</span><span>;</span>
  58. job.<span>setOutputValueClass</span><span>(</span>IntWritable.<span>class</span><span>)</span><span>;</span>
  59. conf.<span>setBoolean</span><span>(</span><span>"mapred.output.compress"</span>, <span>true</span><span>)</span><span>;</span>
  60. conf.<span>setClass</span><span>(</span><span>"mapred.output.compression.codec"</span>, GzipCodec.<span>class</span>,CompressionCodec.<span>class</span><span>)</span><span>;</span>
  61. FileInputFormat.<span>addInputPath</span><span>(</span>job, <span>new</span> Path<span>(</span>otherArgs<span>[</span><span>0</span><span>]</span><span>)</span><span>)</span><span>;</span>
  62. FileOutputFormat.<span>setOutputPath</span><span>(</span>job, <span>new</span> Path<span>(</span>otherArgs<span>[</span><span>1</span><span>]</span><span>)</span><span>)</span><span>;</span>
  63. <span>System</span>.<span>exit</span><span>(</span>job.<span>waitForCompletion</span><span>(</span><span>true</span><span>)</span> <span>?</span> <span>0</span> <span>:</span> <span>1</span><span>)</span><span>;</span>
  64. <span>}</span>
  65. <span>}</span>

6. 在DriverTest上,点击Run As —> Run on Hadoop,选择对应的Hadoop Locaion即可

eclipse安装hadoop插件的更多相关文章

  1. Hadoop学习记录(6)|Eclipse安装Hadoop 插件

    下载 https://skydrive.live.com/redir.aspx?cid=cf7746837803bc50&resid=CF7746837803BC50!1277&par ...

  2. Linux下为Eclipse安装hadoop插件

    前提条件:在Linux系统中已经安装好了jdk和hadoop 本文的安装环境:1.arch Linux 2. hadoop1.0.1本地伪分布模式安装  3. Eclipse 4.5 1. 下载Ecl ...

  3. Eclipse安装Hadoop插件配置Hadoop开发环境

    一.编译Hadoop插件 首先需要编译Hadoop 插件:hadoop-eclipse-plugin-2.6.0.jar,然后才可以安装使用. 第三方的编译教程:https://github.com/ ...

  4. Ubuntu 14.10 下Eclipse安装Hadoop插件

    准备环境 1 安装好了Hadoop,之前安装了Hadoop 2.5.0,安装参考http://www.cnblogs.com/liuchangchun/p/4097286.html 2 安装Eclip ...

  5. Ubuntu13.04 Eclipse下编译安装Hadoop插件及使用小例

    Ubuntu13.04 Eclipse下编译安装Hadoop插件及使用小例 一.在Eclipse下编译安装Hadoop插件 Hadoop的Eclipse插件现在已经没有二进制版直接提供,只能自己编译. ...

  6. Eclipse集成Hadoop插件

    一.Eclipse集成Hadoop插件 1.在这之前我们需要配置真机上的hadoop环境变量 注:在解压tar包的时候普通解压会出现缺文件的现象,所以在这里我们需要用管理员的方式启动我们的解压软件(我 ...

  7. 【Maven】Eclipse安装Maven插件后导致Eclipse启动出错

    本文纯属复制粘贴:具体请参照原文: Eclipse安装Maven插件后,Eclipse启动问题:Maven Integration for Eclipse JDK Warning.  解决方法: 1. ...

  8. Eclipse安装svn插件的几种方式

    Eclipse安装svn插件的几种方式 1.在线安装: (1).点击 Help --> Install New Software... (2).在弹出的窗口中点击add按钮,输入Name(任意) ...

  9. Eclipse安装maven插件报错

    Eclipse安装maven插件,报错信息如下: Cannot complete the install because one or more required items could not be ...

随机推荐

  1. 最优化作业 共轭梯度法 matlab代码

    syms f x1 x2 f=(1/2)*x1^2+x2^2; x=[2;1]; a=[1 0;0 2];% A g1=diff(f,x1); g2=diff(f,x2); g=[g1;g2];%导数 ...

  2. 四、C# 5.0 新特性——Async和Await使异步编程更简单

    一.引言 .NET 4.5 的推出,对于C#又有了新特性的增加--就是C#5.0中async和await两个关键字,这两个关键字简化了异步编程,之所以简化了,还是因为编译器给我们做了更多的工作,下面就 ...

  3. unittest:2 执行多条用例,仅执行一次setUp和tearDown

    对象方法setUp()和tearDown() 每个用例执行前后都会被调用.但是有另外一种场景:setUp之后执行完所有用例,最后调用一次tearDown.比如打开网页,多条用例分别验证网页上的元素正确 ...

  4. 关于javascript的单线程和异步的一些问题

    关于js单线程和异步方面突然就糊涂了,看别人的文章越看越糊涂,感觉这方面是个坑,跳进去就不好跳出来.再去看,看着看着感觉自己明白了一些东西,也不知道对不对,反正是暂时把自己说服了,这样理解能理解的通, ...

  5. python __new__以及__init__

    @[深入Python]__new__和__init__ 1 2 3 4 5 6 7 8 class A(object):     def __init__(self):         print & ...

  6. 通过 lsyncd + rsync 同步文件

    通过rsyncd实现将源服务器上的文件同步到目标服务器,通过lsyncd监控源服务器上的文件是否有变动,若有变动调用rsyncd服务对差异的文件进行同步. 0. lsyncd有三种同步文件的方式: ( ...

  7. LayIM.AspNetCore Middleware 开发日记(二)预备知识介绍

    前言 开发一个AspNetCore的中间件需要理解RequestDelegate.另外,还需要理解.NET Core中的依赖注入.还有一个就是内嵌资源的访问.例如:EmbeddedFileProvid ...

  8. 20165302 预备作业3 Linux安装及学习

    linux系统安装 我在安装VirtualBox时出现了一些小问题,如图 我的电脑只能设置32-bit的Ubuntu版本,但教程上说要选用64-bit的,我通过百度查询得知要进行BIOS设置,设置好后 ...

  9. webpack4配置

    一.安装webpack 需要先在项目中npm init初始化一下,生成package.json 建议node版本安装到8.2以上 // webpack4中除了正常安装webpack之外,需要再单独安一 ...

  10. iOS 二维码生成 记录一下

    #import <CoreImage/CoreImage.h> @property (nonatomic, nonnull , strong) UIImageView *showImage ...