Install hadoop on windows(non-virtual machine, such cygwin)
Download
Before starting make sure you have this two softwares
Extract downloaded tar file
Configuration
Step 1 – Windows path configuration
set HADOOP_HOME path in enviornment variable for windows
Right click on my computer > properties > advanced system settings > advance tab > environment variables > click on new
Set hadoop bin directory path
Find path variable in system variable > click on edit > at the end insert ‘; (semicolon)’ and paste path upto hadoop bin directory in my case it’s a
F:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/bin
Step 2 – Hadoop configuration
Edit hadoop-2.7.1/etc/hadoop/core-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/mapred-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/hdfs-site.xml, paste the following lines and save it, please create data folder somewhere and in my case i have created it in myHADOOP_HOME directory
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/f:/Hortanwork/1gbhadoopram/Software/hadoop-2.7/hadoop-2.7.1/data/datanode</value>
</property>
</configuration>
OR
Edit hadoop-2.7.1/etc/hadoop/yarn-site.xml, paste the following lines and save it.
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
Edit hadoop-2.7.1/etc/hadoop/hadoop-env.cmd, comment existing%JAVA_HOME% using @rem at start, give proper path and save it. (my jdk is in program files to avoid spaces i gave PROGRA~1)
Demo
Step 3 – Start everything
Very Important step!!!!
3.1) Before starting everything you need to add some [dot].dll and [dot].exe files of windows please download bin folder from my github repository –sardetushar_gitrepo_download
or download from https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin
3.2) copy all the files dowloaded into the %HADOOP_HOME%\bin folder (The download file contains .dll and .exe file (winutils.exe for hadoop 2.7.1) which adapte for windows os).
3.3) Open cmd and type ‘hdfs namenode -format’ – after execution you will see below logs
3.4) Open cmd and point to sbin directory and type ‘start-all.cmd’
C:\UserDefined\BigData\hadoop-2.7.1\sbin>start-all.cmd
It will start following process
Namenode
Datanode
YARN resourcemanager
YARN nodemanager
JPS – to see services are running
open cmd and type – jps (for jps make sure your java path is set properly)
GUI
Step 4 – namenode GUI, resourcemanager GUI
Resourcemanager GUI address – http://localhost:8088
Namenode GUI address – http://localhost:50070
In next tutorial we will see how to run mapreduce programs in windows using eclipse and this hadoop setup
Install hadoop on windows(non-virtual machine, such cygwin)的更多相关文章
- [New Portal]Windows Azure Virtual Machine (11) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (1)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (12) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (2)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (13) 在本地使用Hyper-V制作虚拟机模板,并上传至Azure (3)
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,作为自定义的虚拟机模板. 注意:因为在制作VHD的最 ...
- [New Portal]Windows Azure Virtual Machine (14) 在本地制作数据文件VHD并上传至Azure(1)
<Windows Azure Platform 系列文章目录> 之前的内容里,我介绍了如何将本地的Server 2012中文版 VHD上传至Windows Azure,并创建基于该Serv ...
- [New Portal]Windows Azure Virtual Machine (16) 使用Azure PowerShell创建Azure Virtual Machine
<Windows Azure Platform 系列文章目录> 注:本章内容和之前的[New Portal]Windows Azure Virtual Machine (12) 在本地制作 ...
- [New Portal]Windows Azure Virtual Machine (18) Azure Virtual Machine内部IP和外部IP
<Windows Azure Platform 系列文章目录> 在开始本章内容之前,请读者熟悉以下2篇博文: [New Portal]Windows Azure Virtual ...
- [New Portal]Windows Azure Virtual Machine (19) 关闭Azure Virtual Machine与VIP Address,Internal IP Address的关系(1)
<Windows Azure Platform 系列文章目录> 默认情况下,通过Azure Management Portal创建的Public IP和Private IP都是随机分配的. ...
- [New Portal]Windows Azure Virtual Machine (21) 将本地Hyper-V的VM上传至Windows Azure Virtual Machine
<Windows Azure Platform 系列文章目录> 本章介绍的内容是将本地Hyper-V的VHD,上传到Azure数据中心,并且保留OS中的内容. 注意:笔者没有执行Syspr ...
- [New Portal]Windows Azure Virtual Machine (22) 使用Azure PowerShell,设置Virtual Machine Endpoint
<Windows Azure Platform 系列文章目录> 我们可以通过Windows Azure Management Portal,打开Virtual Machine的Endpoi ...
- [New Portal]Windows Azure Virtual Machine (23) 使用Storage Space,提高Virtual Machine磁盘的IOPS
<Windows Azure Platform 系列文章目录> 注意:如果使用Azure Virtual Machine,虚拟机所在的存储账号建议使用Local Redundant.不建议 ...
随机推荐
- xxx/labelKeypoint/utils/qt.py:81: RuntimeWarning: invalid value encountered in double_scalars
原代码: return np.linalg.norm(np.cross(p2 - p1, p1 - p3)) / np.linalg.norm(p2 - p1) 出现报错: xxx/labelKeyp ...
- error: QXcbConnection: Could not connect to display
/********************************************************************************* * error: QXcbConn ...
- 解决安装vmware-tools出现的“The path "" is not a valid path to the 3.2.0-4-amd64 kernel headers”问题
在用虚拟机安装使用64位Crunchbang(一种Debian GNU/Linux 的linux)的过程中出现很多小问题.其中vmware-tools安装就是第一个问题. 在使用终端安装vmware- ...
- ioctl命令
_IO._IOR._IOW._IOWR 宏的使用说明 驱动程序中 ioctl 函数传递的变量 cmd 是应用程序向驱动程序请求处理的命令.cmd 除了用于区别不同命令的数值,还可包含有助于处理的几种 ...
- Connection: keep-alive,Content-Length,Transfer-Encoding: chunked,Content-Encoding: gzip等
Transfer-Encoding,是一个 HTTP 头部字段,字面意思是「传输编码」. 用来改变报文格式,它不但不会减少实体内容传输大小,甚至还会使传输变大,那它的作用是什么呢?下面进行介绍. HT ...
- admin.ModelAdmin 后台管理关联对象,某个字段怎么显示值
admin.ModelAdmin 后台管理关联对象,某个字段如何显示值?对象 WxpAccount: accountName = ... 对象 AccountMenu: ...
- Oracle根据表的大小排序SQL语句
--按照数据行数排序select table_name,blocks,num_rows from dba_tables where owner not like '%SYS%' and table_n ...
- 求幂pow()
pow(x, n) 求x的n次方. 最简单的方法便是计算n个x相乘 public static double pow(double x, int n) { if (n == 0) return 1; ...
- 【shell】grep命令
1.作用Linux系统中grep命令是一种强大的文本搜索工具,它能使用正则表达式搜索文本,并把匹 配的行打印出来.grep全称是Global Regular Expression Print,表示全局 ...
- qt creator 快捷键 (一)
F1 查看帮助F2 跳转到函数定义(和Ctrl+鼠标左键一样的效果)Shift+F2 声明和定义之间切换F4 头文件和源文件之间切换Ctrl+1 ...