Windows 10 上编译 Hadoop
下载源码
源码下载地址（Source download）：https://hadoop.apache.org/releases.html
这里以 2.9.2 为例，查看源码中的编译说明文件 BUILDING.txt，截取 windows 部分
Requirements: * Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site') Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported.
环境准备
1.JDK
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
# 设置系统环境变量
setx /M JAVA_HOME "D:\hadoop\jdk1.8.0_192"
setx /M Path "%Path%;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin"
2.Maven
http://maven.apache.org/download.cgi
# 设置系统环境变量
setx /M M2_HOME "D:\hadoop\apache-maven-3.6.1"
setx /M Path "%Path%;%M2_HOME%\bin"
conf\settings.xml 仓库配置
<!-- 本地仓库路径 -->
<localRepository>D:\hadoop\repo</localRepository> <!-- 网络仓库地址 -->
<mirrors>
<mirror>
<id>central</id>
<mirrorOf>central</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/central</url>
</mirror>
<mirror>
<id>apache.snapshots.https</id>
<mirrorOf>apache.snapshots.https</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/apache-snapshots</url>
</mirror>
</mirrors>
3.ProtocolBuffer
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
注：除了下载 protobuf 源码外，还需要下载相应版本的编译过的用于 Windows 平台的 protoc 命令（protoc-2.5.0-win32.zip）,该命令用于将 .proto 文件转化为 Java 或 C++ 源文件。
将两个压缩包解压，然后将 protoc.exe 复制到 protobuf-2.5.0\src 目录下
# 安装 ProtocolBuffer
cd D:\hadoop\protobuf-2.5.0\java
mvn test
mvn install # 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\protobuf-2.5.0\src" # 查看版本
protoc --version
4.CMake
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\cmake-3.14.3-win64-x64\bin"
5.Visual Studio 2010 Professional
文件名 cn_visual_studio_2010_professional_x86_dvd_532145.iso
SHA1 33D323446131AB9565082D65C9C380BBD7FF228F
文件大小 2.41GB
发布时间 2010-05-26
ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/
6.Zlib
使用 MSVC 方式编译，打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行
cd D:\hadoop\zlib-1.2.11
nmake -f win32/Makefile.msc # 编译完成后在该目录下可看到 zlib1.dll 文件
设置系统环境变量
setx /M ZLIB_HOME "D:\hadoop\zlib-1.2.11"
7.Git（需要 bash 命令）
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\PortableGit\bin"
编译
1.设置 Platform
setx /M Platform "x64"
2.解压源码，开始编译
打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行
cd D:\hadoop\hadoop-2.9.2-src
mvn package -Pdist,native-win -DskipTests -Dtar # 编译完成后，编译好的文件在目录 hadoop-2.9.2-src\hadoop-dist\target\ 中
编译失败时多编译几次，或自己更换下 Maven 源再试，一些依赖的下载不是很稳定。附上编译成功的日志
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.205 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.259 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.421 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.064 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.279 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.839 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.375 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.167 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.589 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.674 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.209 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.911 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.163 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.792 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.042 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 5.462 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.134 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.323 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.160 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.593 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 2.615 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.280 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.254 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.127 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 5.503 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.260 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.355 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.324 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.477 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.219 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.615 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.513 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 5.923 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.596 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.338 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.047 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.554 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.049 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55:18 min
[INFO] Finished at: 2019-04-25T22:47:26+08:00
[INFO] ------------------------------------------------------------------------
Windows 10编译Hadoop 2.6.0源码 - 作业部落 Cmd Markdown 编辑阅读器 （快照）
https://www.cnblogs.com/guoxiaoqian/p/4328812.html
Windows 10 上编译 Hadoop的更多相关文章
- 在Windows 10上安装Oracle 11g数据库出现的问题及解决
在Windows 10上安装Oracle 11g数据库,并且很多次出现过:当安装的进度条进行到快要结束的时候弹出一个提示框.如下: [Java(TM)2 Platform Standard Editi ...
- Windows 10 上，Edge 浏览器不支持插件，因此将不运行 Java
在 Windows 10 上,Edge 浏览器不支持插件,因此将不运行 Java.微软想干嘛?
- DB 查询分析器 6.04 在 Windows 10 上的安装与运行展示
DB查询分析器 6.04 在 Windows 10 上的安装与运行展示 中国本土程序员马根峰(CSDN专访马根峰:海量数据处理与分析大师的中国本土程序员 http://www.csdn.net/art ...
- 如何在Windows 10上运行Docker和Kubernetes？
如何在Windows 10上运行Docker和Kubernetes? 在Windows上学习Docker和Kubernetes,开始的时候会让你觉得无从下手.最起码安装好这些软件都不是一件容易的事情. ...
- Windows 10 上的 Git 如何清除密码？ Git Credential Manager for Windows
Windows 10 上的 Git 如何清除密码? 因为一台新的电脑是 Windows 10 在第一次使用 Git 要求输入密码时把密码给输错了. 之前提交都是说 Token 错了,不再出现提示密码. ...
- Windows 10 上安装 3D Studio Max 2016 报错的解决办法
在 Windows 10 上安装 3D Stuido Max 2016 报错,无法正常安装,查看日志是 VC 运行时安装错误,经过分析发现在 Windows 10 上已经有这些运行时并且版本比安装包中 ...
- 如何在 Windows 10 上安装 WSL 2
翻译自 Joey Sneddon 2020年10月30日的文章<How to Install WSL 2 on Windows 10> [1] 如果您想在最新的 Windows 版本中尝试 ...
- windows 10 上源码编译OpenCV并支持CUDA | compile opencv with CUDA support on windows 10
本文首发于个人博客https://kezunlin.me/post/6580691f/,欢迎阅读! compile opencv with CUDA support on windows 10 Ser ...
- windows 10上源码编译dlib教程 | compile dlib on windows 10
本文首发于个人博客https://kezunlin.me/post/654a6d04/,欢迎阅读! compile dlib on windows 10 Series Part 1: compile ...
随机推荐
- python在sqlite动态创建表源码
代码之余,将开发过程中经常用的代码片段备份一下,如下的代码是关于python在sqlite动态创建表的代码,应该能对各位有所用. import sqlite3 as db conn = db.conn ...
- Linux(CentOS7)压缩和解压缩war包、tar包、tar.gz包命令
一.Linux版本 二.解压缩.tar.gz包到当前目录 tar -xzvf apache-tomcat-7.0.90.tar.gz 三.将指定文件压缩成.tar.gz包 tar -czf apach ...
- LeetCode算法题-Reach a Number（Java实现）
这是悦乐书的第310次更新,第331篇原创 01 看题和准备 今天介绍的是LeetCode算法题中Easy级别的第179题(顺位题号是754).你站在无限数字线的0号位置.在目的地有个target.在 ...
- 用Redis管理Session
maven <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</ar ...
- linux下安装多个Tomcat
编辑环境变量:vi /etc/profile 加入以下代码 ##########first tomcat########### CATALINA_BASE=/usr/local/src/tomcat ...
- OpenStack之Fuel架构及其工作原理
一.Fuel架构 Fuel是一个开源的OpenStack部署和管理的工具,它允许用户通过Web UI进行OpenStack的部署和管理,并可以通过插件来扩展. Fuel的目标:加快和简化不同配置的 ...
- js 简单弹框toast
新建toast.js文件 function Toast(msg,duration){ duration=isNaN(duration)?3000:duration; var m = document. ...
- Net包管理NuGet(3)搭建私服及引用私服的包
1,打开vs创建项目(ASP.NET WEB空项目)假设命名为MyNuGet 空项目解决方案如图 2,右键引用>管理NuGet程序包>切到浏览搜索NuGet.Server然后安装(3.1. ...
- ansible 模块 分享
A a10_server 管理A10 Networks AX / SoftAX / Thunder / vThunder设备 a10_service_group 管理A10网络设备的服务组 a10_v ...
- AOP - 1 基本概念
1.AOP (面向切面编程) AOP,Aspect Oriented Programming,意为:面向切面编程, 通过预编译方式和运行期间动态代理实现程序功能的统一维护的一种技术, AOP是OOP的 ...