Windows 10 上编译 Hadoop
下载源码
源码下载地址(Source download):https://hadoop.apache.org/releases.html
这里以 2.9.2 为例,查看源码中的编译说明文件 BUILDING.txt,截取 windows 部分
Requirements: * Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site') Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported.
环境准备
1.JDK
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
# 设置系统环境变量
setx /M JAVA_HOME "D:\hadoop\jdk1.8.0_192"
setx /M Path "%Path%;%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin"
2.Maven
http://maven.apache.org/download.cgi
# 设置系统环境变量
setx /M M2_HOME "D:\hadoop\apache-maven-3.6.1"
setx /M Path "%Path%;%M2_HOME%\bin"
conf\settings.xml 仓库配置
<!-- 本地仓库路径 -->
<localRepository>D:\hadoop\repo</localRepository> <!-- 网络仓库地址 -->
<mirrors>
<mirror>
<id>central</id>
<mirrorOf>central</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/central</url>
</mirror>
<mirror>
<id>apache.snapshots.https</id>
<mirrorOf>apache.snapshots.https</mirrorOf>
<name>aliyunmaven</name>
<url>https://maven.aliyun.com/repository/apache-snapshots</url>
</mirror>
</mirrors>
3.ProtocolBuffer
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
注:除了下载 protobuf 源码外,还需要下载相应版本的编译过的用于 Windows 平台的 protoc 命令(protoc-2.5.0-win32.zip),该命令用于将 .proto 文件转化为 Java 或 C++ 源文件。
将两个压缩包解压,然后将 protoc.exe 复制到 protobuf-2.5.0\src 目录下
# 安装 ProtocolBuffer
cd D:\hadoop\protobuf-2.5.0\java
mvn test
mvn install # 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\protobuf-2.5.0\src" # 查看版本
protoc --version
4.CMake
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\cmake-3.14.3-win64-x64\bin"
5.Visual Studio 2010 Professional
文件名 cn_visual_studio_2010_professional_x86_dvd_532145.iso
SHA1 33D323446131AB9565082D65C9C380BBD7FF228F
文件大小 2.41GB
发布时间 2010-05-26
ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/
6.Zlib
使用 MSVC 方式编译,打开开始菜单,在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010),用管理员身份运行
cd D:\hadoop\zlib-1.2.11
nmake -f win32/Makefile.msc # 编译完成后在该目录下可看到 zlib1.dll 文件
设置系统环境变量
setx /M ZLIB_HOME "D:\hadoop\zlib-1.2.11"
7.Git(需要 bash 命令)
# 设置系统环境变量
setx /M Path "%Path%;D:\hadoop\PortableGit\bin"
编译
1.设置 Platform
setx /M Platform "x64"
2.解压源码,开始编译
打开开始菜单,在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010),用管理员身份运行
cd D:\hadoop\hadoop-2.9.2-src
mvn package -Pdist,native-win -DskipTests -Dtar # 编译完成后,编译好的文件在目录 hadoop-2.9.2-src\hadoop-dist\target\ 中
编译失败时多编译几次,或自己更换下 Maven 源再试,一些依赖的下载不是很稳定。附上编译成功的日志
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.205 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.259 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.421 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.064 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.279 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.839 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.375 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.167 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.589 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.674 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.209 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.911 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.163 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.792 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.042 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.045 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 5.462 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.134 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.323 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.160 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.593 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 2.615 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.280 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.254 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.127 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 5.503 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.260 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.355 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.324 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.477 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.219 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.615 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.513 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 5.923 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.596 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.338 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.047 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.554 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.049 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 55:18 min
[INFO] Finished at: 2019-04-25T22:47:26+08:00
[INFO] ------------------------------------------------------------------------
Windows 10编译Hadoop 2.6.0源码 - 作业部落 Cmd Markdown 编辑阅读器 (快照)
https://www.cnblogs.com/guoxiaoqian/p/4328812.html
Windows 10 上编译 Hadoop的更多相关文章
- 在Windows 10上安装Oracle 11g数据库出现的问题及解决
在Windows 10上安装Oracle 11g数据库,并且很多次出现过:当安装的进度条进行到快要结束的时候弹出一个提示框.如下: [Java(TM)2 Platform Standard Editi ...
- Windows 10 上,Edge 浏览器不支持插件,因此将不运行 Java
在 Windows 10 上,Edge 浏览器不支持插件,因此将不运行 Java.微软想干嘛?
- DB 查询分析器 6.04 在 Windows 10 上的安装与运行展示
DB查询分析器 6.04 在 Windows 10 上的安装与运行展示 中国本土程序员马根峰(CSDN专访马根峰:海量数据处理与分析大师的中国本土程序员 http://www.csdn.net/art ...
- 如何在Windows 10上运行Docker和Kubernetes?
如何在Windows 10上运行Docker和Kubernetes? 在Windows上学习Docker和Kubernetes,开始的时候会让你觉得无从下手.最起码安装好这些软件都不是一件容易的事情. ...
- Windows 10 上的 Git 如何清除密码? Git Credential Manager for Windows
Windows 10 上的 Git 如何清除密码? 因为一台新的电脑是 Windows 10 在第一次使用 Git 要求输入密码时把密码给输错了. 之前提交都是说 Token 错了,不再出现提示密码. ...
- Windows 10 上安装 3D Studio Max 2016 报错的解决办法
在 Windows 10 上安装 3D Stuido Max 2016 报错,无法正常安装,查看日志是 VC 运行时安装错误,经过分析发现在 Windows 10 上已经有这些运行时并且版本比安装包中 ...
- 如何在 Windows 10 上安装 WSL 2
翻译自 Joey Sneddon 2020年10月30日的文章<How to Install WSL 2 on Windows 10> [1] 如果您想在最新的 Windows 版本中尝试 ...
- windows 10 上源码编译OpenCV并支持CUDA | compile opencv with CUDA support on windows 10
本文首发于个人博客https://kezunlin.me/post/6580691f/,欢迎阅读! compile opencv with CUDA support on windows 10 Ser ...
- windows 10上源码编译dlib教程 | compile dlib on windows 10
本文首发于个人博客https://kezunlin.me/post/654a6d04/,欢迎阅读! compile dlib on windows 10 Series Part 1: compile ...
随机推荐
- Android探究之View的绘制流程
Android中Activity是作为应用程序的载体存在,代表着一个完整的用户界面,提供了一个窗口来绘制各种视图,当Activity启动时,我们会通过setContentView方法来设置一个内容视图 ...
- 阿里云OCR图片转换成文字识别调用
using Newtonsoft.Json; using System; using System.Collections.Generic; using System.Drawing; using S ...
- ChromeDriver截图
一.NuGet安装Selenium.Chrome.WebDriver和Selenium.WebDriver 二.将packages\Selenium.Chrome.WebDriver.2.45\dri ...
- RobotFramework第一篇之环境搭建
定义:是一款python编写的功能自动化测试框架,具备良好的扩展性,可以进行分布性测试 1:对编程能力要求低,容易上手 2:关键字调用方式,已经定义好的功能,只需要去调用它,一个关键字实现了一个功能, ...
- Linux基础学习:目录配置
Linux 目录配置标准:FHS FHS:即Filesystem Hierarchy Stabdard,文件配置标准.其主要目的是希望用户可以了解已安装软件通常放置在哪个目录下. FHS根据文件系统使 ...
- easyui datagrid 表头固定(垂直滚动条)、列固定(水平滚动条)
easyui datagrid 表头固定(垂直滚动条).列固定(水平滚动条),每页显示1000行 最近用多了easyui 之后还是觉得它的功能还是很强大的.它原有的功能就已经能够满足90%以上的界面需 ...
- 11 Django RESTful framework 实现缓存
01-安装 pip install drf-extensions 02-导入 from rest_framework_extensions.cache.mixins import CacheRespo ...
- zoj 3605
链接 [https://vjudge.net/contest/293343#problem/F] 题意 就是有n碗,有一个宝石,知道开始宝石在那个碗下面 进行M次交换,但知道其中的k次,问你最可能在那 ...
- Java规则之条件语句中做空判断时使用||和&&常犯的错误
错误代码示例: public String bar(String string) { //error 1 if (string!=null || !string.equals("" ...
- ArrayBlockingQueue源码分析
ArrayBlockingQueue是一个基于数组实现的有界的阻塞队列. 属性 //底层存储元素的数组.为final说明一旦初始化,容量不可变,所以是有界的. final Object[] items ...