Mac 下编译 Hadoop-2.9.2

系统环境

  1. 系统: Mac OS_10.14.4
  2. maven Apache Maven 3.6.0
  3. jdk: jdk_1.8.0_201
  4. ProtocolBuffer portoc-2.5.0
  5. zlib 1.2.3-lib
  6. OpenSSL 1_0_2r
  7. cmale 3.14.3-win64-x64
  8. Cygwin 2.897_x86_64
  9. hadoop hadoop-2.9.2
  10. snappy:

搭建编译环境


  1. 1.java环境在oracle官网下载安装安装好配置好环境变量
  2. 2.maven我用的是brew直接安装的`brew install maven`
  3. 3.cmake 使用brew安装 `brew install cmake`
  4. 4.openssl 使用brew安装 `brew install openssl`
  5. 5.snappy 使用brew安装 `brew install snappy`
  6. 6.zlib 使用brew 安装 `brew install zlib`
  7. 7.protobuf由于必须使用2.5.0版本需要自行[下载](https://github.com/protocolbuffers/protobuf/releases)安装
  8. 下载后解压后进入protobuf目录
  9. # 配置
  10. $ ./configure --prefix=/usr/local/Cellow/protoc/2.5/
  11. # 编译
  12. $ make
  13. # 安装
  14. $ make install
  15. # 配置环境变量 `~/.bash_profile` 或 `~/.zshrc`
  16. export PATH="/usr/local/Cellow/protoc/2.5/bin:$PATH"
  17. # 验证protoc版本
  18. $ protoc --version

过程中可能遇到的问题

  • 1.[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1

    [ERROR] around Ant part ...... @ 5:153 in /Users/Downloads/hadoop-2.9.2/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

解决办法: 由于这个xml中的编译指令中需要用到一个环境变量OPENSSL_ROOT_DIR由于我们前面没有配置所以他在执行的时候就会报错,需要在~/.bash_profile~/.zshrc中指定OPENSSL_ROOT_DIR的安装目录我的配置是export OPENSSL_ROOT_DIR="/usr/local/Cellor/openssl/1.0.2r"

  • 2.由于OpenSSL爆出有重大的漏洞Mac OS没有使用OpenSSL而是使用了LibreSSl可以在你的Mac下执行which openssl 然后找到/usr/bin/openssl 执行/usr/bin/openssl version 就会出现LibreSSl 2.6.4,但是编译hadoop必须要使用openssl,可以吧openssl的引用改一个名称,但是Mac是不允许你关闭Mac的保护模式但是不建议这样做,我们可以修改系统的shell的环境变量使自己配置的环境变量覆盖系统的环境变量,例如:export PATH="/usr/local/Cellor/openssl/1.0.2r/bin:$PATH"

编译hadoop

  1. # 下载[hadoop源码](https://hadoop.apache.org/releases.html)
  2. # 解压缩后进入hadoop源码路径
  3. # 执行一条maven命令(你的maven需要配置到环境变量中)
  4. $ mvn package -Pdist,native -DskipTests -Dtar
  5. # 编译过程不会太长,但是下载各种依赖就会浪费很长时间,耐心等待!
  6. # 过程中可能会出现各各样的问题我只是列举了我花费比较久时间解决的问题,如果无法解决可以留言。

编译完成

经过漫长的编译终于出现了令人兴奋的画面

  1. [INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
  2. [INFO]
  3. [INFO] Apache Hadoop Main ................................. SUCCESS [ 1.679 s]
  4. [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.158 s]
  5. [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.071 s]
  6. [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.897 s]
  7. [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.297 s]
  8. [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.699 s]
  9. [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.280 s]
  10. [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 2.940 s]
  11. [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.245 s]
  12. [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.548 s]
  13. [INFO] Apache Hadoop Common ............................... SUCCESS [01:05 min]
  14. [INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.707 s]
  15. [INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.596 s]
  16. [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.046 s]
  17. [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.270 s]
  18. [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 46.683 s]
  19. [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 7.154 s]
  20. [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.156 s]
  21. [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 3.215 s]
  22. [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.857 s]
  23. [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.849 s]
  24. [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.100 s]
  25. [INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.073 s]
  26. [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 10.318 s]
  27. [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.624 s]
  28. [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.413 s]
  29. [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.091 s]
  30. [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 8.550 s]
  31. [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 26.322 s]
  32. [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.491 s]
  33. [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 4.873 s]
  34. [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 3.360 s]
  35. [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 16.652 s]
  36. [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.993 s]
  37. [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 4.276 s]
  38. [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.371 s]
  39. [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.464 s]
  40. [INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.747 s]
  41. [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 5.310 s]
  42. [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.894 s]
  43. [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.053 s]
  44. [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.099 s]
  45. [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.366 s]
  46. [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.061 s]
  47. [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.098 s]
  48. [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 7.639 s]
  49. [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.157 s]
  50. [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 15.248 s]
  51. [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 9.780 s]
  52. [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.561 s]
  53. [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 8.028 s]
  54. [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 4.123 s]
  55. [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 4.353 s]
  56. [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.349 s]
  57. [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.550 s]
  58. [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.609 s]
  59. [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.987 s]
  60. [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.530 s]
  61. [INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.456 s]
  62. [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.654 s]
  63. [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.363 s]
  64. [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.833 s]
  65. [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.550 s]
  66. [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.378 s]
  67. [INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.101 s]
  68. [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 6.900 s]
  69. [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.804 s]
  70. [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 7.562 s]
  71. [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.022 s]
  72. [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 9.670 s]
  73. [INFO] Apache Hadoop Client ............................... SUCCESS [ 5.619 s]
  74. [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.638 s]
  75. [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.075 s]
  76. [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 10.540 s]
  77. [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 12.875 s]
  78. [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.670 s]
  79. [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.058 s]
  80. [INFO] Apache Hadoop Distribution ......................... SUCCESS [01:02 min]
  81. [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.473 s]
  82. [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.042 s]
  83. [INFO] ------------------------------------------------------------------------
  84. [INFO] BUILD SUCCESS
  85. [INFO] ------------------------------------------------------------------------
  86. [INFO] Total time: 09:17 min
  87. [INFO] Finished at: 2019-05-06T05:03:29-07:00
  88. [INFO] ------------------------------------------------------------------------
  89. MacBookPro:hadoop-2.9.2-src leone$

大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target目录下。

Mac 下编译 Hadoop的更多相关文章

  1. Windows 10 x64 下编译 Hadoop 源码

    Windows 10 x64 下编译 Hadoop 源码 环境准备 Hadoop并没有提供官方的 Windows 10 下的安装包,所以需要自己手动来编译,官方文档中 BUILDING.txt 文件中 ...

  2. MAC 下编译 ANDROID P 源码 提示 internal error: Could not find a supported mac sdk: ["10.10" "10.11" "10.12" "10.13"]

    MAC 下编译 ANDROID P 源码出现下面的问题: ninja: no work to do. [21/21] out/soong/.bootstrap/bin/soong_build out/ ...

  3. mac下编译FFmpeg-Android

    参考: [史上最傻瓜的]mac下编译FFmpeg-Android http://blog.csdn.net/ashqal/article/details/9381037

  4. 64位centos 下编译 hadoop 2.6.0 源码

    64位os下为啥要编译hadoop就不解释了,百度一下就能知道原因,下面是步骤: 前提:编译源码所在的机器,必须能上网,否则建议不要尝试了 一. 下载必要的组件 a) 下载hadoop源码 (当前最新 ...

  5. eclipse下编译hadoop源代码(转)

    hadoop是一个分布式存储和分布式计算的框架.在日常使用hadoop时,我们会发现hadoop不能完全满足我们的需要,我们可能需要修改hadoop的源代码并重新编译.打包. 下面将详细描述如何从sv ...

  6. 在windows环境下编译hadoop

    1.环境准备 1.1 JDK的安装 下载jdk1.6.0_43(这里务必要使用jdk的1.6版本,因为使用JDK1.7版本编译hadoop的时候,很多Maven依赖下载不完整,最终会报错)解压到,并将 ...

  7. mac下编译安装grafana 4.2.0

    go语言在开发效率和运行效率中的优势让很多人青睐,所以有倾向打算转向go语言的开发. 下面介绍在Mac OS X中golang的开发环境配置. 1.安装brew brew是一个mac下的由ruby开发 ...

  8. Mac下编译libpomelo静态库,并在cocos2dx项目中引用

    最近在学习cocos2dx的过程中需要和服务器进行交互,所以这几天在学习libpomelo静态库的编译和使用.之前在windows系统下编译libpomelo,并在VS中引入比较顺利:但是,目前对Ma ...

  9. Windows下编译 Hadoop

    Windows下编译 Hadoop-2.9.2 系统环境 系统: Windows 10 10.0_x64 maven: Apache Maven 3.6.0 jdk: jdk_1.8.0_201 Pr ...

随机推荐

  1. 使用localStorage写一个简单的备忘录

    使用html+js实现一个简单的备忘录,主要体会一下localStorage的用法. 先看看效果图: 在输入框中输入文字,点击保存按钮,文本内容会在下放展示出来, 然后刷新下浏览器,会发现文本内容不会 ...

  2. Microsoft VBScript 运行时错误 错误 800a005e 无效使用 Null: Replace

    查看数据库   表的字段里面是否有空的字段. where 字段名 is null

  3. 【swoole】PHP+Swoole+Linux实现进程监控

    脚本代码 class Server { const PORT = 8888; public function port() { //netstat -anp 2>/dev/null| grep ...

  4. pca数学原理(转)

    PCA的数学原理 前言 数据的向量表示及降维问题 向量的表示及基变换 内积与投影 基 基变换的矩阵表示 协方差矩阵及优化目标 方差 协方差 协方差矩阵 协方差矩阵对角化 算法及实例 PCA算法 实例 ...

  5. idea能用下划线替换红色报错吗?我色弱,用idea简直太痛苦了

    看看下图的idea,如果某个类的包路径没有引进来,使用颜色来提示,这对于色弱的程序员简直是一种折磨,有没有可以改成eclipse的那种报错提示方式? 个人感觉idea真的没有eclipse友好,也许是 ...

  6. vue-cli3.0和element-ui及axios的安装使用

    一.利用vue-cli3快速搭建vue项目 Vue CLI 是一个基于 Vue.js 进行快速开发的完整系统.有三个组件: CLI:@vue/cli 全局安装的 npm 包,提供了终端里的vue命令( ...

  7. Ubuntu与Window双系统安装的注意事项

    UBUNTU与WINDOW双系统安装的注意事项   Ubuntu与Window双系统安装的注意事项 由 匿名 (未验证) 提交于 2019-05-18 10:07:41 登录 发表评论 29 次浏览 ...

  8. 【ARM-Linux开发】ctrl-xxx的对应的signal含义

    ctrl-c 发送 SIGINT 信号给前台进程组中的所有进程.常用于终止正在运行的程序.ctrl-z 发送 SIGTSTP 信号给前台进程组中的所有进程,常用于挂起一个进程.ctrl-d 不是发送信 ...

  9. 异步I/O、事件驱动、单线程

    异步I/O.事件驱动.单线程 nodejs的特点总共有以下几点 异步I/O(非阻塞I/O) 事件驱动 单线程 擅长I/O密集型,不擅长CPU密集型 高并发 下面是一道很经典的面试题,描述了node的整 ...

  10. 查询SQL时间段内执行过的哪些SQL语句

    QS.creation_time, ) , (( ) ) AS statement_text, ST.text, QS.total_worker_time, QS.last_worker_time, ...