在RockyLinux 9.2环境中编译CockroachDB 23.1

目的

CockroachDB自 23.1开始，编译模式逐步放弃了Makefile，改用Bazel构建。实际上在23.1代码执行make buildshort 能够成功，但make build应该就会因为生成js文件有问题而失败。可惜Bazel构建也不是官网说的那样简单，本文详细描述编译CockroachDB 23.1.x的编译过程，供对CockroachDB感兴趣的同仁参考。

编译环境

Rocky Linux 9.2,原因是Rocky Linux 9.2 自带各种模块版本较高，符合cockroachdb编译要求，并且外围模块趋于稳定，不会出现9.0时因为openssl版本问题导致nodejs需要降级的异常操作

安装RockyLinux 9.2 时，仅需选择Server环境，然后选择开发工具和效率工具即可。

CockroachDB版本：官网的release-23.1分支，今年国内访问github的情况有所改善，虽然不稳定，但是多试几次，尤其是ssh模式还是可以下载代码的

官方参考资料

• 1、Getting and building CockroachDB from source
• 2、Developing with Bazel

准备和检查编译环境

• 1、克隆代码库
  
  采用ssh模式克隆cockroachdb的release-23.1（本文假设您已经有了一个github账号）
  
  git clone -b release-23.1 git@github.com:cockroachdb/cockroach.git
  
  
  
  不要嫌慢，能下载就是好事儿，我的网络是天津联通，github下载速度再10K~2M之间

• 2、下载的同时，可以按官网逐项对比Linux x86平台的环境准备

• 2.1、A C++ compiler that supports C++11. Note that GCC prior to 6.0 doesn't work due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=48891。缺省已经安装的gcc版本是11.3.1，是支持C++11标准的

g++ -v
gcc version 11.3.1 20221121 (Red Hat 11.3.1-4) (GCC)

• 2.2、The standard C/C++ development headers on your system. 同2.1 缺省已经安装头文件
• 2.3、On GNU/Linux, the terminfo development libraries, which may be part of a ncurses development package (e.g. libncurses-dev on Debian/Ubuntu, but ncurses-devel on CentOS). You also need the libresolv-wrapper package on Ubuntu < 22.04. rocky linux缺省没有安装ncurses-devel，需要执行dnf安装

dnf install ncurses-devel

• 2.4、Git 1.9+, we recommend v2.20+，缺省已经安装

git -v
git version 2.39.1

• 2.5、Bash (4+ is preferred) ，缺省已经安装

bash --version
GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

• 2.6、GNU Make (4.2+ is known to work)，缺省已经安装

make -v
GNU Make 4.3

• 2.7、CMake 3.20.* (on Macs 3.21+ are also known to work)。缺省没有安装cmake，需要执行dnf安装

dnf install cmake
cmake --version
cmake version 3.20.2

• 2.8、Autoconf 2.68+

autoconf --version
autoconf (GNU Autoconf) 2.69

• 2.9、Yacc or Bison

bison --version
bison (GNU Bison) 3.7.4

• 2.10、GNU Patch 2.7+

patch -v
GNU patch 2.7.6

• 2.11、Bazelisk (you can use Bazel directly instead of Bazelisk, although Bazelisk makes sure you are using the same version that we use in CI)
  
  安装Bazelisk有多种方式，我们采用安装nodejs，然后用npm安装Bazelisk

dnf install nodejs
node -v
v16.19.1
npm -v
8.19.3

设置国内代理并验证

npm config set registry http://registry.npm.taobao.org/
npm get registry
http://registry.npm.taobao.org/

安装Bazelisk

npm install -g @bazel/bazelisk

安装完成后检查

whereis bazel
bazel: /usr/local/bin/bazel
whereis bazelisk
bazelisk: /usr/local/bin/bazelisk

• 3 开始构建
  
  等待clone代码完成，就开始构建
• 3.1 首先运行 dev doctor 检查代码环境

cd cockroach
./dev doctor
/home/ansible/works/cockroach/bin/dev-versions/dev.84 not found, building...
2023/09/06 10:29:45 Downloading https://github.com/cockroachdb/bazel/releases/download/6.2.1/bazel-6.2.1-linux-x86_64...
长时间等待后
Starting local Bazel server and connecting to it...
INFO: Analyzed target //pkg/cmd/dev:dev (86 packages loaded, 10730 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/dev:dev up-to-date:
  _bazel/bin/pkg/cmd/dev/dev_/dev
INFO: Elapsed time: 254.017s, Critical Path: 12.70s
INFO: 20 processes: 5 internal, 15 linux-sandbox.
INFO: Build completed successfully, 20 total actions
INFO: Invocation ID: 5039d549-7ae7-4c4f-86b6-8695625c41fc
=============================
=== RUNNING DOCTOR CHECKS ===
=============================
INFO: Invocation ID: 27237479-0160-45db-92af-83445ca320d0
INFO: Build option --//build/toolchains:nogo_disable_flag has changed, discarding analysis cache.
INFO: Analyzed target //build/bazelutil:test_nogo_configured (1 packages loaded, 29 targets configured).
INFO: Found 1 target...
Target //build/bazelutil:test_nogo_configured up-to-date:
  _bazel/bin/build/bazelutil/test_nogo_configured.sh
INFO: Elapsed time: 0.465s, Critical Path: 0.01s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
Configuring cache...
INFO: Build option --//build/toolchains:nogo_disable_flag has changed, discarding analysis cache.
INFO: Analyzed target @com_github_buchgr_bazel_remote//:bazel-remote (195 packages loaded, 12422 targets configured).
INFO: Found 1 target...
Target @com_github_buchgr_bazel_remote//:bazel-remote up-to-date:
  _bazel/bin/external/com_github_buchgr_bazel_remote/bazel-remote_/bazel-remote
INFO: Elapsed time: 63.612s, Critical Path: 29.20s
INFO: 206 processes: 7 internal, 199 linux-sandbox.
INFO: Build completed successfully, 206 total actions
Using cache configuration file at /home/ansible/.cache/dev-bazel-remote/config.yml

报告成功

我在一台虚拟机上针对不同提交版本多次执行检查，有可能报告一些错误，但进行交互后都能成功，遇到过如下情况

• DOCTOR >> Which config you want to use (dev,crosslinux)? [dev] > 输入dev
• DOCTOR >> Do you want to use the lintonbuild configuration? [y] > 输入 y
• DOCTOR >> Please set a tmpdir that will be used for storing test artifacts [/tmp/cockroach] > 直接回车
• DOCTOR >> Do you want me to update your ~/.bazelrc for you to configure the loopback cache? [y] > 直接回车
• 3.2 编译主程序

./dev build
$ bazel build //pkg/cmd/cockroach:cockroach
INFO: Invocation ID: c009abbc-9b9f-493a-b491-a86df5a3a2c5
INFO: Build options --//build/toolchains:nogo_disable_flag and --run_under have changed, discarding analysis cache.
INFO: Analyzed target //pkg/cmd/cockroach:cockroach (4313 packages loaded, 48317 targets configured).
INFO: Found 1 target...
ERROR: /home/ansible/works/cockroach/pkg/ccl/gssapiccl/BUILD.bazel:4:11: GoCompilePkg pkg/ccl/gssapiccl/gssapiccl.a failed: (Exit 1): builder failed: error executing command (from target //pkg/ccl/gssapiccl:gssapiccl) bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -tags bazel,gss,bazel,gss -src pkg/ccl/gssapiccl/gssapi.go -src ... (remaining 53 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/usr/bin/ld.gold: error: cannot find -lresolv_wrapper
external/archived_cdep_libkrb5_linux/lib/libkrb5.a(dnsglue.o):dnsglue.c:function krb5int_dns_init: error: undefined reference to '__res_nsearch'
collect2: error: ld returned 1 exit status
compilepkg: error running subcommand /usr/bin/gcc: exit status 1
Target //pkg/cmd/cockroach:cockroach failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 282.874s, Critical Path: 177.14s
INFO: 2913 processes: 38 internal, 2875 linux-sandbox.
INFO: Build Event Protocol files produced successfully.
FAILED: Build did NOT complete successfully
ERROR: exit status 1

可惜，编译失败了，报告libkrb5.a编译失败，原因是rockylinux 9.2 上的makefile对krb5不友好，一些宏的判断有问题，此处问题解决有两个办法：

第一种修改方式

vi .bazelrc

删除gss编译项

第二种修改方式

因为我们仅仅是为了研究cockroachdb的核心技术，也可以简单删除krb5涉及的代码，具体执行如下

• 1、删除工程中唯一一处使用krb5的代码

rm pkg/ccl/gssapiccl/ -rf

• 2、修改ccl中引用gssapiccl的地方

vi pkg/ccl/BUILD.bazel

vi pkg/ccl/ccl_init.go

然后重新执行编译程序

[ansible@xtp2 cockroach]$ ./dev build
$ bazel build //pkg/cmd/cockroach:cockroach
INFO: Invocation ID: 1eb1ac48-7ac9-4d50-8b56-b1b8911c4e8b
INFO: Analyzed target //pkg/cmd/cockroach:cockroach (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/cockroach:cockroach up-to-date:
  _bazel/bin/pkg/cmd/cockroach/cockroach_/cockroach
INFO: Elapsed time: 40.259s, Critical Path: 38.21s
INFO: 4 processes: 1 internal, 3 linux-sandbox.
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 4 total actions
Successfully built binary for target //pkg/cmd/cockroach:cockroach at cockroach

然后验证

./cockroach --version
cockroach version details:
Build Tag:        v23.1.10-dev
Build Time:
Distribution:     CCL
Platform:         linux amd64
Go Version:       go1.19.10
C Compiler:       gcc 11.3.1 20221121 (Red Hat 11.3.1-4)
Build Commit ID:
Build Type:
Enabled Assertions: false
(use './cockroach version --build-tag' to display only the build tag)

再然后就按手册测试更多的功能了

按版本验证的角度，我们应该运行 ./dev test （对应之前的make test）进行完整验证，可惜 ./dev test 运行过程中占满了我的100G虚拟机磁盘，最后执行失败。

总结

- 1、在Rockylinux9.2上，编译libkrb5有问题，需要修改bazelrc文件
- 2、执行单元测试 ./dev test 失败，原因是生成了.cache/bazel 80G以上的文件，导致没有磁盘剩余空间执行终止
- 3、建议，如果不关注cockroach ui，仅仅学习研究cockroach的主程序，还是采用传统的make buildshort， make test 吧