madlib centos yum 包安装

使用centos 测试安装madlib sql 机器学习类库

安装步骤

• 添加pg 10 repo

yum install https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-centos10-10-2.noarch.rpm

• 安装pg python 基本包

yum -y install postgresql10-plpython supervisor

• 安装madlib 依赖包
  注意python 版本,我使用python 2.7 安装时候失败了，修改为了python34

yum update -y && yum install -y \
                    git \
                    gcc \
                    wget \
                    postgresql10-devel \
                    openssl \
                    m4 \
                    vim \
                    flex \
                    bison \
                    graphviz \
                    java \
                    epel-release \
                    python34-devel

• 安装pip 包
  默认一般是包含的

yum install -y python34-pip

• pg_conf 配置(环境变量)

PATH="$PATH:/usr/pgsql-10/bin"

• 安装python 依赖（通过pip）

 pip3 install awscli pygresql paramiko --upgrade

• 安装apache-madlib

下载rpm 包
wget   https://dist.apache.org/repos/dist/release/madlib/1.15.1/apache-madlib-1.15.1-bin-Linux.rpm
安装
yum install -y apache-madlib-1.15.1-bin-Linux.rpm

• 启动pg 数据库

/usr/pgsql-10/bin/postgresql-10-setup initdb
systemctl enable postgresql-10
systemctl start postgresql-10

• 修改pg_hba.conf 添加访问支持
  之后修改之后，需要重启服务，systemctl restart postgresql-10

/var/lib/pgsql/10/data/pg_hba.conf
修改如下：
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust

• 初始化madlib 服务

/usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install

安装结果如下：

madpack.py: INFO : Detected PostgreSQL version 10.6.
madpack.py: INFO : *** Installing MADlib ***
madpack.py: INFO : MADlib tools version = 1.15.1 (/usr/local/madlib/Versions/1.15.1/bin/../madpack/madpack.py)
madpack.py: INFO : MADlib database version = None (host=localhost:5432, db=postgres, schema=madlib)
madpack.py: INFO : Testing PL/Python environment...
madpack.py: INFO : > Creating language PL/Python...
madpack.py: INFO : > PL/Python environment OK (version: 2.7.5)
madpack.py: INFO : > Preparing objects for the following modules:
madpack.py: INFO : > - array_ops
madpack.py: INFO : > - bayes
madpack.py: INFO : > - crf
madpack.py: INFO : > - elastic_net
madpack.py: INFO : > - linalg
madpack.py: INFO : > - pmml
madpack.py: INFO : > - prob
madpack.py: INFO : > - sketch
madpack.py: INFO : > - svec
madpack.py: INFO : > - svm
madpack.py: INFO : > - tsa
madpack.py: INFO : > - stemmer
madpack.py: INFO : > - conjugate_gradient
madpack.py: INFO : > - knn
madpack.py: INFO : > - lda
madpack.py: INFO : > - stats
madpack.py: INFO : > - svec_util
madpack.py: INFO : > - utilities
madpack.py: INFO : > - assoc_rules
madpack.py: INFO : > - convex
madpack.py: INFO : > - glm
madpack.py: INFO : > - graph
madpack.py: INFO : > - linear_systems
madpack.py: INFO : > - recursive_partitioning
madpack.py: INFO : > - regress
madpack.py: INFO : > - sample
madpack.py: INFO : > - summary
madpack.py: INFO : > - kmeans
madpack.py: INFO : > - pca
madpack.py: INFO : > - validation
madpack.py: INFO : Installing MADlib:
madpack.py: INFO : > Created madlib schema
madpack.py: INFO : > Created madlib.MigrationHistory table
madpack.py: INFO : > Wrote version info in MigrationHistory table
madpack.py: INFO : MADlib 1.15.1 installed successfully in madlib schema.

• 检查madlib 服务的安装

/usr/local/madlib/bin/madpack -s madlib -p postgres -c postgres@localhost:5432/postgres install-check

输出结果如下：

madpack.py: INFO : Detected PostgreSQL version 10.6.
TEST CASE RESULT|Module: bayes|bayes.ic.sql_in|PASS|Time: 117 milliseconds

TEST CASE RESULT|Module: crf|crf_train_small.ic.sql_in|PASS|Time: 112 milliseconds
TEST CASE RESULT|Module: crf|crf_test_small.ic.sql_in|PASS|Time: 131 milliseconds
TEST CASE RESULT|Module: elastic_net|elastic_net.ic.sql_in|PASS|Time: 123 milliseconds
TEST CASE RESULT|Module: linalg|linalg.ic.sql_in|PASS|Time: 43 milliseconds
TEST CASE RESULT|Module: linalg|svd.ic.sql_in|PASS|Time: 151 milliseconds
TEST CASE RESULT|Module: linalg|matrix_ops.ic.sql_in|PASS|Time: 231 milliseconds
TEST CASE RESULT|Module: prob|prob.ic.sql_in|PASS|Time: 21 milliseconds
TEST CASE RESULT|Module: svm|svm.ic.sql_in|PASS|Time: 151 milliseconds
TEST CASE RESULT|Module: tsa|arima.ic.sql_in|PASS|Time: 130 milliseconds
TEST CASE RESULT|Module: conjugate_gradient|conj_grad.ic.sql_in|PASS|Time: 35 milliseconds
TEST CASE RESULT|Module: knn|knn.ic.sql_in|PASS|Time: 107 milliseconds
TEST CASE RESULT|Module: lda|lda.ic.sql_in|PASS|Time: 109 milliseconds
TEST CASE RESULT|Module: stats|correlation.ic.sql_in|PASS|Time: 93 milliseconds
TEST CASE RESULT|Module: stats|f_test.ic.sql_in|PASS|Time: 25 milliseconds
TEST CASE RESULT|Module: stats|robust_and_clustered_variance_coxph.ic.sql_in|PASS|Time: 116 milliseconds
TEST CASE RESULT|Module: stats|pred_metrics.ic.sql_in|PASS|Time: 98 milliseconds
TEST CASE RESULT|Module: stats|wsr_test.ic.sql_in|PASS|Time: 29 milliseconds
TEST CASE RESULT|Module: stats|mw_test.ic.sql_in|PASS|Time: 26 milliseconds
TEST CASE RESULT|Module: stats|cox_prop_hazards.ic.sql_in|PASS|Time: 101 milliseconds
TEST CASE RESULT|Module: stats|ks_test.ic.sql_in|PASS|Time: 28 milliseconds
TEST CASE RESULT|Module: stats|chi2_test.ic.sql_in|PASS|Time: 28 milliseconds
TEST CASE RESULT|Module: stats|t_test.ic.sql_in|PASS|Time: 26 milliseconds
TEST CASE RESULT|Module: stats|anova_test.ic.sql_in|PASS|Time: 29 milliseconds
TEST CASE RESULT|Module: utilities|minibatch_preprocessing.ic.sql_in|PASS|Time: 104 milliseconds
TEST CASE RESULT|Module: utilities|pivot.ic.sql_in|PASS|Time: 74 milliseconds
TEST CASE RESULT|Module: utilities|path.ic.sql_in|PASS|Time: 88 milliseconds
TEST CASE RESULT|Module: utilities|sessionize.ic.sql_in|PASS|Time: 74 milliseconds
TEST CASE RESULT|Module: utilities|text_utilities.ic.sql_in|PASS|Time: 78 milliseconds
TEST CASE RESULT|Module: utilities|utilities.ic.sql_in|PASS|Time: 77 milliseconds
TEST CASE RESULT|Module: utilities|transform_vec_cols.ic.sql_in|PASS|Time: 93 milliseconds
TEST CASE RESULT|Module: utilities|encode_categorical.ic.sql_in|PASS|Time: 97 milliseconds
TEST CASE RESULT|Module: assoc_rules|assoc_rules.ic.sql_in|PASS|Time: 114 milliseconds
TEST CASE RESULT|Module: convex|mlp.ic.sql_in|PASS|Time: 226 milliseconds
TEST CASE RESULT|Module: convex|lmf.ic.sql_in|PASS|Time: 118 milliseconds
TEST CASE RESULT|Module: glm|glm.ic.sql_in|PASS|Time: 234 milliseconds
TEST CASE RESULT|Module: graph|graph.ic.sql_in|PASS|Time: 218 milliseconds
TEST CASE RESULT|Module: linear_systems|dense_linear_sytems.ic.sql_in|PASS|Time: 88 milliseconds
TEST CASE RESULT|Module: linear_systems|sparse_linear_sytems.ic.sql_in|PASS|Time: 91 milliseconds
TEST CASE RESULT|Module: recursive_partitioning|random_forest.ic.sql_in|PASS|Time: 155 milliseconds
TEST CASE RESULT|Module: recursive_partitioning|decision_tree.ic.sql_in|PASS|Time: 130 milliseconds
TEST CASE RESULT|Module: regress|clustered.ic.sql_in|PASS|Time: 115 milliseconds
TEST CASE RESULT|Module: regress|robust.ic.sql_in|PASS|Time: 99 milliseconds
TEST CASE RESULT|Module: regress|logistic.ic.sql_in|PASS|Time: 101 milliseconds
TEST CASE RESULT|Module: regress|multilogistic.ic.sql_in|PASS|Time: 98 milliseconds
TEST CASE RESULT|Module: regress|marginal.ic.sql_in|PASS|Time: 279 milliseconds
TEST CASE RESULT|Module: regress|linear.ic.sql_in|PASS|Time: 24 milliseconds
TEST CASE RESULT|Module: sample|train_test_split.ic.sql_in|PASS|Time: 76 milliseconds
TEST CASE RESULT|Module: sample|sample.ic.sql_in|PASS|Time: 21 milliseconds
TEST CASE RESULT|Module: sample|stratified_sample.ic.sql_in|PASS|Time: 76 milliseconds
TEST CASE RESULT|Module: sample|balance_sample.ic.sql_in|PASS|Time: 81 milliseconds
TEST CASE RESULT|Module: summary|summary.ic.sql_in|PASS|Time: 86 milliseconds
TEST CASE RESULT|Module: kmeans|kmeans.ic.sql_in|PASS|Time: 155 milliseconds
TEST CASE RESULT|Module: pca|pca_project.ic.sql_in|PASS|Time: 152 milliseconds
TEST CASE RESULT|Module: pca|pca.ic.sql_in|PASS|Time: 242 milliseconds
TEST CASE RESULT|Module: validation|cross_validation.ic.sql_in|PASS|Time: 110 milliseconds

• 添加数据库测试函数
  创建数据表&&添加数据：

CREATE TABLE array_tbl (
    id integer,
    array1 integer[],
    array2 integer[]
);

INSERT INTO "public"."array_tbl"("id","array1","array2")
VALUES
(1,E'{1,2,3,4,5,6}',E'{6,5,4,3,2,1}'),
(2,E'{1,1,0,0,99,8}',E'{0,0,0,-5,2,1}');

查看插入的结果

select * from array_tbl;
 id | array1 | array2
----+----------------+----------------
  1 | {1,2,3,4,5,6} | {6,5,4,3,2,1}
  2 | {1,1,0,0,99,8} | {0,0,0,-5,2,1}
(2 行记录)

使用madlib API:
说明： 主要是计算数组的最大、最小值

select id,madlib.array_min(array1) min, madlib.array_max(array1) max from array_tbl;
select id,madlib.array_min(array1) min, madlib.array_max(array1) max from array_tbl;
 id | min | max
----+-----+-----
  1 | 1 | 6
  2 | 0 | 99
(2 行记录)

说明

这个只是简单的安装，实际生产还需要好多东西需要调整，同时注意madlib schema 的安装，是对应到数据库的，当然
测试环境我们可以直接使用docker 版本的，尽管有点大，但是还是很方便的

参考资料

https://cwiki.apache.org/confluence/display/MADLIB/Installation+Guide#InstallationGuide-Prerequisites
https://www.postgresql.org/download/linux/redhat/