PostgreSQL性能优化综合案例 - 2
【调优阶段8】
1. 压力测试
pgbench -M prepared -r -c 1 -f /home/postgres/test/login0.sql -j 1 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login0 &
pgbench -M prepared -r -c 1 -f /home/postgres/test/login1.sql -j 1 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login1 &
pgbench -M prepared -r -c 2 -f /home/postgres/test/login2.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login2 &
pgbench -M prepared -r -c 2 -f /home/postgres/test/login3.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login3 &
pgbench -M prepared -r -c 2 -f /home/postgres/test/login4.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login4 &
2. 测试结果
cat log.log*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 1
number of threads: 1
duration: 180 s
number of transactions actually processed: 296485
tps = 1647.130827 (including connections establishing)
tps = 1647.153173 (excluding connections establishing)
statement latencies in milliseconds:
0.003394 \setrandom userid 1 4000000
0.599293 SELECT f_user_login_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 1
number of threads: 1
duration: 180 s
number of transactions actually processed: 270077
tps = 1500.414232 (including connections establishing)
tps = 1500.434330 (excluding connections establishing)
statement latencies in milliseconds:
0.004436 \setrandom userid 4000001 8000000
0.656274 SELECT f_user_login_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 543390
tps = 3018.814281 (including connections establishing)
tps = 3018.901510 (excluding connections establishing)
statement latencies in milliseconds:
0.004553 \setrandom userid 8000001 12000000
0.652033 SELECT f_user_login_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 592774
tps = 3293.147194 (including connections establishing)
tps = 3293.235012 (excluding connections establishing)
statement latencies in milliseconds:
0.003446 \setrandom userid 12000001 16000000
0.599297 SELECT f_user_login_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 593614
tps = 3297.831371 (including connections establishing)
tps = 3297.946707 (excluding connections establishing)
statement latencies in milliseconds:
0.003421 \setrandom userid 16000001 20000000
0.598465 SELECT f_user_login_4(:userid);
总计 :
tps = 12757.337905 (including connections establishing)
tps = 12757.670732 (excluding connections establishing)
3. 瓶颈分析与优化
测试中我们使用的数据库服务器cpu是8核的服务器, 根据以往的经验, 当活跃的进程数等于核数的2倍时可以发挥CPU的最大能力.
所以我们通过增加并发连接来看看到底有多少性能提升.
【调优阶段9】
1. 压力测试
pgbench -M prepared -r -c 2 -f /home/postgres/test/login0.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login0 &
pgbench -M prepared -r -c 2 -f /home/postgres/test/login1.sql -j 2 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login1 &
pgbench -M prepared -r -c 4 -f /home/postgres/test/login2.sql -j 4 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login2 &
pgbench -M prepared -r -c 4 -f /home/postgres/test/login3.sql -j 4 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test/login4.sql -j 4 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login4 &
2. 测试结果
cat log.log*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 375743
tps = 2087.443600 (including connections establishing)
tps = 2087.489913 (excluding connections establishing)
statement latencies in milliseconds:
0.003492 \setrandom userid 1 4000000
0.949744 SELECT f_user_login_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 2
number of threads: 2
duration: 180 s
number of transactions actually processed: 367801
tps = 2043.313370 (including connections establishing)
tps = 2043.386454 (excluding connections establishing)
statement latencies in milliseconds:
0.003710 \setrandom userid 4000001 8000000
0.969828 SELECT f_user_login_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 730267
tps = 4057.007177 (including connections establishing)
tps = 4057.148280 (excluding connections establishing)
statement latencies in milliseconds:
0.003962 \setrandom userid 8000001 12000000
0.976372 SELECT f_user_login_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 738398
tps = 4101.985844 (including connections establishing)
tps = 4102.135039 (excluding connections establishing)
statement latencies in milliseconds:
0.003615 \setrandom userid 12000001 16000000
0.966314 SELECT f_user_login_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 732793
tps = 4070.957105 (including connections establishing)
tps = 4071.200533 (excluding connections establishing)
statement latencies in milliseconds:
0.003882 \setrandom userid 16000001 20000000
0.973208 SELECT f_user_login_4(:userid);
总计 :
tps = 16360.707096 (including connections establishing)
tps = 16361.360219 (excluding connections establishing)
3. 瓶颈分析与优化
继续增加连接,tps还可以再提高吗? : 不可以.
8核的机器16个活动的会话基本上就到达它的上限了.
因此要提高tps还可以加CPU.
下面增加连接到30个的测试结果证明了上面的结论.
pgbench -M prepared -r -c 6 -f /home/postgres/test/login0.sql -j 6 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login0 &
pgbench -M prepared -r -c 6 -f /home/postgres/test/login1.sql -j 6 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login1 &
pgbench -M prepared -r -c 6 -f /home/postgres/test/login2.sql -j 6 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login2 &
pgbench -M prepared -r -c 6 -f /home/postgres/test/login3.sql -j 6 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login3 &
pgbench -M prepared -r -c 6 -f /home/postgres/test/login4.sql -j 6 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login4 &
结果
cat log.log*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 6
number of threads: 6
duration: 180 s
number of transactions actually processed: 544811
tps = 3026.494301 (including connections establishing)
tps = 3026.608244 (excluding connections establishing)
statement latencies in milliseconds:
0.003768 \setrandom userid 1 4000000
1.973230 SELECT f_user_login_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 6
number of threads: 6
duration: 180 s
number of transactions actually processed: 544485
tps = 3024.298399 (including connections establishing)
tps = 3024.468785 (excluding connections establishing)
statement latencies in milliseconds:
0.003735 \setrandom userid 4000001 8000000
1.974466 SELECT f_user_login_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 6
number of threads: 6
duration: 180 s
number of transactions actually processed: 544778
tps = 3025.262019 (including connections establishing)
tps = 3025.469901 (excluding connections establishing)
statement latencies in milliseconds:
0.003707 \setrandom userid 8000001 12000000
1.973661 SELECT f_user_login_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 6
number of threads: 6
duration: 180 s
number of transactions actually processed: 542008
tps = 3010.921306 (including connections establishing)
tps = 3011.146550 (excluding connections establishing)
statement latencies in milliseconds:
0.003662 \setrandom userid 12000001 16000000
1.983714 SELECT f_user_login_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 6
number of threads: 6
duration: 180 s
number of transactions actually processed: 539505
tps = 2996.511493 (including connections establishing)
tps = 2996.874239 (excluding connections establishing)
statement latencies in milliseconds:
0.003768 \setrandom userid 16000001 20000000
1.992923 SELECT f_user_login_4(:userid);
总计 :
tps = 15083.487518 (including connections establishing)
tps = 15084.567719 (excluding connections establishing)
连接数超过2倍核数后根本不会有性能提升了, 这台服务器的潜力基本上挖掘得差不多了.
接下来就需要通过增加服务器来提升数据库的整体性能了.
首先要用到的是PostgreSQL的流复制, 通过hot standby可以进行读写分离, 也就是将SELECT的请求分发到hot standby上.
(需要注意跨库事务的问题, 如standby的延时, 这里不详细阐述)
新建查询函数和插入更新函数 :
create or replace function f_user_login_sel_0
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_0 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_sel_1
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_1 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_sel_2
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_2 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_sel_3
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_3 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_sel_4
(i_userid int,
OUT o_userid int,
OUT o_engname text,
OUT o_cnname text,
OUT o_occupation text,
OUT o_birthday date,
OUT o_signname text,
OUT o_email text,
OUT o_qq numeric
)
as $BODY$
declare
begin
select userid,engname,cnname,occupation,birthday,signname,email,qq
into o_userid,o_engname,o_cnname,o_occupation,o_birthday,o_signname,o_email,o_qq
from user_info_4 where userid=i_userid;
return;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_insupd_0
(i_userid int
)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_0 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_insupd_1
(i_userid int
)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_1 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_insupd_2
(i_userid int
)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_2 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_insupd_3
(i_userid int
)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_3 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_insupd_4
(i_userid int
)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
update user_session_4 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
hot standby库也需要将数据加载到内存, 具体操作略.
【调优阶段10】
1. 测试脚本
cat log*
\setrandom userid 1 4000000
SELECT f_user_login_insupd_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_login_insupd_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_login_insupd_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_login_insupd_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_login_insupd_4(:userid);
\setrandom userid 1 4000000
SELECT f_user_login_sel_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_login_sel_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_login_sel_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_login_sel_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_login_sel_4(:userid);
2. 压力测试
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel0.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login_sel0 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel1.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login_sel1 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel2.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login_sel2 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel3.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login_sel3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test_zsplit/login_sel4.sql -j 4 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login_sel4 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_insupd0.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_insupd0 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_insupd1.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_insupd1 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_insupd2.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_insupd2 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_insupd3.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_insupd3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test_zsplit/login_insupd4.sql -j 4 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_insupd4 &
3. 测试结果
hot standby的测试数据 :
cat log.login_sel*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 552618
tps = 3012.767914 (including connections establishing)
tps = 3012.877330 (excluding connections establishing)
statement latencies in milliseconds:
0.003166 \setrandom userid 1 4000000
0.988247 SELECT f_user_login_sel_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 750314
tps = 4089.671930 (including connections establishing)
tps = 4089.771337 (excluding connections establishing)
statement latencies in milliseconds:
0.003030 \setrandom userid 4000001 8000000
0.726462 SELECT f_user_login_sel_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 727839
tps = 3967.242817 (including connections establishing)
tps = 3967.364415 (excluding connections establishing)
statement latencies in milliseconds:
0.003260 \setrandom userid 8000001 12000000
0.748466 SELECT f_user_login_sel_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 715952
tps = 3903.028278 (including connections establishing)
tps = 3903.130455 (excluding connections establishing)
statement latencies in milliseconds:
0.003077 \setrandom userid 12000001 16000000
0.761439 SELECT f_user_login_sel_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 964366
tps = 5257.974345 (including connections establishing)
tps = 5258.120849 (excluding connections establishing)
statement latencies in milliseconds:
0.003153 \setrandom userid 16000001 20000000
0.753196 SELECT f_user_login_sel_4(:userid);
总计 :
tps = 20230.685284 (including connections establishing)
tps = 20231.264386 (excluding connections establishing)
primary的测试数据 :
cat log.login_insupd*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 745415
tps = 4141.145602 (including connections establishing)
tps = 4141.250129 (excluding connections establishing)
statement latencies in milliseconds:
0.003236 \setrandom userid 1 4000000
0.716912 SELECT f_user_login_insupd_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 737761
tps = 4098.582645 (including connections establishing)
tps = 4098.704693 (excluding connections establishing)
statement latencies in milliseconds:
0.003360 \setrandom userid 4000001 8000000
0.723997 SELECT f_user_login_insupd_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 761171
tps = 4228.709500 (including connections establishing)
tps = 4228.817139 (excluding connections establishing)
statement latencies in milliseconds:
0.003333 \setrandom userid 8000001 12000000
0.701648 SELECT f_user_login_insupd_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 761960
tps = 4233.031271 (including connections establishing)
tps = 4233.166856 (excluding connections establishing)
statement latencies in milliseconds:
0.003306 \setrandom userid 12000001 16000000
0.700967 SELECT f_user_login_insupd_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 999167
tps = 5550.893825 (including connections establishing)
tps = 5551.246720 (excluding connections establishing)
statement latencies in milliseconds:
0.003385 \setrandom userid 16000001 20000000
0.712689 SELECT f_user_login_insupd_4(:userid);
总计 :
tps = 22252.362843 (including connections establishing)
tps = 22253.185537 (excluding connections establishing)
QPS :
qps = 20230.685284 + (22252.362843 * 2) (including connections establishing)
qps = 20231.264386 + (22253.185537 * 2) (excluding connections establishing)
4. 瓶颈分析与优化
主节点 :
avg-cpu: %user %nice %system %iowait %steal %idle
56.30 0.00 21.72 4.24 0.00 17.73
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 2781.50 0.00 93.50 0.00 22876.00 244.66 0.09 0.93 0.90 8.40
sdd 0.00 10656.50 0.00 2302.50 0.00 105300.00 45.73 108.00 27.85 0.43 100.05
dm-0 0.00 0.00 0.00 2875.50 0.00 23004.00 8.00 2.56 0.89 0.03 8.30
dm-1 0.00 0.00 0.00 12943.00 0.00 103544.00 8.00 569.00 34.94 0.08 100.10
dm-2 0.00 0.00 0.00 2832.50 0.00 22660.00 8.00 2.55 0.90 0.03 8.05
dm-3 0.00 0.00 0.00 41.50 0.00 332.00 8.00 0.02 0.54 0.06 0.25
dm-4 0.00 0.00 0.00 1.50 0.00 12.00 8.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 1.00 0.00 8.00 8.00 0.01 0.00 4.00 0.40
dm-6 0.00 0.00 0.00 11545.50 0.00 92364.00 8.00 505.23 33.04 0.08 91.75
dm-7 0.00 0.00 0.00 1396.50 0.00 11172.00 8.00 63.54 50.65 0.15 20.65
standby节点 :
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.31 12.87 0.00 86.82
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 0.00 1222.39 0.00 996.52 0.00 19136.32 19.20 113.22 116.63 1.00 99.55
cciss/c0d0p1 0.00 2.99 0.00 1.00 0.00 31.84 32.00 0.10 101.50 101.50 10.10
cciss/c0d0p2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
cciss/c0d0p3 0.00 1219.40 0.00 995.52 0.00 19104.48 19.19 113.12 116.64 1.00 99.55
cciss/c0d1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
cciss/c0d2 0.00 1384.08 0.00 251.74 0.00 13297.51 52.82 142.31 522.75 3.95 99.55
cciss/c0d3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
cciss/c0d4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
cciss/c0d5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 1638.81 0.00 13110.45 8.00 946.36 538.61 0.61 99.55
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 2193.03 0.00 17544.28 8.00 275.53 132.89 0.45 99.55
显然IO到达瓶颈了. 为什么每次IO都顶不住呢? 是的, 机械硬盘的随机IOPS能力就是这么差, 不要有太高的奢望.
要提升IOPS要么就用高端存储要么就选择SSD硬盘. 下次有机会找块ssd硬盘来测试一下它的iops能力到底有多强.
那么这些IO是怎么产生的呢?
1. 主库的IO来自insert和update请求.
2. hot standby的IO来自stream data recovery.
因为我的测试环境没有办法扩存储, 所以这里就不通过扩存储来解决这个瓶颈了, 还是加服务器.
但是这次加2台服务器, 1台用来做hot standby. 另一台我要把insert请求剥离过去.
也就是总共用4台服务器.
具体的操作如下 :
初始化新增的日志库 :
create table user_login_rec
(userid int,
login_time timestamp without time zone,
ip inet
);
create table user_logout_rec
(userid int,
logout_time timestamp without time zone,
ip inet
);
create or replace function f_user_login_ins
(i_userid int)
returns int as $BODY$
declare
begin
insert into user_login_rec (userid,login_time,ip) values (i_userid,now(),inet_client_addr());
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
主库新增函数 :
create or replace function f_user_login_upd_0
(i_userid int
)
returns int as $BODY$
declare
begin
update user_session_0 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_upd_1
(i_userid int
)
returns int as $BODY$
declare
begin
update user_session_1 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_upd_2
(i_userid int
)
returns int as $BODY$
declare
begin
update user_session_2 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_upd_3
(i_userid int
)
returns int as $BODY$
declare
begin
update user_session_3 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
create or replace function f_user_login_upd_4
(i_userid int
)
returns int as $BODY$
declare
begin
update user_session_4 set logintime=now(),login_count=login_count+1 where userid=i_userid;
return 0;
exception
when others then
return 1;
end;
$BODY$
language plpgsql;
再增加一台standby, 流复制过程略, 请参考我写过的流复制环境搭建BLOG.
《PostgreSQL HOT STANDBY using Stream》
http://blog.163.com/digoal@126/blog/static/16387704020110442050808/
优化当前环境如下,
primary : 172.16.3.150
standby1 : 172.16.3.33
standby2 : 172.16.3.39
logdb : 172.16.3.40
【调优阶段11】
1. 测试脚本
postgres@db5-> cat login_ins.sql
\setrandom userid 1 20000000
SELECT f_user_login_ins(:userid);
postgres@db5-> cat login_sel*
\setrandom userid 1 4000000
SELECT f_user_login_sel_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_login_sel_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_login_sel_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_login_sel_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_login_sel_4(:userid);
postgres@db5-> cat login_upd*
\setrandom userid 1 4000000
SELECT f_user_login_upd_0(:userid);
\setrandom userid 4000001 8000000
SELECT f_user_login_upd_1(:userid);
\setrandom userid 8000001 12000000
SELECT f_user_login_upd_2(:userid);
\setrandom userid 12000001 16000000
SELECT f_user_login_upd_3(:userid);
\setrandom userid 16000001 20000000
SELECT f_user_login_upd_4(:userid);
2. 压力测试
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel0.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login33_sel0 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel1.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login33_sel1 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel2.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login33_sel2 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel3.sql -j 3 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login33_sel3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test_zsplit/login_sel4.sql -j 4 -n -T 180 -h 172.16.3.33 -p 1921 -U digoal digoal >./log.login33_sel4 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel0.sql -j 3 -n -T 180 -h 172.16.3.39 -p 1921 -U digoal digoal >./log.login39_sel0 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel1.sql -j 3 -n -T 180 -h 172.16.3.39 -p 1921 -U digoal digoal >./log.login39_sel1 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel2.sql -j 3 -n -T 180 -h 172.16.3.39 -p 1921 -U digoal digoal >./log.login39_sel2 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_sel3.sql -j 3 -n -T 180 -h 172.16.3.39 -p 1921 -U digoal digoal >./log.login39_sel3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test_zsplit/login_sel4.sql -j 4 -n -T 180 -h 172.16.3.39 -p 1921 -U digoal digoal >./log.login39_sel4 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_upd0.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd0 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_upd1.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd1 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_upd2.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd2 &
pgbench -M prepared -r -c 3 -f /home/postgres/test_zsplit/login_upd3.sql -j 3 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd3 &
pgbench -M prepared -r -c 4 -f /home/postgres/test_zsplit/login_upd4.sql -j 4 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd4 &
pgbench -M prepared -r -c 16 -f /home/postgres/test_zsplit/login_ins.sql -j 16 -n -T 180 -h 172.16.3.40 -p 1921 -U digoal digoal >./log.login_ins &
3. 测试结果
cat log.login33_sel*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1534211
tps = 8523.315651 (including connections establishing)
tps = 8523.524318 (excluding connections establishing)
statement latencies in milliseconds:
0.002438 \setrandom userid 1 4000000
0.346514 SELECT f_user_login_sel_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1533785
tps = 8520.894378 (including connections establishing)
tps = 8521.168645 (excluding connections establishing)
statement latencies in milliseconds:
0.002423 \setrandom userid 4000001 8000000
0.346564 SELECT f_user_login_sel_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1544585
tps = 8580.974433 (including connections establishing)
tps = 8581.260902 (excluding connections establishing)
statement latencies in milliseconds:
0.002448 \setrandom userid 8000001 12000000
0.344071 SELECT f_user_login_sel_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1482080
tps = 8233.719776 (including connections establishing)
tps = 8234.138037 (excluding connections establishing)
statement latencies in milliseconds:
0.002435 \setrandom userid 12000001 16000000
0.358877 SELECT f_user_login_sel_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 1982503
tps = 11013.842899 (including connections establishing)
tps = 11014.329592 (excluding connections establishing)
statement latencies in milliseconds:
0.002422 \setrandom userid 16000001 20000000
0.357698 SELECT f_user_login_sel_4(:userid);
cat log.login39_sel*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1534696
tps = 8526.005287 (including connections establishing)
tps = 8526.221472 (excluding connections establishing)
statement latencies in milliseconds:
0.002436 \setrandom userid 1 4000000
0.346352 SELECT f_user_login_sel_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1542513
tps = 8569.192037 (including connections establishing)
tps = 8569.392061 (excluding connections establishing)
statement latencies in milliseconds:
0.002416 \setrandom userid 4000001 8000000
0.344625 SELECT f_user_login_sel_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1508389
tps = 8379.888796 (including connections establishing)
tps = 8380.257897 (excluding connections establishing)
statement latencies in milliseconds:
0.002426 \setrandom userid 8000001 12000000
0.352536 SELECT f_user_login_sel_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 1491690
tps = 8287.124725 (including connections establishing)
tps = 8287.453198 (excluding connections establishing)
statement latencies in milliseconds:
0.002464 \setrandom userid 12000001 16000000
0.356436 SELECT f_user_login_sel_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 2014650
tps = 11192.426565 (including connections establishing)
tps = 11192.867173 (excluding connections establishing)
statement latencies in milliseconds:
0.002418 \setrandom userid 16000001 20000000
0.351905 SELECT f_user_login_sel_4(:userid);
cat log.login_ins
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 7091331
tps = 39394.952222 (including connections establishing)
tps = 39397.035365 (excluding connections establishing)
statement latencies in milliseconds:
0.002984 \setrandom userid 1 20000000
0.399208 SELECT f_user_login_ins(:userid);
cat log.login_upd*
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 968016
tps = 5377.633815 (including connections establishing)
tps = 5377.769568 (excluding connections establishing)
statement latencies in milliseconds:
0.002434 \setrandom userid 1 4000000
0.552395 SELECT f_user_login_upd_0(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 965529
tps = 5363.841108 (including connections establishing)
tps = 5364.017826 (excluding connections establishing)
statement latencies in milliseconds:
0.002461 \setrandom userid 4000001 8000000
0.553797 SELECT f_user_login_upd_1(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 969904
tps = 5388.302421 (including connections establishing)
tps = 5388.476038 (excluding connections establishing)
statement latencies in milliseconds:
0.002436 \setrandom userid 8000001 12000000
0.551348 SELECT f_user_login_upd_2(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 3
number of threads: 3
duration: 180 s
number of transactions actually processed: 990833
tps = 5504.605729 (including connections establishing)
tps = 5504.844893 (excluding connections establishing)
statement latencies in milliseconds:
0.002448 \setrandom userid 12000001 16000000
0.539510 SELECT f_user_login_upd_3(:userid);
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 4
number of threads: 4
duration: 180 s
number of transactions actually processed: 1316258
tps = 7312.497604 (including connections establishing)
tps = 7312.837009 (excluding connections establishing)
statement latencies in milliseconds:
0.002405 \setrandom userid 16000001 20000000
0.541622 SELECT f_user_login_upd_4(:userid);
总计 :
QPS :
158169.217446 ( select 89827.384547, insert 39394.952222, update 28946.880677 )
158175.593994 ( select 89830.613295, insert 39397.035365, update 28947.945334 )
4. 瓶颈分析与优化
如果前面的拆库是纵向的拆的话, 那么接下来要提升性能就得横向的来拆了.
例如 :
select能力可以通过数据库流复制扩展, 9.2以后可以级联复制因此基本上可以做到不影响主库性能的情况下无限扩展.
insert能力可以通过增加logdb服务器扩展, 无限扩展.
update能力可以通过将表拆分到多个服务器上, 无限扩展.
横向分库,需要考虑跨库事务的问题, 1.plproxy
【调优阶段12】
本文的最后一个阶段, 由于服务器有限, 所以我这里测试的是一个节点的性能, 以前测试过plproxy, 性能是线性扩展的.
因此测试一个节点基本上就可以推算出多节点的性能.
1. 压力测试
pgbench -M prepared -r -c 16 -f /home/postgres/test_zsplit/login_upd0.sql -j 16 -n -T 180 -h 172.16.3.150 -p 1921 -U digoal digoal >./log.login_upd0 &
2. 测试结果
cat log.login_upd0
transaction type: Custom query
scaling factor: 1
query mode: prepared
number of clients: 16
number of threads: 16
duration: 180 s
number of transactions actually processed: 6015759
tps = 33416.574452 (including connections establishing)
tps = 33419.030898 (excluding connections establishing)
statement latencies in milliseconds:
0.002152 \setrandom userid 1 4000000
0.473792 SELECT f_user_login_upd_0(:userid);
因此5个节点的性能约等于 :
tps = 167082.872260 ( 33416.574452 * 5 ) (including connections establishing)
tps = 167095.154490 ( 33419.030898 * 5 ) (excluding connections establishing)
3. 瓶颈分析与优化
同11阶段 :
select能力可以通过数据库流复制扩展, 9.2以后可以级联复制因此基本上可以做到不影响主库性能的情况下无限扩展.
insert能力可以通过增加logdb服务器扩展, 无限扩展.
update能力可以通过将表拆分到多个服务器上, 无限扩展.
横向分库,需要考虑跨库事务的问题, 1.plproxy
【其他可优化点补充】
1. 批量提交,降低IO请求量, 并发请求很高的场景. 但是当并发场景这么高的时候已经可以考虑增加服务器分库了.
相关参数
#commit_delay = 0
#commit_siblings = 5 2. 连接池,如pgbouncer(适用于短连接, 大量空闲连接的情况.)
3. 绑定变量, 性能提升参考
4. user_session中记录了用户的登陆统计信息和退出统计信息, 由于MVCC特性, 每次更新都会新产生一条tuple, 因此如果将登陆和退出的统计拆开, 就能减少新增的tuple的大小. 一定程度上提升性能.
user_session_login (userid, logintime, login_count)
user_session_logout (userid, logouttime, online_interval)
5. OS级别也有可以优化的地方, 比如文件系统的mount参数可以加上noatime.
6. 服务器硬件也有可以优化的地方, 比如numa.
7. PostgreSQL也还有可以微调的参数, 比如bgwriter_lru_maxpages和bgwriter_lru_multiplier它们的值也将影响数据库和文件系统交互的频率以及每次交互产生的io请求数.
8. 在做分表优化的时候, 本例使用的是按userid分段拆分成了5个表. 其实还可以按hash取模拆, 按时间段拆等等. 拆分的关键是尽量按照常用的条件字段进行拆分. 另外需要注意的是, 我这里没有提到PostgreSQL的partition table的实现, 而是直接使用应用端来识别数据在哪个分区. 原因是PostgreSQL的partition table需要通过rule或者触发器来实现, 大量的消耗数据库服务器的CPU, 不推荐使用.
【小结】
1. 诊断角度
操作系统层面: 查看CPU, IO.
数据库层面:
查看pg_stat_statements
Column | Type | Modifiers
---------------------+------------------+-----------
userid | oid |
dbid | oid |
query | text |
calls | bigint |
total_time | double precision |
rows | bigint |
shared_blks_hit | bigint |
shared_blks_read | bigint |
shared_blks_written | bigint |
local_blks_hit | bigint |
local_blks_read | bigint |
local_blks_written | bigint |
temp_blks_read | bigint |
temp_blks_written | bigint |
其他pg_stat性能视图
日志中的long SQL,
2. 优化角度
参数, SQL, 架构, 连接池, 表空间拆分, 存储cache, 分表, 分库
PostgreSQL性能优化综合案例 - 2的更多相关文章
- 连接postgres特别消耗cpu资源而引发的PostgreSQL性能优化考虑
由于是开发阶段,所以并没有配置postgres的参数,都是使用安装时的默认配置,以前运行也不见得有什么不正常,可是前几天我的cpu资源占用突然升高.查看进程,发现有一个postgres的进程占用CPU ...
- 性能优化实战案例——助力某移动OA系统
前言 最近连续接触了4个OA系统,均存在着不同的性能问题,本文记述对某移动OA系统的优化全过程,让看官们对数据库优化流程有一个了解,并揭开隐式转换这无情杀手的神秘面纱. 本文使用的工具:SQL专家云平 ...
- SQL性能优化案例分析
这段时间做一个SQL性能优化的案例分析, 整理了一下过往的案例,发现一个比较有意思的,拿出来给大家分享. 这个项目是我在项目开展2期的时候才加入的, 之前一期是个金融内部信息门户, 里面有个功能是收集 ...
- 常见性能优化策略的总结 good
阅读目录 代码 数据库 缓存 异步 NoSQL JVM调优 多线程与分布式 度量系统(监控.报警.服务依赖管理) 案例一:商家与控制区关系的刷新job 案例二:POI缓存设计与实现 案例三:业务运营后 ...
- Go 性能分析之案例一
思考 相信大家在实际的项目开发中会遇到这么一个事,有的程序员写的代码不仅bug少,而且性能高:而有的程序员写的代码能否流畅的跑起来,都是一个很大问题.而我们今天要讨论的就是一个关于性能优化的案例分析. ...
- 《Android开发艺术探索》读书笔记 (13) 第13章 综合技术、第14章 JNI和NDK编程、第15章 Android性能优化
第13章 综合技术 13.1 使用CrashHandler来获取应用的Crash信息 (1)应用发生Crash在所难免,但是如何采集crash信息以供后续开发处理这类问题呢?利用Thread类的set ...
- 老李案例分享：Weblogic性能优化案例
老李案例分享:Weblogic性能优化案例 POPTEST的测试技术交流qq群:450192312 网站应用首页大小在130K左右,在之前的测试过程中,其百用户并发的平均响应能力在6.5秒,性能优化后 ...
- PostgreSQL之性能优化（转）
转载自:https://blog.csdn.net/huangwenyi1010/article/details/72853785 解决问题 前言 PostgreSQL的配置参数作为性能调优的一部分, ...
- [转帖]PostgreSQL 参数调整(性能优化)
PostgreSQL 参数调整(性能优化) https://www.cnblogs.com/VicLiu/p/11854730.html 知道一个 shared_pool 文章写的挺好的 还没仔细看 ...
- PostgreSQL 参数调整(性能优化)
昨天分别在外网和无外网环境下安装PostgreSQL,有外网环境下安装的相当顺利.但是在无外网环境下就是两个不同的概念了,可谓十有八折.感兴趣的同学可以搭建一下. PostgreSQL安装完成后第一件 ...
随机推荐
- form表单里的button 等元素不能使用margin: 0 auto;
记得把form和button都设为display:block; 就能用margin: 0 auto;水平居中了
- 《HTTP权威指南》– 7.缓存
Web缓存的概念 Web缓存是可以自动保存常见文档副本的HTTP设备. 使用缓存的优点: 减少了数据的数据传输,节省了网络费用: 缓解了网络瓶颈的问题,不需要更多的带宽就能更快地加载页面: 降低了原始 ...
- 搭建漏洞环境及实战——搭建DVWA漏洞环境
DVWA是一款开源的渗透测试漏洞练习平台,其中内涵XSS.SQL注入.文件上传.文件包含.CSRF和暴力破解等各个难度的测试环境. 1.在安装时需要在数据库里创建一个数据库名,进入MySQL管理中的p ...
- VC实例和VM实例的区别!!!
1.内置关系是什么 VueComponent.prototype.__proto__ === Vue.prototype 2.为什么要有这个关系 让组件实例对象可以访问到 Vue原型上的属性.方法.
- Redis学习整理
目录 1.Redis基本概念 2.Redis的5种基本类型 3.Jedis整合redis操作 4.Springboot整合redis 5.Redis主从复制 5.1.概念 5.2.原理 6.开启主从复 ...
- 贪心算法Dijkstra
Dijkstra 最短路径问题 : 给定一个带权有向图 G = (V, E, W),同时给定一个源点 u (u ∈ V),我们要找出从源点 u 出发到其它各点的最短路径距离,并得出这些最短路径的具体路 ...
- Python网络爬虫get方法出现乱码的解决的三种方案
给大家祭出网络爬虫过程中三种中文乱码的处理方案,希望对大家的学习有所帮助. 方案一 将requests.get().text改为requests.get().content 我们可以看到通过text( ...
- CTF-MISC方向涉及技术导图
MISC方向涉及技术导图
- Nexus私有maven库部署和使用
原文地址:Nexus私有maven库部署和使用 - Stars-One的杂货小窝 前段圣诞节前后,Jitpack网站突然崩溃了,无法下载依赖,然后过了一个星期才解决了,好在没啥紧急的Android开发 ...
- pip19.2.3升级到20.3.3版本升级失败问题
2021-01-06 macOS版本:11.1 安装pip: sudo easy_install pip 话不多说,直接上问题 一行命令搞定 终端:sudo pip install --upg ...