Q:
I have a table in pg like so:
CREATE TABLE t (
a BIGSERIAL NOT NULL, -- 8 b
b SMALLINT, -- 2 b
c SMALLINT, -- 2 b
d REAL, -- 4 b
e REAL, -- 4 b
f REAL, -- 4 b
g INTEGER, -- 4 b
h REAL, -- 4 b
i REAL, -- 4 b
j SMALLINT, -- 2 b
k INTEGER, -- 4 b
l INTEGER, -- 4 b
m REAL, -- 4 b
CONSTRAINT a_pkey PRIMARY KEY (a)
);

The above adds up to 50 bytes per row. My experience is that I need another 40% to 50% for system overhead, without even any user-created indexes to the above. So, about 75 bytes per row. I will have many, many rows in the table, potentially upward of 145 billion rows, so the table is going to be pushing 13-14 terabytes. What tricks, if any, could I use to compact this table? My possible ideas below ...

Convert the real values to integer. If they can stored as smallint, that is a saving of 2 bytes per field.

Convert the columns b .. m into an array. I don't need to search on those columns, but I do need to be able to return one column's value at a time. So, if I need column g, I could do something like

SELECT a, arr[] FROM t;

Would I save space with the array option? Would there be a speed penalty?

Any other ideas?

A:

"Column Tetris"

Actually, you can do something, but this needs deeper understanding. The keyword is alignment paddingEvery data type has specific alignment requirements.

You can minimize space lost to padding between columns by ordering them favorably. The following (extreme) example would waste a lot of physical disk space:

CREATE TABLE t (
e int2 -- 6 bytes of padding after int2
, a int8
, f int2 -- 6 bytes of padding after int2
, b int8
, g int2 -- 6 bytes of padding after int2
, c int8
, h int2 -- 6 bytes of padding after int2
, d int8)

To save 24 bytes per row, use instead:

CREATE TABLE t (
a int8
, b int8
, c int8
, d int8
, e int2
, f int2
, g int2
, h int2) -- 4 int2 occupy 8 byte (MAXALIGN), no padding at the end

As a rule of thumb, if you put 8-byte columns first, then 4-bytes, 2-bytes and 1-byte columns last you can't go wrong. text or boolean do not have alignment restrictions like that, some other types do. Some types can be compressed or "toasted" (stored out of line).(注:在pg9.6版本中,text是有对齐限制的,对齐要求是4 bytes,boolean仍不无对齐要求)。

Normally, you may save a couple of bytes per row at best playing "column tetris". None of this is necessary in most cases. But with billions of rows it can mean a couple of gigabytes easily.

You can test the actual column / row size with the function pg_column_size().
Be aware that some data types can use more space in RAM than on disk (compressed format). So you can get bigger results for constants (RAM format) than for table columns (disk format) when testing the same value (or row of values vs. table row) with pg_column_size().

For example:

CREATE TABLE t1 (
e int2 -- 6 bytes of padding after int2
, a int8
, f int2 -- 6 bytes of padding after int2
, b int8
, g int2 -- 6 bytes of padding after int2
, c int8
, h int2 -- 6 bytes of padding after int2
, d int8); INSERT INTO t1 VALUES (1,1,1,1,1,1,1,1); CREATE TABLE t2 (
a int8
, b int8
, c int8
, d int8
, e int2
, f int2
, g int2
, h int2); INSERT INTO t2 VALUES (1,1,1,1,1,1,1,1);
swrd=# SELECT pg_column_size(t1) AS not_optimized FROM t1 LIMIT 1;
not_optimized
---------------
88
(1 row) swrd=#
swrd=# SELECT pg_column_size(t2) AS optimized FROM t2 LIMIT 1;
optimized
-----------
64
(1 row) swrd=#
swrd=# SELECT pg_column_size('{1}'::int[]) AS int_plus_array_overhead;
int_plus_array_overhead
-------------------------
28
(1 row)

Overhead per tuple (row)

4 bytes per row for the item pointer - not subject to above considerations.
And at least 24 bytes (23 + padding) for the tuple header. The manual on Database Page Layout:

There is a fixed-size header (occupying 23 bytes on most machines), followed by an optional null bitmap, an optional object ID field, and the user data.

For the padding between header and user data, you need to know MAXALIGN on your server - typically 8 bytes on a 64-bit OS (or 4 bytes on a 32-bit OS). If you are not sure, check out pg_controldata.

The manual:

The actual user data (columns of the row) begins at the offset indicated by t_hoff, which must always be a multiple of the MAXALIGN distance for the platform.

So you typically get the storage optimum by packing data in multiples of 8 bytes.

There is nothing to gain in the example you posted. It's already packed tightly. 2 bytes of padding after the last int2, 4 bytes at the end. You could consolidate the padding to 6 bytes at the end, which wouldn't change anything.

Overhead per data page

Some overhead per data page (typically 8 KB): Remainders not big enough to fit another tuple, and more importantly dead rows or a percentage reserved with the FILLFACTOR setting.

There are a couple of other factors for size on disk to take into account:

Array types?

With array like you were evaluating, you would add 24 bytes of overhead for the array type alone. Plus, elements of an array occupy space as usual. Nothing to gain there.

注:

1、pg在存储层,特别是行的存储层面,存储字段时,对一些字段要求对齐填充,不同的类型要求不同。下面是常用类型的对齐填充要求(typalign)、类型长度(typlen)、存储类型(typstorage)

swrd=# select typname,typlen,typalign,typstorage from pg_type where typname in ('int4','int8','varchar','text','timestamp','numeric','bool');
typname | typlen | typalign | typstorage
-----------+--------+----------+------------
bool | 1 | c | p
int8 | 8 | d | p
int4 | 4 | i | p
text | -1 | i | x
varchar | -1 | i | x
timestamp | 8 | d | p
numeric | -1 | i | m
(7 rows)

对齐类型的官方说明:

c = char alignment, i.e., no alignment needed.
s = short alignment (2 bytes on most machines).
i = int alignment (4 bytes on most machines).
d = double alignment (8 bytes on many machines, but by no means all).

关于类型存储的官方说明:

p: Value must always be stored plain.
e: Value can be stored in a “secondary” relation (if relation has one, see pg_class.reltoastrelid).
m: Value can be stored compressed inline.
x: Value can be stored compressed inline or stored in “secondary” storage.
2、在平时使用时,不必在意类型的对齐填充问题,但如果表非常大,可以考虑一下,也许可以节省大量空间
3、上面的例子中通过使用pg_column_size展示的结果,比较明显
4、在pg的每行行头有23 bytes的固定长度,剩余1byte用来标记行中是否有null值。所以每行一共被占去了24 bytes,另外在当前页中还有4 bytes的item pointer,用来指向数据存放的位置,所以一行实际上一共占去了28bytes。而数据的存放pg要求必须是maxalign的整数倍,maxalign在64位操作系统中是8bytes,在32位系统中是4位,可以通过命令pg_controldata来查看:
postgres@db-> pg_controldata |grep align
Maximum data alignment: 8
5、每页的开销:除了存放数据行,而且还要存放dead rows和考虑fillfactor因素。
6、至于pg为什么要记性对齐填充,后面再整理一篇文章。不过至于mysql或oracle有无类型对齐不得而知。
下面是列出的pg页和行的结构图:
 
 
 
参考:
http://www.postgres.cn/news/viewone/1/220
https://www.postgresql.org/docs/10/static/catalog-pg-type.html

Calculating and saving space in PostgreSQL的更多相关文章

  1. How to get the free disk space in PostgreSQL (PostgreSQL获取磁盘空间)

    Get the current free disk space in PostgreSQL PostgreSQL获取磁盘空间 from eshizhan Here has a simple way t ...

  2. Measure the size of a PostgreSQL table row

    Q:   I have a PostgreSQL table. select * is very slow whereas select id is nice and quick. I think i ...

  3. PostgreSQL源码安装文档

    This document describes the installation of PostgreSQL using the source    code distribution. (If yo ...

  4. 记一次CUDA编程任务

    这个月6号开始,着手解决一个具有实际意义的计算任务.任务数据有9879896条,每条包含30个整数,任务是计算每两条数据之间的斯皮尔相关系数及其P值.原始数据只有500+MB,因此我并不认为这是个多么 ...

  5. Level shifting a +/- 2.5V signal to 0 - 5V

    Google : Op-Amp Level Shifter Level shifting a +/- 2.5V signal to 0 - 5V I have a front end module t ...

  6. L364 Should Your Resume Be One Page or Two?

    Should Your Resume Be One Page or Two? Conventional wisdom suggests that you should keep it short: A ...

  7. what are stop words

    what are stop words 一.总结 一句话总结:就是在seo的关键词中不要有stop words,不然的话搜索引擎会直接忽略 stop words  most common  words ...

  8. python学习1 ---range()函数

    奇怪的现象 在paython3中 print(range(10)) 得出的结果是 range(0,10) ,而不是[0,1,2,3,4,5,6,7,8,9] ,为什么呢? 官网原话: In many ...

  9. Non-Inverting Level Shifter : +/-5V signal into a 0 to 3.3V

    http://electronicdesign.com/boards/non-inverting-level-shifter-requires-only-one-op-amp-one-supply-v ...

随机推荐

  1. 城市规模越大,工资、GDP、犯罪率越高:4.5星|《规模》

    规模 信息浓度非常高的一本书.篇幅也不小,纸书有568页,致谢与注释只占7%. 全书讲各种复杂的东西中存在的普遍规律:哺乳动物体重每增加一倍,心率降低25%:城市人口每增加一倍,加油站只增加85%:城 ...

  2. shell基础 -- grep、sed、awk命令简介

    在 shell 编程中,常需要处理文本,这里介绍几个文本处理命令. 一.grep 命令 grep 命令由来已久,用 grep 命令来查找 文本十分方便.在 POSIX 系统上,grep 可以在两种正则 ...

  3. day09,10 函数

    一.函数 什么是函数 函数: 对代码块和功能的封装和定义 定义一个事情或者功能. 等到需要的时候直接去用就好了. 那么这里定义的东西就是一个函数. 语法: def 函数名(形参): 函数体 函数名(实 ...

  4. java 乐观锁 vs 悲观锁

    在数据库的锁机制中介绍过,数据库管理系统(DBMS)中的并发控制的任务是确保在多个事务同时存取数据库中同一数据时不破坏事务的隔离性和统一性以及数据库的统一性. 悲观锁其实就是 完全同步 比如 sync ...

  5. loadrunner之analysis详解

    本文原出处:http://blog.csdn.net/lykangjia/article/details/56009750 一.常用到的性能测试术语 1.事务(Transaction) 在web性能测 ...

  6. scrum立会报告+燃尽图(第三周第七次)

    此作业要求参见:https://edu.cnblogs.com/campus/nenu/2018fall/homework/2286 项目地址:https://coding.net/u/wuyy694 ...

  7. 四则运算——单元测试(测试方法:Right-BICEP )

    一.测试的具体部位 Right-结果是否正确? B-是否所有的边界条件都是正确的? I-能查一下反向关联吗? C-能用其他手段交叉检查一下结果吗? E-你是否可以强制错误条件发生? P-是否满足性能要 ...

  8. java 中的 i=i++

    记得大学刚开始学C语言时,老师就说:自增有两种形式,分别是i++和++i,i++表示的是先赋值后加1,++i是先加1后赋值,这样理解了很多年也没出现问题,直到遇到如下代码,我才怀疑我的理解是不是错了: ...

  9. yum源中默认好像是没有mysql的。为了解决这个问题,我们要先下载mysql的repo源。

    CentOS7的yum源中默认好像是没有mysql的.为了解决这个问题,我们要先下载mysql的repo源. 1. 下载mysql的repo源 $ wget http://repo.mysql.com ...

  10. rabbitmq 配置用户信息

    本文摘自:http://my.oschina.net/hncscwc/blog/262246 1. 用户管理 用户管理包括增加用户,删除用户,查看用户列表,修改用户密码. 相应的命令 (1) 新增一个 ...