How to speed up insertion performance in PostgreSQL
- Disable any triggers on the table
- Drop indexes before starting the import, re-create them afterwards. (It takes much less time to build an index in one pass than it does to add the same data to it progressively, and the resulting index is much more compact).
Change table to UNLOGGED
table without indexes, then change it to logged and add the indexes.Unfortunately in PostgreSQL 9.4 there's no support for changing tables fromUNLOGGED
to logged. 9.5 addsALTER TABLE ... SET LOGGED
to permit you to do this.- Remove Foreign Key Constraints
- If doing the import within a single transaction, it's safe to drop foreign key constraints, do the import, and re-create the constraints before committing. Do not do this if the import is split across multiple transactions as you might introduce invalid data.
- If possible, use
COPY
instead ofINSERT
s - If you can't use
COPY
consider using multi-valuedINSERT
s if practical. Don't try to list too many values in a singleVALUES
though; those values have to fit in memory a couple of times over, so keep it to a few hundred per statement. - Batch your inserts into explicit transactions, doing hundreds of thousands or millions of inserts per transaction. There's no practical limit AFAIK, but batching will let you recover from an error by marking the start of each batch in your input data.
- Increase maintenance_work_mem:This will help to speed up CREATE INDEX commands and ALTER TABLE ADD FOREIGN KEY commands. It won't do much for COPY itself, so this advice is only useful when you are using one or both of the above techniques.
- Use
synchronous_commit=off
and a hugecommit_delay
to reduce fsync() costs. This won't help much if you've batched your work into big transactions, though. INSERT
orCOPY
in parallel from several connections. How many depends on your hardware's disk subsystem; as a rule of thumb, you want one connection per physical hard drive if using direct attached storage.- Set a high
checkpoint_segments
value and enablelog_checkpoints
. Look at the PostgreSQL logs and make sure it's not complaining about checkpoints occurring too frequently. - If and only if you don't mind losing your entire PostgreSQL cluster (your database and any others on the same cluster) to catastrophic corruption if the system crashes during the import, you can stop Pg, set
fsync=off
, start Pg, do your import, then (vitally) stop Pg and setfsync=on
again. See WAL configuration. Do not do this if there is already any data you care about in any database on your PostgreSQL install. If you setfsync=off
you can also setfull_page_writes=off
; again, just remember to turn it back on after your import to prevent database corruption and data loss. See non-durable settings in the Pg manual. - Run ANALYZE Afterwards
参考:
http://stackoverflow.com/questions/12206600/how-to-speed-up-insertion-performance-in-postgresql
https://www.postgresql.org/docs/9.4/static/populate.html
How to speed up insertion performance in PostgreSQL的更多相关文章
- PostgreSQL相关的软件,库,工具和资源集合
PostgreSQL相关的软件,库,工具和资源集合. 备份 wal-e - Simple Continuous Archiving for Postgres to S3, Azure, or Swif ...
- 【转载】利用window.performance.timing进行性能分析
利用window.performance.timing进行性能分析 性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备 ...
- 利用window.performance.timing进行性能分析
性能分析... window.performance.timing中相关属性语义: // .navigationStart 准备加载页面的起始时间 // .unloadEventStart 如果前一个 ...
- 数据库新秀 postgresql vs mongo 性能PK
前几天看了一篇文章<High Performance JSON PostgreSQL vs. MongoDB> 发布在Percona Live Europe 2017 作者是<Dom ...
- Index-Only Scans and Covering Indexes
小结: 1.覆盖索引 回表 2. All indexes in PostgreSQL are secondary indexes, meaning that each index is stored ...
- Doherty Threshold
Prior to the publication of the IBM technical paper behind what commonly known today as the Doherty ...
- Android WIFI 分析(二)
本文介绍Wifi 分析线路二:在Setting中打开WiFi功能.扫描网络以及连接网络的流程. WifiSettings 无线网络设置界面 WifiEnabler 相当于无线网络设置开关 WifiDi ...
- sandy bridge
SANDY BRIDGE SPANS GENERATIONS Intel Focuses on Graphics, Multimedia in New Processor Design By Li ...
- 39. Volume Rendering Techniques
Milan Ikits University of Utah Joe Kniss University of Utah Aaron Lefohn University of California, D ...
随机推荐
- <cfloat> (float.h)
头文件: <cfloat> (float.h) 浮点类型的特性 这个头文件为特殊系统和编译器的实现描述了浮点类型的特征. 一个浮点数包含四个元素: 一个标志(a sign):正或负; 一个 ...
- Microsoft Edge 浏览器开始支持webkit私有样式
微软表示新版的浏览器Edge(spartan)不会再增加新的私有属性,同时移除了部分-ms-属性,但很多标准在没有支持到之前,会使用webkit的api.Edge开发工程师Jacob Rossi列出了 ...
- Intro to Probabilistic Model
概率论复习 概率(Probability) 频率学派(Frequentist):由大量试验得到的期望频率(致命缺陷:有些事情无法大量试验,例如一封邮件是垃圾邮件的概率,雷达探测的物体是一枚导弹的概率) ...
- IntelliJ IDEA 2018 for MAC安装及破解
---------------------说在前面-------------------------- IntelliJ IDEA 2018 版本为2018.1.4 教程按照下载安装sdk.破解两部分 ...
- 软件工程第二周PSP
- 20162328蔡文琛week01
学号20162328 <程序设计与数据结构>第1周学习总结 教材学习内容总结 通过练习课本上给出的代码并结合老师所提供教程,熟悉并初步了解Java的基本编辑 教材学习中的问题和解决过程 无 ...
- iOS开发CABasicAnimation动画理解
1.CALayer简介 CALayer是个与UIView很类似的概念,同样有backgroundColor.frame等相似的属性,我们可以将UIView看做一种特殊的CALayer.但实际上UIVi ...
- <Effective C++>读书摘要--Introduction
Introduction 1.Learning the fundamentals of a programming language is one thing; learning how to des ...
- 代码编写规范Asp.Net(c#)
1 目的 为了统一公司软件开发的设计过程中关于代码编写时的编写规范和具体开发工作时的编程规范,保证代码的一致性,便于交流和维护,特制定此规范. 2 范围 本规范适用于开发组 ...
- Linux中实现在系统启动时自动加载模块
下面是以前学习Linux时写的,后来仔细研究rc.sysinit后发现,只需要修改下列地方就可以了,不必这么麻烦的: rc.sysinit中有这样的一段代码: # Load other user-de ...