想往某个表中插入几百万条数据做下测试,

原先的想法,直接写个循环10W次随便插入点数据试试吧,好吧,我真的很天真....

DROP PROCEDURE IF EXISTS proc_initData;--如果存在此存储过程则删掉
DELIMITER $
CREATE PROCEDURE proc_initData()
BEGIN
DECLARE i INT DEFAULT 1;
WHILE i<=100000 DO
INSERT INTO text VALUES(i,CONCAT('姓名',i),'XXXXXXXXX');
SET i = i+1;
END WHILE;
END $
CALL proc_initData();

执行CALL proc_initData()后,本来想想,再慢10W条数据顶多30分钟能搞定吧,结果我打了2把LOL后,回头一看,还在执行,此时心里是彻底懵逼的....待我打完第三把结束后,终于执行完了,这种方法若是让我等上几百万条数据,是不是早上去上班,下午下班回来还没结束呢?10W条数据,有图有真相

aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXsAAAByCAIAAAA9JOYgAAAT/ElEQVR4nO2deXQUx53Ha8nb54f3j9X+kZiss7Yh2dg5drNm8MZZJza+Yuzn3bzYydrWyN5wmCx27GjDjbE0kkBcxiE+MBgJNCMggwEjMJcAGaTRCcICIXShYzSHRkIgNUgIS5qe2T9aNK3uqprq7lHPqPX7vO+bV1Pzq+NX1f2dGjESyOv1chLmzZvHAQAAjA4IHAcAAMPAO07v9YApJeTY090znmX6RTB3ggaaw6iAcRy7wwkCgeJZsfIL/eDPOH29nZrVey0Qt+I4zu5wXr1yVbNi/x6nW6ZfBHMnKGQXA6uIEnjHudF7WbP6rnfErTiOszucV7quaFZ399WxLtMvgrkT5MzpOH1dmtV3vTNuxQnX4uUuzdL+vhk3Mv0imDtBzpSO0993BStPoHN/uW/9Afe6/a37y32eQKcyRs/5aLTFcZzd4ezqvKxZ2t8340amXwRzJ8iZ03FuXFVqX7lvp6vLffnrQM9AoGegpfPmrpKufeU+WdiN3q64FcdxdofzckenqOBQkGcgOBQU4ru6Lo91KRdBrWKewnhOkDOp43TLtGJPS/55zt89cMnbXdvsr28JtLRz/u6B/PM9q/a2jAgmnI/iQRzH2R3OzkCHqFAoNP3tAw//bx5Fj711IBQKCfFdlzvHupSLIOh4bYWyEquYp6AtQUGLlr67cfNW0tP4T5AzpePc7O+Ran+5L32P53RT76GvunfnnwmFQqFQaO/xymPnucrmvpRdbYfO+MRg7PkoTsQJ12J7QFSI5x97i8FxeF6Iv9zZMdalXITO9kCd+9I/b5nl93pl9VjFPAUNCSbMPXDn7IN/N+fgnN//ITU981vJJya/U/TDzHLh6X98XP2zTeenbz4X/wlyJnWcEXpzc32Ks21LQWfabu/8tc5wOBwOhxet/2zl576tJzuXO9ve+LThdvyNbj1CCCGElOWIwSziOM7ucHb420WFeH5N7um6Js8qR0Vdk6e2yZNpL69pdK/MKTtX15yxrfSrC40rt5WGeF6I7+wMREsIIcZXsZH05hQpF6HD355RvGPCuhm5pQdl9d1fftWbdbw/u6Av58uek+eivgijIWyCE2cdOOe5fqDq8m8TX3v9jeQfpZT8JL38oTWVv375tVnzkp/f2fSfu1qmvn/G+ATV7iNnSsf5+uY1qZ5JP//8qotvZLVYP2icMX/PY/+z+tHXVj27YM9rHzX9Mac18YPG36yrFYNl5yO1GjYRwlNpJQlK5xzH2R3OgM8vipc4zipHRaa9fKW9fGVO2YptpRlbS9OzS9KzizO2lvI8L8R3dLRHSwghSr3ykdRcuQL0cZWLEPD5f7pn/oQ1M57ekiytvOI6f33dvhsbj/ZnF/RnHe/94NCV4uroLgJR5WnTEEJoWnq56rbYBCfOOvDAUpegHywr/uE7xT9OLX1wRcVP3/vqFx9VP5VV+1R2/XdTi/TvMv3iZL8M6NnFwCqiBMlxrktlmV/x4Pyzj6fVPJle++CLqcIZ58EXU6fbLj6dUfvLjAsvra2+Hd/PEZX3O4QQmrWbFEDfLYSQNIzUnDIBjuPsDme71ydK6jgXmzwXGt0rt5Wer23O2FpyprrBtsVVdrY2LauY53khPhDwU7QjSZypJa0UE0BJTRmJLSsrZa9ig6VSLkJh3dm7dsz8xpoZE1OermmoFesDth1Xk7OuL995fcXua6l/vbowpz19h5pFQEk7lAG5SeKLxE5K0qbhF5BFygTbvb6ysrLwLYQfCwSDwcHBwf7+/t7e3p6enkAgkJ+fz7LLYha47CJvnKrLgJSd8U4RLfCOM/D1dake+H3hI4vP/Gpt3bOZdQ+/kiFs2/efW/LgovPT0y6+sOrcixkVYrDsfHRLe2aLizp7DyHm2rBlEF6VhRFdidyQE65Fj1eU1HEkp5uStKwS2xZX6qdF724uTP20iOf54SbtPrJyk6bZSoTydisSywQhhLCVMmSV9KdiJUXKRfhjwab79s6bsHYGWvLotcp7+LN381Xf4au+Eyy7i6+YFCy95+ofPu2a94lv5p8vPLkw0iIUpyXdSrzENg1Zt2MXCl/PHqAuwXaP9/Tp08J1293dbbFYgsHgwoUL77zzzuTk5CtXrnR0dLS2th46dIhhl33t7cVp0yzTpqGk7RG2lbQXpJiIeydmFwOriBIkx+mV6pmlRU8tLvqvzOrnMmt/npQp7NxD/5329Iral96vfXV1+dz1Zbfjb14nKm+m4DjYV8UbRlqWIosk9UCZAMdxdofT3+YRJXWcCw2t5+uaV2wtPXOh0bbFVXq2ZvmmwoKSquWbTvE8L8S3+71sciRZbCW4l7CpYWPEsqxAqlf2g5VyEX7w2Zv3f5E8Yd2zaNmjA0WTQie/HSq8pZP/yJ+62/1s6qXpiyvvn3nkW79SsQi5VoRfBEcSsm6XlJNuHXum2Vztfu928ZQkNM+1iguVlKslQX+bp6rqnHC6mTJlypQpUwYHBxMSEqqrqxMSEvx+v9vtrq+vz8vLY9nl7UkoKXf4Ebu/EXeZpUzPLgZWESXwjjM40CeV41jjz+cdnD7/5EMLTt//StaUXy763ozFDyRmP7yk6pmlruf+dGjn8QYxWOZWI7RfcJzPSQHDlkHpQRJGgtKQ4zi7w+lzt4kSHEf4gfGKbaUrtpVmZJekZRWnflq0/JNTyz7+cvFHJ5Z+XMDzvBDv93lYVGKzIKudHoMQotQLuchqsM2VK0AfV7YIOyvz790771+OLprw3rPo3en9x/4pdOTeUP49ofx7Q/n3hg5PDh6eXIieOoGe2IcePWyZybQIgkdYbCX4AHsSsubeLqOkXI/f5/EX2yzD9ZKAYpsFWWzFYrBYZk1QUF19vfBhiuf5J5988ubNm2lpaRMnTly2bFlLS0tDQ0N1dfXevXsj7nKJzWKxFfl9nlzrrWmTt1VPmZ5dDKwiSpAc54ZMv17w+SOzdk2d88X3Zp349syiSTOLJ88+NXXu4emv73p1+b6RwX1E7Z+FEEJzPicFCDeMWJChDCM1J4njOLvD6W11ixIdZ6W9/Hxdc9XFpvTsktKzte9uLiwoPrv4wxMHjhUv+uA4z/NCvM/bFlmORGRJLVbU040SISTEiMHYR1lvyiYRJVuEN4988OPDS6adSvmb959DaU/0ue4OHbkvdHRy6MiU0OEpocOTh47fswf9Ygd6JHvCz6o27mJdBG+bz5VqQYm5mJdyrLfrpeVCm8Vic42sdCQia47YNteKrA51CQpqbm4Wf3zzxBNP9PX1IYQ6OjoQQvX19dXV1ZWVlXv37I2wy5LJkGYi3QVKmXINsGQXA6uIEgTHGexX6tXkT/49ceu/vbL9/pf/+v2XnP/68vaHrdm/S/5QHqlwq9sadpx9pIBblkHuQRJGgtKQE67FllZRguNcvNQm/gTHllWcsrnwnU9OLfmoYOFfjv3p/aMLNuTzPD/cxOOmy2WbiiwprkhhCCHxkfQqKVjMVPZUVk+SbBHyiwte3JP2ky1z/3b98/dl/ra3JiFUmxCq/YdhXUwY+Cph9Temrp/0eN6iNYyLcEuFNstUW5GyfpsVJTowZTFeUmlPRNZtYluHFVntEcZV7rK3pbWtrY3n+aGhoYGBgccff5zjuLfeemvixIlz586tqampqqqqqKjYvXs3PUGXbapsqS22QsrekcqUvY4ozpSOMzR4k6Q5b697YfaGF2ZveP3tdYSYfqIOCI6TRwoQdlEsyKA3YRHHcXaH09PcIkpwnJrGtpU5pedqm89eaLRtcblOVy/bePJQQfn8DfnO/Sf+b/0RnueHm3haKbJbEbJuo8d4PK0IIVlBLMtSlr1E6kTZD0XKRZAq6EsIt98RDtwRDtwRbr8j7JsYbPt7eRil/6IUi7gCRSkW9IodE7bNerteWj5ls0y1FY2sLEqxIKFyZFllgm632+fzeb1et7utpaW1oaGxpuZiVdW58oqKIperoKDg4MGD+/PyWHZZ3GurnXVzSU8jbhYpuxhYRZQgOM7Q19qFt6G8OSNupYfX12LCbtnH7YKsXny6ds1qZbAsTCmO4+wOZ1tTsyie5//srFxlL8+0l6/MKcvYWmrLcqVsLnxn48nFH55YsCE/ef3h+e8fCYVCw03aWogqfNcy0i8SczBhwkvSp+KjtIZUiW1Ij5RJuQhSDXi/y/u/yXsn8Z5JQfddwdZvDrZOkYfR+j+ZensVpqYWKgO2Jg6/+krO8FOhILQVmkgrRywsdklZEnSVVly52tPZ1d1xucsX6HR7/A2XWi9crK+sqi4/c9ZVWn78y6KSU4UMCQ4rJ5G4v9iNwO5gxM0iZRcDq4gSeMcJDg3o0NeaJeyBWJAhDVu3drWyctOmjbJKmTiOszuc7ktNosLMCPFt7mbNGr7C3M3SR2lBGSkNkIXJupJCn4ZyEdRKzyIYIGyCrtKKwuKyglPFh0+cPHD42OdfHPls3xeOnZ9tzs7d8NGm1e9tSM9ce/zI0ajsMmlPsU9ZtkyZXQysIkoQHCc4qEMDMmHtAyFEipQWZPXYssimTZ+QehbECddi4yXtam0iCZsjJRJbI31JrJGGycaS1cvKWMXJIoyesAmeOn6i4Gj+yWPHSCo4ml9W5KInGDE77JbJwrBrwr5cnCkdhw8O6dBg3IrjOLvD2drQqF0tl8a6TL8I5k6QM6fj8EEdGopbcRxndzhb6hu0q7lxrMv0i2DuBDlTOk6I53UoGLfiOM7ucDbX1WtXU8NYl+kXwdwJcuZ0HF3wcSuO42L1332AQCC7w0n8PzlzAAAAog3Ncdj/5RgAAIAFcBwAAIwDHAcAAONQ5zgIIeOnyDh6bOc2GpgvIwBQ4TjKG8DIW4JlLP3zEb/3GbFylJANBKYDmAztjmPYTYgdXXMMS3PsPa+tcw2twHEAE8PqONhL37D7gX0gPVMajRx1Oo7+CQBAXDHqjiMehZSfF6Q10jBsMLZb7CegiFMijc5SiW2OnZLO5mozAoAxgRFnHPpdJL3DZY+kgShmxDIlUnPGHCmTZ+lTVXPSrABgjGKQ41BqNDhOmPxTJPb7U3mgYHcc7IGINIra5uA4gIkZq47D0jPLlHSecaI1Q3AcYJxgxL+OMzpOmGA3LE00zyfOP1WB3QAmQ++/jke8JZSfICjN2R2H0pwF5UcbbJ9qJ49dELXNpS+B4wAmA75zHL+YLyMAgN+rAgDAOMBxAAAwDnAcAACMAxwHAADjAMcBAMA4wHEAADAOcBwAAIwjguMgBbGZZVTHpWQUw6/AqPpWIUtzVRsX9dHFesZIYJwQ+Ywju4hHdzaGXIiI/AtcsQK7yOwrH7G58aOHCV+2ZuwTMCvqHGfUZ2PSsSISW8chdaVn9IgN42r9AcNQ4TjSgnhAIJ3kR4wx8mwvDVB2xdKhsh7beYTMI32CIE2JMivsKIxhpIlhAyg3tmxNSJNnnCpLc8rk2QcCxglMjoO9G2UFUllbK/ksmW8wej+UPikDSR8ZJywN0O842u55yiJTpqTH7+gTYxkIMD1azjjiU9I7f8Sbf5QcJ2I/lD4pAykdB+vCelB7TGBvzt6JnubiUoDjAHS0/xyH8Q3NlI4TsXO1qF0QluYa+iFFam4OjgPI0PhvVSxHd7WVYfJVGLeOw/7mzz6ZiNZG93qWRR6l5soJswwEjCu0fB9HWb7dneQlxkpsD4xh0mtd+ZSWNrlDZWrKTLEZUQaKGMY4JUqfalc+is1Jq8HeJzB+gO8cAwBgHOA4AAAYBzgOAADGAY4DAIBxgOMAAGAc4DgAABgHOA4AAMYBjgMAgHGo+01OA764Fd2BKJOPty+h6fy+3FhpDt8AHOew/paD9HEUZxPt310ycvJ6QCO/UkypHNPN2fsEzArrpypjro9RGiX+L+6xYhk6m4PjABodRzwYy07IsjMzKUwaKQ1gPIdTusUnydYh4+iq0DDDiPcndpJjojk4DqD9jMN+5SkvOMrVrByFVGa/ZLGR2MlLHyPOjXFocBxwHEBEl+Pgexx5KFBlGSRfUx40VF2vehwHO3rU0Xl/jpXm4DhAlB2H/TiDrVHla9h69mC1Z5xRZaxYhs7m4DhANB1H/5XH2Cc2kkJUHEfbHcJ4OFK7ShSnjufm4DgAk+MoP1aQPmsgAtJXSc0j9hkxEp8htbl0PmGJ48jmrPn2YG/LmDupz7HVXPN6AmMd+M4xAADGAY4DAIBxgOMAAGAc4DgAABgHOA4AAMYBjgMAgHGA4wAAYBzgOAAAGEds/iIXe2+U0U32LTKd35dTRqrauKiPLtYzRgLjhCj/RS5VVxLjpcw++thF5hSUSm3NjR89zPz7EMC4Isp/kSvqjqOh27FIbB2H1JWe0SM2NP2eAlj0/n0c6fVNOsOzH7mJs1Tzm1aaPxqMBuyfSrBlUgDL2YG+IIxTZWmu1nFiuB1AzNH+NwAjliO+qvOMg+1N+hhxbgYQFcfRds9T9ogyJT1+R58Yy0CA6Rl1xxHrVV2OyuYRK0mOQ3qrjyt0+rKqIxJ7V+zNxeUFxwHoGOE42HddUjB+ltE448Qzah2cpbm28110DQscB5ABn6pGFz2fqugHRlXNGT8W6WxO331Kn8D4QeNf5CJVhglXpOyjjbKGNkVFGKnDsMRxpLcB420/GrAPzZImpU/2PYp6c9IKs/cJjB/gO8cAABgHOA4AAMYBjgMAgHGA4wAAYBzgOAAAGAc4DgAAxgGOAwCAcYDjAABgHOA4AAAYBzgOAADGAY4DAIBxgOMAAGAc4DgAABgHOA4AAMYBjgMAgHGA4wAAYBzgOAAAGAc4DgAAxgGOAwCAcYDjAABgHOA4AAAYBzgOAADG8f+qvno4ZGekbwAAAABJRU5ErkJggg==" alt="" />

JDBC往数据库中普通插入方式

后面查了一下,使用JDBC批量操作往数据库插入100W+的数据貌似也挺快的,

先来说说JDBC往数据库中普通插入方式,简单的代码大致如下,循环了1000条,中间加点随机的数值,毕竟自己要拿数据测试,数据全都一样也不好区分

    private String url = "jdbc:mysql://localhost:3306/test01";
private String user = "root";
private String password = "123456";
@Test
public void Test(){
Connection conn = null;
PreparedStatement pstm =null;
ResultSet rt = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(url, user, password);
String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
pstm = conn.prepareStatement(sql);
Long startTime = System.currentTimeMillis();
Random rand = new Random();
int a,b,c,d;
for (int i = 1; i <= 1000; i++) {
pstm.setInt(1, i);
pstm.setInt(2, i);
a = rand.nextInt(10);
b = rand.nextInt(10);
c = rand.nextInt(10);
d = rand.nextInt(10);
pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d); pstm.executeUpdate();
}
Long endTime = System.currentTimeMillis();
System.out.println("OK,用时:" + (endTime - startTime));
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}finally{
if(pstm!=null){
try {
pstm.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
if(conn!=null){
try {
conn.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
}

输出结果:OK,用时:738199,单位毫秒,也就是说这种方式与直接数据库中循环是差不多的。

在讨论批量处理之前,先说说遇到的坑,首先,JDBC连接的url中要加rewriteBatchedStatements参数设为true是批量操作的前提,其次就是检查mysql驱动包时候是5.1.13以上版本(低于该版本不支持),因网上随便下载了5.1.7版本的,然后执行批量操作(100W条插入),结果因为驱动器版本太低缘故并不支持,导致停止掉java程序后,mysql还在不断的往数据库中插入数据,最后不得不停止掉数据库服务才停下来...

那么低版本的驱动包是否对100W+数据插入就无力了呢?实际还有另外一种方式,效率相比来说还是可以接受的。

使用事务提交方式

先将命令的提交方式设为false,即手动提交conn.setAutoCommit(false);最后在所有命令执行完之后再提交事务conn.commit();

     private String url = "jdbc:mysql://localhost:3306/test01";
private String user = "root";
private String password = "123456";
@Test
public void Test(){
Connection conn = null;
PreparedStatement pstm =null;
ResultSet rt = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(url, user, password);
String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
pstm = conn.prepareStatement(sql);
conn.setAutoCommit(false);
Long startTime = System.currentTimeMillis();
Random rand = new Random();
int a,b,c,d;
for (int i = 1; i <= 100000; i++) {
pstm.setInt(1, i);
pstm.setInt(2, i);
a = rand.nextInt(10);
b = rand.nextInt(10);
c = rand.nextInt(10);
d = rand.nextInt(10);
pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
pstm.executeUpdate();
}
conn.commit();
Long endTime = System.currentTimeMillis();
System.out.println("OK,用时:" + (endTime - startTime));
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}finally{
if(pstm!=null){
try {
pstm.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
if(conn!=null){
try {
conn.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
}

以上代码插入10W条数据,输出结果:OK,用时:18086,也就十八秒左右的时间,理论上100W也就是3分钟这样,勉强还可以接受。

批量处理

接下来就是批量处理了,注意,一定要5.1.13以上版本的驱动包。

 private String url = "jdbc:mysql://localhost:3306/test01?rewriteBatchedStatements=true";
private String user = "root";
private String password = "123456";
@Test
public void Test(){
Connection conn = null;
PreparedStatement pstm =null;
ResultSet rt = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(url, user, password);
String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
pstm = conn.prepareStatement(sql);
Long startTime = System.currentTimeMillis();
Random rand = new Random();
int a,b,c,d;
for (int i = 1; i <= 100000; i++) {
pstm.setInt(1, i);
pstm.setInt(2, i);
a = rand.nextInt(10);
b = rand.nextInt(10);
c = rand.nextInt(10);
d = rand.nextInt(10);
pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
pstm.addBatch();
}
pstm.executeBatch();
Long endTime = System.currentTimeMillis();
System.out.println("OK,用时:" + (endTime - startTime));
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}finally{
if(pstm!=null){
try {
pstm.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
if(conn!=null){
try {
conn.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
}

10W输出结果:OK,用时:3386,才3秒钟.

批量操作+事务

然后我就想,要是批量操作+事务提交呢?会不会有神器的效果?

 private String url = "jdbc:mysql://localhost:3306/test01?rewriteBatchedStatements=true";
private String user = "root";
private String password = "123456";
@Test
public void Test(){
Connection conn = null;
PreparedStatement pstm =null;
ResultSet rt = null;
try {
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(url, user, password);
String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
pstm = conn.prepareStatement(sql);
conn.setAutoCommit(false);
Long startTime = System.currentTimeMillis();
Random rand = new Random();
int a,b,c,d;
for (int i = 1; i <= 100000; i++) {
pstm.setInt(1, i);
pstm.setInt(2, i);
a = rand.nextInt(10);
b = rand.nextInt(10);
c = rand.nextInt(10);
d = rand.nextInt(10);
pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
pstm.addBatch();
}
pstm.executeBatch();
conn.commit();
Long endTime = System.currentTimeMillis();
System.out.println("OK,用时:" + (endTime - startTime));
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}finally{
if(pstm!=null){
try {
pstm.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
if(conn!=null){
try {
conn.close();
} catch (SQLException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
}
}

以下是100W数据输出对比:(5.1.17版本MySql驱动包下测试,交替两种方式下的数据测试结果对比)

批量操作(10W) 批量操作+事务提交(10W) 批量操作(100W) 批量错作+事务提交(100W)

OK,用时:3901

OK,用时:

OK,用时:44242

OK,用时:

OK,用时:4142

OK,用时:

OK,用时:44248

OK,用时:

OK,用时:3664

OK,用时:

OK,用时:44389

OK,用时:

可见有一定的效率提升,但是并不是太明显,当然因为数据差不算太大,也有可能存在偶然因数,毕竟每项只测3次。

预编译+批量操作

网上还有人说使用预编译+批量操作的方式能够提高效率更明显,但是本人亲测,效率不高反降,可能跟测试的数据有关吧。

预编译的写法,只需在JDBC的连接url中将写入useServerPrepStmts=true即可,

如:

private String url = "jdbc:mysql://localhost:3306/test01?useServerPrepStmts=true&rewriteBatchedStatements=true"

好了,先到这里...

JDBC实现往MySQL插入百万级数据的更多相关文章

  1. 利用JDBC或者事物或者调用存储过程实现往MySQL插入百万级数据

    转自:http://www.cnblogs.com/fnz0/p/5713102.html 想往某个表中插入几百万条数据做下测试, 原先的想法,直接写个循环10W次随便插入点数据试试吧,好吧,我真的很 ...

  2. 教你几招,快速创建 MySQL 五百万级数据,愉快的学习各种优化技巧

    我是风筝,公众号「古时的风筝」,一个兼具深度与广度的程序员鼓励师,一个本打算写诗却写起了代码的田园码农! 文章会收录在 JavaNewBee 中,更有 Java 后端知识图谱,从小白到大牛要走的路都在 ...

  3. mysql 插入百万条数据

    利用mysql内存表插入速度快的特点,先存储过程在内存表中生成数据,然后再从内存表插入普通表中 一.创建内存表 CREATE TABLE `vote_record_memory` ( `id` ) N ...

  4. 百万级数据 MySQL处理(转)

    转自 http://www.cnblogs.com/win7xt/p/3156334.html 使用MySQL处理百万级以上数据时,不得不知道的几个常识   最近一段时间参与的项目要操作百万级数据量的 ...

  5. MYSQL百万级数据,如何优化

    MYSQL百万级数据,如何优化     首先,数据量大的时候,应尽量避免全表扫描,应考虑在 where 及 order by 涉及的列上建立索引,建索引可以大大加快数据的检索速度.但是,有些情况索引是 ...

  6. 使用JDBC向数据库中插入一条数据

    原谅我是初学者,这个方法写的很烂,以后不会改进,谢谢 /** * 通过JDBC向数据库中插入一条数据 1.Statement 用于执行SQL语句的对象 1.1 通过Connection 的 * cre ...

  7. PHP MySQL 插入多条数据

    PHP MySQL 插入多条数据 使用 MySQLi 和 PDO 向 MySQL 插入多条数据 mysqli_multi_query() 函数可用来执行多条SQL语句. 以下实例向 "MyG ...

  8. 使用POI导出百万级数据到excel的解决方案

    1.HSSFWorkbook 和SXSSFWorkbook区别 HSSFWorkbook:是操作Excel2003以前(包括2003)的版本,扩展名是.xls,一张表最大支持65536行数据,256列 ...

  9. 使用 MySQLi 和 PDO 向 MySQL 插入多条数据

    PHP MySQL 插入多条数据 使用 MySQLi 和 PDO 向 MySQL 插入多条数据 mysqli_multi_query() 函数可用来执行多条SQL语句. 以下实例向 "MyG ...

随机推荐

  1. bzoj 3196 Tyvj 1730 二逼平衡树(线段树套名次树)

    3196: Tyvj 1730 二逼平衡树 Time Limit: 10 Sec  Memory Limit: 128 MBSubmit: 1807  Solved: 772[Submit][Stat ...

  2. CSS書寫規範及CSS Hack

    基本原则: CSS样式可细分为3类:自定义样式.重新定义HTML样式.链接状态样式. 样式为设计师自定义的新 CSS 样式,影响被使用本样式的区域,用于完成网页中局部的样式设定. 样式名 “.”+“相 ...

  3. json里的日期字符串 怎么 转换成 javascript 的 Date 对象?

    “/Date(1232035200000)/” 怎么转换成  javascript 的 Date 对象 做法:new Date(+/\d+/.exec(value)[1]); value就是json字 ...

  4. CM5(Cloudera Manager 5) + CDH5(Cloudera's Distribution Including Apache Hadoop 5)的安装详细文档

    参考 :http://www.aboutyun.com/thread-9219-1-1.html Cloudera Manager5及CDH5在线(cloudera-manager-installer ...

  5. python selenium启动浏览器打开百度搜索

    python selenium打开百度搜索 #!usr/bin/python from selenium import webdriver import time browser = webdrive ...

  6. Mac 解决SSH登录服务器终端乱码

    一.Mac自带的终端 ssh 连接Linux 乱码,可用如下方法解决 终端 --> 偏好设置 --> 描述文件 --> 高级  --> 设为GBK 即可 二.secureCRT ...

  7. 使用ajax代替iframe

    相信大多数程序员都跟iframe打过交道,iframe简单,好用.在我用的过程中比较苦逼的是关于iframe高度的设置. 由于子页面内容不确定,页面高度也不确定.于是开始网上的各种搜索,一般有两种:一 ...

  8. 一秒钟速算(WINFORM)

    WIN7以上的系统可以直接运行,XP的系统要按装.NET 2.0 程序下载 http://files.cnblogs.com/xe2011/%E4%B8%80%E7%A7%92%E9%92%9F%E5 ...

  9. linux系统禁止root用户通过ssh登录及ssh的访问控制

    Linux系统默认情况下,是可以通过ssh以root权限登录的.但出于安全考虑,这样的权限是不合适的,因为黑客可能通过暴力破解你的root密码,然后进入你的系统,oh,damn it..... 同样是 ...

  10. 谋哥:App排行榜的秘密

    App在改变世界,改变人们的生活.       如今购物大家都用淘宝.京东,吃饭你会用饭否,看天气预报你用墨迹天气,看视频用优酷.K歌你用唱吧,聊天联系你用微信,看新闻你用今日头条等等.你的生活由你自 ...