C# .NET - Sql Bulk Insert from multiple delimited Textfile using c#.net
SqlBulkCopy.WriteToServer has 4 overloads:
SqlBulkCopy.WriteToServer (DataRow[])
Copies all rows from the supplied DataRow array to a destination table specified by the
DestinationTableName property of the SqlBulkCopy object.
SqlBulkCopy.WriteToServer (DataTable)
Copies all rows in the supplied DataTable to a destination table specified by the
DestinationTableName property of the SqlBulkCopy object.
SqlBulkCopy.WriteToServer (IDataReader)
Copies all rows in the supplied IDataReader to a destination table specified by the
DestinationTableName property of the SqlBulkCopy object.
SqlBulkCopy.WriteToServer (DataTable, DataRowState)
Copies only rows that match the supplied row state in the supplied DataTable to a
destination table specified by the DestinationTableName property of the SqlBulkCopy object.
When importing text files with this method you have to create a DataTable first, import the text file
to the created DataTable and then write this DataTable to server.
With this we're acctually performing 2 tasks in .net:
1. Fill data from text file to DataTable in memory
2. Fill data from DataTable in memory to SQL server
Compared to SQL servers native bulk import methods where we just import the text file directly.
I used the same file and the same table structure as in previous bulk import methods described in Last.
The time it took to complete the whole process was around 30 seconds.
This is the code i used for import:
private void StartImport()
{
Stopwatch sw = new Stopwatch();
sw.Start();
SqlBulkCopy bulkCopy = new SqlBulkCopy("Server=ServerName;Database=test;Trusted_Connection=True;",
SqlBulkCopyOptions.TableLock);
bulkCopy.DestinationTableName = "dbo.testSQLBulkCopy";
bulkCopy.WriteToServer(CreateDataTableFromFile());
sw.Stop();
txtResult.Text = (sw.ElapsedMilliseconds/1000.00).ToString();
}
private DataTable CreateDataTableFromFile()
{
DataTable dt = new DataTable();
DataColumn dc;
DataRow dr; dc = new DataColumn();
dc.DataType = System.Type.GetType("System.Int32");
dc.ColumnName = "c1";
dc.Unique = false;
dt.Columns.Add(dc);
dc = new DataColumn();
dc.DataType = System.Type.GetType("System.Int32");
dc.ColumnName = "c2";
dc.Unique = false;
dt.Columns.Add(dc);
dc = new DataColumn();
dc.DataType = System.Type.GetType("System.Int32");
dc.ColumnName = "c3";
dc.Unique = false;
dt.Columns.Add(dc);
dc = new DataColumn();
dc.DataType = System.Type.GetType("System.Int32");
dc.ColumnName = "c4";
dc.Unique = false;
dt.Columns.Add(dc);
StreamReader sr = new StreamReader(@"d:\work\test.txt");
string input;
while ((input = sr.ReadLine()) != null)
{
string[] s = input.Split(new char[] { '|' });
dr = dt.NewRow();
dr["c1"] = s[];
dr["c2"] = s[];
dr["c3"] = s[];
dr["c4"] = s[];
dt.Rows.Add(dr);
}
sr.Close();
return dt;
}
Bulk Import Methods are ad below..:-
1. BCP
2. Bulk Insert
3. OpenRowset with BULK option
4. SQL Server Integration Services - SSIS
I ran each bulk import option 12 times, disregarded best and worst time and averaged the remaining ten times.
Results are:
1. | SSIS - FastParse ON | = | 7322 ms |
2. | SSIS - FastParse OFF | = | 8387 ms |
3. | Bulk Insert | = | 10534 ms |
4. | OpenRowset | = | 10687 ms |
5. | BCP | = | 14922 ms |
So speed gain is quite large when using FastParse.
I was also surprised that SSIS - FastParse OFF method was faster by 20% to Bulk Insert and OpenRowset
and around 40% faster than BCP.
Since my desire was to test how much faster is importing flat files when FastParse option is used
I created a text file containing 4 bigint columns with 1,000,000 rows.
The script i used to create a sample test file in C#:
string str;
StreamWriter sw = new StreamWriter(@"d:\work\test.txt");
for (int i = ; i <= ; i++)
{
str = i.ToString() + "|" + Convert.ToString(i * ) + "|" + Convert.ToString(i * ) + "|" + Convert.ToString(i / );
sw.WriteLine(str);
}
sw.Close();
I also created this format file for use with BCP, Bulk Insert and OpenRowset:
9.0
4
1 SQLBIGINT 0 8 "|" 1 c1 ""
2 SQLBIGINT 0 8 "|" 2 c2 ""
3 SQLBIGINT 0 8 "|" 3 c3 ""
4 SQLBIGINT 0 8 "\r\n" 4 c4 ""
SSIS Package was a very simple one with a Flat File source and SQL server destination objects.
The sql script i used is:
create database test
go
use test
go
-- ran for each SSIS test run
-- SSIS data type for each column was "eight-byte signed integer [DT_I8]"
drop table testFastParse
create table testFastParse(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
go
-- insert data using OPENROWSET
create table testOpenRowset(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
go
DBCC DROPCLEANBUFFERS
declare @start datetime
set @start = getdate()
insert into testOpenRowset(c1, c2, c3, c4)
SELECT t1.c1, t1.c2, t1.c3, t1.c4
FROM OPENROWSET( BULK 'd:\work\test.txt',
FORMATFILE = 'd:\work\testImport-f-n.Fmt') AS t1(c1, c2, c3, c4);
select getdate() - @start as ElapsedTime
drop table testOpenRowset
-- insert data using Bulk Insert
create table testBulkInsert(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
go
DBCC DROPCLEANBUFFERS
declare @start datetime
set @start = getdate()
BULK INSERT testBulkInsert
FROM 'd:\work\test.txt'
WITH (FORMATFILE='d:\work\testImport-f-n.Fmt')
select getdate() - @start as ElapsedTime
drop table testBulkInsert
go
-- insert data using BCP
create table testBCP(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
go
DBCC DROPCLEANBUFFERS
exec master..xp_cmdshell 'bcp test.dbo.testBCP in d:\work\test.txt -T -b1000000 -fd:\work\testImport-f-n.Fmt'
drop table testBCP
go
drop database test
C# .NET - Sql Bulk Insert from multiple delimited Textfile using c#.net的更多相关文章
- SQL SERVER – Import CSV File Into SQL Server Using Bulk Insert – Load Comma Delimited File Into SQL Server
CSV stands for Comma Separated Values, sometimes also called Comma Delimited Values. Create TestTabl ...
- SQL Server Bulk Insert批量数据导入
SQL Server的Bulk Insert语句可以将本地或远程的数据文件批量导入到数据库中,速度非常的快.远程文件必须共享才行,文件路径须使用通用约定(UNC)名称,即"\\服务器名或IP ...
- SQL Server批量数据导出导入Bulk Insert使用
简介 Bulk insert命令区别于BCP命令之处在于它是SQL server脚本语句,它可以将本地或远程的文件数据批量导入数据库,速度非常之快:远程文件必须共享才行, 文件路径须使用通用约定(UN ...
- BULK INSERT如何将大量数据高效地导入SQL Server
转载自:http://database.51cto.com/art/201108/282631.htm BULK INSERT如何将大量数据高效地导入SQL Server 本文我们详细介绍了BULK ...
- SQL SERVER 使用BULK Insert将txt文件中的数据批量插入表中(1)
1/首先建立数据表 CREATE TABLE BasicMsg( RecvTime FLOAT NOT NULL , --接收时间,不存在时间相同的数据 AA INT NOT NULL, --24位地 ...
- Sql server bulk insert
Bulk Insert Sql server 的bulk insert语句可以高效的导入大数据量的平面文件(txt,csv文件)到数据库的一张表中,其用法如下: bulk insert test fr ...
- [转]UTF-8 encoding support for the BCP utility and BULK INSERT Transact-SQL command in SQL Server 2014 SP2
本文转自:https://support.microsoft.com/en-us/help/3136780/utf-8-encoding-support-for-the-bcp-utility-and ...
- Bulk Insert:将文本数据(csv和txt)导入到数据库中
将文本数据导入到数据库中的方法有很多,将文本格式(csv和txt)导入到SQL Server中,bulk insert是最简单的实现方法 1,bulk insert命令,经过简化如下 BULK INS ...
- 笔记整理之 Bulk Insert
之前2篇日志整理了BCP大致的用法,这次整理一下它的兄弟 Bulk Insert 的写法以及和bcp那边的结合的用法. 首先,Bulk Insert 语句要在连接了Sql Server 服务器之后才执 ...
随机推荐
- eclipse CDT写c++使用文件作为输入源(输入重定向)
在main函数第一句添加下面. freopen("inputfile","r",stdin); 创建一个inputfile,放project根文件夹下. 注意添 ...
- Atitit 找人软福利建设 技术团队建设大概流程
Atitit 找人软福利建设 技术团队建设大概流程 火车公司有免费车座,餐馆有免费饭吃.. 软件公司嘛,就是软件资源,知识了...技术... 培训体系 大概的知识库体系..让他知道来到我们团队有着很高 ...
- 【Unity】12.1 基本概念
开发环境:Win10.Unity5.3.4.C#.VS2015 创建日期:2016-05-09 一.简介 导航网格(Navmesh)是世界坐标系中几何体的简化表示,被游戏代理用来进行全局导航.通常,代 ...
- HTML5学习笔记(二):HTML基础学习之二
表单 表单用来传递用户数据,多用来与后端进行数据交互. 前端: <!DOCTYPE html> <html lang="en"> <head> ...
- MyBean-关于plugMap共享对象
plugMap实现了对象的存储,使用setObject,和getObject来对对象进行存储 内部其实是一个列表,而且他会在释放的时候会情况尝试释放所有的对象,所以如果你共享的对象提前进行了释放, ...
- 【驱动】DM9000A网卡驱动框架源码分析
Linux网络设备结构 首先看一下Linux网络设备的结构,如下图: 网络协议接口层向网络层协议提供提供统一的数据包收发接口,不论上层协议为ARP还是IP,都通过dev_queue_xmit()函数发 ...
- java向mysql插入数据乱码
修改jdbc的链接,将原来的 jdbc:mysql://localhost:3306/demo改为 jdbc:mysql://localhost:3306/demo?us ...
- Animation.setFillAfter and Animation.setFillBefore的作用
转:http://blog.csdn.net/yangweigbh/article/details/9788531 setFillAfter(boolean fillAfter) 在Android ...
- 【Linux】crontab 每隔1小时 2小时的执行job写法
crontab -l crontab -e 每五分钟执行 */5 * * * * 每小时执行 0 * * * * 每2小时执行 0 */2 * * * 每天执行 0 0 ...
- 在python3.3后urllib2已经不能再用,只能用urllib.request来代替
版权声明:本文为博主原创文章,未经博主允许不得转载. 在python3.3后urllib2已经不能再用,只能用urllib.request来代替 response=urllib2.urlopen(' ...