Database Sharding Challenges DATABASE SHARDING
w分布式查询、数据聚合、跨碎片join是可且应避免的、自增主键管理、基于-会话/事务/语句-选择碎片、通过-主键/模块/碎片索引-碎片化数据
http://www.agildata.com/database-sharding/
Database Sharding Challenges
Due to the distributed nature of individual databases, a number of key elements must be taken into account:
- Reliability. First and foremost, any production business application must be reliable and fault-tolerant, and cannot be subject to frequent outages. The database tier is often the single most critical element in any reliability design, and therefore an implementation of Database Sharding is no exception. In fact, due to the distributed nature of multiple shard databases, the criticality of a well-designed approach is even greater. To ensure a fault-tolerant and reliable approach, the following items are required:
- Automated backups of individual Database Shards.
- Database Shard redundancy, ensuring at least 2 “live” copies of each shard are available in the event of an outage or server failure. This requires a high-performance, efficient, and reliable replication mechanism.
- Cost-effective hardware redundancy, both within and across servers.
- Automated failover when an outage or server failure occurs.
- Disaster Recovery site management.
- Distributed queries. Many types of queries can be processed far faster using distributed queries, performing parallel processing of interim results on each shard server. This technique can achieve order-of-magnitude improvements in performance, in many cases 10X or more. To enable distributed queries in a seamless manner for the application, it is important to have a facility that can process a segment of the query on each individual shard, and then consolidate the results into a single result set for the application tier. Common queries that can benefit from distributed processing are:
- Aggregation of statistics, requiring a broad sweep of data across the entire system. Such an example is the computation of sales by product, which ordinarily requires evaluation of the entire database.
- Queries that support comprehensive reports, such as listings of all individual customers that purchased a given product in the last day, week or month.
- Avoidance of cross-shard joins. In a sharded system, queries or other statements that use inner-joins that span shards are highly inefficient and difficult to perform. In the majority of cases, it has been found that such inner-joins are not actually required by an application, so long as the correct techniques are applied. The primary technique is the replication of Global Tables, the relatively static lookup tables that are common utilized when joining to much larger primary tables. Tables containing values as Status Codes, Countries, Types, and even Products fall into this category. What is required is an automated replication mechanism that ensures values for Global Tables are in synch across all shards, minimizing or eliminating the need for cross-shard joins.
- Auto-increment key management. Typical auto-increment functionality provided by database management systems generate a sequential key for each new row inserted into the database. This is fine for a single database application, but when using Database Sharding, keys must be managed across all shards in a coordinated fashion. The requirement here is to provide a seamless, automated method of key generation to the application, one that operates across all shards, ensuring that keys are unique across the entire system.
- Support for multiple Shard Schemes. It is important to note that Database Sharding is effective because it offers an application specific technique for massive scalability and performance improvements. In fact it can be said that the degree of effectiveness is directly related to how well the sharding algorithms themselves are tailored to the application problem at hand. What is required is a set of multiple, flexible shard schemes, each designed to address a specific type of application problem. Each scheme has inherent performance and/or application characteristics and advantages when applied to a specific problem domain. In fact, using the wrong shard scheme can actually inhibit performance and the very results you are trying to obtain. It is also not uncommon for a single application to use more than one shard scheme, each applied to a specific portion of the application to achieve optimum results. Here is a list of some common shard schemes:
- Session-based sharding, where each individual user or process interacts with a specific shard for the duration of the user or process session. This is the simplest technique to implement, and adds virtually zero overhead to overall performance, since the sharding decision is made only once per session. Applications which can benefit from this approach are often customer-centric, where all data for a given customer is contained in a single shard, and that is all the data that the customer requires.
- Transaction-based sharding determines the shard by examining the first SQL Statement in a given database transaction. This is normally done by evaluating the “shard key” value used in the statement (such as an Order Number), and then directing all other statements in the transaction to the same shard.
- Statement-based sharding is the most process intensive of all types, evaluating each individual SQL Statement to determine the appropriate shard to direct it to. Again, evaluation of the shard key value is required. This option is often desirable on high-volume, granular transactions, such as recording phone call records.
- Determine the optimum method for sharding the data. This is another area that is highly variable, change from application to application. It is closely tied with the selection of the Database Shard Scheme described above. There are numerous methods for deciding how to shard your data, and its important to understand your transaction rates, table volumes, key distribution, and other characteristics of your application. This data is required to determine the optimum sharding strategy:
- Shard by a primary key on a table. This is the most straightforward option, and easiest to map to a given application. However, this is only effective if your data is reasonably well distributed. For example, if you elected to shard by Customer ID (and this is a sequential numeric value), and most of your transactions are for new customers, very little if anything will be gained by sharding your database. On the other hand, if you can select a key that does adequately and naturally distribute your transactions, great benefits can be realized.
- Shard by the modulus of a key value. This option works in a vast number of cases, by applying the modulus function to the key value, and distributing transactions based on the calculated value. In essence you can predetermine any number of shards, and the modulus function effectively distributes across your shards on a “round-robin” basis, creating a very even distribution of new key values.
- Maintain a master shard index table. This technique involves using a single master table that maps various values to specific shards. It is very flexible, and meets a wide variety of application situations. However, this option often delivers lower performance as it requires an extra lookup for each sharded SQL Statement.
As you can see, there are many things to consider and many capabilities required in order to ensure that a Database Sharding implementation is successful and effective, delivering on its objectives of providing new levels of scalability and performance in a cost-effective manner.
Database Sharding Challenges DATABASE SHARDING的更多相关文章
- Azure SQL Database (19) Stretch Database 概览
<Windows Azure Platform 系列文章目录> Azure SQL Database (19) Stretch Database 概览 Azure SQL Da ...
- 使用duplicate target database ... from active database复制数据库
使用duplicate target database ... from active database复制数据库 source db:ora11auxiliary db:dupdb 1.修改监听文件 ...
- Oracle® Database Patch 19121551 - Database Patch Set Update 11.2.0.4.4 (Includes CPUOct2014) - 傲游云浏览
Skip Headers Oracle® Database Patch 19121551 - Database Patch Set Update 11.2.0.4.4 (Includes CPUOct ...
- Oracle Database 12c Using duplicate standby database from active database Created Active DataGuard
primary database db_name=zwc, db_unique_name=zwc standby database db_name=zwc, db_unique_name=standb ...
- Teradata Delete Database and Drop Database
DELETE DATABASE and DELETE USER statements delete all data tables, views, and macros from a database ...
- Cannot connect to database because the database client
问题描述: arcgis server10.1 arcgis sde10出现下面问题 Cannot connect to database because the database client ...
- Database Partitioning Options DATABASE SHARDING
w主写从读.集群节点间时时内存复制.单表横切纵切.分析报表系统通过服务器联表 http://www.agildata.com/database-sharding/ Database Partition ...
- Database Corruption ->> Fix Database In Suspect State
昨天在工作中遇到一个情况,就是Development环境中的某台服务器上的某个数据库进入了Suspect状态.以前看书倒是知道说这个状态,不过实际工作当中从来没有遇到过.那么一些背景情况是这样的. 环 ...
- What is the difference between database table and database view?
The database table has a physical existence in the database. A view is a virtual table, that is one ...
随机推荐
- WF4.0——升级技能:泛型应用
前提: 在项目的开发中.我们知道,增加泛型,通过对类型的封装,进行抽象后.能够大大降低我们代码量,在项目中,泛型能够说是高级project师必备的技能之中的一个.也是面向对象的核心"抽象&q ...
- CentOS 6.5 下Vim 配置图解
分享个CentOS 6.5 下Vim 配置图文详解,希望对大家有所帮助. 1. 登录并进入你常用的用户名下,查看其主目录 命令: # su xxx $ cd xxx $ ls -a 2.查看并建立目录 ...
- JS的事件冒泡和事件捕获
先上结论:他们是描述事件触发时序问题的术语.事件捕获指的是从document到触发事件的那个节点,即自上而下的去触发事件.相反的,事件冒泡是自下而上的去触发事件.绑定事件方法的第三个参数,就是控制事件 ...
- XP 终端服务组件 恢复补丁包 terminal service patch
terminal 终端服务组件恢复包 下载地址(点击) winconnect server xp软件 下载地址(点击)
- XLua系统学习
官方网站:https://github.com/Tencent/xLua 学习手册:http://manual.luaer.cn/ 技术博客: http://blog.csdn.net/column/ ...
- IIS 使用多个https和通配证书解决方案
环境:OS :WINDOWS 2008 IIS: IIS7 域名:三个二级域名 问题:由于一个网站只支持一个443,但可以通过更改配置得到绑定不同域名.但由于公用证书,所以问题出来.只能为一个二级域名 ...
- 【java】java 设计模式(4):建造者模式(Builder)
工厂类模式提供的是创建单个类的模式,而建造者模式则是将各种产品集中起来进行管理,用来创建复合对象,所谓复合对象就是指某个类具有不同的属性,其实建造者模式就是前面抽象工厂模式和最后的Test结合起来得到 ...
- CRC-16校验C#代码
[csharp] view plaincopyprint? using System; using System.Collections.Generic; using System.Text; usi ...
- Javascript 字符串替换
<!DOCTYPE html><html><head> <meta http-equiv="Content-Type" content=& ...
- ionic安装及测试
官方教程: http://ionicframework.com/getting-started/ 官方教程写得比较简单,简单来说就是 1)安装nodejs(安装方法:http://www.cnblog ...