w分布式查询、数据聚合、跨碎片join是可且应避免的、自增主键管理、基于-会话/事务/语句-选择碎片、通过-主键/模块/碎片索引-碎片化数据

http://www.agildata.com/database-sharding/

Database Sharding Challenges

Due to the distributed nature of individual databases, a number of key elements must be taken into account:

  • Reliability. First and foremost, any production business application must be reliable and fault-tolerant, and cannot be subject to frequent outages. The database tier is often the single most critical element in any reliability design, and therefore an implementation of Database Sharding is no exception. In fact, due to the distributed nature of multiple shard databases, the criticality of a well-designed approach is even greater. To ensure a fault-tolerant and reliable approach, the following items are required:

    • Automated backups of individual Database Shards.
    • Database Shard redundancy, ensuring at least 2 “live” copies of each shard are available in the event of an outage or server failure. This requires a high-performance, efficient, and reliable replication mechanism.
    • Cost-effective hardware redundancy, both within and across servers.
    • Automated failover when an outage or server failure occurs.
    • Disaster Recovery site management.
  • Distributed queries. Many types of queries can be processed far faster using distributed queries, performing parallel processing of interim results on each shard server. This technique can achieve order-of-magnitude improvements in performance, in many cases 10X or more. To enable distributed queries in a seamless manner for the application, it is important to have a facility that can process a segment of the query on each individual shard, and then consolidate the results into a single result set for the application tier. Common queries that can benefit from distributed processing are:
    • Aggregation of statistics, requiring a broad sweep of data across the entire system. Such an example is the computation of sales by product, which ordinarily requires evaluation of the entire database.
    • Queries that support comprehensive reports, such as listings of all individual customers that purchased a given product in the last day, week or month.
  • Avoidance of cross-shard joins. In a sharded system, queries or other statements that use inner-joins that span shards are highly inefficient and difficult to perform. In the majority of cases, it has been found that such inner-joins are not actually required by an application, so long as the correct techniques are applied. The primary technique is the replication of Global Tables, the relatively static lookup tables that are common utilized when joining to much larger primary tables. Tables containing values as Status Codes, Countries, Types, and even Products fall into this category. What is required is an automated replication mechanism that ensures values for Global Tables are in synch across all shards, minimizing or eliminating the need for cross-shard joins.
  • Auto-increment key management. Typical auto-increment functionality provided by database management systems generate a sequential key for each new row inserted into the database. This is fine for a single database application, but when using Database Sharding, keys must be managed across all shards in a coordinated fashion. The requirement here is to provide a seamless, automated method of key generation to the application, one that operates across all shards, ensuring that keys are unique across the entire system.
  • Support for multiple Shard Schemes. It is important to note that Database Sharding is effective because it offers an application specific technique for massive scalability and performance improvements. In fact it can be said that the degree of effectiveness is directly related to how well the sharding algorithms themselves are tailored to the application problem at hand. What is required is a set of multiple, flexible shard schemes, each designed to address a specific type of application problem. Each scheme has inherent performance and/or application characteristics and advantages when applied to a specific problem domain. In fact, using the wrong shard scheme can actually inhibit performance and the very results you are trying to obtain. It is also not uncommon for a single application to use more than one shard scheme, each applied to a specific portion of the application to achieve optimum results. Here is a list of some common shard schemes:
    • Session-based sharding, where each individual user or process interacts with a specific shard for the duration of the user or process session. This is the simplest technique to implement, and adds virtually zero overhead to overall performance, since the sharding decision is made only once per session. Applications which can benefit from this approach are often customer-centric, where all data for a given customer is contained in a single shard, and that is all the data that the customer requires.
    • Transaction-based sharding determines the shard by examining the first SQL Statement in a given database transaction. This is normally done by evaluating the “shard key” value used in the statement (such as an Order Number), and then directing all other statements in the transaction to the same shard.
    • Statement-based sharding is the most process intensive of all types, evaluating each individual SQL Statement to determine the appropriate shard to direct it to. Again, evaluation of the shard key value is required. This option is often desirable on high-volume, granular transactions, such as recording phone call records.
  • Determine the optimum method for sharding the data. This is another area that is highly variable, change from application to application. It is closely tied with the selection of the Database Shard Scheme described above. There are numerous methods for deciding how to shard your data, and its important to understand your transaction rates, table volumes, key distribution, and other characteristics of your application. This data is required to determine the optimum sharding strategy:
    • Shard by a primary key on a table. This is the most straightforward option, and easiest to map to a given application. However, this is only effective if your data is reasonably well distributed. For example, if you elected to shard by Customer ID (and this is a sequential numeric value), and most of your transactions are for new customers, very little if anything will be gained by sharding your database. On the other hand, if you can select a key that does adequately and naturally distribute your transactions, great benefits can be realized.
    • Shard by the modulus of a key value. This option works in a vast number of cases, by applying the modulus function to the key value, and distributing transactions based on the calculated value. In essence you can predetermine any number of shards, and the modulus function effectively distributes across your shards on a “round-robin” basis, creating a very even distribution of new key values.
    • Maintain a master shard index table. This technique involves using a single master table that maps various values to specific shards. It is very flexible, and meets a wide variety of application situations. However, this option often delivers lower performance as it requires an extra lookup for each sharded SQL Statement.

As you can see, there are many things to consider and many capabilities required in order to ensure that a Database Sharding implementation is successful and effective, delivering on its objectives of providing new levels of scalability and performance in a cost-effective manner.

Database Sharding Challenges DATABASE SHARDING的更多相关文章

  1. Azure SQL Database (19) Stretch Database 概览

    <Windows Azure Platform 系列文章目录>  Azure SQL Database (19) Stretch Database 概览      Azure SQL Da ...

  2. 使用duplicate target database ... from active database复制数据库

    使用duplicate target database ... from active database复制数据库 source db:ora11auxiliary db:dupdb 1.修改监听文件 ...

  3. Oracle® Database Patch 19121551 - Database Patch Set Update 11.2.0.4.4 (Includes CPUOct2014) - 傲游云浏览

    Skip Headers Oracle® Database Patch 19121551 - Database Patch Set Update 11.2.0.4.4 (Includes CPUOct ...

  4. Oracle Database 12c Using duplicate standby database from active database Created Active DataGuard

    primary database db_name=zwc, db_unique_name=zwc standby database db_name=zwc, db_unique_name=standb ...

  5. Teradata Delete Database and Drop Database

    DELETE DATABASE and DELETE USER statements delete all data tables, views, and macros from a database ...

  6. Cannot connect to database because the database client

    问题描述: arcgis server10.1  arcgis sde10出现下面问题 Cannot connect to  database because the database client ...

  7. Database Partitioning Options DATABASE SHARDING

    w主写从读.集群节点间时时内存复制.单表横切纵切.分析报表系统通过服务器联表 http://www.agildata.com/database-sharding/ Database Partition ...

  8. Database Corruption ->> Fix Database In Suspect State

    昨天在工作中遇到一个情况,就是Development环境中的某台服务器上的某个数据库进入了Suspect状态.以前看书倒是知道说这个状态,不过实际工作当中从来没有遇到过.那么一些背景情况是这样的. 环 ...

  9. What is the difference between database table and database view?

    The database table has a physical existence in the database. A view is a virtual table, that is one ...

随机推荐

  1. linux -- 查看Ubuntu命令行调用的文件

    which 如: 输入:which reboot 输出:/sbin/reboot 输入:which shutdown -h now 输出:/sbin/shutdown

  2. Struts2 ajax json使用介绍

    一.jar包首先引入Struts和json所需的jar包. watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvaXRteWhvbWUxOTkw/font/5a6 ...

  3. 搭建LNMP+CI环境

    首先搭建 LNMP 的服务器环境 安装 Nginx, MySQL 和 PHP 软件包,执行以下命令 yum install -y nginx mariadb-server mariadb php ph ...

  4. linux中,查看某个命令是来自哪个RPM包或者是通过哪个RPM包安装的

    需求描述: 今天在测试ssh命令到底是哪个RPM包,安装之后生成的,找了一些文档 在这里进行记录下,主要是rpm -qf命令的使用,查询文件在哪个包里. 操作过程: 1.通过whereis 定位ssh ...

  5. 合格前端系列第六弹-从指向看JavaScript

    https://my.oschina.net/qiangdada/blog/1484001

  6. 把Excel中的数据转换成Sql语句

    假如excel表格中有A.B.C三列数据,希望导入到数据库users表中,相应的字段各自是name,sex,age ,在你的excel表格中添加一列.利用excel的公式自己主动生成sql语句,方法例 ...

  7. java中被遗忘的native关键字

    我是无意间看见JNI( java调用动态链接库dll )这块的东西. 所有记下来:本地声明方法  装载完成dll文件后,将使用的方法用native关键字声明. public native static ...

  8. C文件流

    在Linux系统中,系统默认认为每个进程打开了3个文件,即每个进程默认可以操作3 个流,即标准输入了流(/dev/stdin),标准输出流(/dev/stdout),标准错误输出流(/dev/stde ...

  9. Android底层有一定的认识,研究过相关的Android源码

    一.系统架构: 一).系统分层:(由下向上)[如图] 1.安卓系统分为四层,分别是Linux内核层.Libraries层.FrameWork层,以及Applications层: 其中Linux内核层包 ...

  10. 面试题思考:Servlet 生命周期、工作原理

    Servlet 生命周期:Servlet 加载--->实例化--->服务--->销毁. init():在Servlet的生命周期中,仅执行一次init()方法.它是在服务器装入Ser ...