DBLINK引起的SQL性能问题
最近发现报表系统上有一存储过程越来越慢，在数据库中查询后，发现有以下条SQL
--优化前：耗时>1h
select c.policyno,
c.endorseno,
r.item_code,
sum(r.outstanding_amount - r.settled_amount - r.offset_amount) OS_amount
from c_reserve_list@aclaim r
left join c_claim@aclaim c
on c.claimno = r.claim_no
where r.busi_phase in
(select max(busi_phase)
from rpt_st.c_reserve_list
where claim_no = r.claim_no
and last_modify_date =
(select max(last_modify_date)
from rpt_st.c_reserve_list
where claim_no = r.claim_no
and last_modify_date < to_date(V_ENDDATE, 'yyyymmdd')))
and r.count =
(select max(count)
from rpt_st.c_reserve_list
where claim_no = r.claim_no
and busi_phase = r.busi_phase
and count in
(select
count
from rpt_st.c_reserve_list
where claim_no = r.claim_no
and busi_phase = r.busi_phase
and last_modify_date < to_date(V_ENDDATE, 'yyyymmdd')))
group by c.policyno, c.endorseno, r.item_code;
--执行计划
Plan hash value: 3027187983
-------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Inst |IN-OUT|
-------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 22383 | 6251K| | 6751M (1)|999:59:59 | | |
| 1 | HASH GROUP BY | | 22383 | 6251K| 408G| 6751M (1)|999:59:59 | | |
|* 2 | FILTER | | | | | | | | |
|* 3 | HASH JOIN | | 1417M| 377G| 23M| 97M (1)|324:16:14 | | |
|* 4 | HASH JOIN RIGHT OUTER| | 144K| 22M| | 605 (2)| 00:00:08 | | |
| 5 | REMOTE | C_CLAIM | 5527 | 356K| | 153 (1)| 00:00:02 | AUTOC~ | R->S |
| 6 | REMOTE | C_RESERVE_LIST | 144K| 12M| | 450 (2)| 00:00:06 | AUTOC~ | R->S |
| 7 | VIEW | VW_SQ_1 | 1417M| 166G| | 88M (1)|293:20:25 | | |
| 8 | HASH GROUP BY | | 1417M| 248G| 601G| 88M (1)|293:20:25 | | |
|* 9 | HASH JOIN | | 1735M| 303G| 14M| 19266 (88)| 00:03:52 | | |
| 10 | REMOTE | C_RESERVE_LIST | 144K| 12M| | 450 (2)| 00:00:06 | NEWAU~ | R->S |
| 11 | REMOTE | C_RESERVE_LIST | 144K| 12M| | 450 (2)| 00:00:06 | NEWAU~ | R->S |
| 12 | SORT AGGREGATE | | 1 | 61 | | | | | |
|* 13 | FILTER | | | | | | | | |
| 14 | REMOTE | C_RESERVE_LIST | 1 | 61 | | 3 (0)| 00:00:01 | NEWAU~ | R->S |
| 15 | SORT AGGREGATE | | 1 | 51 | | | | | |
| 16 | REMOTE | C_RESERVE_LIST | 32 | 1632 | | 3 (0)| 00:00:01 | NEWAU~ | R->S |
-------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("R"."BUSI_PHASE"= (SELECT MAX("BUSI_PHASE") FROM "A1" WHERE "LAST_MODIFY_DATE"= (SELECT
MAX("LAST_MODIFY_DATE") FROM "A1" WHERE "CLAIM_NO"=:B1 AND "LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26
00:00:00', 'syyyy-mm-dd hh24:mi:ss')) AND "CLAIM_NO"=:B2))
3 - access("R"."COUNT"="VW_COL_1" AND "CLAIM_NO"="R"."CLAIM_NO" AND "BUSI_PHASE"="R"."BUSI_PHASE" AND
"CLAIM_NO"="R"."CLAIM_NO" AND "BUSI_PHASE"="R"."BUSI_PHASE")
4 - access("C"."CLAIMNO"(+)="R"."CLAIM_NO")
9 - access("COUNT"="COUNT")
13 - filter("LAST_MODIFY_DATE"= (SELECT MAX("LAST_MODIFY_DATE") FROM "A1" WHERE "CLAIM_NO"=:B1 AND
"LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))
Remote SQL Information (identified by operation id):
----------------------------------------------------
5 - SELECT "CLAIMNO","POLICYNO","ENDORSENO" FROM "C_CLAIM" "C" (accessing 'aclaim' )
6 - SELECT "CLAIM_NO","ITEM_CODE","OUTSTANDING_AMOUNT","SETTLED_AMOUNT","OFFSET_AMOUNT","BUSI_PHASE","CO
UNT" FROM "C_RESERVE_LIST" "SYS_ALIAS_2" (accessing 'aclaim' )
10 - SELECT "CLAIM_NO","LAST_MODIFY_DATE","BUSI_PHASE","COUNT" FROM "aclaim"."C_RESERVE_LIST" "A2"
WHERE "LAST_MODIFY_DATE"<:1 (accessing 'NEWaclaim' )
11 - SELECT "CLAIM_NO","BUSI_PHASE","COUNT" FROM "aclaim"."C_RESERVE_LIST" "A1" (accessing
'NEWaclaim' )
14 - SELECT "CLAIM_NO","LAST_MODIFY_DATE","BUSI_PHASE" FROM "aclaim"."C_RESERVE_LIST" "A1" WHERE
"CLAIM_NO"=:1 (accessing 'NEWaclaim' )
16 - SELECT "CLAIM_NO","LAST_MODIFY_DATE" FROM "aclaim"."C_RESERVE_LIST" "A1" WHERE "CLAIM_NO"=:1 AND
"LAST_MODIFY_DATE"<:2 (accessing 'NEWaclaim' )
分析
--在以上执行计划中可以看出该SQL，查询的都是远程表。所以可以改写成以下形式,然后在远程库直接执行。
select c.policyno,
c.endorseno,
r.item_code,
sum(r.outstanding_amount - r.settled_amount - r.offset_amount) OS_amount
from aclaim.c_reserve_list r
left join aclaim.c_claim c
on c.claimno = r.claim_no
where r.busi_phase in
(select max(busi_phase)
from aclaim.c_reserve_list
where claim_no = r.claim_no
and last_modify_date =
(select max(last_modify_date)
from aclaim.c_reserve_list
where claim_no = r.claim_no
and last_modify_date < to_date(20170626, 'yyyymmdd')))
and r.count =
(select max(count)
from aclaim.c_reserve_list
where claim_no = r.claim_no
and busi_phase = r.busi_phase
and count in
(select
count
from aclaim.c_reserve_list
where claim_no = r.claim_no
and busi_phase = r.busi_phase
and last_modify_date < to_date(20170626, 'yyyymmdd')))
group by c.policyno, c.endorseno, r.item_code;
--执行计划如下：
Plan hash value: 897376705
------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 22383 | 1508K| | 15M (2)| 52:55:06 |
| 1 | HASH GROUP BY | | 22383 | 1508K| 22M| 15M (2)| 52:55:06 |
|* 2 | FILTER | | | | | | |
|* 3 | HASH JOIN RIGHT OUTER | | 144K| 9723K| | 536 (3)| 00:00:07 |
| 4 | VIEW | index$_join$_002 | 5527 | 161K| | 83 (0)| 00:00:02 |
|* 5 | HASH JOIN | | | | | | |
| 6 | INDEX FAST FULL SCAN| IDX_C_CLAIM_03 | 5527 | 161K| | 46 (0)| 00:00:01 |
| 7 | INDEX FAST FULL SCAN| PK_C_CLAIM | 5527 | 161K| | 36 (0)| 00:00:01 |
| 8 | TABLE ACCESS FULL | C_RESERVE_LIST | 144K| 5496K| | 450 (2)| 00:00:06 |
| 9 | SORT AGGREGATE | | 1 | 24 | | | |
|* 10 | FILTER | | | | | | |
|* 11 | TABLE ACCESS FULL | C_RESERVE_LIST | 32 | 768 | | 449 (2)| 00:00:06 |
| 12 | SORT AGGREGATE | | 1 | 21 | | | |
|* 13 | TABLE ACCESS FULL | C_RESERVE_LIST | 32 | 672 | | 449 (2)| 00:00:06 |
| 14 | SORT AGGREGATE | | 1 | 46 | | | |
|* 15 | HASH JOIN | | 3 | 138 | | 901 (3)| 00:00:11 |
|* 16 | TABLE ACCESS FULL| C_RESERVE_LIST | 2 | 54 | | 450 (2)| 00:00:06 |
|* 17 | TABLE ACCESS FULL| C_RESERVE_LIST | 2 | 38 | | 450 (2)| 00:00:06 |
------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("R"."BUSI_PHASE"= (SELECT MAX("BUSI_PHASE") FROM "aclaim"."C_RESERVE_LIST"
"C_RESERVE_LIST" WHERE "LAST_MODIFY_DATE"= (SELECT MAX("LAST_MODIFY_DATE") FROM
"aclaim"."C_RESERVE_LIST" "C_RESERVE_LIST" WHERE "CLAIM_NO"=:B1 AND
"LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd hh24:mi:ss')) AND
"CLAIM_NO"=:B2) AND "R"."COUNT"= (SELECT MAX("COUNT") FROM "aclaim"."C_RESERVE_LIST"
"C_RESERVE_LIST","aclaim"."C_RESERVE_LIST" "C_RESERVE_LIST" WHERE "CLAIM_NO"=:B3 AND
"BUSI_PHASE"=:B4 AND "LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd
hh24:mi:ss') AND "COUNT"="COUNT" AND "CLAIM_NO"=:B5 AND "BUSI_PHASE"=:B6))
3 - access("C"."CLAIMNO"(+)="R"."CLAIM_NO")
5 - access(ROWID=ROWID)
10 - filter("LAST_MODIFY_DATE"= (SELECT MAX("LAST_MODIFY_DATE") FROM
"aclaim"."C_RESERVE_LIST" "C_RESERVE_LIST" WHERE "CLAIM_NO"=:B1 AND
"LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))
11 - filter("CLAIM_NO"=:B1)
13 - filter("CLAIM_NO"=:B1 AND "LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00',
'syyyy-mm-dd hh24:mi:ss'))
15 - access("COUNT"="COUNT")
16 - filter("CLAIM_NO"=:B1 AND "BUSI_PHASE"=:B2 AND "LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26
00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
17 - filter("CLAIM_NO"=:B1 AND "BUSI_PHASE"=:B2)
--执行计划中有filter关键字且有两个子级，一般来说是很耗费性能。
--如果不进行改写，根据自己的经验，最有效的方式就是在执行计划中谓词信息里找出有绑定变量的字段并建立索引。
--所以建立如下索引：
create index IDX_C_RESERVE_LIST_TEST on C_RESERVE_LIST(CLAIM_NO,BUSI_PHASE,LAST_MODIFY_DATE);
执行SQL，10s内能返回全部结果。
但是，在报表库上运行时，发现SQL的性能还是没改善。
再次分析执行计划：
发现此sql运行时，采用多个dblink访问远程库。查看dblink的元数据：
CREATE OR REPLACE SYNONYM "rpt_st"."C_RESERVE_LIST" FOR "aclaim"."C_RESERVE_LIST"@"NEWaclaim";
create public database link aclaim connect to aclaim using 'DOICLC';
create public database link NEWaclaim connect to aclaim using 'DOICLC';发现两个DBLINK都是访问AUTOCLAIM用户下的表。
这种方式很容易造成sql不能走正确执行计划，所以可以把sql中的dblink改写成同一个。
再次执行，10s内能返回全部结果。
SELECT C.POLICYNO,
C.ENDORSENO,
R.ITEM_CODE,
SUM(R.OUTSTANDING_AMOUNT - R.SETTLED_AMOUNT - R.OFFSET_AMOUNT) OS_AMOUNT
FROM rpt_st.C_RESERVE_LIST R
LEFT JOIN rpt_st.C_CLAIM C
ON C.CLAIMNO = R.CLAIM_NO
WHERE R.BUSI_PHASE IN
(SELECT MAX(BUSI_PHASE)
FROM rpt_st.C_RESERVE_LIST
WHERE CLAIM_NO = R.CLAIM_NO
AND LAST_MODIFY_DATE =
(SELECT MAX(LAST_MODIFY_DATE)
FROM rpt_st.C_RESERVE_LIST
WHERE CLAIM_NO = R.CLAIM_NO
AND LAST_MODIFY_DATE < TO_DATE('20170626', 'yyyymmdd')))
AND R.COUNT =
(SELECT MAX(COUNT)
FROM rpt_st.C_RESERVE_LIST
WHERE CLAIM_NO = R.CLAIM_NO
AND BUSI_PHASE = R.BUSI_PHASE
AND COUNT IN
(SELECT COUNT
FROM rpt_st.C_RESERVE_LIST
WHERE CLAIM_NO = R.CLAIM_NO
AND BUSI_PHASE = R.BUSI_PHASE
AND LAST_MODIFY_DATE < TO_DATE('20170626', 'yyyymmdd')))
GROUP BY C.POLICYNO, C.ENDORSENO, R.ITEM_CODE
Plan hash value: 2524269579
-------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Inst |
-------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT REMOTE | | 22383 | 1508K| | 109K (1)| 00:21:55 | |
| 1 | HASH GROUP BY | | 22383 | 1508K| 22M| 109K (1)| 00:21:55 | |
|* 2 | FILTER | | | | | | | |
|* 3 | HASH JOIN RIGHT OUTER | | 144K| 9723K| | 536 (3)| 00:00:07 | |
| 4 | VIEW | index$_join$_002 | 5527 | 161K| | 83 (0)| 00:00:02 | DOICLC |
|* 5 | HASH JOIN | | | | | | | |
| 6 | INDEX FAST FULL SCAN | IDX_C_CLAIM_03 | 5527 | 161K| | 46 (0)| 00:00:01 | DOICLC |
| 7 | INDEX FAST FULL SCAN | PK_C_CLAIM | 5527 | 161K| | 36 (0)| 00:00:01 | DOICLC |
| 8 | TABLE ACCESS FULL | C_RESERVE_LIST | 144K| 5496K| | 450 (2)| 00:00:06 | DOICLC |
| 9 | SORT AGGREGATE | | 1 | 24 | | | | |
| 10 | FIRST ROW | | 1 | 24 | | 3 (0)| 00:00:01 | |
|* 11 | INDEX RANGE SCAN (MIN/MAX) | IDX_C_RESERVE_LIST_TEST | 1 | 24 | | 3 (0)| 00:00:01 | DOICLC |
| 12 | SORT AGGREGATE | | 1 | 21 | | | | |
|* 13 | INDEX RANGE SCAN | IDX_C_RESERVE_LIST_TEST | 32 | 672 | | 3 (0)| 00:00:01 | DOICLC |
| 14 | SORT AGGREGATE | | 1 | 46 | | | | |
|* 15 | HASH JOIN | | 3 | 138 | | 9 (12)| 00:00:01 | |
| 16 | TABLE ACCESS BY INDEX ROWID| C_RESERVE_LIST | 2 | 54 | | 4 (0)| 00:00:01 | DOICLC |
|* 17 | INDEX RANGE SCAN | IDX_C_RESERVE_LIST_TEST | 2 | | | 3 (0)| 00:00:01 | DOICLC |
| 18 | TABLE ACCESS BY INDEX ROWID| C_RESERVE_LIST | 2 | 38 | | 4 (0)| 00:00:01 | DOICLC |
|* 19 | INDEX RANGE SCAN | IDX_C_RESERVE_LIST_TEST | 2 | | | 3 (0)| 00:00:01 | DOICLC |
-------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter("A2"."BUSI_PHASE"= (SELECT MAX("A4"."BUSI_PHASE") FROM "aclaim"."C_RESERVE_LIST" "A4" WHERE
"A4"."CLAIM_NO"=:B1 AND "A4"."LAST_MODIFY_DATE"= (SELECT MAX("A5"."LAST_MODIFY_DATE") FROM
"aclaim"."C_RESERVE_LIST" "A5" WHERE "A5"."CLAIM_NO"=:B2 AND "A5"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26
00:00:00', 'syyyy-mm-dd hh24:mi:ss'))) AND "A2"."COUNT"= (SELECT MAX("A3"."COUNT") FROM "aclaim"."C_RESERVE_LIST"
"A3","aclaim"."C_RESERVE_LIST" "A6" WHERE "A6"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd
hh24:mi:ss') AND "A6"."BUSI_PHASE"=:B3 AND "A6"."CLAIM_NO"=:B4 AND "A3"."BUSI_PHASE"=:B5 AND "A3"."CLAIM_NO"=:B6 AND
"A3"."COUNT"="A6"."COUNT"))
3 - access("A1"."CLAIMNO"(+)="A2"."CLAIM_NO")
5 - access(ROWID=ROWID)
11 - access("A4"."CLAIM_NO"=:B1)
filter("A4"."LAST_MODIFY_DATE"= (SELECT MAX("A5"."LAST_MODIFY_DATE") FROM "aclaim"."C_RESERVE_LIST" "A5"
WHERE "A5"."CLAIM_NO"=:B1 AND "A5"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))
13 - access("A5"."CLAIM_NO"=:B1 AND "A5"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd
hh24:mi:ss'))
filter("A5"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26 00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
15 - access("A3"."COUNT"="A6"."COUNT")
17 - access("A6"."CLAIM_NO"=:B1 AND "A6"."BUSI_PHASE"=:B2 AND "A6"."LAST_MODIFY_DATE"<TO_DATE(' 2017-06-26
00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
19 - access("A3"."CLAIM_NO"=:B1 AND "A3"."BUSI_PHASE"=:B2)
Note
-----
- fully remote statement
DBLINK引起的SQL性能问题的更多相关文章
- SQL性能优化常见措施(Lock wait timeout exceeded)
SQL性能优化常见措施 目 录 1.mysql中explain命令使用 2.mysql中mysqldumpslow的使用 3.mysql中修改my.ini配置文件记录日志 4.mysql中如何加索引 ...
- SQL性能优化案例分析
这段时间做一个SQL性能优化的案例分析, 整理了一下过往的案例,发现一个比较有意思的,拿出来给大家分享. 这个项目是我在项目开展2期的时候才加入的, 之前一期是个金融内部信息门户, 里面有个功能是收集 ...
- SQL性能优化：如何定位网络性能问题
一同事跟我反馈他遇到了一个SQL性能问题,他说全表只有69条记录,客户端执行耗费了两分多钟,这不科学呀.要我分析一下原因并解决.我按照类似表结构,构造了一个案例,测试截图如下所示 这个表有13800K ...
- SQL性能优化
引言: 以前在面试的过程中,总有面试官问道:你做过sql性能优化吗?对此,我的答复是没有.一次没有不是自己的错误,两次也不是,但如果是多次呢?今天痛下决心,把有关sql性能优化的相关知识总结一下,以便 ...
- Oracle 数据库SQL性能查看
作为一个开发/测试人员,或多或少都得和数据库打交道,而对数据库的操作归根到底都是SQL语句,所有操作到最后都是操作数据,那么对sql性能的掌控又成了我们工作中一件非常重要的工作.下面简单介绍下一些查看 ...
- 如何进行正确的SQL性能优化
在SQL查询中,为了提高查询的效率,我们常常采取一些措施对查询语句进行SQL性能优化.本文我们总结了一些优化措施,接下来我们就一一介绍. 1.查询的模糊匹配 尽量避免在一个复杂查询里面使用 LIKE ...
- 使用show profiles分析SQL性能
如何查看执行SQL的耗时 使用show profiles分析sql性能. Show profiles是5.0.37之后添加的,要想使用此功能,要确保版本在5.0.37之后. 查看数据库版本 mysql ...
- Oracle DB SQL 性能分析器
• 确定使用SQL 性能分析器的优点 • 描述SQL 性能分析器工作流阶段 • 使用SQL 性能分析器确定数据库更改所带来的性能改进 SQL 性能分析器:概览 • 11g 的新增功能 • 目标用户:D ...
- SQL Select count(*)和Count（1）的区别和执行方式及SQL性能优化
SQL性能优化:http://www.cnblogs.com/CareySon/category/360333.html Select count(*)和Count(1)的区别和执行方式 在SQL S ...
随机推荐
- bzoj 2599: [IOI2011]Race【点分治】
点分治,用一个mn[v]数组记录当前root下长为v的链的最小深度,每次新加一个儿子的时候都在原来儿子更新过的mn数组里更新ans(也就是查一下mn[m-dis[p]]+de[p]) 这里注意更新和初 ...
- 【CodeForces - 651C 】Watchmen（map）
Watchmen 直接上中文 Descriptions: 钟表匠们的好基友马医生和蛋蛋现在要执行拯救表匠们的任务.在平面内一共有n个表匠,第i个表匠的位置为(xi, yi). 他们需要安排一个任务计划 ...
- c++ const的使用
const是用来声明一个常量的,当你不想让一个值被改变时就用const,const int max && int const max 是没有区别的,都可以.不涉及到指针const很好理 ...
- UvaLive6442（思维、结论）
结论是:按位置排序好以后,对于真正的答案,走法应该是:依次走向第0个等分点,第1个等分点……这样对于这种等分情况,是最优的调度. /* 先假设一个终点位置然后按位站好 这个位置不一定是最优所以要调 调 ...
- Windows下DVWA安装指南
注意:DVWA需要依赖httpd.PHP.MySQL.php-mysql等应用或组件,最简单的方法是安装wampserver(http://www.wampserver.com/),安装完了所需的各种 ...
- magento layout xml 小结
基础概念: http://magebase.com/magento-tutorials/demystifying-magentos-layout-xml-part-1/ 调试方案函数: $this-& ...
- [转]Java中Date转换大全，返回yyyy-MM-dd的Date类型
/** * 获取现在时间,这个好用 * * @return返回长时间格式 yyyy-MM-dd HH:mm:ss */ public static Date getSqlDate() { Date s ...
- 【前端】模拟微信上传图片（带预览，支持预览gif）
一.Html <style type="text/css"> #previewDiv{width:50px;height:50px;overflow:hidden;po ...
- flex和box兼容性写法
display: -webkit-box; /* Chrome 4+, Safari 3.1, iOS Safari 3.2+ */ display: -moz-box; /* Firefox 17- ...
- Angular JS中变量定义的基本原则
在Angular JS开发中,经常需要定义一些变量,关于这些变量的定义方法及作用域应该注意以下几点: 1. 如果能用局部变量解决问题,尽量不要用全局变量. 2. 需要与界面双向绑定的变量采用$scop ...