DB异常状态：Recovery Pending，Suspect，估计Recovery的剩余时间

一，RECOVERY PENDING状态

今天修改了SQL Server的Service Account的密码，然后重启SQL Server的Service，发现有db处于Recovery Pending状态。

Recovery Pending状态是指：数据库在还原（recovery）时遇到跟资源相关的错误，虽然数据库没有损坏，但是文件可能丢失，或者系统资源的限制，导致该数据库不能开始还原进程。数据库处于Recovery Pending 状态，表明还原进程被挂起，数据库不能开始数据库的数据和日志的还原进程；这种情况，不能说慢Recovery失败，因为Recovery还没有开始。这种情况下，最可能的原因是丢失数据文件或日志文件。

对于Recovery Pending状态，应该如何修复：

ALTER DATABASE [DB_Name] SET  SINGLE_USER WITH NO_WAIT
ALTER DATABASE [DB_Name] SET EMERGENCY;
DBCC checkdb ([DB_Name], REPAIR_ALLOW_DATA_LOSS  )
ALTER DATABASE [DB_Name] SET online;
ALTER DATABASE [DB_Name] SET  Multi_USER WITH NO_WAIT

在使用CheckDB命令Repair之前，查看DB的大小

select DB_NAME(mf.database_id) as DatabaseName,
    mf.type_desc as FileType,
    mf.name as FileLogicName,
    mf.physical_name as FilePhysicalName,
    mf.size as PagesCount,
    mf.size*8/1024  as Size_MB,
    mf.size*8/1024/1024.0 as Size_GB
from sys.master_files mf
where mf.database_id= db_id(N'dbname')

在执行时，出现各种问题：

1，User does not have permission to alter database 'Office365', the database does not exist, or the database is not in a state that allows access checks.

2，Database 'Office365' cannot be opened due to inaccessible files or insufficient memory or disk space.  See the SQL Server errorlog for details.

最后，我到File 的 Physical path下，找不到相应的MDF文件，但是Log文件是存在的，并且log文件最后修改的时间离现在有2年，可能是被遗弃的DB。修改 Service Account ，不会删除一个18GB的MDF文件，向Leader询问，Leader说这是一个被废弃的DB。虚惊一场，像这种，MDF文件被删除，Log文件还保存的情况，数据文件肯定是被强制删除。

有惊无险，血泪的教训：在Service Restart 之前，一定确保DB没有在运行更新操作，并使用checkpoint保存脏数据。

二，估计Recovery的剩余时间

当一个DB处于 In Recovery 状态时，用户是不能访问的，如果Recovery时间很长，那么对一个DBA来说，等待的过程是虐心的，DBA需要知道剩余的还原时间。如何预测一个DB从In Recovery 状态，还原到正常Online状态所需的时间？ SQL Server 没有直接给出答案，但是，在Recovery的过程中SQL Server将还原进程记录到ErrorLog中，可以通过Recovery的历史记录来估计剩余的完成时间。

DECLARE @DBName VARCHAR(64) = 'databasename'

DECLARE @ErrorLog AS TABLE
(
[LogDate] CHAR(24),
[ProcessInfo] VARCHAR(64),
[TEXT] VARCHAR(MAX)
)

INSERT INTO @ErrorLog
EXEC master..sp_readerrorlog 0, 1, 'Recovery of database', @DBName

SELECT TOP 11
     [LogDate]
    ,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
    ,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
    ,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining
    ,[TEXT]

FROM @ErrorLog
ORDER BY [LogDate] DESC

在SQL Server的Log中，记录的消息是：

Recovery of database 'database name' (16) is 0% complete (approximately 303767 seconds remain). Phase 1 of 3. This is an informational message only. No user action is required.

Recovery of database 'database name' (16) is 0% complete (approximately 396166 seconds remain). Phase 2 of 3. This is an informational message only. No user action is required.

三，Database 处于Suspect状态

在物理机安装Windows更新，重启之后，发现该Server上有一个DB处于Suspect状态，该DB的Files分布在不同的Server上，我怀疑是在Remote Server重启时，导致该DB不能访问Remote Files，因此，SQL Server 进入 Suspect状态。

查看Windows 日志报告，发现一下错误信息：

The operating system returned error 53(The network path was not found.) to SQL Server during a read at offset 0x000001bed08000 in file '\\RemoteServerName\ShareFolder\xxxx.ndf'. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

这个错误是由于Remote Server重启，导致该DB不能访问位于Remote Server上的Files，数据库的文件并没有损坏。所以，解决方法是：等到所有的Remote Server都重启之后，只需要使该DB先脱机（offline），再联机（Online），SQL Server会自动检测该数据库的完整性，如果该DB的所有Files都能正常访问，该DB就会恢复到正常的Online状态。

alter database database_name
set offline
--wait for some seconds
alter database database_name
set online

附件：

数据库的状态和描述：

• ONLINE:Database is available for access. The primary filegroup is online, although the undo phase of recovery may not have been completed.
• OFFLINE:Database is unavailable. A database becomes offline by explicit user action and remains offline until additional user action is taken. For example, the database may be taken offline in order to move a file to a new disk. The database is then brought back online after the move has been completed.
• RESTORING:One or more files of the primary filegroup are being restored, or one or more secondary files are being restored offline. The database is unavailable.
• RECOVERING:Database is being recovered. The recovering process is a transient state; the database will automatically become online if the recovery succeeds. If the recovery fails, the database will become suspect. The database is unavailable.
• RECOVERY PENDING:SQL Server has encountered a resource-related error during recovery. The database is not damaged, but files may be missing or system resource limitations may be preventing it from starting. The database is unavailable. Additional action by the user is required to resolve the error and let the recovery process be completed.
• SUSPECT:At least the primary filegroup is suspect and may be damaged. The database cannot be recovered during startup of SQL Server. The database is unavailable. Additional action by the user is required to resolve the problem.
• EMERGENCY:User has changed the database and set the status to EMERGENCY. The database is in single-user mode and may be repaired or restored. The database is marked READ_ONLY, logging is disabled, and access is limited to members of the sysadmin fixed server role. EMERGENCY is primarily used for troubleshooting purposes. For example, a database marked as suspect can be set to the EMERGENCY state. This could permit the system administrator read-only access to the database. Only members of the sysadmin fixed server role can set a database to the EMERGENCY state.

推荐阅读：

How to resolve the issue of a database that was in Recovery Pending mode

Troubleshooting: SCOM DW Database is in a Suspect State

Search Engine Q&A #4: Using EMERGENCY mode to access a RECOVERY PENDING or SUSPECT database

Corruption: Last resorts that people try first…

How To Repair A Suspect Database In MSSQL

Recovering a SQL Server Database from Suspect Mode