INF 311

Exchange 2003 Backup Restore & Recovery

Ronen Gabbay Microsoft Exchange Regional Director Exchange MVP Microsoft Secure and Well Managed Infrastructure Specialist Microsoft Israel Microsoft Exchange Server User Group leader E-mail v-roneng@microsoft.com

Agenda
Are you aware Exchange is a mission critical application How to prepare: Exchange Database Architecture Backup and Restore Recovery Storage Group Dial Tone Recovery Exchange Snapshot Architecture How to avoid and what to do when things go bad Exchange Errors and how to avoid them ESEUtil & ISInteg and all in between Server & Alternate Server Restore procedure

Exchange is a mission critical application

Imagine how would life be without Exchange No inbound nor outbound mails No Calendar No mails between company users Have you considered how much money your company loses How long can you bear to be without Exchange Have you have any SLA Have you got any written policy in case Are you prepared for the worst

Storage Groups
STORE Storage Group 1 ESE Instance Storage Group 2 ESE Instance

LOG LOG LOG

LOG LOG LOG

EDB
STM

EDB
STM

EDB
STM

EDB
STM

EDB
STM

Exchange 2003 Databases

*.EDB Files Properties Database MAPI Messages and Attachments Headers for STM pages *.STM Files Raw streaming data (MIME, documents, multimedia, etc) Current database = EDB + STM + Unflushed Log entries

Storing Data
Transaction Log File
7 15 5 25 4 15 8 4 1 10

Database File
4 KB

Memory
4 8 7 17 15 8 25 3 4 1

1 5

2 6

3 7

9
13 18 22

10
14 19 23

11
15 20 24

12
16 21 25

10

ESE Database Consistency

Normal Operation Improper Normal Shutdown Operation Shutdown Normal Transaction Log File Database File Memory
7 7 15 15 5 5 25 25 4 4 15 15 8 8 4 4 1 1 10 10
4 KB 4 KB

1 4 KB 1 5 5 9 9 13 13

2 2 6 6 10 10 14 14

3 3 7 7 11 11 15 15

4 4 8 8 12 12 16 16

7 17

15 8

25 3

4 1

10

18 18
22 22

19 19
23 23

20 20
24 24

21 21
25 25

Previous Log Files

Current Logs Are Renamed After 5 MB of Data Is Accepted
Current Log Previous Logs
5 3 MB
Renamed

EXX00001.LOG EXX00002.LOG EXX00003.LOG . . .

Edb.log

New EXX.log (5 MB)

Circular Logging
EXX0005.log EXX0001.log EXX0002.log

EXX0003.log 14 14 40 40 16 16 99 99 19 19 33 33 31 31 20 20 52 52 21 21

EXX0004.log

277 7 15 30 15 625 5 25 53 25 364 4 17 70 17 948 8 473 3 821 1 10 71 10

99 44 44 22 18 18 66 13 13 11 11 22 22 12 12 35 35

44 15 15 23 23 42 42 26 26 34 34 46 46 29 29 61 61 48 48

In most cases Circular Logging should NOT be enabled

Checkpoint File

Transaction Log Entries Written to the Database

EXX.chk

Transaction Log Entries Not Yet Written to the Database

EXXnnnnn.log

Starting an ESE Database

Locating the Checkpoint Check File Signatures match Check Database File Consistency Recovering a Database at Startup

Online Maintenance
Not clash with backup schedules Database size, Backup/Restore SLAs Purge Indices (public and mailbox stores) Tombstone Maintenance (public and mailbox stores) Dumpster Cleanup (public and mailbox stores) Cleanup Deleted Mailboxes (mailbox stores) Events for each Database processed 700 Defrag Started 701 Defrag Ended 1221 The Defrag ended and the amount of free space in file

Exchange & Active Directory 2003

Active Directory Directory stores Exchange server information The Exchange configuration is stored in the Active Directory configuration partition Mailboxes are not directory objects A Mailbox is an attribute of a user Mailboxes can be reconnected to other users

Mailbox retention time Deleted Items retention time Dumping Headers using ESEUtil Configure Exchange EVS on Cluster

Exchange 2003 Backup Process flow

Backup APIs Called Backup program calls the ESE Backup Mode Exchange backup API Store informs ESE The checkpoint file freeze each database to Online maintenance pauses backup Begin Backup Agent requests DB pages sequentially 64K at time Pages are check-summed as read

Backup Complete

ESE Normal Mode

End Backup

The checkpoint file unfreezes Online maintenance resume

Page read completed Logs copied to tape Truncate logs Backup set closed

Backup
The IS must be running and all databases must be mounted Support for storage groups and for a single store Best practice backup the entire storage group for logs truncation Concurrent backup/restore are supported Truncate Log Files (*.log) Check-summing Database files (*.edb) Stream files (*.stm)

Backup Types
Type Full(Normal) Copies DB Copy Logs X X X X Truncates Logs X

X
X

Copy/Daily
Incremental Differential Snapshot Offline

X
X X X

Not recommended

Exchange 2003 Restore Process Flow

Dismount Database Backup application or Administrator dismounts database ESE Restore Mode Store informs ESE and restore mode entered Restore SG created Begin Restore Agent copies EDB/STM from Tape to DB path Log files from backup set are copied To temp restore location

Restore Complete

ESE Normal Mode

End Restore

DB is mounted by SG Data deleted from temporary directory

Logs are processed By ESE restore instance Current logs processed By ESE restore instance Cleanup/Restore SG killed

Restore
MSExchangeIS must be running to restore Databases to restore must be dismounted and the Db can be overridden by restore. Remaining databases can be mounted. System Attendant not used in restore. Restore.env file created Hard recovery via command line Eseutil /cc

Restore: Restoring Files

Databases Backup Agent asks the Store where to place the database based on the database GUID Store has the databases placed on top of the existing databases

Log Placed in a temporary log directory Location of temporary directory is specified by user Restore.env file is created

Restore.env
Replaces Restore In Progress Key Placed in temporary log directory during restore Data included in Restore.env Restore path Restore log file path Storage group System parameters for the restore storage group Log file range Restore time

Restore: Log File Replay

Log file signature checked Log files replayed First from temporary log file location Then from running storage group No Logs can be played if Circular Logging is enabled
In this case the database will be rolled back to the date of backup

If there are multiple DBs in a storage group only the log records applicable to the failed DB are replayed, the others are ignored

On-line vs. Off-line Backups

The IS must be running and the database mounted Page level checksum, during on-line backup Automatic Log files truncation indicates succesfull backup Only online backups are Supported VSS or Off-line Backup needs manual integrity check TIP: Moving large files (Eseutil or ESEFILE) NTBackup to disk use /FU switch file unbuffered
http://www.msexchange.org/tutorials/OptimizeExchange-2003-Performance-Part2.html

Recovery Storage Group

Enable the Administrator to verify backup consistency Enables to recover a single item out of backup In case of a long recovery process enables an instant solution for mail connectivity (Dial Tone) It becomes the default location for the Backup API for restore purpose After using The RSG remove it ,or set the registry key to ignore it by the Backup API
Location: HKEY_LOCAL_MACHINE \ System \ CurrentControlSet \ Services \ MSExchangeIS \ ParametersSystem Parameter: Recovery SG Override Type: REG_DWORD Value: 0x00000001

Dial Tone Recovery Explained

Restore mail service immediately; restore data later. New e-mail vs. historical data Why swap databases? When should you use this method? Q282496: Considerations and Best Practices When Resetting an Exchange Mailbox Database Dramatically reduce EXMerged data Preserve single instance storage and avoid database bloat

Setting End User Expectations

Send immediate status email Data is being recovered, but service has been restored in the meantime Time expectations If you intend to swap back in the original database, users should not reconfigure rules, views, offline files Merge process

Outlook 2003 Exchange Recovery Mode

The Message Exchange is currently in recovery mode. You can either connect to your Exchange server using the network, work offline, or cancel this logon The OST problem Linked to a specific mailbox. Dependent on a matching key in the current user profile. Key is destroyed by connecting to a different mailbox. Q282496: Considerations and Best Practices When Resetting an Exchange Mailbox Database Exchange Recovery Mode Options Offline Access to current (Old) OST only Online _ Access to new (Rest) OST only

Creating the Recovery Storage Group

Stop the entire storage group Why? So you can preserve all log files COPY all log files and MOVE affected database files Start blank store Send status message to all affected mailboxes

Restoring to the Recovery Storage Group

If possible, set paths to the same logical drive as the original database Recover the database in the Recovery Storage Group Mount and dismount the database in the Recovery Storage Group Notify users of an impending outage Swap database methods: Move database files between folders on the same drive Move database files between drives Swap logical paths

Simple Backup & Restore Dump the restore.env file Using the Recovery Storage Group Dial Tone Restore using RSG

Exchange Snapshot Support

VSS components
Windows Server 2003 (VSS service and framework) Requestor Third-party backup applications NTBackup does not use VSS to back up Exchange Writer Application-specific logic for participating in snapshot process, and restore/recovery Provider Third-party hardware control software Windows includes a software provider

Snapshot services

Exchange VSS Basics

ExWriter.dll Installed with Exchange 2003 Exwriter and file system allows for a completely safe snap Fast restores (sometimes minutes) During an extended snap: Clients may hour-glass during submits Outlook 2003 in cached mode experiences zero production impact

VSS and Exchange 2003

Two restore types: Point in time restore (victimized restore) (Recovery is to time of backup) Roll forward recovery (Recovery is to time of failure) VSS can restore to: Same location Alternate forest and server

What can VSS back up and restore?

Backup Only Read-only access is allowed Storage Group Administration functions prohibited Backup Choices Minimum selection is the storage group (SG) (to truncate log files) Can snap multiple storage groups at the same time (best practice will be to snap individual SG) On Backup Complete Requestor validates Shadow image consistency using ESEUtil /k Log files are truncated Storage Group Administration functions resume Writing to the database is allowed Restore Restore choices Entire storage group Single database Multiple databases from a single SG

VSS Best Practice

Snapshot is not a complete backup replacement but an alternative to streaming you still need to backup the databases Makes sure requester does page check summing to identify corruption Using Eseutil /K switch Exchange 2003 supports full backups and copy backups Must restore to same logical drive letters Put each database at its own LUN This allows snapshot restore of a single database No native support for VSS restores to RSG NTbackup does not support Exchange VSS

Using OPC to extract data from snapshot

Going from Pages to Mailboxes

Database is composed of pages. Pages are linked together into B+-Trees B+-Trees are collected into tables. Tables cross-reference each other and store folders, messages, mailboxes and database metadata

Eseutil ISInteg and all in between

ESE Level Database is seen as tables and indexes, not folders, messages, attachments. Application (Information Store) level Database is seen as folders, mailboxes, messages, etc. Eseutil understands the database at the ESE level only. ISInteg understands the database at the application level only.

ESEUtil switches
/D = Defrag Mode New signatures Log files mismatch, immediate backup required /T can be used to specify alternate location for the temp files /M = Dump headers (/ML, /MK, /MH) /R = Recovery performs soft recovery /K = Integrity Check for Snapshots /G = Checks Integrity at the ESE Level No Changes Are Made, This is a reporting Tool /F = Copy Mode /C = Hard Recovery Mode (direct to restore.env) /CC = Force Hard Recovery /CM Dump the restore.env file header /P = Repair /P
/CreateSTM

When to perform a repair /p /d Isinteg fix test alltests

Dumping File Headers using ESEUtil

Database Header Database signature, log signature State: Consistent or Inconsistent Checkpoint Header Shows checkpoint log file and signature Transaction Log File Headers Generation number Log file signature The attached databases

ISInteg
-Dump: Dumps Database folders and indexes Isinteg dump -Test: Test for integrity level errors -Fix: Fixes integrity level errors Isinteg fix test alltests.

Logical vs. Physical Corruption

You must understand the cause of the failure Log file are missing or corrupted Hardware / Virus software etc Database file is corrupted There are three layers of Information Store database corruption Page level ESE level Store level

Strategies for removing corruption

Restore an uncorrupted backup of the database If possible In case the backup is to old or can not be used Repair the database Expunge the corrupted pages from the database Salvage data and generate a new database

Repairing the Database

Remember that if the cause of the problems are corrupted or lost log files then the assumption is that the database is undamaged Sometimes simple check-disk would do the trick Running the Repair Function Attempts to repair links If it finds physical corruption, it will delete the page Database signature is changed if fixes are made After Running Repair Run ISInteg -fix Perform a full backup of the database

Lesson: Error -1018, -1019 and 1022

Error -1018: JET_errReadVerifyFailure Bad checksum Wrong page number Error -1019: JET_errPageNotInitialized Page expected in use is un-initialized (pgno = 0x00000000) Error -1022: JET_errDiskIO Generic disk I/O failure

-1018 Error Reporting

Exchange 2003 SP1
Event ID : 474 Raw Event ID : 474 Record Nr. : 34715 Category : Database Page Cache Source : ESE Type : Error Generated : 10/10/2005 01:00:00 Written : 10/10/2005 01:00:00 Machine : Exchange Message : Information Store (4884) The database page read from the file "C:\EXCHSRVR\MDBDATA\C4SG2DB2.edb" at offset 14080122880 (0x00000003473da000) for 4096 (0x00001000) bytes failed verification due to a page checksum mismatch. The expected checksum was 1506336388 (0x59c8de84) and the actual checksum was 1237900932 (0x49c8de84). The read operation will fail with error 1018 (0xfffffc06). If this condition persists then please restore the database from a previous backup.

-1018 Error Reporting

Exchange 2003 SP2

Root Causes of Error -1018?

Hardware Firmware File system corruption Virus Protection software Not Exchange! With some exceptions--having to do with false positives and negatives, not actually causing -1018s A -1018 would be Exchanges fault if it: Constructed the wrong checksum for a page Dropped a page in the wrong place in the database This does not mean that Exchange has no corrupting bugsbut these errors are not -1018. How serious is a -1018? When do you see it? Whats on the page? During normal operation? (somewhat serious) During startup? (likely fatal) During backup (may be minor)

What causes Error -1022?

Any disk I/O failure File damage or truncation File locked by another process Anti-virus software Almost always fatal to the service, but does not necessarily indicate database damage Troubleshooting Check file size/lock status Reboot to clear locks and other problems Do not conclude the database is damaged until you see that it is

Perform Soft Recovery Using Eseutil Repair Exchange Database Using Eseutil Running ISInteg /Fix Exchange DRA tool

Restore AD Requirements
Exchange 2003 disaster recovery assumes that the Active Directory is available and if necessary fully recovered as well Running Setup /DisasterRecovery rebuilds the local box and does not re-write ANY data to the AD Note: Exchange setup does not enforce or check that objects already exist in the AD

Full Exchange Server Restore

Reconfigure hardware drives similar to original server Reinstall operating system using the old server name Install service packs and fixes Install Exchange Setup /DisasterRecovery Install Exchange service packs and fixes Restore Exchange databases Any certificates must be restored separately

Using the Setup /Disasterrecovery switch

Alternate Server Restores

Active Directory Forest Production Server

Active Directory Forest

Restore

Restore Server

Copy .pst

Copy

Alternate Server Restores

Configuring the restore server
Configure new Windows 2003 Server with the latest hotfixes/service packs DCPROMO to create a new forest Only one Exchange org per AD forest alternatedomain.corp.mycompany.com Install DNS as a standard primary, allowing dynamic updates, and point to self as DNS server Install Exchange using same org name and Administrative Group name

Alternate Server Restores

LegacyExchangeDNs must match /O=organization/OU=site/CN= container/CN=object First Exchange 2003 admin group usually is First Administrative Group AG display names can be changed, but this doesnt change the legacyExchangeDN To match DNs Use Event 1088 on the Eventvwr or use ISInteg /DUMP Use LegacyDN tool to modify legacyExchangeDNs

Alternate Server Restores

Create a new storage group and database with matching display names Or rename existing ones Dismount database and mark This database can be overwritten by a restore Restore backup sets and mount database Run Cleanup Agent and reconnect Log on and extract data To .PST using Outlook Or, use Exmerge
Ronen Gabbay

Alternate Server Restore Rename the Legacy DN using LegcyDN tool Mounting the database Create all users using Mailbox Recovery Center Bulk reconnect all users to their mailboxes

Disaster Recovery Strategies

Service Level Agreements (SLA) Monitoring and Notifications Exchange Technical Expertise Use relative small database files Build Documentation / Change Control / Patch Management Firmware Updates Software Updates Documented Disaster Recovery Plan Regular Recovery Testing

Review
Exchange is mission critical application Exchange Database Architecture Deleted items & Mailbox retention time. Backup and Restore Architecture Recovery Storage Group Dial Tone Restore Exchange Snapshot Architecture Eseutil & ISInteg and all in between Full Server Restore procedure Alternate Server Restore procedure

Thank You !

Ronen Gabbay Microsoft Exchange Regional Director Exchange MVP Microsoft Secure and Well Managed Infrastructure Specialist Microsoft Israel Microsoft Exchange Server User Group leader E-mail v-roneng@microsoft.com

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. 2002 Microsoft Corporation. All rights reserved. Microsoft, BackOffice, the BackOffice logo, Microsoft Internet Explorer logo, the Office logo, Where do you want to go today?, the Windows logo, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other product and company names mentioned herein may be the trademarks of their respective owners.

 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.