Active Directory Replication Troubleshooting

Active Directory Replication Troubleshooting
Troubleshooting Lingering Objects
DRAFT V9.3
Released: October 17, 2011
About the Authors Author: Bio: Justin Turner Justin is a Sr. Support Escalation Engineer with the Directory Services group based in Irving Texas with over 10 years of support and Active Directory experience. Justin has created or contributed too many training courses and KB articles for the Microsoft Knowledgebase. Justin Turner
Project Lead: Bio:
Table of Contents
1.0 TAP .......................................................................................................................................................... 1
1.1 Topic ................................................................................................................................................................1 1.2 Audience ..........................................................................................................................................................1 1.3 Purpose ...........................................................................................................................................................1 1.4 Format .............................................................................................................................................................1
2.0 Problem ................................................................................................................................................... 2

2.1 The Problem ....................................................................................................................................................2 2.2 Potential Challenges ........................................................................................................................................2 2.3 Learners Needs ...............................................................................................................................................2 2.4 Instructors Needs ...........................................................................................................................................3
3.0 Learning Expectations ............................................................................................................................. 4

3.1 Learning Goals and Objectives ........................................................................................................................4 3.2 Lesson Components ........................................................................................................................................4 3.3 Resources ........................................................................................................................................................5
4.0 Learning Activities ................................................................................................................................... 6

Focus on goals .......................................................................................................................................................6 Connect to prior knowledge ..................................................................................................................................6 Gain and integrate content knowledge .................................................................................................................6 Take action and monitor learning progress ..........................................................................................................6 Synthesize and evaluation .....................................................................................................................................6 Extend and transfer ...............................................................................................................................................7
5.0 Assessment ............................................................................................................................................. 8

5.1 Assessment Objectives ....................................................................................................................................8 5.2 Post-course exam ............................................................................................................................................9 5.3 Post-course exam Answer Key ......................................................................................................................13 5.4 Performance Assessment ..............................................................................................................................14 5.5 Performance Assessment Rubric ...................................................................................................................15
6.0 Evaluation ............................................................................................................................................. 16

6.1 Survey Questions ...........................................................................................................................................16
7.0 Timeline................................................................................................................................................. 18 8.0 Job Aid ................................................................................................................................................... 20 8.1 Instructor Job Aid .................................................................................................................................. 21
Course Parameters ..............................................................................................................................................21 Note to Trainers ..................................................................................................................................................22 Obtaining Access to Virtual Machines .................................................................................................................23 Activities ..............................................................................................................................................................24 8.2 Learner Job Aid ..................................................................................................................................................25 Lingering Object Terminology .............................................................................................................................25 Tombstone Lifetime Default Values ....................................................................................................................26 Replication Consistency Settings .........................................................................................................................26 Troubleshooting Overview ..................................................................................................................................29 Repadmin /removelingeringobjects Quick Reference ........................................................................................29 Un-hosting a partition .........................................................................................................................................30 Manually adding a replication connection using repadmin.exe .........................................................................31 Repldiag quick reference .....................................................................................................................................32
9.0 Course Workbook ................................................................................................................................. 36

Document Conventions ...........................................................................................................................................36 Program Code and Commands............................................................................................................................36 Notes ...................................................................................................................................................................37 Tables and Figures ...............................................................................................................................................37 Course Document and Slide Numbering .............................................................................................................37
Lesson 1: Lingering Objects Fundamentals ................................................................................................. 39

What You Will Learn ............................................................................................................................................39 Terminology associated with Lingering Object issues .............................................................................................39 Lingering Objects .................................................................................................................................................39 Tombstone ..........................................................................................................................................................39 Tombstone Lifetime (TSL)....................................................................................................................................39 Strict and Loose Replication Constancy ..............................................................................................................42 Loose Replication Consistency ............................................................................................................................42 Strict Replication Consistency .............................................................................................................................43 Abandoned object ...............................................................................................................................................46 Abandoned delete ...............................................................................................................................................46
Lesson 2: Symptoms and Cause .................................................................................................................. 48

What You Will Learn ............................................................................................................................................48 Symptoms of Lingering Objects ...............................................................................................................................48 Detection of Domain Controllers That Have Not Replicated in the Tombstone Lifetime ...................................48
Replication Errors Caused by Lingering Objects ..................................................................................................50 Cause of Lingering Objects ......................................................................................................................................51 How lingering objects occur ................................................................................................................................51 Five Causes of Lingering Objects .........................................................................................................................51 Lingering Object Prevention ................................................................................................................................53
Lesson 3: Identification and Classification .................................................................................................. 54

What You Will Learn ............................................................................................................................................54 Create a replication health report ...........................................................................................................................54 Try This: Generate an AD Replication report using repadmin .............................................................................55 Use AD Replication report and repadmin to determine the scope of the problem ................................................55
Lesson 4: Lingering Object Removal ........................................................................................................... 57

What You Will Learn ............................................................................................................................................57 Methods to Remove Lingering Objects ...................................................................................................................57 Removing Lingering Objects with Repadmin.......................................................................................................57 Events Associated with Lingering Object Removal..............................................................................................58 Details of Repadmins Lingering Object Removal Mechanism ............................................................................59 Remove Lingering Objects Using Repldiag ..........................................................................................................59 Remove Lingering Objects Using Replfix .............................................................................................................63 Remove Lingering Object using LDP or Script .....................................................................................................63 Remove Lingering Objects by partition re-host operation ..................................................................................63
Lesson 5: Real World Application ............................................................................................................... 67

What You Will Learn ............................................................................................................................................67 Determining What to Do with a Lingering Object ...............................................................................................67
10.0 Lab Guide ............................................................................................................................................ 69 Lab Sessions ................................................................................................................................................ 70

Setting Up Your Lab Environment ...........................................................................................................................70 Hardware .............................................................................................................................................................71 Software ..............................................................................................................................................................71 Network Layout ...................................................................................................................................................72 Computer Names and IP Addresses ....................................................................................................................72 Configuring Your Computer(s).............................................................................................................................73 Accounts and Group Membership ......................................................................................................................74 Domain Membership...........................................................................................................................................74 Shares on Instructor Computer(s) .......................................................................................................................75
Using the Keyboard and Mouse in a Virtual Machine .............................................................................................75 Using the Keyboard .............................................................................................................................................75 Using the Mouse .................................................................................................................................................76
Lab 1: Exploring Lingering Object Fundamentals........................................................................................ 78

Configuring Your Computer(s) .................................................................................................................................78 Configuring Your Virtual Machine Environment .................................................................................................78 Accounts and Group Membership ......................................................................................................................78 Domain Membership...........................................................................................................................................79 Exercise 1: Determine Tombstone Lifetime Setting ................................................................................................79 Exercise 2: Determine forest and DC replication consistency settings ...................................................................81
Lab 2: Lingering Object Diagnosis and Documentation .............................................................................. 83

Exercise 1: Lingering Object Diagnosis ....................................................................................................................83 Exercise 2: Lingering Object Documentation ..........................................................................................................84
Lab 3: Lingering Object removal using repadmin ....................................................................................... 86

Exercise 1: <Problem Solving Exercise Title> ...........................................................................................................86 Exercise 2: <Simulation Exercise Title> ....................................................................................................................87
Lab 4: Lingering Object removal using ldp and repldiag ............................................................................. 89 Lab 5: Abandoned Object and Abandoned Deleted object remediation ................................................... 90 Lab 6: Lingering Link identification and cleanup......................................................................................... 91 10.0 Presentation Slides.............................................................................................................................. 92
DRAFT V9.3
1.0 TAP
This will be a half-day course covering Troubleshooting Lingering Objects. The proposed solution will consist of lecture, classroom discussion, case study and a hands-on laboratory environment using virtualized domain controllers on a Hyper-V server. Client: Stacy Raynor | Support Escalation Manager | Microsoft Corporation Problem: High case TMPI and escalation rate for AD Replication (lingering object) issues Solution: 6 hour training module
1.1 Topic
Troubleshooting Lingering Objects: Symptom, Cause and Resolution
1.2 Audience
Support Engineers at Microsoft Corporation
1.3 Purpose
The purpose of this workshop is to equip Microsoft Support Engineers with the necessary background knowledge and skills required to troubleshoot and resolve Active Directory Replication failures involving Lingering Objects.
1.4 Format
Instructor Led in classroom and remotely through Live Meeting consisting of: Lecture Classroom discussion Case study Lab Assessment
justin.turner@microsoft.com Microsoft Corporation
Troubleshooting Lingering Objects 2.0 Problem
DRAFT V9.3
2.0 Problem
Analysis of over 3,000 cases revealed that the Total Minutes per Incident (TMPI) for Active Directory replication issues involving lingering objects is more than twice the TMPI average of standard Active Directory replication cases. Interviews of SMEs and other engineers who work these issues revealed the following as likely contributors to the higher TMPI metric: Lack of consolidated documentation Complicated terminology, troubleshooting and remediation methods
2.1 The Problem

There is one technology area within Active Directory replication that has a higher than normal TMPI statistic: Lingering Objects. Cases that fall into this area are escalated to the next level of engineers frequently and take longer to resolve. Engineers will escalate cases for a number of reasons, one of them being that they do not feel they have the skills to resolve the problem. While there are a number of factors that can increase a cases TMPI and escalation rate, case analysis and engineer interviews reveal that targeted training is the right approach for this particular area. A targeted 3-5 course module should be sufficient.
2.2 Potential Challenges

Active Directory (AD) Replication is a somewhat broad support topic and the particular issues that occur within that support topic can vary greatly. Training on such a broad topic in the past is usually conducted over the course of several days. Targeted, in-depth training on the more complicated scenarios is preferred over the standard, which is typically broad in scope with little technical depth. Additionally, support for Microsoft is handled world-wide so this solution would need to consider options available for remote delivery and/or some type of self-study component. Challenges that we may have to deal with: Consolidation of existing resources Creation of a comprehensive lab environment in Hyper-V Course length and modality
2.3 Learners Needs

Interviews with SMEs and many engineers that routinely work these issues revealed the following needs: Consolidated documentation o Too many sources of information exist o I have over 30 articles to look through when working these issues Updated documentation (there are several scenarios un-accounted for in existing documentation)
2011 Microsoft Corporation. All rights reserved.
DRAFT V9.3
o
Repldiag was created several years ago to make lingering object cleanup faster and easier. Case data and SME interviews suggest that this tool is rarely used. o The SMEs ask if Ive already tried X. How would I know to try something when its not documented? Terminology is well defined and easy to understand o There are a lot of different terms used when SMEs discuss lingering objects. The terminology is difficult to grasp. How can I understand your action plan if I dont know what youre saying? Practice performing the different clean-up procedures. o Lab materials that support the course (Hands-on experience with analysis and resolution steps) To be able to understand the full scope of a lingering object problem in a large environment o I understand how to fix one or two DCs, but its a little scary when the customer has hundreds of servers and most of them have problems. To be able to understand which method to use o There are five or more methods that do the same thing. Which one should I use?
2.4 Instructors Needs

Supporting materials: o Documentation o Visual Aids o Well defined lab materials o Available Resources
Troubleshooting Lingering Objects 3.0 Learning Expectations
DRAFT V9.3
3.0 Learning Expectations

3.1 Learning Goals and Objectives
1.0 To understand the cause, identify the symptoms, and identify ways to resolve lingering object issues 1.1 The learner will be able summarize seven terms commonly used in lingering object scenarios. 1.2 The learner will be able to explain three ways in which lingering objects are created. 1.3 The learner will be able to list four symptoms of lingering objects. 1.4 The learner will be able to identify the currently configured tombstone lifetime and replication consistency settings in a lab environment. 2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in the future 2.1 The learner will be able to list at least three methods to prevent lingering objects 3.0 To be able to accurately determine the scope of a lingering object problem 3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis 3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by listing all partitions and all servers containing lingering objects. 4.0 To be able to document which method to use to resolve the issue and why 4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will remove the lingering objects in all partitions on all servers. 4.2 Given five different scenarios, the leaner will be able to recommend the correct method to remove lingering objects. 4.3 Given a subpar action plan, the learner will be able to recommend changes that will result in a better solution 5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object scenario 5.1 The learner will be able to execute the steps in an action plan in order to remove lingering objects 5.2 The learner will be able to remove lingering objects using five different methods.
3.2 Lesson Components

The course will consist of PowerPoint slides, supporting documentation in Microsoft Word, and a laboratory environment where the methods and procedures can be practiced on virtualized domain controllers running on a Windows Server 2008 R2 Hyper-V server. The
DRAFT V9.3
course workbook will contain all necessary supporting documentation and will include realworld examples of actual cases in a "Did you know?" format.
3.3 Resources
The instructor and students will have pre-requisite knowledge of Active Directory replication troubleshooting The instructor and students will have a computer running Windows 7 with Microsoft Office 2010 and remote desktop access to a Server running Windows Server 2008 R2 with HyperV. Hyper-V will contain the required virtualized domain controllers. The classroom will have a project, screen, and whiteboard
Troubleshooting Lingering Objects 4.0 Learning Activities
DRAFT V9.3
4.0 Learning Activities

Focus on goals
Each lesson: Begins with an overview and explanation of the goals of the lesson Instructor will ask questions to generate curiosity and judge prior knowledge
Connect to prior knowledge

Classroom discussion
Instructor will facilitate discussion of student's prior knowledge
Gain and integrate content knowledge

Case study
Present problems and demonstrate how to solve, explicitly stating the strategies that were used.
Real-world examples
Present new information in context in which it will be used
Lecture with slides, workbook and hands-on lab

Present information through multiple modes of representation Allow learners to revisit information as needed Provide adequate resources
Take action and monitor learning progress

Hands-on lab
Provide support and coaching as needed when learners are performing tasks Ask learners to demonstrate skill; provide corrective feedback
Synthesize and evaluation

Short-answer, matching, multiple choice, and free recall format exam
Posttest on knowledge
Performance based assessment

Have leaners demonstrate procedure or skill
DRAFT V9.3
Have learners demonstrate their own summaries
Case study
Present case studies, role lays, or simulations in which learners demonstrate skills, knowledge, attitudes
Extend and transfer

Hands-on labs
Provide practice in a variety of situations Gradually remove prompts and cues Provide opportunity to apply skills in realistic contexts
Workbook and Quick-reference handouts "Cube-note"

Provide job aids Provide access to additional information on the topic
Troubleshooting Lingering Objects 5.0 Assessment
DRAFT V9.3
5.0 Assessment
There are two different assessments: One is accessible via an Intranet web page and consists of a short-answer, matching, multiple choice, and free recall format exam. The other assessment is a performance-based lab assessment where the student is presented with a common lingering object scenario and has to document the issue, action plan and perform the procedure to correctly remove the lingering objects.
5.1 Assessment Objectives

1.1 The learner will be able identify seven terms commonly used in lingering object scenarios and match them to the corresponding definition. (exam) 1.2 The learner will be able to explain three ways in which lingering objects are created. (Performance assessment) 1.3 The learner will be able to identify four symptoms of lingering objects. (exam) 1.4 The learner will be able to identify the currently configured tombstone lifetime and replication consistency settings in a lab environment. (Performance assessment) 2.0 To be able to explain how an Active Directory Administrator can avoid lingering objects in the future 2.1 The learner will be able to list at least three methods to prevent lingering objects (exam) 3.0 To be able to accurately determine the scope of a lingering object problem 3.1 The learner will be able to use repadmin.exe to generate diagnostic data for analysis (exam and Performance assessment) 3.2 Given diagnostic data, the learner will be able to identify the scope of the problem by listing all partitions and all servers containing lingering objects. (exam and Performance assessment) 4.0 To be able to document which method to use to resolve the issue and why 4.1 Given diagnostic data, the learner will be able to create a detailed action plan that will remove the lingering objects in all partitions on all servers. (exam and Performance assessment) 4.2 Given five different scenarios, the leaner will be able to recommend the correct method to remove lingering objects. (exam and performance assessment) 4.3 Given a subpar action plan, the learner will be able to recommend changes that will result in a better solution (performance assessment) 5.0 To be able to apply that knowledge in a lab environment and resolve a lingering object scenario
DRAFT V9.3
5.1 The learner will be able to execute the steps in an action plan in order to remove lingering objects (performance assessment) 5.2 The learner will be able to remove lingering objects using five different methods. (Performance assessment)
5.2 Post-course exam

Multiple Choice: For each of the following questions, circle the letter of the answer that best answers the question. (5 points each)
1. Which of the following commands would generate a forest-wide replication status report to be used to aid in lingering object analysis? [Objective 3.1] A. B. C. D. E. F. G. 2. repadmin /replsum /xls >repl.xls repadmin /replsum /verbose >repl.xml repadmin /showrepl * /csv >repl.csv repadmin /showrepl /verbose >repl.txt ldp | removelingeringobjects A and D All of the above
Which of the following lingering object removal methods automates the removal of lingering objects?
[Objective 4.2]
A. B. C. D. E. F. G. 3.
repadmin /unhost repadmin /removelingeringobjects repadmin /rehost repldiag /removelingeringobjects ldp removelingeringobjects primitive replfix None of the above
Which of the following lingering object removal methods will remove objects on Windows 2000 Windows 2008 R2 and will remove abandoned objects? [Objective 4.2] A. B. C. D. E. F. G. repadmin /unhost repadmin /removelingeringobjects repadmin /rehost repldiag /removelingeringobjects ldp removelingeringobjects primitive replfix None of the above
4.
Which of the following lingering object removal methods allow you to review which objects will be removed prior to actually removing the objects? [Objective 4.2] A. B. C. D. E. F. repadmin /unhost repadmin /removelingeringobjects repadmin /rehost repldiag /removelingeringobjects ldp removelingeringobjects primitive replfix
DRAFT V9.3
G. B and F H. D and E
True or False: For each statement, circle True or False. (2 points each) [Objective 1.3]
True True True True True True False False False False False False 5. 6. 7. 8. 9. Replication status 8606 indicates that lingering objects are present on the source DC in a replication report. Event ID 1988 indicates that the source DC contains one or more lingering objects. Replication status 8453 indicates that lingering objects are present on the destination DC. Event ID 1388 indicates a lingering object was purged from the database.
Event ID 1945 indicates that a lingering object was detected after running repadmin /removelingeringobjects. 10. Abandoned objects can be removed using repadmin /removelingeringobjects.
Fill in the Blank and Matching: Into each sentence below, copy a term from the word bank that correctly completes the sentence. (5 points each) [objective 1.1]
Lingering Links Abandoned Object Abandoned Delete Lingering Object Loose Replication Consistency Strict Replication Consistency Tombstone Tombstone Lifetime
11. The length of time that a deleted object will remain in the database is referred to as _______. 12. A _________ is an object that is present on one replica, but has been deleted and garbage collected on another replica. 13. A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as ___________. 14. An object that has been deleted but not yet garbage collected. _________ 15. An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. _________ 16. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be reanimated. _________ 17. An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition. ____________
10
DRAFT V9.3
18. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked with the source DC for the partition where the lingering object was detected. __________
19. Essay Question: List three or more methods to prevent lingering objects (8 points) (objective 2.1)
Use Figure 1 Replication Status to answer the remaining questions.
Figure 1 Replication Status
20. Essay Question: Use Figure 1 Replication Status, document every DC containing lingering objects and for which partition. (10 points) (objective 3.2)
11
DRAFT V9.3
21. Essay Question: Using Figure 1 Replication Status and the following information, provide the exact command line syntax to log all lingering objects on DC 5thWardCorpDC to the event log, and the syntax to remove those lingering objects. (10 points) (objective 4.1)
Repadmin /removelingeringobjects <Dest_DSA_LIST> <Source DSA GUID> <NC> [/ADVISORY_MODE] The following DCs host writable copies of the partition in question: Dallas\DALCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 87ccb4f8-1057-4cfa-aed6-79b5626db9fd DC invocationID: 56f7cb84-0a67-43c1-93de-9d01f53e02c5 Dallas\NYCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 4009aef6-b279-43d2-82f6-4298f02505e8 DC invocationID: a29c83ab-5dea-4829-bbbf-1343f037098d Liverpool\LONCONTOSODC DC Options: IS_GC Site Options: (none) DC object GUID: a29bbfda-8425-4cb9-9c66-8e07d505a5c6 DC invocationID: d58a6322-6a28-4708-82d3-53b7dcc13c1a Liverpool\LONEMEADC DC Options: IS_GC Site Options: (none) DC object GUID: ba9bcfb2-7445-2cd9-8c66-9b27d534a4b3 DC invocationID: e38b6355-fb31-3785-71b1-42c6ddc23f8e Houston\5THWARDCORPDC DC Options: IS_GC Site Options: (none) DC object GUID: 9653cb84-7aa2-4a59-ab46-382e5dc1d3a8 DC invocationID: e0cb69c0-5d24-4254-b830-99b0c9b4da1f
12
DRAFT V9.3
5.3 Post-course exam Answer Key

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. C D E G True True False False True False Tombstone Lifetime Lingering Object Lingering Link Tombstone Abandoned Object Loose Replication Consistency Abandoned Delete Strict Replication Consistency At least 3 of the following: o o o o o o Resolve replication failures within TSL Ensure Strict Replication Consistency is enabled Ensure large jumps in system time are blocked via registry key or policy Don't remove replication quarantine with "allowDivergent" setting without removing LOs first Don't restore system backups that are near TSL number of days old Don't bring DCs back online that haven't replicated within TSL
20. LONCONTOSODC: DomainDNSZones, Configuration

5THWARDCORPDC: Configuration DALCORPDC: Configuration FOURTHDC1: Configuration, ForestDNSZones NYCORPDC: Configuration CONTOSOROOTDC1: Configuration FOURTHDC2: Configuration
21.
Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c669b27d534a4b3 cn=configuration,dc=contoso,dc=com /advisory_mode
13
DRAFT V9.3
Repadmin /removelingeringobjects 5thwardCorpDC ba9bcfb2-7445-2cd9-8c669b27d534a4b3 cn=configuration,dc=contoso,dc=com
5.4 Performance Assessment

Students take their performance assessment in the hands-on lab environment. The performance assessment is a culmination of all prior lab tasks without the benefit of step-bystep guidance. The lab environment is broken via several scripts. After the scripts run, both lingering objects and abandoned objects are present. The students receive a handout with intentionally vague problem descriptions. They are instructed to document the issue thoroughly and then resolve the problems. Good documentation consists of symptoms, cause, and resolution. The symptoms section should contain a list of all "problematic objects." The resolution section should have a thoroughly documented action plan. Here is the text they are prompted with: You are the consultant for Adatum Corporation. Please help resolve the following problems in our environment. Changes are not propagated amongst DCs for the Adatum domain. Unable to create the following user account in the West domain: Mike Miller Ann Wallace's account in the East domain does not show up on any other domain's GC Users that send email to the CorpVP mail-enabled universal group receive NDRs on occasion. Additionally, our Exchange 2010 mailbox server cannot generate an Offline Address Book. This worked on our Exchange 2007 mailbox server.
Please ensure that you document each problem thoroughly. This documentation should include forest and DC environment settings (tombstone lifetime and replication consistency), symptom, cause and resolution sections. The symptoms section should contain a list of all "problematic objects." The resolution section should have a thoroughly documented action plan. Implement your action plan after documenting the issue.
14
DRAFT V9.3
5.5 Performance Assessment Rubric

Hands-on Assessment Rubric: Troubleshooting Lingering Objects Student Name: _____________________________________
Assessment Criteria Max. Points
10
Exceptional (all points)

Symptom, cause and resolution sections The symptoms section contains a list of all objects The resolution section has a thoroughly documented action plan
Average (65-85%)
Symptom, cause and resolution sections are mostly documented The symptoms section contains a partial list of all objects The action plan is missing one to two steps Most (greater than 75%) of lingering objects are removed.
Poor (0 - 65%)
Symptom, cause and resolution sections is inadequate Less than 25% of all objects are listed The action plan will not resolve the issue or will make things worse Less than 25% of lingering objects are removed AD Replication is not successful Abandoned object is still present on most DCs
Comments & Points Earned
Documentation (objectives 1.4, 3.1, 3.2, 4.1, 4.2)
AD Replication and Lingering object cleanup (5.1, 5.2) Abandoned object cleanup (5.1, 5.2)
25
All lingering objects are removed from the environment AD Replication is successful Abandoned object is no longer present on any DC new object is created in its place Object completely removed from the environment CorpVP group contains correct group membership on all DCs Group still has the same objectSID
25
Abandoned object is no longer present on most DCs (greater than 75%) Object mostly removed from the environment CorpVP group contains correct group membership on all DCs Group does not have the same ObjectSID
Abandoned delete resolution (5.1, 5.2)
25
Object is still present on most DCs in the environment CorpVP group has inconsistent group membership on most DCs
Lingering Link cleanup (5.1, 5.2)
15
TOTAL:
100
15
Troubleshooting Lingering Objects 6.0 Evaluation
DRAFT V9.3
6.0 Evaluation
Following the conclusion of the course, the students are emailed a link to a survey to take online.
6.1 Survey Questions

Strongly Agree Agree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree
# Question
1 I was provided with the information I needed Strongly Agree Agree (logistics, pre-work) for the training in a timely manner. 2 The classroom setup and hardware (if supplied) functioned appropriately to support face-to-face learning.
Strongly Agree Agree
3 The instructor was knowledgeable about the Strongly Agree Agree subject matter. 4 The instructor's presentation skills helped me better understand the content.
5 The instructor consistently linked the course Strongly Agree content to Microsofts business and/or my Agree role. 6 The length of the course was appropriate
7 Overall, I was satisfied with this course.
8 This course builds skills improving how I sell, Strongly Agree Agree market, and/or provide services to our customers and partners. 9 This course was a valuable use of my time.
16
DRAFT V9.3
10 I would recommend this course.
Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree
11 The messaging in this course is relevant to Microsoft's customers and/or partners. 12 If not, please provide additional feedback. 13 How soon will you be able to apply this learning?
Neither Disagree Strongly Don't Agree Disagree Know nor Disagree Neither Disagree Strongly Don't Agree Disagree Know nor Disagree
14 My manager and I have discussed how I will Strongly Agree Agree apply this training to my job. 15 What are you going to do differently as a result of this course? 16 What was the most useful portion of this course? (Please provide specifics, e.g. instructor effectiveness, content quality, materials usefulness). 17 What was the least useful portion of this course? (Please provide specifics, e.g. instructor effectiveness, content quality, materials usefulness). 18 Please provide any additional comments (e.g. learning environment, instructor effectiveness, content/materials quality, content level, relevance, application).
17
Troubleshooting Lingering Objects 7.0 Timeline
DRAFT V9.3
7.0 Timeline
The following proposed timeline should allow for sufficient coverage of the course material. Time Objectives Activities / Training Methods
Intro and Classroom discussion Lecture and discussion
Materials
9:00 AM 15 minutes
Welcome and Instructor Introduction 1.2 Lingering Object Fundamentals 1.1 Exploring Lingering Object Fundamentals 1.4 Symptoms and Cause 1.2, 1.3, 2.1 Identification and Classification 3.1, 3.2
Slide 1: Course Title and Instructor Name
9:15 AM 20 minutes
Lesson 1 Slides
9:35 AM 15 minutes
Lab 1exercise
Lab 1 guide and lab environment
9:50 AM 20 minutes 10:10 AM 20 minutes
Lecture and discussion Lecture and discussion
Lesson 2 Slides Provide real-word scenarios Lesson 3 Slides Show prior case action plans
10:30 AM 10 minutes 10:40 AM 45 minutes
Break
Lingering Object Diagnosis and Documentation 3.1, 3.2, 4.1 Lingering Object Removal 5.1, 5.2 Lunch
Lab 2 exercise
Lab 2 guide and lab environment
11:25 AM 20 minutes 11:45 AM 60 minutes 1:00 PM 90 minutes
Lecture and discussion
Lesson 4 Slides
Lingering Object removal labs 5.1, 5.2
Lab exercise 4 6
LabLab documentation, Hyper-V images
18
DRAFT V9.3
2:30 PM 10 minutes 2:40 PM 10 minutes 2:50 PM 30 Minutes
Real World Application 4.2, 4.3 Real-world case study 4.2, 4.3 Break
Lecture and discussion Case Study
Lesson 5 Slides
Case data in instructor share Case Details, Diagnostic Data Present the high-level symptoms. What data do you want to see? Show the data What is the action plan?
3:20 PM 10 minutes 3:30 PM 30 minutes 4:00 PM 10 minutes 4:10 PM 60 minutes 10 minutes
Question Time
Ask if there are any questions Post-course test Share assessment URL on-screen
Assessment
Break
Performance assessment
Lab-based assessment
VMAS connection instructions for postcourse performance assesment Slide
Summary and questions 1.1 - 5.2
Course Summary and wrap-up
19
Troubleshooting Lingering Objects 8.0 Job Aid
DRAFT V9.3
8.0 Job Aid
20
DRAFT V9.3
8.1 Instructor Job Aid

Course Parameters
Course title Course Length Course Objectives Troubleshooting Lingering Objects 6 hours (1 day) At the completion of this workshop, the engineer shall be able to: 1. Explain how a user's group membership is stored in Active Directory 2. Explain what happens during a user deletion 3. Understand why a special procedure is needed to restore users along with their group membership. 4. Explain the three methods of recovery after deletion 5. Identify recommendations and considerations for a better recovery experience 6. Perform the most common (and preferred) method of recovery for our customers Microsoft Support Engineers (Platforms, Directory Services) Trainee: 1. Knowledge of Active Directory replication 2. Familiarity with Active Directory concepts and terminology 3. Experience with Hyper-v for the lab session Instructor: 1. Real world experience with Active Directory replication and Lingering Object troubleshooting procedures 2. Hyper-V user experience for demonstration session 3. PowerPoint user experience Room arrangement Materials/equipment Classroom setting PowerPoint setup, whiteboard and markers, Computer for demonstration, and one computer for each workshop participant. On the computers: Microsoft Windows 7, Microsoft Office 2010, Intranet access, PowerPoint presentation, and supporting reference documentation Learning exercises for participants and online Instructor/classroom evaluation form Justin Turner is a Sr. Support Escalation Engineer on the Microsoft Platforms Directory Services Support team where he obtained his firsthand knowledge of the material. He has been with Microsoft for over ten years, and is currently pursuing his MS in Computed Education and Cognitive Systems degree from the University of North Texas.
Target Audience Prerequisites
Evaluation/ Assignments Instructor
21
Troubleshooting Lingering Objects 8.1 Instructor Job Aid
DRAFT V9.3
Note to Trainers
Checklist of Supplies
Print out slides with notes pages. The notes pages provide the necessary material to help explain the contents of each slide. Alternatively, you can have the students copy the course materials to their computer and print out the slides to a new Microsoft OneNote notebook. The student lab guide is stored electronically on the hyper-v image: DC1
Room Arrangement
Standard Microsoft classroom configuration: Classroom style with whiteboard and projector screen at the front of the room
Handouts / Visual Aids

Print out one copy of the slide deck in "Handouts" format for each student (or print to OneNote). Course workbook and lab guide are available on the Instructor computer.
Lab Computer setup

Microsoft Windows 7 Office 2010 Connection to the corporate Intranet
Preparation
Before Class starts: 1. Have PowerPoint slide deck opened up 2. On instructor machine: Launch Hyper-V, and launch DC1s image 3. Ensure classroom has intranet connectivity
22
DRAFT V9.3
Obtaining Access to Virtual Machines

To access VMs provisioned for your use during this course, perform the following steps: 4. Log onto the physical computer using your Corpnet credentials. 5. Access the VMAS server that hosts your VMs using the link provided by your instructor. 6. Open the VMAS menu and select Manage VMAS VMs.
7. Use Manage My VMs to access virtual machines referenced in lab exercises.
Note:
For more information, click links in the Documents section on the right to open course documents included in the VM package.
23
DRAFT V9.3
Activities
Introduction
Welcome the students to the course. Ask them to share the following: Name Role Time at Microsoft Something that no one (at work) else knows about them or something unique
Classroom Discussion
After the introduction, lead a discussion to gauge student's prior knowledge. Ask probing questions like: What is a lingering object? Why do I care about removing them from my environment? What does tombstone lifetime have to do with this? Who can explain the different between strict and loose replication consistency? What is an abandoned object? How is that different from a lingering object? What is a lingering linked value? Who here has worked a lingering object issue? Were you able to resolve it? How long did it take? Who here has used repldiag? What did you think about it?
Real-world examples
Where appropriate, provide examples of actual cases worked. Highlight the successes and failures (what went right and what went wrong). Present new information in context in which it will be used
Case Study
The case study within the course includes real diagnostics data from an actual customer case. The data was scrubbed to remove personally identifiable information (PII). Present the facts of the case and encourage the students to play the role of engineer. There is an action plan included in the case study. The action plan is intentionally poor in quality and if implemented would result in disastrous results. Together come up with the appropriate action plan to resolve the problem.
24
DRAFT V9.3

Present case studies, role plays, or simulations in which learners demonstrate skills, knowledge, attitudes Present problems and demonstrate how to solve, explicitly stating the strategies that were used.
Lab Activities
Students have access to their lab environment through the VMAS site. Each lab activity corresponds to a lesson in the course. You may be tempted to do the entire lecture at once and then all lab activities at the end of the course. It is important not to do this. Please have the students complete the lab activities along with the appropriate lesson in the course.
Hands-on lab
If unfamiliar with the lab environment and lab material, you should work through each lab activity at least one time prior to the course Provide support and coaching as needed when learners are performing tasks Ask learners to demonstrate skill; provide corrective feedback
8.2 Learner Job Aid

Lingering Object Terminology
Table 1: Lingering Object Terminology
Term
Abandoned delete
Definition
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition. An object created on one DC that never got replicated to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links.
Abandoned object
Lingering link
25
DRAFT V9.3
An object that is present on one replica, but has been deleted and garbage collected on another replica. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be reanimated. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked with the source DC for the partition where the lingering object was detected
Lingering Object
Loose Replication Consistency
Strict Replication Consistency
Tombstone
An object that has been deleted but not yet garbage collected The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.
Tombstone Lifetime (TSL)
Tombstone Lifetime Default Values

Table 2: Default TSL Values
OS Install Path Windows 2000 RTM Windows 2003 RTM, 2003 R2 Windows 2000RTM upgrade to Windows 2003 SP1 Windows 2003SP1, 2003SP2, 2008, 2008R2 NT4 upgrade to Windows 2003 SP1
Default TSL 60 days 60 days 60 days 180 days 180 days
Replication Consistency Settings

Defines how a destination DC behaves if a source DC sends updates to an object that does not exist in the destination DCs local copy of Active Directory.
26
DRAFT V9.3
o o o
Destination DCs should see USN for creates before object is modified Only modifies for lingering objects arrive for object not on destination DC Only destination DCs enforce strict replication and log events
Destination DCs stop replicating from source DCs partitions containing LOs Lingering objects are quarantined on source DCs where they can be detected End-to-end replication may be impacted for partitions containing lingering objects Administrators must remove lingering objects to restore replication
Enabling Strict Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command prompt: For all domain controllers, type: repadmin /regkey * +strict For all global catalog servers, type: repadmin /regkey gc: +strict
You can also enable strict replication by manually setting the Strict Replication Consistency registry value to 1. HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency (Reg_DWORD) to 1 1 (enabled): Inbound replication of the specified directory partition from the source is stopped on the destination.
Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency due to the existence of lingering objects.

If you enable Loose Replication Consistency, if a destination receives a change to an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This behavior causes a lingering object to be reapplied to all domain controllers in the replication topology.
Enable Loose Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command prompt:
For all domain controllers, type: repadmin /regkey * -strict
27
DRAFT V9.3
For all global catalog servers, type: repadmin /regkey gc: -strict
You can also enable strict replication by manually setting the Strict Replication Consistency registry value to 0.
HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object. Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.
Default Settings for Strict Replication Consistency

Upgrade Path Windows NT 4.0 Windows 2000 RTM Root Default Loose Loose A post-SP2 NTDSA.DLL defaulted to strict replication consistency but was quickly recalled. Windows 2000 Services 1 through 4 all default to loose replication consistency. Notes
Windows NT 4.0 to Windows 2000 Root Windows 2000 to Windows Server 2003 SP1
Loose Loose Upgrading a Windows 2000 forest to Windows Server 2003 slipstreamed with SP1 does not enabled strict replication consistency. DCPROMO creates an operational GUID that causes Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers. Same as above. DCPROMO creates an operational GUID that causes
Windows Server 2003 RTM Root
Strict
Windows Server 2003 SP1 root Windows NT 4.0 to Windows Server 2003 root
Strict Strict
28
DRAFT V9.3
Windows Server 2003 domain controllers to inherit strict replication mode but is ignored by Windows 2000 domain controllers.
The default value for the strict replication consistency registry entry is determined by the conditions under which the domain controller was installed into the forest. Note: Raising the domain or forest functional level does not change the replication consistency setting on any domain controller.
More Information: For more information about this topic, see: http://blogs.technet.com/b/askds/archive/2010/02/15/strict-replication-consistencymyth-versus-reality.aspx
Troubleshooting Overview
Common methods to remove lingering objects include: Repadmin /Removelingeringobjects Replfix Repldiag Manually through LDP or using script Rehost the partition: Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the DC containing lingering objects) Un-GC (but you dont really have control over who the DCs sources the partition from) Demote and Promote (DCPromo)
Repadmin /removelingeringobjects Quick Reference

Have the customer run the following command: repadmin /showrepl * /csv >showrepl.csv Once you have this, filter column K for 8606, so that you know exactly which DCs have lingering objects and in which partitions. The DCs in the SourceDC column contain lingering objects. You can use the repadmin /removelingeringobjects command to remove lingering objects. In some cases it may make sense to just rehost the partition with the repadmin /rehost command. In order to use the /removelingeringobjects command you need to know three things: 1. You need to know which dc's contain lingering objects 2. Which partition the lingering object resides in 3. A good reference DC that hosts that partition that does not contain lingering objects
29
DRAFT V9.3
Repadmin RLO example usage:

The command is: repadmin /removelingeringobjects LingeringDC ReferenceDC_DSA_GUID Partition Where: LingeringDC: FQDN of DC that has the lingering objects ReferenceDC_DSA_GUID: The DSA GUID of a domain controller that hosts a writeable copy of the partition Partition: The distinguished name of the directory partition where the lingering objects exist So for example: We have a server named DC1.contoso.com that contains lingering objects. We know that the lingering object is in the childdomain.contoso.com partition. We know that DC3.childdomain.contoso.com hosts a writeable copy of the partition and doesn't contain any lingering objects. We need to find the DSA GUID of DC3 is, so we run: repadmin /showrepl DC3.childdomain.contoso.com At the top of the output, locate the DC Object GUID entry. This is the GUID you need to enter in the command for the reference DC. The command would be repadmin /removelingeringobjects DC1.contoso.com 5ed02b33-a6ab-4576-b109bb688221e6e3 dc=childdomain,dc=contoso,dc=com
-------------------------------------------------------------------------------------------------
Detailed troubleshooting guidance is located here:

2028495 Troubleshooting Active Directory operations that fail with error 8606: Insufficient attributes were given to create an object. http://support.microsoft.com/default.aspx?scid=kb;en-US;2028495
Un-hosting a partition
It is sometimes necessary to remove a partition from the database of a DC temporarily. Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows you to exercise more control over the procedure. Take note that /unhost only allows you to remove a read-only copy of the partition. With the exception of application partitions, you cannot remove a writable copy of a partition from a DC without using DCPROMO.
Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com
Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until event ID 1660 is logged in the Directory Services event log. The re-host operation may fail with error 8339 if you attempt to re-add the partition too soon after the un-host.
30
DRAFT V9.3
Manually adding a replication connection using repadmin.exe

The add command will create a RepsFrom attribute on the destination domain controller for the specified naming context and initiate a replication request. During a normal replication cycle, the destination domain controller will request updates from the source domain controller. When creating temporary replication links between replication partners, the process could fail if the KCC starts while you are performing the procedure. The KCC will delete any replication links for which no corresponding connection object exists. Since these commands can take a very long time to complete as they trigger the replication of the corresponding naming context, it is important to ensure that KCC do not disturb the process. This is where you would use +DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate connection objects to replication links.
Disable KCC connection translation so that KCC doesnt remove our temporary replication connection:
Repadmin /options ContosoDC1 +disable_ntdsconn_xlate
Then add a replication connection for the configuration partition of the server we want to source the partition from:
Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
Add a replication connection to the server for the domain partition that we need to source from (/readonly is specified if the partition is a GC non-writable
partition /selsecrets needs to be specified if the destination DC is an RODC):
repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly
One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
31
DRAFT V9.3
If you need to replicate the other way, then just reverse the order of the server names in the commands. To begin a normal sync of the partition using the new replication connection:
Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly
To begin a full sync of that partition using the new replication connection:
repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full
Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.
Turn KCC connection translation back on when you no longer need the connection:
Repadmin /options ContosoDC1 -disable_ntdsconn_xlate
Repldiag quick reference

Removing lingering objects from a forest with repldiag is as simple as running repldiag /removelingeringobjects. However, it is usually best to exercise some control over the process in larger environments. The option /OverRideReferenceDC allows you to select which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to see what a forest-wide cleanup looks like using repadmin.
Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax

This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new cleaned up reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.
Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=forestdnszones,dc=contoso,dc=com
32
DRAFT V9.3
repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com justin.turner@microsoft.com Microsoft Corporation
33
DRAFT V9.3
repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.
This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)
More control: /OverRideReferenceDC

This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange facing DC Other DCs dont report replication failures with reference DC as the source: filter repadmin /showrepl * /csv ouput, or use the topology report created by repldiag /save.
repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp. contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso. com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc .corp.contoso.com /outputrepadmincommandlinesyntax Replication topology analyzer. Written by kenbrumf@microsoft.com Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.co ntoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.co m Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.c orp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden
/UseRobustDCLocation
34
DRAFT V9.3
Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. Weve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.
35
Troubleshooting Lingering Objects 9.0 Course Workbook
DRAFT V9.3
9.0 Course Workbook

Document Conventions
The following conventions are used in the course materials: Acronyms appear in all uppercase letters. Path and file names may appear in a combination of uppercase and lowercase letters. Unless otherwise indicated, paths and file names entered in dialog boxes or at a command prompt are not case-senstitive File extensions without a file name appear in all lower-case letters. Book titles and URLs appear in Italic. Window, dialog box, menu titles, menu items, and section titles appear in Bold.
Other document conventions are described below.
Program Code and Commands

Program code listings, diagnostic output, entries typed at a command prompt or in scripts or initialization files, and other text mode content appear in a console font with a grey background formatted as shown in the following example. Descriptive comments may be inserted in line with the listing.
d:\%systemroot%>dir /ad
where: d: is the drive letter where the operating system is installed. %systemroot% is the folder where the operating system is installed.
Volume in drive C is Main Volume Serial Number is 000A-BCDE Directory of C:\Windows 12/19/2004 11:56 AM <DIR> 12/19/2004 11:56 AM <DIR> 07/07/2003 06:57 AM <DIR> 11/17/2004 02:45 PM <DIR> 11/17/2004 02:47 PM <DIR> 11/17/2004 02:42 PM <DIR> ...
. .. addins Application Compatibility Scripts AppPatch Cache
The ellipsis (...) on the last line indicates a partial listing. The following conventions apply to all commands and program code listings: Type command statement elements that appear in Bold exactly as they appear in the example, including quotation marks. Italic elements in command statements indicate placeholders for variable information.
36
DRAFT V9.3
Braces ({ }) enclose required items as shown by {parameter1, parameter2, title} in the example. Commas separate multiple items. Type quotation marks as shown; do not type the braces. Square brackets ([ ]) enclose optional items as shown by [option1 | option2] in the example. Pipe symbols ( | ) indicate alternate choices. If multiple options are listed, only type one option. Do not type the brackets or pipe symbols.
Notes
Icons and labels call attention to informational notes and reader alerts as shown in the following table.
Table 3. Note Icons and Labels
Icon
Label Note/Important Important Tip Critical Warning Do Not More Information More Help Trends
Description Emphasizes content and provides additional information. Strongly emphasizes key content. Highlights a best practice. Indicates strongly recommended actions. Indicates strongly recommended actions required to prevent data loss or other undesirable results. Warns against actions that may cause system failure or data loss. Link to reference material. Link to guides, white papers, or KB articles. Indicates industry trends, top support issue trends, etc.
Tables and Figures

Each table and figure is preceded by Caption. Captions are numbered sequentially throughout each module.
Course Document and Slide Numbering

Modules may be numbered sequentially within a course. Lessons, demonstrations, and videos may be numbered sequentially within a module. Topic and subtopic headings are not numbered. Lab sessions may be numbered sequentially throughout the course. Individual exercises are numbered sequentially within each lab session.
37
Troubleshooting Lingering Objects 9.0 Course Workbook
DRAFT V9.3
In each module, slide number paragraphs shown in the following figure identify the presentation slide that accompanies the topic.
Figure 2. Slide Number Paragraph Slide ##
The first slide in each presentation is unnumbered. Subsequent slides and slide indicator paragraphs in each module are numbered sequentially starting with 1.
Each presentation slide corresponds to a topic section in the module. Topic sections that include supplemental information may not be referenced on corresponding presentation slides.
Note:
38
DRAFT V9.3
Lesson 1: Lingering Objects Fundamentals

There is a lot of technical jargon associated with Lingering Object issues that you will need to understand. The following section provides a definition for each term with context to enable you to speak confidently when dealing with lingering object issues.
What You Will Learn

After completing this lesson, you will be able to: Summarize seven terms commonly used in lingering object scenarios.
Terminology associated with Lingering Object issues

Lingering Objects
A lingering object is an object that is present on one replica, but has been deleted and garbage collected on another replica.
Tombstone
When Active Directory deletes an object from the directory, it does not physically remove the object from the database. Instead, Active Directory marks the object as deleted by setting the object's isDeleted attribute to TRUE, stripping most of the attributes from the object, renaming the object, and then moving * the object to a special container in the object's naming context (NC) named CN=Deleted Objects. The object, now called a tombstone, is invisible to normal directory operations. Some objects dont get moved upon deletion and will therefore not be moved into the Deleted Objects container.
Note:

When an object is deleted, Active Directory replicates the deletion as a tombstone object. By inbound-replicating this object, other domain controllers in the domain and forest become aware of the deletion. The tombstone is retained in Active Directory for a specified period called the tombstone lifetime. At the end of the tombstone lifetime, the tombstone is deleted from the directory permanently.
More Help: For more help on this topic, see: Determine the tombstone lifetime for the forest http://technet.microsoft.com/en-us/library/cc784932(WS.10).aspx
39
Troubleshooting Lingering Objects Lesson 1: Lingering Objects Fundamentals
DRAFT V9.3
In most cases, the default value is 60 days. If the forest was built on 2008 or later, it should be 180. The minimum setting is 2 days.
Do Not: Do not reduce TSL to 2 days. (Unless directed to do so by a senior AD Replication SME)
Refer to the following table to determine TSL default values

Table 4: Default TSL Values
OS Install Path Windows 2000 RTM Windows 2003 RTM, 2003 R2 Windows 2000RTM upgrade to Windows 2003 SP1 Windows 2003SP1, 2003SP2, 2008, 2008R2 NT4 upgrade to Windows 2003 SP1
Default TSL 60 days 60 days 60 days 180 days 180 days
Removing Outdated Objects Following Expiration of Tombstone Lifetime

If a domain controller fails to replicate for a number of days exceeding the tombstone lifetime, replicas of objects that have been deleted from a writable partition might remain in that domain controller's directory. Because the tombstones of the deleted objects are permanently removed from the directory at the end of the tombstone lifetime, a domain controller that fails to replicate changes for tombstoned objects never deletes or garbage collects deleted objects. This condition can occur for a variety of reasons, including the following: Prolonged misconfigurations (such as those that cause 1311 events); Prolonged errors in name resolution, authentication, or the replication engine, each of which blocks inbound replication; Turning on a domain controller that has been offline for more than 60 days; and, Advancing system time or reducing TSL values in an attempt to accelerate garbage collection before end-to-end replication has occurred for all naming contexts in the forest.
To avoid such conditions, incorporate monitoring regimens that detect domain controller replication problems.
40
DRAFT V9.3
Outdated objects can also occur due to hardware and software problems that render the domain controller unreachable. Regardless of the reason, a deleted object can remain on a domain controller in either of the following circumstances. A domain controller goes offline immediately before the deletion of an object on another domain controller, and remains offline for a period that exceeds the tombstone lifetime. A domain controller goes offline immediately after the deletion of an object on another domain controller, but before receiving replication of the tombstone, and remains offline for a period that exceeds the tombstone lifetime.
The following provides information for a legacy operating system but is included here as it is still relevant. Additionally, some pre-Windows 2000 SP3 domain controllers experience a replication error condition after a non-authoritative restore. A large number of objects created after the restore may never be considered for replication.
More Information: For more information about this topic, see: Microsoft Knowledge Base Article 316829, Possible Active Directory Inconsistency after You Restore a Domain Controller.
On domain controllers that are running Windows Server 2003 or later, you can use the Repadmin support tool to analyze and remove lingering objects from a domain controller that you suspect or know has not replicated for a tombstone lifetime. This tool includes the RemoveLingeringObjects command. This command removes objects that are outdated (do not exist in a replica of the same directory partition on the source domain controller).
Problems with Lingering Objects

In Windows 2000, if an attribute for a lingering object had been replicated, the inbound domain controller that had previously processed the deletion would re-animate the entire object. However, this is undesirable for a number of reasons. The lingering object is holding a value on a unique attribute, such as samAccountName, that another object wants to use. This commonly occurs when the lingering object exists in the read-only naming context but not the domain naming context. The lingering object is a security risk. For example, it might represent a user that should be deleted. The lingering object only exists in the read-only naming context (global catalog). This behavior makes the object difficult to delete in Windows 2000.
A deleted user or group account remains in the global address list (GAL) on Exchange servers. Therefore, although the account name appears in the GAL, attempts to send e-mail messages result in errors. Multiple copies of an object appear in the object picker or GAL for an object that should be unique in the forest. Duplicate objects sometimes appear with altered
Important:
41
DRAFT V9.3
names, causing confusion on directory searches. For example, if the relative distinguished name of two objects cannot be resolved, conflict resolution appends "*CNF:GUID" to the name, where * represents a reserved character, CNF is a constant that indicates a conflict resolution, and GUID represents the objectGUID attribute value. E-mail messages are not delivered to a user whose Active Directory account appears to be current. After an outdated domain controller or global catalog server becomes reconnected, both instances of the user object appear in the global catalog. Because both objects have the same e-mail address, e-mail messages cannot be delivered. A universal group that no longer exists continues to appear in a users access token. Although the group no longer exists, if a user account still has the group in its security token, the user might have access to a resource that you intended to be unavailable to that user. A new object or Exchange mailbox cannot be created, but you do not see the object in Active Directory. An error message reports that the object already exists. Searches that use attributes of an existing object incorrectly find multiple copies of an object of the same name. One object has been deleted from the domain, but it remains in an isolated global catalog server.
Strict and Loose Replication Constancy

If the attributes on a lingering object never change, the object is never considered for replication. However, if an attribute changes, the attribute is considered for outbound replication. The problem is that the receiving domain controller does not hold the object for the attribute being replicated. An update cannot be performed because the entire object does not exist on the partner domain controller. What happens next depends on the replication consistency set on the domain controller.

When replication consistency is set to loose, the receiving domain controller detects that it does not have the object for the attribute that is being replicated. The inbound partner requests the entire object from the outbound partner, and reanimates the object on its copy of the directory. The same process repeats on all domain controllers that do not have a copy of the object. This mechanism can be used to reanimate lingering objects across the entire forest. If a lingering object is discovered and its presence is appropriate, then you may perform any update to that object. As long as replication consistency is set to loose on all domain controllers, the object will be reanimated as it replicates around the forest. Loose replication consistency is the default for Windows 2000 domain controllers (except on domain controllers that have the Security Rollup Package installed from November 2001).
42
DRAFT V9.3

Because of the issues outlined above in the Problems section, the default behavior for Windows Server 2003 (and upgraded Windows NT 4.0 domain controllers) is to block inbound replication per naming context when a domain controller receives an update to an object that it does not have. Replication is halted in the naming context for the object until the lingering object is removed or the replication mode is set to loose. Storage for Consistency Setting The setting for replication consistency is in the registry on each domain controller.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters Strict Replication Consistency Value: 1 (Set to 0 to disable) Data type: REG_DWORD
Note A post-SP2 hot fix (also included in the SRP) from November of 2001 used a different registry value. A setting of 0 will not recreate the missing object (strict), and a setting of 1 will create the missing object. This value is only needed with the November version of the hot fix.
Value Name: Correct Missing Objects Data type: REG_DWORD Value: 1
Defines how a destination DC behaves if a source DC sends updates to an object that does not exist in the destination DCs local copy of Active Directory. o o o Destination DCs should see USN for creates before object is modified Only modifies for lingering objects arrive for object not on destination DC Only destination DCs enforce strict replication and log events
Destination DCs stop replicating from source DCs partitions containing LOs Lingering objects are quarantined on source DCs where they can be detected End-to-end replication may be impacted for partitions containing lingering objects Administrators must remove lingering objects to restore replication
Enabling Strict Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command prompt: For all domain controllers, type: repadmin /regkey * +strict
43
DRAFT V9.3
For all global catalog servers, type: repadmin /regkey gc: +strict
You can also enable strict replication by manually setting the Strict Replication Consistency registry value to 1. HKLM\System\CurrentControlSet\Services\NTDS\Parameter Strict Replication Consistency (Reg_DWORD) to 1 1 (enabled): Inbound replication of the specified directory partition from the source is stopped on the destination.
Warning: Ensure you are prepared to deal with replication failures after enabling strict replication consistency.

If you enable Loose Replication Consistency, if a destination receives a change to an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This behavior causes a lingering object to be reapplied to all domain controllers in the replication topology.
Enable Loose Replication

Use Repadmin from Window Server 2003 SP1 or later to set strict replication via command prompt:

For all domain controllers, type: repadmin /regkey * -strict For all global catalog servers, type: repadmin /regkey gc: -strict
You can also enable strict replication by manually setting the Strict Replication Consistency registry value to 0.
HKLM\System\CurrentControlSet\Services\NTDS\Parameters Value: Strict Replication Consistency Type: (Reg_DWORD) Value Data: 0 0 (disabled): The destination requests the full object from the source domain controller, and the lingering object is revived in the directory as a new object. Critical: The Loose Replication Consistency setting will cause the undesirable behavior of reanimation of lingering objects.
44
DRAFT V9.3
Ensure Strict Replication Consistency Is Enabled On Newly Promoted Domain Controllers

If you are upgrading a forest that was originally created using a computer running Windows 2000 Server, you should ensure that the forest is configured to enable strict replication consistency on newly promoted domain controllers to help avoid lingering objects. After you update the forest, all new domain controllers that you subsequently add to the forest are created with strict replication consistency disabled. However, you can implement a forest configuration change that causes new domain controllers to have strict replication consistency enabled. To ensure that new domain controllers that you add to the forest have strict replication consistency enabled, you can use Ldifde.exe to create an object in the configuration directory partition of the forest. This object is responsible for enabling strict replication consistency on any Windows Server 2003 domain controller that is promoted into the forest. The object that you create is an operational GUID with the following name:
CN=94fdebc6-8eeb-4640-80deec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDoma in>
Perform the following procedure on any domain controller in the forest to add this object to the configuration directory partition. Requirements: Administrative credentials: To complete this procedure, you must be a member of the Domain Admins group. Tools: Ldifde.exe, Notepad To create the object that ensures strict replication consistency on new domain controllers 1. In a text editor such as Notepad, create the following text file:
dn: CN=94fdebc6-8eeb-4640-80deec52b9ca17fa,CN=Operations,CN=ForestUpdates,CN=Configuration,DC=<ForestRootDo main> changetype: add objectClass: container showInAdvancedViewOnly: TRUE name: 94fdebc6-8eeb-4640-80de-ec52b9ca17fa objectCategory: CN=Container,CN=Schema,CN=Configuration,DC=<ForestRootDomain>
Where <ForestRootDomain> contains all domain components (DC=) of the forest root domain. For example, for the contoso.com forest, DC=contoso,DC=com; for the fineartschool.net forest, DC=fineartschool,DC=net.
45
DRAFT V9.3
2. Open a Command Prompt as an administrator: On the Start menu, right-click Command Prompt, and then click Run as administrator. If the User Account Control dialog box appears, provide Enterprise Admins credentials, if required, and then click Continue. 3. At the command prompt, type the following command and then press ENTER:
ldife -i f <Path\FileName>
Value -i -f <Path\FileName>
Description Specifies import mode. If not specified, the default mode is export. Identifies the import or export file name. The path and name of the import file that you created in step 1. For example, C:\ldifde.txt.
More Information:
For more information about this topic, see: http://technet.microsoft.com/en-us/library/cc780362(WS.10).aspx
Abandoned object
An object created on one DC that is not replicated to other DCs hosting a writable copy of the NC but is replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. The net effect is the object exists only in read-only copies of the partition. The object is present on RODCs or GCs hosting a read-only copy of the partition.
Abandoned delete
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
Table 5: Lingering Object Terminology
Term
Abandoned delete
Definition
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition. An object created on one DC that never got replicated
Abandoned object
46
DRAFT V9.3
to other DCs hosting a writable copy of the NC but does get replicated to DCs/GCs hosting a read-only copy of the NC. The originating DC goes offline prior to replicating the originating write to other DCs that contain a writable copy of the partition. Lingering link A linked attribute contains the DN of an object that no longer exists in Active Directory. These stale references are referred to as lingering links. An object that is present on one replica, but has been deleted and garbage collected on another replica. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, the entire object is replicated to the target for the sake of replication consistency. This undesirable behavior causes a lingering object to be reanimated. With this behavior enabled, if a destination DC receives a change to an attribute for an object that it does not have, replication is blocked with the source DC for the partition where the lingering object was detected Tombstone An object that has been deleted but not yet garbage collected The amount of time tombstones are retained in Active Directory before being garbage collected and permanently purged from the database.
Lingering Object
47
Troubleshooting Lingering Objects Lesson 2: Symptoms and Cause
DRAFT V9.3
Lesson 2: Symptoms and Cause

It is uncommon for an Administrator to be aware of and want to resolve a lingering object problem without first experiencing some other problem in their environment that leads them to discover the lingering object issue. This lesson will present common symptoms and causes of lingering objects.
What You Will Learn

After completing this lesson, you will be able to: Identify four symptoms of lingering object issues Explain three ways in which lingering objects are created List at least three methods to prevent lingering objects.
Symptoms of Lingering Objects

Detection of Domain Controllers That Have Not Replicated in the Tombstone Lifetime
Windows Server 2003 records the last time a domain controller has replicated (directly or transitively). Each domain controller will periodically compare the last time a domain controller replicated with the forests tombstone lifetime. If a domain controller does not replicate within the tombstone lifetime, event 1864 is posted to the directory service (DS) log. Event ID: 1864 NTDS Replication This is the replication status for the following directory partition on the local domain controller. The local domain controller has not recently received replication information from a number of domain controllers. The count of domain controllers is shown, divided into the following intervals. More than 24 hours: 1 More than a week: 1 More than one month: 1 More than two months: 1 More than a tombstone lifetime: 1 Tombstone lifetime (days): 60 If a domain controller in this state attempts to replicate, the inbound domain controller will block replication and alert the administrator with the message below (event 2042). In this case, the administrator has the following options. 1. Forcefully demote or reinstall the domain controllers that have not replicated, and then perform a metadata cleanup.
48
DRAFT V9.3
2. Remove any lingering objects on the non-replicating domain controller, and then enable replication with divergent or corrupt partners (as follows). a. Run repadmin /removelingeringobjects (see Removing Lingering Objects with Repadmin for instructions). b. Enable replication with divergent or corrupt partners by adding the following registry key.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters Allow Replication With Divergent and Corrupt Partner Value: 1 (Set to 0 to disable) Data type: REG_DWORD
Important Before using the above-mentioned key to override this replication safeguard, be sure to use repadmin /removelingeringobjects command to prevent the spread of unwanted lingering objects. Once replication has succeeded, be sure to remove the Replication With Divergent and Corrupt Partner value, or set it to zero.
Event Source: NTDS Replication Event Type: Error Event Category: Replication Event ID: 2042 Description: It has been too long since this machine last replicated with the named source machine. The time between replications with this source has exceeded the tombstone lifetime. Replication has been stopped with this source. The reason that replication is not allowed to continue is that the two machine's views of deleted objects may now be different. The source machine may still have copies of objects that have been deleted (and garbage collected) on this machine. If they were allowed to replicate, the source machine might return objects which have already been deleted. Time of last successful replication: <date and time of last replication> Invocation ID of source: <invocation ID of the source DC> Name of source: <replication guid._msdcs.forest.root of source DC> Tombstone lifetime (days): 60 The replication operation has failed. User Action: Determine which of the two machines was disconnected from the forest and is now out of date. You have three options: 1. Demote or reinstall the machine(s) that were disconnected.
49
DRAFT V9.3
2. Use the "repadmin /removelingeringobjects" tool to remove inconsistent deleted objects and then resume replication. 3. Resume replication. Inconsistent deleted objects may be introduced. You can continue replication by using the following registry key. Once the systems replicate once, it is recommended that you remove the key to reinstate the protection. Registry Key: HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Allow Replication With Divergent and Corrupt Partner
Replication Errors Caused by Lingering Objects

If a domain controller has replicated within the tombstone lifetime and replication consistency is set to loose, administrators cannot be alerted to the presence or replication of lingering objects. If an attribute is changed, the object is re-animated by all participating domain controllers without notification. If strict replication is enabled in the domain, replication of the partition hosting the object is halted on all inbound domain controllers. Replication for the partition stops until the object is removed or replication consistency is set to loose. When the replication is halted, the following error message is reported in the DS log on the inbound domain controller.
Event ID: 1988 Event Type: Error Event Source: NTDS Replication Event Category: Replication Description: Another domain controller has attempted to replicate into this domain controller an object which is not present on this domain controller. The object may have been deleted and already garbage collected (a tombstone lifetime or more has past since the object was deleted) on this domain controller. Replication will not continue with the source domain controller until the situation has been resolved. Source DC:<DC guid>._msdcs.<forestroot> Object:<dn of object> Object GUID: <guid of object> User Action: Verify that the object was deleted on this domain controller or in the forest. If object restoration is desired, authoritatively restore the object on the source domain controller. If restoration isn't desired, install the support tools included on the installation CD and use "repadmin /removelingeringobjects" on the source domain controller to remove the object from the forest and continue replication. To allow automatic restoration of this object and future similar objects on this domain controller, the following registry key can be deleted. Registry Key: HKLM\System\CurrentControlSet\Services\NTDS\Parameters\Strict Replication Consistency
Repadmin /showreps
50
DRAFT V9.3
In addition to the above-mentioned event, repadmin reports the following.

Sitename\<DC sending lingering object> via RPC DC object GUID: <dsa guid of dc> Last attempt @ 2002-07-19 19:14:43 failed, result 8606 (0x219e): Insufficient attributes were given to create an object. This object may not exist because it may have been deleted and already garbage collected.
Cause of Lingering Objects

How lingering objects occur
When a domain controller is disconnected for a period that is longer than the TSL, one or more objects that are deleted from Active Directory on all other domain controllers may remain on the disconnected domain controller. Such objects are called lingering objects. Because the domain controller is offline during the time that the tombstone is alive, the domain controller never receives replication of the tombstone. When this domain controller is reconnected to the replication topology, it acts as a source replication partner that has an object that its destination partner does not have. Replication problems occur when the object on the source domain controller is updated. In this case, when the destination partner tries to inbound-replicate the update, the destination domain controller responds in one of two ways:
If the destination domain controller has Strict Replication Consistency enabled, the controller recognizes that it cannot update the object. The controller locally stops inbound replication of the directory partition from the source domain controller.
If the destination domain controller has Strict Replication Consistency disabled, the controller requests the full replica of the updated object. In this case, the object is reintroduced into the directory.
Five Causes of Lingering Objects

Cause 1: The source DC sends updates to object that have already been garbage collected on the destination Dc either because the source DC has been offline or has failed replicati An object deleted on
one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
51
DRAFT V9.3
An object deleted on one DC that never got replicated to other DCs hosting a writable copy of the NC for that object. The deletion replicates to DCs/GCs hosting a read-only copy of the NC. The DC that originated the object deletion goes offline prior to replicating the change to other DCs hosting a writable copy of the partition.
on for TSL elapsed # of days

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC2 experiences a motherboard failure. Meanwhile, DC1 makes originating deletes for stale security groups over each of the next 90 days. After being offline for 90 days , DC2 gets its a replacement motherboard, powers up then originates an ACL change on all user accounts before it inbound replicates knowledge of originating deletes from DC1. DC1 logs 8606 errors for updates security groups purged on DC1 for the 1st 30 days that DC2 was offline.
Cause 2: The Source DC sends updates to objects @ the cusp of TSL expiration that have already been garbage collected by a strict mode destination DC
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. DC1 is in-place upgraded to W2K8 R2 which stamps new attributes on all objects in the configuration and writable domain partitions, including objects currently in the deleted objects container, some of which were deleted 60 days ago and now at the cusp of tombstone expiration. DC2 garbage collects some of the objects deleted TSL days ago before the replication schedule opens with DC2. Error 8606 is logged until DC1 garbage collects the blocking objects. Any updates to the partial attribute set can cause temporary lingering objects that, like the addition of the 1st W2K8 R2 DC to an existing forest, will clear themselves up once source DCs garbage collect deleted objects @ the cusp of TSL expiration.
Cause 3: A time jump on a destination DC prematurely accelerates the garbage collection of deleted objects on a destination DC
The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. The reference time source used by DC1 (but not DC2) rolls forward to calendar year 2039, causing DC2 to also adopt a system time in CY2039 which causes DC1 to prematurely purge objects deleted today from its deleted objects container. DC2 meanwhile originates changes to attributes on users, computers and groups that are live on DC2 but deleted and now prematurely garbage collected on DC1. DC1 will log error 8606 when it next inbound-replicates changes for the premature deleted objects.
Cause 4: An object is reanimated at the cusp of TSL expiration

The CONTOSO.COM domain contains two DCs in the same domain. Tombstone lifetime = 60 days. Strict replication is enabled on both DCs. DC1 and DC2 replicate every 24 hours. DC1 originates deletes on a daily basis. An OU containing users, computers and groups is
52
DRAFT V9.3
accidentally deleted. A system state backup made at the cusp of TSL in the past is auth restored on DC2. The backup contains objects that are live on DC2 but already deleted and garbage collected DC1.
Cause 5: A USN bubble is triggered the logging of the 8606

Say you create an object in a USN bubble, such that it doesnt outbound replicate because the destination DC "thinks" it has the object due to the bubble. Now, after the bubble closes and new changes start replicating again, a change is created for that object on the source DC and appears as a lingering object to the destination DC which logs the 8606 event.
Lingering Object Prevention

It's easy to come up with methods to prevent lingering objects, now that you know how they are caused. Keep the following in mind the next time someone asks you what they need to do to ensure they hit this issue again.
Important:
o o o o o o
Resolve replication failures within TSL Ensure Strict Replication Consistency is enabled Ensure large jumps in system time are blocked via registry key or policy Don't remove replication quarantine with "allowDivergent" setting without removing LOs first Don't restore system backups that are near TSL number of days old Don't bring DCs back online that haven't replicated within TSL
53
Troubleshooting Lingering Objects Lesson 3: Identification and Classification
DRAFT V9.3
Lesson 3: Identification and Classification

What You Will Learn
After completing this lesson, you will be able to: Use repadmin.exe to generate diagnostic data for analysis Use diagnostic data to determine the scope of the problem by listing all partitions and all servers containing lingering objects.
Create a replication health report

A good first step in tracking down the cause of Active Directory replication failures is to get a list of the replication errors encountered. This is a very simple procedure using repadmin /showrepl with the /csv option. For every domain controller in the forest, the spreadsheet shows the source replication partner, the time that replication last occurred, and the time that the last replication failure occurred for each naming context (directory partition). By using Autofilter in Excel, you can view the replication health for working domain controllers only, failing domain controllers only, or domain controllers that are the least or most current, and you can see the replication partners that are replicating successfully. To generate a forest-wide replication status spreadsheet for domain controllers: 1. Open a Command Prompt as an administrator: On the Start menu, right-click Command Prompt, and then click Run as administrator. If the User Account Control dialog box appears, provide Enterprise Admins credentials, if required, and then click Continue. 2. At the command prompt, type the following command, and then press ENTER
repadmin /showrepl * /csv >showrepl.csv
3. Open Microsoft Excel. 4. Click the Office button (File menu for versions prior to Excel 2010), click Open, navigate to showrepl.csv, and then click Open. 5. Hide or delete column A and column G, as follows: To hide a column, right click the column header then click Hide To delete a column, right click the column header then click Delete 6. Select a column that you want to hide or delete. 7. Select row 1 beneath the column heading row. On the View tab, click Freeze Panes, and then click Freeze Top Row. 8. Select any cell. On the Data tab, click Filter.
54
DRAFT V9.3
9. In the Last Failure Status column, click the filter down arrow, deselect the value 0. You now have a filtered report showing only the replication failures. Deselect all values except value 8606 to display just the replication failures caused by lingering objects..
Try This: Generate an AD Replication report using repadmin

Take what you have learned and try to use repadmin.exe to generate a forest-wide AD Replication report 1. Connect to DC1 in your lab environment. 2. Use the steps documented above to generate a filtered report. 3. Save the report to the desktop as showrepltimestamp.xls
Use AD Replication report and repadmin to determine the scope of the problem
The list of DCs in the Source DC column contain lingering objects when the replication report is filtered on value 8606 in column K. This display gives you the following information: DC containing lingering objects Partition where lingering objects exist
This is two of the three data points needed for repadmin /removelingeringobjects.
Important: Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid DirectoryPartition (Optional switch /advisory_mode)
DC containing lingering objects = DestinationDC Partition where lingering objects exist = DirectoryPartition
A common misconception is that the list you have just generated is comprehensive and once you remove lingering objects from the DCs in the Source DC column your job is done. However, that may not be the case as this is only a list of DCs where replication is currently blocked. It is entirely possible that once you remove lingering objects from these DCs, replication will begin failing with these now-clean DCs as the destination and a new list of DCs as the source. Once you have a list of DCs containing lingering objects
To save time, act as if all DC / GCs contain lingering objects for the partition in question. Tip:
55
Troubleshooting Lingering Objects Lesson 3: Identification and Classification
DRAFT V9.3
Run repadmin /removelingeringobjects in /advisory_mode first to see what objects are considered lingering on the DC. Event ID 1946 is logged once per lingering objects on the destination DCs Directory Services event log.
Increase the size of the Directory Services event log prior to running repadmin /removelingeringobjects with the /advisory_mode option. It is common to see the event log wrap when this command is run and the event log is the default size.
Tip:
You can also use ldifde and replfix.exe to generate a list of lingering objects. This process is describe in Lesson 4.
56
DRAFT V9.3
Lesson 4: Lingering Object Removal

What You Will Learn
After completing this lesson, you will be able to: Execute the steps in an action plan in order to remove lingering objects Remove lingering objects using five different methods
Methods to Remove Lingering Objects

Common methods to remove lingering objects include: Repadmin /Removelingeringobjects Repldiag Replfix Manually through LDP or using script Rehost the partition: o Repadmin /rehost (or /unhost and /add) (only if the partition is not-writable on the DC containing lingering objects) o Un-GC (but you dont really have control over who the DCs sources the partition from) o Demote and Promote (DCPromo)
Removing Lingering Objects with Repadmin

Repadmin includes an advanced switch (view using /experthelp) to remove lingering objects from a specific server. To remove outdated (lingering) objects from a directory partition on a domain controller that has not replicated for a tombstone lifetime, perform the following. 1. Using Repadmin, type the following at the command line:
Repadmin /RemoveLingeringObjects DestinationDC SourceDC_Guid DirectoryPartition (Optional switch /advisory_mode)
where DestinationDC is the DNS name or IP address of the domain controller that has outdated objects; and, SourceDC_Guid is the domain controllers object GUID. To obtain the objects GUID, do one of the following.
57
Troubleshooting Lingering Objects Lesson 4: Lingering Object Removal
DRAFT V9.3
o -or-
Use Repadmin /showrepl SourceDCName. The domain controllers object GUID is listed as domain controller object GUID.
In Active Directory Sites and Services, find the Source domain controller under Sites\<the domain controllers Site>\ Servers\ DCname\ NTDS Settings\ Properties. Look in the DNS Alias box. The GUID prior to _msdcs.forestrootname.com is the domain controllers Object GUID. Repadmin only needs the GUID. Omit _msdcs.forestrootname.com from the Repadmin syntax. DirectoryPartition is the distinguished name of the directory partition from which to remove outdated objects. 2. Repeat the procedure for the following partitions, as needed. Domain directory partition dc=DomainName,dc=ForestRootDomainName o Configuration directory partition cn=configuration,dc=DomainName,dc=ForestRootDomainName Application directory partition or partitions cn=ApplicationDirectoryPartitionName,dc=DomainName,dc=ForestRootDomainName Schema directory partition cn=schema,cn=configuration,dc=ForestRootDomainName
The following is an example of the command syntax.

C:\>repadmin /removelingeringobjects lonemeadc.emea.contoso.com B0AE6093-15F54DB8-836B-4495F3B19493 dc=contoso,dc=com /advisory_mode RemoveLingeringObjects successful on lonemeadc.emea.contoso.com
Events Associated with Lingering Object Removal

When removing lingering objects, the target domain controller (the domain controller with the lingering objects) will record all removal information, including source domain controller, objects removed, and a total count of all objects removed. Event ID 1937: NTDS Replication. Lingering Object Removal has been initiated on this domain controller. All objects on this DC will have their existence verified on the following source domain controller. Objects that have been deleted and garbage collected from the source domain controller will be DELETED from this domain controller if they still exist. Subsequent event logs will list all deleted objects. Source DC: <source DC guid ._msdcs.<forest root> Event ID 1945: NTDS Replication. Lingering Object Removal will DELETE the following object. Its deletion and garbage collection was detected on the source domain controller without replicating the deletion to this domain controller.
58
DRAFT V9.3
Object:DC= <dn of lingering object> Object GUID:<objectGUID> Source DC: <dc guid> ._msdcs.<forest root>
Event ID 1939: NTDS Replication. Lingering Object Removal has executed successfully on this domain controller. All objects on this domain controller have had their existence verified on the source domain controller. Objects that had been deleted and garbage collected from the source domain controller were DELETED from this domain controller. Previous event logs list all such objects. Source DC: <source DC guid> ._msdcs.<forest root> Lingering Objects Deleted 23
Details of Repadmins Lingering Object Removal Mechanism

To be added after external reviews are complete.
Remove Lingering Objects Using Repldiag

Removing lingering objects from a forest with repldiag is as simple as running repldiag /removelingeringobjects. However, it is usually best to exercise some control over the process in larger environments. The option /OverRideReferenceDC allows you to select which DC is used for cleanup. The option /outputrepadmincommandlinesyntax allows you to see what a forest-wide cleanup looks like using repadmin.
Tip:
Repldiag is by far the easiest and fastest way to remove lingering objects. The other methods are important to know when repldiag is not an option.
Help
Replication topology analyzer. Written by kenbrumf@microsoft.com Version: 2.0.3397.24022 Command Line Options: ReplDiag [/Save] [/CheckForStableReplTopology] [/RemoveLingeringObjects] [/ImportData:<FileName.XML>] [/ShowTestCases] [/OverrideDefaultReferenceDC:"dc=namingcontext,dc=com":domainController.namingcontext.com] /UseRobustDCLocation -Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. /Save -Save out the data from the current environment to XML. File is named "ReplicationData.xml" and is located in the current directory. /ImportData -Import the XML that was saved during a prior execution of this utility. Run one of the other options to do something with the data. /ShowTestCases -Show detail about test cases. Lingering Object Cleanup: /RemoveLingeringObjects -Use the current forest topology to clean all the
59
Troubleshooting Lingering Objects Lesson 4: Lingering Object Removal NCs in the forest. WILL NOT CLEAN WINDOWS 2000 SYSTEMS!!! /AdvisoryMode -Check for lingering objects only, do not clean. Must be used with /RemoveLingeringObjects. /OverrideDefaultReferenceDC -Specify reference DC for a naming context when when removing lingering objects, can be used multiple times for different NCs. Only functional if using /RemoveLingeringObjects. /OutputRepadminCommandLineSyntax -Output the command line syntax for repadmin. Only active in conjunction with /RemoveLingeringObjects. Example syntax: ReplDiag /Save - Collect the AD replication topology from the environment and save it. ReplDiag /ImportData:"ReplicationData.xml" - Load in previously collected data and check replication status. ReplDiag /RemoveLingeringObjects /OverrideDefaultReferenceDC:"cn=Configuration,dc=forestroot,dc=com":dc1.forestroot.com /OverrideDefaultReferenceDC:"dc=forestroot,dc=com":dc2.forestroot.com
DRAFT V9.3
Sample output
Repldiag.exe /save
Open ReplicationData.xml in Excel
Repldiag /removelingeringobjects /outputrepadmincommandlinesyntax

This will give you output of corresponding repadmin /removelingeringobjects syntax. It will first select one DC per partition to be used as a reference DC. It will then clean the reference DCs up against all other DCs for the partition(s) it was selected to be used as a reference for. Finally it cleans up all other DCs in the forest with the new cleaned up reference DCs as sources. The /outputrepadmincommandlinesyntax option does not actually attempt object cleanup. You would need to leave this option off if you want to execute lingering object cleanup.
Number Complete,Status,Server Name,Naming Context,Reference DC,Duration,Error Code,Error Message repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=forestdnszones,dc=contoso,dc=com
60
DRAFT V9.3
repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 87ccb4f8-1057-4cfa-aed679b5626db9fd dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com 4009aef6-b279-43d2-82f64298f02505e8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com b3ff6e2e-6025-4782-9d7b54b0431a374a dc=corp,dc=contoso,dc=com Reference NCs cleaned in 0h:0m:0s. Cleaning everything else against reference NCs. repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=forestdnszones,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 cn=configuration,dc=contoso,dc=com repadmin /removelingeringobjects 5thwardcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com a29bbfda-8425-4cb9-9c668e07d505a5c6 dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=domaindnszones,dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects loncontosodc.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects dalcorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects nycorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com repadmin /removelingeringobjects seacorpdc.corp.contoso.com 9653cb84-7aa2-4a59-ab46382e5dc1d3a8 dc=corp,dc=contoso,dc=com All NCs cleaned in 0h:0m:0s.
61
DRAFT V9.3
This output can also be viewed in Excel: Copy commands to a text file. Modify the text file to include only the command portion of the output. Then open up the text file in Exel. (space delimited)
More control: /OverRideReferenceDC

This option allows you to specify a DC that you want to be used as a reference DC for the partition specified. In a large distributed environment, take careful consideration when choosing the reference DC. Things to consider when choosing a suitable reference DC: Well connected: Fast WAN link. Performance: Excellent server class hardware: Disk, RAM, CPU and NIC Critical Network Applications / Services do not depend on this DC: Such as an Exchange facing DC Other DCs dont report replication failures with reference DC as the source: filter repadmin /showrepl * /csv ouput, or use the topology report created by repldiag /save.
repldiag /removelingeringobjects /overridedefaultreferencedc:"cn=configuration,dc=contoso,dc=com":nycorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=corp,dc=contoso,dc=com":seacorpdc.corp.contoso.com /overridedefaultreferencedc:"dc=forestdnszones,dc=contoso,dc=com":5thwardcorpdc.corp.conto so.com /outputrepadmincommandlinesyntax

Replication topology analyzer. Written by kenbrumf@microsoft.com Version: 2.0.3397.24022 Command Line Switch: /removelingeringobjects Command Line Switch: /overridedefaultreferencedc:cn=configuration,dc=contoso,dc=com:nycorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=corp,dc=contoso,dc=com:seacorpdc.corp.contoso.com Command Line Switch: /overridedefaultreferencedc:dc=forestdnszones,dc=contoso,dc=com:5thwardcorpdc.corp.contoso.com Command Line Switch: /outputrepadmincommandlinesyntax Attempting to override NC cn=configuration,dc=contoso,dc=com with DC nycorpdc.corp.contoso.com... Overriden Attempting to override NC dc=corp,dc=contoso,dc=com with DC seacorpdc.corp.contoso.com... Overriden Attempting to override NC dc=forestdnszones,dc=contoso,dc=com with DC 5thwardcorpdc.corp.contoso.com... Overriden
/UseRobustDCLocation
Query each and every DC for a list of DCs in forest. Ensures replication instability does not cause any to be missed. Weve had cases where we clean up lingering objects in the forest but do to an AD topology problem some DCs were not cleaned up. This option is almost always recommended if you want it to do a thorough job.
62
DRAFT V9.3
Remove Lingering Objects Using Replfix Remove Lingering Object using LDP or Script
Removing Lingering Objects in Windows 2000
Unfortunately, Windows 2000 provides no easy way to detect and remove lingering objects. A supported method to delete these objects is documented in MSKB 314282: Lingering Objects May Remain After You Bring an Out-of-Date Global Catalog Server Back Online In Windows 2000 SP3 (and in the post-SP2 hot fix), enhancements were made that allow an administrator to enable strict replication. This will help identify lingering objects and prevent them from replicating. However, lingering objects will not be detected unless an attribute on the object is changed.
Even though this method was first used for Windows 2000, it is still sometimes needed in certain scenarios.
Note:
Remove Lingering Objects by partition re-host operation

When one of the other methods is not an option, it is sometimes necessary to re-host the partition from a DC containing a good clean writable copy of the partition. This may be a temporary solution if the problem is widespread since the DC may later replicate with a DC that is not clean.
If re-host is necessary, it is usually best to identify all GCs needing the procedure and clean them up at the same time to prevent recurrence.
Tip:
Un-hosting a partition
It is sometimes necessary to remove a partition from the database of a DC temporarily. Repadmin includes a /rehost option that allows you to do this, but the /unhost option allows you to exercise more control over the procedure. Take note that /unhost only allows you to remove a read-only copy of the partition. With the exception of application partitions, you cannot remove a writable copy of a partition from a DC without using DCPROMO.
Repadmin /?:unhost Remove a specific read-only partition from a GC. [SYNTAX] /unhost DSA <Naming Context> Repadmin /unhost ContosoDC1 dc=corp,dc=contoso,dc=com
63
DRAFT V9.3
Event ID 1659 indicates the status of the un-host operation. Do not re-add the partition until event ID 1660 is logged in the Directory Services event log.
Warning: The re-host operation may fail with error 8339 if you attempt to re-add the partition too soon after the un-host.
Manually adding a replication connection using repadmin.exe

The add command will create a RepsFrom attribute on the destination domain controller for the specified naming context and initiate a replication request. During a normal replication cycle, the destination domain controller will request updates from the source domain controller. When creating temporary replication links between replication partners, the process could fail if the KCC starts while you are performing the procedure. The KCC will delete any replication links for which no corresponding connection object exists. Since these commands can take a very long time to complete as they trigger the replication of the corresponding naming context, it is important to ensure that KCC do not disturb the process. This is where you would use +DISABLE_NTDSCONN_XLATE which effectively disables KCC's capability to translate connection objects to replication links.
Disable KCC connection translation so that KCC doesnt remove our temporary replication connection:
Repadmin /options ContosoDC1 +disable_ntdsconn_xlate
Then add a replication connection for the configuration partition of the server we want to source the partition from:
Repadmin /add <Naming Context> <Dest DSA> <Source DSA> [/readonly] [/selsecrets] <Source DSA> The source DSA must be specified by fully qualified computername. repadmin /add cn=configuration,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
Add a replication connection to the server for the domain partition that we need to source from (/readonly is specified if the partition is a GC non-writable
partition /selsecrets needs to be specified if the destination DC is an RODC):
64
DRAFT V9.3
repadmin /add dc=emea,dc=contoso,dc=com ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com /readonly

One-way replication from source:LONEMEADC.Emea.contoso.com to dest:ContosoDC1.contoso.com established.
If you need to replicate the other way, then just reverse the order of the server names in the commands. To begin a normal sync of the partition using the new replication connection:
Repadmin /replicate <Dest_DSA_LIST> <Source DSA_NAME> <Naming Context> [/force] [/async] [/full] [/addref] [/readonly] repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly
To begin a full sync of that partition using the new replication connection:
repadmin /replicate ContosoDC1.contoso.com LONEMEADC.Emea.contoso.com dc=emea,dc=contoso,dc=com /readonly /full
Sync from LONEMEADC.Emea.contoso.com to ContosoDC1.contoso.com completed successfully.
Turn KCC connection translation back on when you no longer need the connection:
Repadmin /options ContosoDC1 -disable_ntdsconn_xlate
DSASTAT
Dsastat can be used to compare the number of objects that exist on two domain controllers. However, it cannot report on which objects exist on one and not the other. Likewise, it cannot make an intelligent determination about the differences. Replication latency or other factors might result in valid cases where an object exists but has not replicated out. Some objects are set to not replicate (like the Universal group membership cache). For this reason, DSASTAT can only be used as a guideline for comparisons between naming contexts hosted on different domain controllers.
Read-only Naming Context (Global Catalogs)

The global catalog is particularly susceptible to problems caused by lingering objects. This is because an object can exist on a read-only naming context, but not in the domain naming context from which it originally replicated. If it still existed in the domain naming context, it could be deleted there, and the tombstone could remove it from the global catalog. The other problem is that global catalogs can replicate from each other. The global catalog function might be removed from a computer, and then reinstated in an attempt to re-replicate the partial attribute set from a domain controller hosting the writable copy of the naming
65
DRAFT V9.3
context. In this case, the global catalog might replicate from another global catalog. This would return the object you were trying to delete. A better solution is to determine whether the object exists on all global catalogs. If it does not, remove the global catalog function from all servers that contain the object. Then reinstate the global catalog function on all of them, and let a clean copy of the directory replicate in. In larger environments, removing and reinstating the global catalog function might be undesirable and prohibited. Applications such as Microsoft Exchange Server depend on the global catalog to operate. Moreover, the additional traffic incurred as the domains re-replicate into the global catalog might be undesirable. In this case, use the post-SP2 hot fix and process described in the following article. MSKB 314282: Lingering Objects May Remain After You Bring an Out-of-Date Global Catalog Server Back Online
66
DRAFT V9.3
Lesson 5: Real World Application

What You Will Learn
After completing this lesson, you will be able to: Create a detailed action plan that will remove the lingering objects in all partitions on all servers Recommend the correct method to remove lingering objects given five different scenarios Recommend changes that will result in a better solution given a subpar action plan
Determining What to Do with a Lingering Object

In most cases, a lingering object results from a missed tombstone. In other words, the object was intentionally deleted because one or more domain controllers received the instruction to delete the object. In rare cases, the object was not actually deleted. In such cases, the existence of the object may be intended. Determining what to do with a deleted object depends on whether or not it was intentionally deleted. First, lets look at some common lingering object scenarios, and then discuss the recommended corrective action. Known Cause: Domain controller has not replicated beyond the tombstone lifetime. If a domain controller has not replicated within the tombstone lifetime, it will likely have missed the deletions of some attributes (or inbound replicated tombstones). Left alone, the objects will persist without any replication. If any of the attributes are changed, or if another domain controller is doing a full sync (as when a global catalog is populating its copy of the domain partition), the objects will attempt to replicate out and cause problems. These lingering objects are unintended, and should be removed using repadmin (see below). Unknown Cause: Security principal is attempting to replicate in. In some cases, a user or computer object has become a lingering object without any known cause. These are almost always undesirable. However, before removing them, check the event log and see what object is being replicated in. If the object is desired, enable loose consistency. (Refer to the section Intended Objects below.) Unknown Cause: Deletion is replicating in (tombstone replicating in). If the inbound object is a deletion (the object will include DEL in the name), it is probably harmless and not needed. However, if the deleted object still exists on another domain controller somewhere else in the forest, removing this lingering object will actually turn the other .live. copy into a lingering object.
The next section examines what to do with intended and unintended objects.
Unintended Objects
Use Repadmin to delete these lingering objects (see below).
67
Troubleshooting Lingering Objects Lesson 5: Real World Application
DRAFT V9.3
Intended Objects
Change the replication consistency on the inbound domain controller. The object will be reanimated on this domain controller. When using this method, the following things should be considered. After the object has been reanimated and replicated into the domain controller, it will replicate out to the domain controllers other partners. It is not likely that the other partners will have the object, and inbound replication will be blocked until the consistency setting is changed. This might result in the lingering object or re-animation moving throughout the domain. To animate the object fully, you might have to .chase. the replication failures throughout the forest. Use Eventcomb to monitor for the lingering object detection event. While the idea of chasing a lingering object around a forest might not seem like much fun, there is a good reason to do it. It is possible to turn off replication consistency in a domain or forest (using scripts or custom ADM files with Group Policy). However, this could have some unwanted side effects: for example, replication would be blocked for the first lingering object.
68
DRAFT V9.3
10.0 Lab Guide

This lab manual describes the environment required to perform practice exercises in this course and lab sessions included in this manual.
Before You Begin

Before starting this course you should: Complete... course prerequisite. Review... course prerequisite.
Practice exercises are performed on physical and virtual machines on one computer per participant. To complete the exercises, your computer hardware and software must be configured as described in this section. For additional details, refer to the Classroom Setup Guide that accompanies this course.
Critical: Lab sessions that accompany this course use a preconfigured virtual machine environment. If you start or modify VMs in any way prior to use in lab exercises, exercise tasks and steps will not work as intended. DO NOT start or modify any VM until instructed to do so in the lab exercises. Preconfigured VMs use lab environment scripts to complete certain steps at first launch based on the computer name entered in mini-setup. Failure to enter computer names specified in the lab exercises exactly as shown will incorrectly configure VMs, which will cause lab exercise tasks and steps to fail.
What You Will Learn

After completing the labs in this course, you will be able to: Describe | Explain course objective. Install | Configure course objective. Analyze | Troubleshoot... course objective.
69
Troubleshooting Lingering Objects Lab Sessions
DRAFT V9.3
Lab Sessions
This manual includes the following lab sessions. Each lab includes step-by-step instructions to complete the exercises. You can use the problem solving lab exercises in your workbook to challenge your understanding of course material and refer to the Lab Manual for detailed steps if needed.
Lab 1: Exploring Lingering Object Fundamentals

During this lab, you will identify the forest's configured tombstone lifetime and replication consistency settings Estimated time to complete this lab: 15 minutes
Lab 2: Lingering Object Diagnosis and Documentation

During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will then analyze that data and document all lingering objects in the environment. Estimated time to complete this lab: 30 minutes
Lab 3: Lingering Object removal using repadmin

During this lab, you will remove lingering objects from the environment using repadmin /removelingeringobjects. Estimated time to complete this lab: 30 minutes
Lab 4: Lingering Object removal using ldp and repldiag

During this lab, you will remove a single lingering object using ldp. You will then remove the remaining lingering objects using repldiag. Estimated time to complete this lab: 30 minutes
Lab 5: Abandoned Object and Abandoned Deleted object remediation

During this lab, you will identify and remove an abandoned object. You will then remediate and abandoned deleted object scenario. Estimated time to complete this lab: 30 minutes
Lab 6: Lingering Link identification and cleanup

During this lab, you will identify all lingering-linked values in the environment. You will them remove them in order to ensure group membership consistency. Estimated time to complete this lab: 45 minutes
Setting Up Your Lab Environment

To complete this lab, you will need the hardware and software configuration described in this section.
70
DRAFT V9.3
Hardware
Practice exercises assume that all lab hardware is listed on the Hardware Compatibility List (HCL) as compatible with operating systems and applications described later in this section. The following table describes minimum hardware requirements for practice exercises.
Table 6: Minimum Hardware Requirements
Minimum System Requirements Computer/Processor Operating System Memory Storage Display Peripherals Computer with a 2.4 GHz processor or higher (If available, disable hyperthreading and enable hardware virtualization) <Host OS>; see Classroom Setup Guide for details 4 GB RAM 160 GB hard drive CD or DVD drive (DVD drive recommended) Super VGA (800 x 600) or higher-resolution monitor with 256 color (Recommended: 1024 x 768 with 16-bit or higher color) Microsoft Mouse or compatible pointing device Microsoft or compatible keyboard
Software
Operating systems and applications listed in the following table must be installed on all computers.
Table 7: Lab Computer requirements
Software Microsoft Windows 7, Enterprise Edition Current Microsoft Windows 7, Enterprise Edition Service Pack 1 and Critical Updates Office 2010 Professional Microsoft Office 2010 OneNote Current Office 2010 Service Pack 1 and Critical Updates Microsoft .NET Framework Version 2.0 Current .NET Framework 2.0 Service Pack and Critical Updates Microsoft .NET Framework Version 3.0 Current .NET Framework 3.0 Service Pack and Critical Updates
Version tested and notes Service Pack 1
Service Pack 1 Service Pack 1
Retail
Retail
71
DRAFT V9.3
Version tested and notes Retail
Software Current Adobe Reader Version Current Adobe Reader Critical Updates
Network Layout
The following figure illustrates the lab network. The lab network must be isolated from production networks.
Figure 3: Network Layout
Individual computer configurations are described in detail in the next section.
Computer Names and IP Addresses

Table 8: Lab Computer Names and IP Addresses on page 73 lists computer configurations for the classroom lab network. Replace <Host> in Computer Name with <site>-<room> as follows:
<site> <room> Site name abbreviation (Example: For Las Colinas, use LC1 or LC2) Room number (Example: For Rio Grande classroom, use 1693)
72
DRAFT V9.3
For example, participant computer 1 in the Rio Grande classroom in Las Colinas Building 2 would be named LC2-1693-1 or LC2-1693-1A (see table). Replace x in IP address with the classroom number or any representative number that is unique on the overall classroom subnet and reference this number in all lab exercises.
This computer naming convention eliminates potential issues when multiple classrooms are connected to the same subnet during classroom configuration or course delivery.
Important:
Table 8: Lab Computer Names and IP Addresses
Computer Name <Host>- Instr-1 <Host>-Instr-2 <Host>-1 <Host>-2 <Host>-3 <Host>-4 <Host>-5 <Host>-6 <Host>-7 <Host>-8 <Host>-9 <Host>-10 <Host>-11 <Host>-12 <Host>-13 <Host>-14 <Host>-15 <Host>-16
IP Address 172.168.1.200 172.168.x.201 172.168.x.101 172.168.x.102 172.168.x.103 172.168.x.104 172.168.x.105 172.168.x.106 172.168.x.107 172.168.x.108 172.168.x.109 172.168.x.110 172.168.x.111 172.168.x.112 172.168.x.113 172.168.x.114 172.168.x.115 172.168.x.116
Preferred DNS Server 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200 192.168.x.200
Role Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server Stand-alone Server
Configuring Your Computer(s)

Each student requires one physical machine with a fully configured virtual machine environment. Before starting this lab, make sure your computer is configured as follows: <Operating System and Version> installed and started
73
DRAFT V9.3
Virtual Server 2005 R2 SP1 (may be preinstalled on classroom computers)

Virtual Server 2005 R2 SP1 may already be installed on your computer. If these applications are not installed, you may obtain a free download of the installation files from: http://www.microsoft.com/downloads/details.aspx?FamilyID=bc49c7c8-4840-4e67-8dc41e6e218acce4&DisplayLang=en
Note:
Windows Server 2008 DVD media or installation ISO file in <path>. Virtual machines installed or created on the computer: o o o <VMName>: <OS | Role | description> <VMName>: <OS | Role | description> <VMName>: <OS | Role | description>
Course files located in the C:\Labfiles and C:\VS folders on your computer or accessible from a network share on the instructor computer.
Accounts and Group Membership

Important: You must log on as an administrative user in order to perform some of the tasks in this lab.
The following user accounts and passwords must be configured on the physical computer and in all virtual machines: Administrative username and password
Administrator Local Administrators Password: LS1setup!
Username: Member of:
Normal username and password

Studentn Local Users Password: LS1setup!
Username: Member of:
Replace n in Studentn with the number assigned to your classroom computer by the instructor.
Domain Membership
Your physical computer is not joined to a domain. Lab exercises may require you to join the following virtual domain(s): Contoso.com
74
DRAFT V9.3
Virtual machines joined to a virtual domain require group and account configurations as shown in the following table.
Table 9: Groups and Accounts
Group Domain Groups Domain Administrators Domain Users Local Groups Administrators Users
Members
Administrator Studentn
Administrator, Domain Administrators Studentn, Domain Users
Shares on Instructor Computer(s)

During lab exercises, you may be required to access the following shares on the instructor computer(s): \\<Host>-Instr-1\Labfiles Includes lab files installed on participant host computers \\<Host>-Instr-1\VS Includes files required for virtual machine environment
Using the Keyboard and Mouse in a Virtual Machine

This course includes virtual machines for lab exercises. You use the keyboard and a mouse to control a virtual machine much as you would a physical computer. This section explains how to use the keyboard and mouse in virtual machines and describes special keys and menu items.
Using the Keyboard

In general, the keyboard works the same for a virtual machine as it does for a physical computer. However, some keyboard shortcuts such as Ctrl+Alt+Delete do not work within a virtual machine because of the interaction between the host operating system and the guest operating system. Virtual Server 2005 provides much of the required keyboard functionality with a Host key and keyboard shortcuts. By default, the Host key is the Right-Alt key. You can use the Host key in two ways: If a virtual machine has captured the pointer, press the Host key to return control of the mouse to the host operating system.
75
DRAFT V9.3
Use the Host key in combination with other keys for specific functions as described in the following table.
Table 10. Keyboard Shortcuts for Virtual Machines
Key Combination Host Key+Delete Host Key+C Host Key+A Host Key+I Host Key+V Host Key+H Host Key+Enter
Description Sends Ctrl+Alt+Delete functionality to the virtual machine operating system. Connects the Remote Control or VMRC to the VMRC server. Switches the Remote Control or VMRC to the Administrator Display. Displays connection information. Sets the virtual machine so that the guest operating system cannot be manipulated. You can only view the virtual machine window. Displays the control to set the Host key. Switches the virtual machine window to full-screen display. This option is available only when you connect to a virtual machine using the VMRC client. Switches to the previous virtual machine. This option is available only when you connect to a virtual machine using the VMRC client. Switches to the next virtual machine. This option is available only when you connect to a virtual machine using the VMRC client.
Host Key+Left Arrow Host Key+Right Arrow
Tip:
As shown in the preceding table, you can use Host Key+Delete to send the functionality of the Ctrl+Alt+Delete keyboard shortcut to a guest operating system running in a virtual machine. You can also use Send Ctrl+Alt+Del from the Remote Control menu of either the VMRC or Remote View page.
Using the Mouse

The way you use the mouse depends on whether Virtual Machine Additions is installed. If Virtual Machine Additions is installed on the virtual machine, you can move the pointer freely between the virtual machine window and the host operating system. This simplifies switching among virtual machines and the host operating system. If Virtual Machine Additions is not installed on the virtual machine, the virtual machine must capture the pointer before the mouse can be used within the virtual machine window. The virtual machine captures the pointer when you click the pointer inside the virtual machine window.
If a pointer is captured by a virtual machine on which Virtual Machine Additions is not installed, the virtual machine must release it before you can use the mouse on the host
76
DRAFT V9.3
operating system or in another virtual machine window. You can use the Host key to return the use of the mouse to the host operating system.
77
Troubleshooting Lingering Objects Lab 1: Exploring Lingering Object Fundamentals
DRAFT V9.3
Lab 1: Exploring Lingering Object Fundamentals

During this lab, you will identify the forest's configured tombstone lifetime and replication consistency settings Estimated time to complete this lab: 15 minutes
Before You Begin

To complete this lab: Complete Lesson 1
What You Will Learn

After completing this lab, you will be able to determine the Active Directory settings that govern how it handles tombstones and lingering objects. Exercise 1 Determine tombstone lifetime. Exercise 2 Determine DC replication consistency setting.
Scenario
You are assisting a customer who is having issues with
Configuring Your Computer(s)

Each student requires at least one physical computer and a fully configured local or remote hosted virtual machine environment. Before starting this lab, make sure your computer is configured as described in About This Lab.
Configuring Your Virtual Machine Environment

Exercises in this Lab require the following virtual machines: VMname: DC1
Exercises may also require files located in the C:\Labfiles folder on your computer or accessible from a network share on the instructor computer.
Accounts and Group Membership

Important: You must log on as an administrative user in order to perform some of the tasks in this lab.
The following user accounts and passwords must be configured on the physical computer and in all virtual machines:
78
DRAFT V9.3
Administrative username and password Username: Administrator
Member of: Local Administrators
Password:
LS1Setup!
Normal username and password Username: Usern

Member of: Local Users
Password:
LS1Setup!
Replace n in Usern with the number assigned to your classroom computer by the instructor.
Domain Membership
Lab exercises may require virtual machines to be joined to the following virtual domain(s): Contoso.com
Virtual machines joined to a virtual domain require group and account configurations as shown in the following table.
Table 11. Groups and Accounts
Group Domain Groups Domain Administrators Domain Users Local Groups Administrators Users
Members
Administrator Usern
Administrator, Domain Administrators Usern, Domain Users
Exercise 1: Determine Tombstone Lifetime Setting

In this exercise, you will attempt to determine the tombstone lifetime setting of the forest.
Scenario
You are assisting a customer that is having issues
Task Detailed Steps Complete these steps by connecting to DC1
79
DRAFT V9.3
Task Task Description
Detailed Steps 1. Step. a. Sub-step. Setting | Parameter Item 1 Item 2 b. Sub-step. c. Sub-step. 2. Step. Value
Task Description
1. Step. a. Sub-step. b. Edit the registry as shown below:

Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReporting\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
c. 2. Step. Task Description
Sub-step
1. Step. a. Sub-step. b. Sub-step. 2. Step. 1. Step. a. Sub-step. b. Sub-step. 2. Step.
Task Description
Review
1. <Question> Answer
80
DRAFT V9.3
Exercise 2: Determine forest and DC replication consistency settings

In this exercise, you will identify the replication consistency settings for each DC in the environment and will determine if the f
Scenario
You are assisting a customer that is having issues <add scenario here>.
Task Detailed Steps Complete these steps by connecting to <VM name> Task Description 3. Step. a. Sub-step. Setting | Parameter Item 1 Item 2 b. Sub-step. c. Sub-step. 4. Step. Task Description 3. Step. a. Sub-step. b. Edit the registry as shown below:
Key Name: HKEY_CURRENT_USER\Software\Microsoft\PCHealth\ErrorReportin g\DW Value: Name: DWAllQueuesHeadless Type: REG_DWORD Data: 0x1
Value
c. 4. Step. Task Description
Sub-step
3. Step. a. Sub-step. b. Sub-step. 4. Step.
81
DRAFT V9.3
Task Description
3. Step. a. Sub-step. b. Sub-step. 4. Step.
Review
82
DRAFT V9.3
Lab 2: Lingering Object Diagnosis and Documentation

During this lab, you will generate diagnostic data via repadmin, ldifde and replfix. You will then analyze that data and document all lingering objects in the environment. Estimated time to complete this lab: 30 minutes
Before You Begin

To complete this lab: Complete Lesson 1 and Lesson 2
What You Will Learn

After completing this lab you will be able to <Lab terminal objective>. After completing the exercises you will be able to: Exercise 1 enabling objective. Exercise 2 enabling objective.
Scenario
Exercise 1: Lingering Object Diagnosis

<Briefly describe the goal of the exercise>
Scenario
Tasks
<Define starting conditions, including virtual machines and lab files required>. 1. <Task>. a. <Step>. i. <Sub-step>. ii. <Sub-step>. b. <Step>. 2. <Task>.
83
Troubleshooting Lingering Objects Lab 2: Lingering Object Diagnosis and Documentation
DRAFT V9.3
Value
Setting | Parameter Item 1 Item 2
3. <Task>. a. <Step>. Edit the registry as shown below:

b. <Step>.
Review
Exercise 2: Lingering Object Documentation

Scenario
Tasks
<Define starting conditions, including virtual machines and lab files required>.
84 2011 Microsoft Corporation. All rights reserved.
DRAFT V9.3
4. <Task>. a. <Step>. i. <Sub-step>. ii. <Sub-step>. b. <Step>. 5. <Task>.
Setting | Parameter Item 1 Item 2
Value
6. <Task>. a. <Step>. Edit the registry as shown below:

b. <Step>.
Review
85
Troubleshooting Lingering Objects Lab 3: Lingering Object removal using repadmin
DRAFT V9.3
Lab 3: Lingering Object removal using repadmin

During this lab, you will remove lingering objects from the environment using repadmin /removelingeringobjects. Estimated time to complete this lab: 30 minutes
Before You Begin

To complete this lab: Complete lessons 1-4
What You Will Learn

After completing this lab you will be able to remove lingering objects using repadmin. After completing the exercises you will be able to: Exercise 1 enabling objective. Exercise 2 enabling objective.
Scenario
Exercise 1: <Problem Solving Exercise Title>

Scenario
You have completed recovering files from a back up and now need to restore the files.
Tasks
<Define starting conditions, including virtual machines and lab files required>. 1. <Task>. a. <Step>. b. <Step>.
Setting | Parameter Item 1 Item 2 Value
2. <Task>.
86
DRAFT V9.3
Sample solution
Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab Manual that accompanies this course. For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that accompanies this course.
Review
Exercise 2: <Simulation Exercise Title>

Scenario
You have received email from your manager requesting a maintenance action.
Tasks
1. Read Email from your manager explaining the situation. <Add email text here> 2. Review supporting documents in <local path>: a. Company organization chart. b. Company ____ data. c. Report on problems with the ____ system.
87
Troubleshooting Lingering Objects Lab 3: Lingering Object removal using repadmin
DRAFT V9.3
3. Open the VM containing the company system and resolve the issues.
Sample solution
Your result should look something like the Sample in <Lab Title>, <Exercise Title> in the Lab Manual that accompanies this course. For step by step instructions, see <Lab Title>, <Exercise Title> in the Lab Manual that accompanies this course.
Review
88
DRAFT V9.3
Lab 4: Lingering Object removal using ldp and repldiag

During this lab, you will remove a single lingering object using ldp. You will then remove the remaining lingering objects using repldiag. Estimated time to complete this lab: 30 minutes
Before You Begin

To complete this lab: Complete <list lesson(s) etc.>.
What You Will Learn

After completing this lab, you will be able to <Lab terminal objective>. Exercise 1 enabling objective. Exercise 2 enabling objective.
Scenario
89
Troubleshooting Lingering Objects Lab 5: Abandoned Object and Abandoned Deleted object remediation
DRAFT V9.3
Lab 5: Abandoned Object and Abandoned Deleted object remediation

During this lab, you will identify and remove an abandoned object. You will then remediate and abandoned deleted object scenario. Estimated time to complete this lab: 30 minutes
Before You Begin

To complete this lab: Complete 1-4 Configure | verify your lab environment: o o o Virtual machines <VM name(s)> installed and configured. <Application name> installed and configured. <List and link to specific lab files if needed>.
What You Will Learn

Scenario
90
DRAFT V9.3
Lab 6: Lingering Link identification and cleanup

During this lab, you will identify all lingering-linked values in the environment. You will them remove them in order to ensure group membership consistency. Estimated time to complete this lab: 45 minutes
Before You Begin

To complete this lab: Complete lessons 1-4 Configure | verify your lab environment: o o o Virtual machines <VM name(s)> installed and configured. <Application name> installed and configured. <List and link to specific lab files if needed>.
What You Will Learn

Scenario
91
Troubleshooting Lingering Objects 10.0 Presentation Slides
DRAFT V9.3
10.0 Presentation Slides
92
DRAFT V9.3
93
DRAFT V9.3
94
DRAFT V9.3
95
DRAFT V9.3
96
DRAFT V9.3
97
DRAFT V9.3
98

Active Directory Replication Troubleshooting

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Active Directory Replication Troubleshooting

Încărcat de

Drepturi de autor:

Formate disponibile

Active Directory Replication Troubleshooting

Troubleshooting Lingering Objects

Released: October 17, 2011

Project Lead: Bio:

2.0 Problem ................................................................................................................................................... 2

3.0 Learning Expectations ............................................................................................................................. 4

4.0 Learning Activities ................................................................................................................................... 6

5.0 Assessment ............................................................................................................................................. 8

6.0 Evaluation ............................................................................................................................................. 16

9.0 Course Workbook ................................................................................................................................. 36

Lesson 1: Lingering Objects Fundamentals ................................................................................................. 39

Lesson 2: Symptoms and Cause .................................................................................................................. 48

Lesson 3: Identification and Classification .................................................................................................. 54

Lesson 4: Lingering Object Removal ........................................................................................................... 57

Lesson 5: Real World Application ............................................................................................................... 67

Lab 1: Exploring Lingering Object Fundamentals........................................................................................ 78

Lab 2: Lingering Object Diagnosis and Documentation .............................................................................. 83

Lab 3: Lingering Object removal using repadmin ....................................................................................... 86