AD Replication Issues Server not replicated in a long time

hello

i  have server dc1 has not replicated log time.

rebuilding not option it's remotely located

here repadmin /showreps

site1dc1

dsa options: is_gc

site options: (none)

dsa object guid: a0a03b2f-3cef-4fae-b721-786ef49d24b0

dsa invocationid: 0bdc582b-7a5b-4308-99fb-e81ad6350040

source: site2\dc3
******* 10 consecutive failures since 2014-09-23 20:50:57

last error: -2146893022 (0x80090322):

            target principal name incorrect.

naming context: cn=configuration,dc=company,dc=local

source: site2\dc3
******* warning: kcc not add replica link due error.

naming context: dc=forestdnszones,dc=company,dc=local

source: site2\dc3
******* warning: kcc not add replica link due error.

naming context: dc=domaindnszones,dc=company,dc=local

source: site2\dc3
******* warning: kcc not add replica link due error.

naming context: dc=company,dc=local

source: site2\dc3
******* warning: kcc not add replica link due error.

source: site3\dc2

******* 10 consecutive failures since 2014-09-23 20:51:00

last error: -2146893022 (0x80090322):

            target principal name incorrect.

source: site2\dc6

******* 1 consecutive failures since 2014-09-23 22:52:44

last error: 1722 (0x6ba):

            rpc server unavailable.

naming context: cn=configuration,dc=company,dc=local

source: site2\dc6

******* warning: kcc not add replica link due error.

naming context: dc=forestdnszones,dc=company,dc=local

source: site2\dc6

******* warning: kcc not add replica link due error.

naming context: dc=domaindnszones,dc=company,dc=local

source: site2\dc6

******* warning: kcc not add replica link due error.

naming context: dc=company,dc=local

source: site2\dc6

******* warning: kcc not add replica link due error.

and dc diag /c

directory server diagnosis

performing initial setup:

   trying find home server...

   home server = dc1

   * identified ad forest.
   done gathering initial info.

doing initial required tests

  
   testing server: site1\dc1

      starting test: connectivity

         ......................... dc1 passed test connectivity

doing primary tests

  
   testing server: site1\dc1

      starting test: advertising

         ......................... dc1 passed test advertising

      starting test: checksecurityerror

         [dc1] no security related replication errors found on

         dc!  target connection specific source dc use

         /replsource:<dc>.

         ......................... dc1 passed test checksecurityerror

      starting test: cutoffservers

         ......................... dc1 passed test cutoffservers

      starting test: frsevent

         ......................... dc1 passed test frsevent

      starting test: dfsrevent

         there warning or error events within last 24 hours after the

         sysvol has been shared.  failing sysvol replication problems may cause

         group policy problems.
         ......................... dc1 failed test dfsrevent

      starting test: sysvolcheck

         ......................... dc1 passed test sysvolcheck

      starting test: frssysvol

         ......................... dc1 passed test frssysvol

      starting test: kccevent

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:19

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:19

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:20

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:20

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:21

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:22

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:22

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:23

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:24

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:14:25

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:44

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:45

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:45

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:46

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:46

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:47

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:48

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:48

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:49

            event string:

            attempt establish replication link following writable directory partition failed.

         warning event occurred.  eventid: 0x80000785

            time generated: 09/23/2014   22:21:50

            event string:

            attempt establish replication link following writable directory partition failed.

         ......................... dc1 passed test kccevent

      starting test: knowsofroleholders

         [dc3] dsbindwithspnex() failed error -2146893022,

         target principal name incorrect..
         warning: dc3 schema owner, not responding ds

         rpc bind.

         [dc3] ldap bind failed error 8341,

         directory service error has occurred..
         warning: dc3 schema owner, not responding ldap

         bind.

         warning: dc3 domain owner, not responding ds

         rpc bind.

         warning: dc3 domain owner, not responding ldap

         bind.

         warning: dc3 pdc owner, not responding ds rpc

         bind.

         warning: dc3 pdc owner, not responding ldap

         bind.

         warning: dc3 rid owner, not responding ds rpc

         bind.

         warning: dc3 rid owner, not responding ldap

         bind.

         warning: dc3 infrastructure update owner, not

         responding ds rpc bind.

         warning: dc3 infrastructure update owner, not

         responding ldap bind.

         ......................... dc1 failed test knowsofroleholders

      starting test: machineaccount

         ......................... dc1 passed test machineaccount

      starting test: ncsecdesc

         ......................... dc1 passed test ncsecdesc

      starting test: netlogons

         ......................... dc1 passed test netlogons

      starting test: objectsreplicated

         ......................... dc1 passed test objectsreplicated

      starting test: outboundsecurechannels

         ** did not run outbound secure channels test because /testdomain: was

         not entered

         ......................... dc1 passed test

         outboundsecurechannels

      starting test: replications

         replication-received latency warning

         dc1:  current time 2014-09-23 22:22:08.

            dc=forestdnszones,dc=company,dc=local
               last replication received dc2 @
          2014-06-21 16:56:38
               last replication received dc4 @
          2014-06-21 17:08:38
               last replication received dc5 @
          2014-06-21 18:53:35
               last replication received dc6 @
          2014-06-21 18:53:34
               last replication received dc7 @
          2014-06-21 17:08:38
               last replication received dc3 @
          2014-06-21 18:56:46
            dc=domaindnszones,dc=company,dc=local
               last replication received dc2 @
          2014-06-21 16:56:38
               last replication received dc4 @
          2014-06-21 17:08:37
               last replication received dc5 @
          2014-06-21 18:53:35
               last replication received dc6 @
          2014-06-21 18:56:58
               last replication received dc7 @
          2014-06-21 17:08:37
               last replication received dc3 @
          2014-06-21 18:56:58
            cn=schema,cn=configuration,dc=company,dc=local
               last replication received dc2 @
          2014-06-21 16:56:38
               last replication received dc4 @
          2014-06-21 17:08:37
               last replication received dc5 @
          2014-06-21 18:53:35
               last replication received dc6 @
          2014-06-21 18:53:34
               last replication received dc7 @
          2014-06-21 17:08:37
               last replication received dc3 @
          2014-06-21 18:56:43
            cn=configuration,dc=company,dc=local
               last replication received dc2 @
          2014-06-21 17:05:10
               last replication received dc4 @
          2014-06-21 17:08:36
               last replication received dc5 @
          2014-06-21 18:53:35
               last replication received dc6 @
          2014-06-21 18:53:34
               last replication received dc7 @
          2014-06-21 17:08:35
               last replication received dc3 @
          2014-06-21 18:56:43
            dc=company,dc=local
               last replication received dc2 @
          2014-06-21 16:56:38
               last replication received dc4 @
          2014-06-21 17:08:37
               last replication received dc5 @
          2014-06-21 18:53:35
               last replication received dc6 @
          2014-06-21 18:57:19
               last replication received dc7 @
          2014-06-21 17:08:35
               last replication received dc3 @
          2014-06-21 19:01:08
         ......................... dc1 passed test replications

      starting test: ridmanager

         ......................... dc1 failed test ridmanager

      starting test: services

         ......................... dc1 passed test services

      starting test: systemlog

         error event occurred.  eventid: 0xc0001b63

            time generated: 09/23/2014   21:26:09

            event string:

            timeout (30000 milliseconds) reached while waiting transaction response umrdpservice service.

         error event occurred.  eventid: 0xc0001b63

            time generated: 09/23/2014   21:26:39

            event string:

            timeout (30000 milliseconds) reached while waiting transaction response scdeviceenum service.

         error event occurred.  eventid: 0xc0001b58

            time generated: 09/23/2014   21:26:39

            event string:

            smart card device enumeration service service failed start due following error:

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:28:34

            event string:

            kerberos client received krb_ap_err_modified error server dc2$. target name used cifs/dc2.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:36:48

            event string:

            kerberos client received krb_ap_err_modified error server . target name used host/dc2.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain () different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:44:30

            event string:

            kerberos client received krb_ap_err_modified error server dc6$. target name used ldap/4db3f8ca-a1b8-47fe-9edf-f07a4f6f506a._msdcs.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:47:56

            event string:

            kerberos client received krb_ap_err_modified error server dc3$. target name used ldap/dc3.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:50:28

            event string:

            kerberos client received krb_ap_err_modified error server dc3$. target name used company\dc3$. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:50:43

            event string:

            kerberos client received krb_ap_err_modified error server dc2$. target name used company\dc2$. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:51:28

            event string:

            kerberos client received krb_ap_err_modified error server dc3$. target name used e3514235-4b06-11d1-ab04-00c04fc2dcd2/fb138164-6f72-452f-a911-fd03e47c3b10/company.local@company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   21:51:31

            event string:

            kerberos client received krb_ap_err_modified error server dc2$. target name used e3514235-4b06-11d1-ab04-00c04fc2dcd2/484f72cd-dc70-41d7-a9fe-b2b9941a179c/company.local@company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   22:06:35

            event string:

            kerberos client received krb_ap_err_modified error server dc2$. target name used ldap/484f72cd-dc70-41d7-a9fe-b2b9941a179c._msdcs.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   22:06:36

            event string:

            kerberos client received krb_ap_err_modified error server dc3$. target name used ldap/fb138164-6f72-452f-a911-fd03e47c3b10._msdcs.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         error event occurred.  eventid: 0x40000004

            time generated: 09/23/2014   22:16:22

            event string:

            kerberos client received krb_ap_err_modified error server dc3$. target name used ldap/dc3.company.local. indicates target server failed decrypt ticket provided client. can occur when target server principal name (spn) registered on account other account target service using. ensure target spn registered on account used server. error can happen if target service account password different configured on kerberos key distribution center target service. ensure service on server , kdc both configured use same password. if server name not qualified, , target domain (company.local) different client domain (company.local), check if there identically named server accounts in these 2 domains, or use fully-qualified name identify server.

         ......................... dc1 failed test systemlog

      starting test: topology

         ......................... dc1 passed test topology

      starting test: verifyenterprisereferences

         ......................... dc1 passed test

         verifyenterprisereferences

      starting test: verifyreferences

         ......................... dc1 passed test verifyreferences

      starting test: verifyreplicas

         ......................... dc1 passed test verifyreplicas

  
      starting test: dns

        

         dns tests running , not hung. please wait few minutes...

         ......................... dc1 passed test dns

  
   running partition tests on : forestdnszones

      starting test: checksdrefdom

         ......................... forestdnszones passed test checksdrefdom

      starting test: crossrefvalidation

         ......................... forestdnszones passed test

         crossrefvalidation

  
   running partition tests on : domaindnszones

      starting test: checksdrefdom

         ......................... domaindnszones passed test checksdrefdom

      starting test: crossrefvalidation

         ......................... domaindnszones passed test

         crossrefvalidation

  
   running partition tests on : schema

      starting test: checksdrefdom

         ......................... schema passed test checksdrefdom

      starting test: crossrefvalidation

         ......................... schema passed test crossrefvalidation

  
   running partition tests on : configuration

      starting test: checksdrefdom

         ......................... configuration passed test checksdrefdom

      starting test: crossrefvalidation

         ......................... configuration passed test crossrefvalidation

  
   running partition tests on : company

      starting test: checksdrefdom

         ......................... company passed test checksdrefdom

      starting test: crossrefvalidation

         ......................... company passed test crossrefvalidation

  
   running enterprise tests on : company.local

      starting test: dns

         test results domain controllers:

           
            dc: dc1.company.local

            domain: company.local

           

                 
               test: dynamic update (dyn)
                  warning: failed delete test record dcdiag-test-record in zone company.local
        
         summary of test results dns servers used above domain

         controllers:

        

            dns server: 128.8.10.90 (d.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.127.in-addr.arpa. failed on dns server 128.8.10.90              
            dns server: 2001:500:1::803f:235 (h.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:1::803f:235              
            dns server: 2001:500:2::c (c.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:2::c              
            dns server: 2001:500:2d::d (d.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:2d::d              
            dns server: 2001:500:2f::f (f.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:2f::f              
            dns server: 2001:500:3::42 (l.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:3::42              
            dns server: 2001:500:84::b (b.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:500:84::b              
            dns server: 2001:503:ba3e::2:30 (a.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:503:ba3e::2:30              
            dns server: 2001:503:c27::2:30 (j.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:503:c27::2:30              
            dns server: 2001:7fd::1 (k.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:7fd::1              
            dns server: 2001:7fe::53 (i.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:7fe::53              
            dns server: 2001:dc3::35 (m.root-servers.net.)

               1 test failure on dns server

               ptr record query 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa failed on dns server 2001:dc3::35              
               dc1                  pass pass pass pass warn pass n/a 
         ......................... company.local passed test dns

      starting test: locatorcheck

         ......................... company.local passed test locatorcheck

      starting test: fsmocheck

         ......................... company.local passed test fsmocheck

      starting test: intersite

         ......................... company.local passed test intersite

any thoughts appreciated

thank you

hiya,

first , foremost, might stating obvious. domain kerberos traffic failing, hence domain related errors. domain services rely on kerberos, if not working, domain severed.

basically kerberos requires 2 things in order function:

1: correct spns.

2: correct dns. (looking @ tests attached, last part indicates dns issues)

so should verify dc's.

Windows Server  >  Directory Services