state of network interfaces fluctuating in cluster netint command
the problem have cluster architecture in there 2 node
connectivity between 2 node in cluster netint commands show unreachable 5-10 mins , automatically 20mins or so. , process continues
during unreachable state node ips pinged successfully.
c:\winnt\profiles\administrator>cluster netint
listing status available network interfaces:
node network status
-------------------------- -------------------------- -------
m101ap1b ipn100-2 unreachable
m101ap1a local maintenance failed
m101ap1a heartbeat 2 up
m101ap1b ipn100-1 up
m101ap1a ipn100-2 unreachable
m101ap1a heartbeat 1 up
m101ap1b heartbeat 2 up
m101ap1b local maintenance failed
m101ap1b heartbeat 1 up
m101ap1b public up
m101ap1a public up
m101ap1a ipn100-1 up
after 10 mins or so
c:\winnt\profiles\administrator>cluster netint
listing status available network interfaces:
node network status
-------------------------- -------------------------- -------
m101ap1b ipn100-2 up
m101ap1a local maintenance failed
m101ap1a heartbeat 2 up
m101ap1b ipn100-1 up
m101ap1a ipn100-2 up
m101ap1a heartbeat 1 up
m101ap1b heartbeat 2 up
m101ap1b local maintenance failed
m101ap1b heartbeat 1 up
m101ap1b public up
m101ap1a public up
m101ap1a ipn100-1 up
have put trace on ether ports .packets go through during unreachable , state. following logs attached when state unreachable ,
00000544.00000f98::2009/06/01-08:59:50.887 info [nm] received connectivity report node 1 (interface 1) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000f98::2009/06/01-08:59:50.887 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b28::2009/06/01-08:59:53.251 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-08:59:53.251 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] interface 1 , interface 0 not agree on connectivity on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] aborting state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 until connectivity data valid.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b24::2009/06/01-08:59:54.813 info [clmsg] received interface unreachable event node 1 network 1
00000544.0000091c::2009/06/01-08:59:54.813 warn [nm] communication lost interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2)
00000544.0000091c::2009/06/01-08:59:54.813 info [nm] updating local connectivity info network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.0000091c::2009/06/01-08:59:54.813 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b28::2009/06/01-08:59:57.156 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-08:59:57.156 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] started failure isolation timer (3600ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 in state 1
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumsendupdate: locker waiting type 2 context 15
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] thread 0xb1c updatelock wait on type 2
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumpdolockingupdate: lock free, granted 2
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumpdolockingupdate successful, sequence=2021661 generation=0
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumsendupdate: locker dispatching seq 2021661 type 2 context 15
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] received update set state network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 warn [nm] interface 31f083f9-2eef-4553-82b2-e1c639703a6b unreachable (node: m101ap1b, network: ipn100-2).
....
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:03:02.132 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:03:02.132 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.343 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.343 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:03:02.353 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:03:02.373 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:03:02.373 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:04:02.309 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:04:02.309 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.529 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:04:02.529 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.739 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.739 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:04:02.749 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:04:02.769 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:04:02.769 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:05:02.696 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:05:02.696 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.906 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:05:02.906 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.116 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.116 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:05:03.126 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:05:03.146 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:05:03.146 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.000009d0::2009/06/01-09:05:20.221 info [qfs] getdiskfreespaceex i:\mscs\, status 0
00000544.00000b28::2009/06/01-09:06:03.082 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:06:03.082 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.293 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:06:03.293 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.513 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.513 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:06:03.523 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:06:03.543 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:06:03.543 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:07:03.556 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:07:03.556 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.766 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:07:03.766 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.977 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.977 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:07:03.987 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:07:04.007 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:07:04.007 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:08:03.943 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:08:03.943 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:04.153 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:08:04.153 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.364 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.364 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:08:04.374 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:08:04.384 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:08:04.384 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:09:04.330 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:09:04.330 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.540 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:09:04.540 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.750 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.750 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:09:04.760 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:09:04.781 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:09:04.781 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000f98::2009/06/01-09:09:51.558 info [nm] received connectivity report node 1 (interface 1) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000f98::2009/06/01-09:09:51.558 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b24::2009/06/01-09:09:52.439 info [clmsg] received interface event node 1 network 1
00000544.0000091c::2009/06/01-09:09:52.439 info [nm] communication (re)established interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2)
00000544.0000091c::2009/06/01-09:09:52.439 info [nm] updating local connectivity info network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:09:53.901 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:09:53.901 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 1 reports interface 0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 0 reports interface 1 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] unavailable=0, failed = 0, unreachable=0, reachable=2, up=2 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 in state 3
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker waiting type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] thread 0xb1c updatelock wait on type 2
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdolockingupdate: lock free, granted 2
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdolockingupdate successful, sequence=2021662 generation=0
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker dispatching seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] received update set state network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 31f083f9-2eef-4553-82b2-e1c639703a6b (node: m101ap1b, network: ipn100-2).
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2).
00000544.00000b1c::2009/06/01-09:09:53.901 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2) up.
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: dispatching seq 2021662 type 2 context 15 node 1
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker updating seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdounlockingupdate releasing lock ownership
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: completed update seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
connectivity between 2 node in cluster netint commands show unreachable 5-10 mins , automatically 20mins or so. , process continues
during unreachable state node ips pinged successfully.
c:\winnt\profiles\administrator>cluster netint
listing status available network interfaces:
node network status
-------------------------- -------------------------- -------
m101ap1b ipn100-2 unreachable
m101ap1a local maintenance failed
m101ap1a heartbeat 2 up
m101ap1b ipn100-1 up
m101ap1a ipn100-2 unreachable
m101ap1a heartbeat 1 up
m101ap1b heartbeat 2 up
m101ap1b local maintenance failed
m101ap1b heartbeat 1 up
m101ap1b public up
m101ap1a public up
m101ap1a ipn100-1 up
after 10 mins or so
c:\winnt\profiles\administrator>cluster netint
listing status available network interfaces:
node network status
-------------------------- -------------------------- -------
m101ap1b ipn100-2 up
m101ap1a local maintenance failed
m101ap1a heartbeat 2 up
m101ap1b ipn100-1 up
m101ap1a ipn100-2 up
m101ap1a heartbeat 1 up
m101ap1b heartbeat 2 up
m101ap1b local maintenance failed
m101ap1b heartbeat 1 up
m101ap1b public up
m101ap1a public up
m101ap1a ipn100-1 up
have put trace on ether ports .packets go through during unreachable , state. following logs attached when state unreachable ,
00000544.00000f98::2009/06/01-08:59:50.887 info [nm] received connectivity report node 1 (interface 1) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000f98::2009/06/01-08:59:50.887 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b28::2009/06/01-08:59:53.251 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-08:59:53.251 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] interface 1 , interface 0 not agree on connectivity on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] aborting state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 until connectivity data valid.
00000544.00000b1c::2009/06/01-08:59:53.251 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b24::2009/06/01-08:59:54.813 info [clmsg] received interface unreachable event node 1 network 1
00000544.0000091c::2009/06/01-08:59:54.813 warn [nm] communication lost interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2)
00000544.0000091c::2009/06/01-08:59:54.813 info [nm] updating local connectivity info network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.0000091c::2009/06/01-08:59:54.813 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b28::2009/06/01-08:59:57.156 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-08:59:57.156 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] started failure isolation timer (3600ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 in state 1
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumsendupdate: locker waiting type 2 context 15
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] thread 0xb1c updatelock wait on type 2
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumpdolockingupdate: lock free, granted 2
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumpdolockingupdate successful, sequence=2021661 generation=0
00000544.00000b1c::2009/06/01-08:59:57.156 info [gum] gumsendupdate: locker dispatching seq 2021661 type 2 context 15
00000544.00000b1c::2009/06/01-08:59:57.156 info [nm] received update set state network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-08:59:57.156 warn [nm] interface 31f083f9-2eef-4553-82b2-e1c639703a6b unreachable (node: m101ap1b, network: ipn100-2).
....
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:03:01.922 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:03:02.132 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:03:02.132 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.343 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.343 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:03:02.353 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:03:02.363 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:03:02.373 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:03:02.373 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:03:02.373 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:04:02.309 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:04:02.309 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.309 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:04:02.529 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:04:02.529 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.739 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.739 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:04:02.749 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:04:02.759 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:04:02.769 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:04:02.769 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:04:02.769 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:05:02.696 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:05:02.696 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.696 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:05:02.906 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:05:02.906 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.116 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.116 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:05:03.126 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:05:03.136 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:05:03.146 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:05:03.146 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:05:03.146 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.000009d0::2009/06/01-09:05:20.221 info [qfs] getdiskfreespaceex i:\mscs\, status 0
00000544.00000b28::2009/06/01-09:06:03.082 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:06:03.082 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.082 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:06:03.293 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:06:03.293 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.513 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.513 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:06:03.523 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:06:03.533 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:06:03.543 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:06:03.543 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:06:03.543 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:07:03.556 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:07:03.556 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.556 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:07:03.766 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:07:03.766 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.977 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.977 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:07:03.987 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:07:03.997 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:07:04.007 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:07:04.007 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:07:04.007 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:08:03.943 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:08:03.943 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:03.943 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:08:04.153 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:08:04.153 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.364 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.364 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:08:04.374 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:08:04.384 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:08:04.384 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:08:04.384 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:09:04.330 info [nm] failure isolation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:09:04.330 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 1 reports interface 0 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 0 reports interface 1 unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) unreachable on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unavailable=0, failed = 0, unreachable=2, reachable=0, up=0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] trying determine scope of connectivity failure 2 interfaces on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unreachable interface 1 = 31f083f9-2eef-4553-82b2-e1c639703a6b on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.330 info [nm] unreachable interface 2 = 96c40e5e-190e-48af-9256-471bd94433df on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:04.540 info [nm] found address 192.168.170.4 interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:09:04.540 info [nm] online enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.750 info [nm] online enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.750 info [nm] total online address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:09:04.760 info [nm] ping enum interface 31f083f9-2eef-4553-82b2-e1c639703a6b contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] ping enum interface 96c40e5e-190e-48af-9256-471bd94433df contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] total ping address count network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 2
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] union ping list network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 contains 1 addresses
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] issuing rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] pinging targets interface 31f083f9-2eef-4553-82b2-e1c639703a6b.
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] pinging host 192.168.170.128
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] ping of host 192.168.170.128 succeeded.
00000544.00000b1c::2009/06/01-09:09:04.770 info [nm] waiting rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df complete
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] wait rpcdointerfaceping interface 96c40e5e-190e-48af-9256-471bd94433df completed.
00000544.00000b1c::2009/06/01-09:09:04.781 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 state indeterminate, scheduling failure isolation poll
00000544.00000b1c::2009/06/01-09:09:04.781 warn [nm] failed read isolationpolltimervalue or value out of range,status 2 using default 60000 ms
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] started failure isolation timer (60000ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] connectivity failure scope determination complete network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:04.781 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000f98::2009/06/01-09:09:51.558 info [nm] received connectivity report node 1 (interface 1) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000f98::2009/06/01-09:09:51.558 info [nm] started state recalculation timer (2400ms) network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2)
00000544.00000b24::2009/06/01-09:09:52.439 info [clmsg] received interface event node 1 network 1
00000544.0000091c::2009/06/01-09:09:52.439 info [nm] communication (re)established interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2)
00000544.0000091c::2009/06/01-09:09:52.439 info [nm] updating local connectivity info network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b28::2009/06/01-09:09:53.901 info [nm] state recalculation timer expired network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2).
00000544.00000b28::2009/06/01-09:09:53.901 info [nm] scheduled worker thread service network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] worker thread processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] beginning phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] examining connectivity data interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 1 reports interface 0 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 0 (31f083f9-2eef-4553-82b2-e1c639703a6b) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] examining connectivity data interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 0 reports interface 1 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 1 (96c40e5e-190e-48af-9256-471bd94433df) on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] completed phase 1 of state computation network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] unavailable=0, failed = 0, unreachable=0, reachable=2, up=2 on network 0741cf2a-c41d-4872-bfd6-3241cb4c2789
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 in state 3
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker waiting type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] thread 0xb1c updatelock wait on type 2
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdolockingupdate: lock free, granted 2
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdolockingupdate successful, sequence=2021662 generation=0
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker dispatching seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] received update set state network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 31f083f9-2eef-4553-82b2-e1c639703a6b (node: m101ap1b, network: ipn100-2).
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] interface 96c40e5e-190e-48af-9256-471bd94433df (node: m101ap1a, network: ipn100-2).
00000544.00000b1c::2009/06/01-09:09:53.901 warn [nm] network 0741cf2a-c41d-4872-bfd6-3241cb4c2789 (ipn100-2) up.
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: dispatching seq 2021662 type 2 context 15 node 1
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: locker updating seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumpdounlockingupdate releasing lock ownership
00000544.00000b1c::2009/06/01-09:09:53.901 info [gum] gumsendupdate: completed update seq 2021662 type 2 context 15
00000544.00000b1c::2009/06/01-09:09:53.901 info [nm] worker thread finished processing network 0741cf2a-c41d-4872-bfd6-3241cb4c2789.
Windows Server > High Availability (Clustering)
Comments
Post a Comment