#12922 vHMC network issues
Closed: Fixed with Explanation by kevin. Opened by kevin.

Our vHMC (virtual hardware management console) appliance vm is having network issues. ;(

It all started when I moved it's disk from a file to a logical volume, but I can't see why that would have caused it, and indeed, I have moved it back and it's still doing it.

symptoms:

  • the vHMC ( 10.16.163.140 ) gets about 50% packet loss to the power10 systems in the mgmt vlan.
  • from vmhost-x86-02 (where it's running) 0% packet loss to power10 systemd in mgmt vlan
  • to/from vmhost-x86-02, 0% packet loss.

The appliance itself is AIX? or some weird linux variant. I only have a restricted shell on it via ssh, so it's hard to get really any debugging out of it.
The network settings are all the same as they were before.

The problem really seems to be isolated to it, so I don't think it's really the virthost or the firewall or other items.

I guess the only things I can think of left to do are to update it, but that will require some tricks since it cant seem to talk to the ibm site to get the update.

CC: @arrfab


kevin@vHMC:~> ping 10.16.160.140 -c 10
PING 10.16.160.140 (10.16.160.140) 56(84) bytes of data.
64 bytes from 10.16.160.140: icmp_seq=3 ttl=61 time=0.358 ms
64 bytes from 10.16.160.140: icmp_seq=4 ttl=61 time=0.371 ms
64 bytes from 10.16.160.140: icmp_seq=7 ttl=61 time=0.389 ms
64 bytes from 10.16.160.140: icmp_seq=9 ttl=61 time=0.367 ms
--- 10.16.160.140 ping statistics ---
10 packets transmitted, 4 received, 60% packet loss, time 9231ms
rtt min/avg/max/mdev = 0.358/0.371/0.389/0.017 ms
[root@vmhost-x86-02 ~][PROD-RDU3]# ping -c 10 10.16.160.140
PING 10.16.160.140 (10.16.160.140) 56(84) bytes of data.
64 bytes from 10.16.160.140: icmp_seq=1 ttl=61 time=0.256 ms
64 bytes from 10.16.160.140: icmp_seq=2 ttl=61 time=0.254 ms
64 bytes from 10.16.160.140: icmp_seq=3 ttl=61 time=0.259 ms
64 bytes from 10.16.160.140: icmp_seq=4 ttl=61 time=0.277 ms
64 bytes from 10.16.160.140: icmp_seq=5 ttl=61 time=0.274 ms
64 bytes from 10.16.160.140: icmp_seq=6 ttl=61 time=0.263 ms
64 bytes from 10.16.160.140: icmp_seq=7 ttl=61 time=0.540 ms
64 bytes from 10.16.160.140: icmp_seq=8 ttl=61 time=0.252 ms
64 bytes from 10.16.160.140: icmp_seq=9 ttl=61 time=0.254 ms
64 bytes from 10.16.160.140: icmp_seq=10 ttl=61 time=0.272 ms
--- 10.16.160.140 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9218ms
rtt min/avg/max/mdev = 0.252/0.290/0.540/0.083 ms

Metadata Update from @james:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, high-trouble

We rebooted the virthost this vm is on as part of our updates/reboots today and after that... it's been stable again.

No idea why it was happening, but it's not now, so I guess I will call it a win. ;)

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Metadata