Not sure what is going on but vpn connections for various tools are very slow with 20 to 40 second to start
local:
[root@proxy02 xinetd.d][PROD]# time rsync localhost:: log Server Logs real 0m0.012s user 0m0.002s sys 0m0.005s
from log01 to proxy02
[smooge@log01 fedora_stats][PROD-IAD2]$ time rsync proxy02.vpn.fedoraproject.org:: log Server Logs real 2m0.382s user 0m0.000s sys 0m0.003s
Tried other connections and got fast time
time curl --connect-to fedoraproject.org:443:proxy02.vpn.fedoraproject.org:443 "https://fedoraproject.org/" <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="https://getfedora.org/">here</a>.</p> <hr> <address>Apache Server at fedoraproject.org Port 443</address> </body></html> real 0m0.346s user 0m0.018s sys 0m0.009s
This seems to be caused by SSH and our default ifcfg using DNS from IAD2 for all systems.
Metadata Update from @smooge: - Issue assigned to smooge - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: dns, medium-gain, medium-trouble, ops
This was caused by duh duh duhhhh DNS. systemd-resolve was firing up before every connection to xinetd and the DNS configuration for system-resolve was using 10.3.163.33 versus /etc/resolv.conf. From looking at verbose logs connections were doing: * try to look up host on 10.3.163.33 UDP and timeout * try to look up host on 10.3.163.33 TCP and timeout * try to look up host on 10.3.164.34 UDP and timeout * try to look up host on 10.3.164.34 TCP and timeout. * go to /etc/resolv.conf and get the hostname * repeat.
Changing all the networkmanager DNS to use the same as /etc/resolv.conf fixed the issue.
Metadata Update from @smooge: - Issue untagged with: dns, medium-gain, medium-trouble, ops - Issue close_status updated to: Fixed - Issue priority set to: Needs Review (was: Waiting on Assignee) - Issue status updated to: Closed (was: Open)