SSSD / sssd

#3400 sss_ssh_knownhostsproxy connections blocked mysql connection
Closed: worksforme Opened by matrixzj.

Closed: worksforme
Reopen Issue

Background:
server A running with mysql database. And it will simultaneously send a 1.3GB file to 14 clients.

With 'ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h':
mysql connection will be blocked by those 14 rsync connections.
from 'netstat -tupnlo' result, we can find that send-queue is higher and higher, looks like it has sent has been blocked. Finally, after mysql 'net_write_timeout', connection will be closed since no data can be sent from this connection.

without 'ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h':
mysql connection can be worked as normal.

sss_ssh_knownhostsproxy version: sssd-common-1.14.0-43.el7_3.11.x86_64
rsync version: rsync-3.0.9-17.el7.x86_64
kernel version: 3.10.0-229.el7.x86_64

If anything info needed, I can collect for further investigation.

Is the problem only with sss_ssh_knownhostproxy or also with any other ProxyCommand (netcat ...)
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#ProxyCommand_with_Netcat

BTW I assume it is not related to mysql. So it would be good to provide simpler reproducer maybe with scp/rsync/...

BTW I assume it is not related to mysql. So it would be good to provide simpler reproducer maybe with scp/rsync/...

tried to reproduce it with rsync, but failed. rsync works fine.

tried to reproduce it with rsync, but failed. rsync works fine.

hmm,
I am not really sure how mysql uses ssh connection. I was able to look something up but it needn't be your case. https://www.digitalocean.com/community/tutorials/how-to-secure-mysql-replication-using-ssh-on-a-vps

If you use mysql + port forwarding then it should be possible to reproduce with cat /dev/urandom or something similar. It would really be helpful for us to find a simpler reproducer which we can try to debug on our side. Because I have no clue what can be a problem.

thanks for your update

mysql connection is not configured with port forwarding :(

According to 'netstat -tupnao' result, i found that
At 'May 17 07:39:13 GMT 2017', send-queue of mysql connection (server port 3306) was 52128.
After 120 seconds (mysql config net_write_timeout) , at 'May 17 07:41:13 GMT 2017', connection was closed, and everything in send-queue has been discarded. Here are some snippets from netstat result.

......snippet.....
Wed May 17 07:39:12 GMT 2017
tcp        0      0 10.0.11.192:3306        10.0.11.229:38605       ESTABLISHED 21656/mysqld         keepalive (7175.55/0/0)
Wed May 17 07:39:13 GMT 2017
tcp        0  52128 10.0.11.192:3306        10.0.11.229:38605       ESTABLISHED 21656/mysqld         on (0.26/1/0)
Wed May 17 07:39:14 GMT 2017
tcp        0  52128 10.0.11.192:3306        10.0.11.229:38605       ESTABLISHED 21656/mysqld         on (0.05/2/0) 
......snippet.....
Wed May 17 07:41:12 GMT 2017
tcp        0  52128 10.0.11.192:3306        10.0.11.229:38605       ESTABLISHED 21656/mysqld         on (85.79/9/0)
Wed May 17 07:41:13 GMT 2017
tcp        0  52129 10.0.11.192:3306        10.0.11.229:38605       FIN_WAIT1   -                    on (84.78/9/0)
Wed May 17 07:41:14 GMT 2017
tcp        0      0 10.0.11.192:3306        10.0.11.229:38605       TIME_WAIT   -                    timewait (0.00/0/0)
Edited by matrixzj

I still miss a big picture. You found out that removing ProxyCommand from ssh_config make it works. But I cannot see any relation between ssh and mysql.
Could you try to describe your setup? And what is a interaction between mysql and ssh (and thus ProxyCommand)?

I also think there is no any relationship between ssh_config and mysql.

In my thinking, since network bandwidth are exhausted by 14 rsync connections. mysql connection was broken is expected behavior.

But now, what I see is very weird.
Without 'ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h' config in ssh_config, some data can be sent via mysql connection. I can identify it from netstat result. And new connections can be established from mysql clients.
With 'ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h' config in ssh_config, no data can be sent and new connections can not be established.

Edited by matrixzj

more findings:

server A bandwidth was exhausted by rsync, try to run a 'mysql' cmd. mysql will be failed with following error:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0

mysql strace shown:

29861 16:46:30 futex(0x984d30, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000012>
29861 16:46:30 write(3,"W\0\0\1\205\242\17\0\0\0\0\1!\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0backend_new\0\24\301\333z\34VUu\301\325R\224SB\345}\206\3=t\27mysql_native_password\0", 91) = 91 <0.000013>
29861 16:46:30 read(3, "", 16384)       = 0 <10.008160>
29861 16:46:40 read(3, "", 16384)       = 0 <0.000078>
29861 16:46:40 shutdown(3, SHUT_RDWR)   = 0 <0.000019>
29861 16:46:40 close(3)                 = 0 <0.000060>

and from tcpdump, we found that packets can not be sent 

~ tshark -r Downloads/mysql.pcap
1   0.000000  10.0.11.229 ? 10.0.11.192  TCP 74 43599 ? 3306 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=499217280 TSecr=0 WS=512
2   0.000907  10.0.11.192 ? 10.0.11.229  TCP 74 3306 ? 43599 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=797307839 TSecr=499217280 WS=512
3   0.014459  10.0.11.229 ? 10.0.11.192  TCP 66 43599 ? 3306 [ACK] Seq=1 Ack=1 Win=14848 Len=0 TSval=499217295 TSecr=797307839
4   0.015437  10.0.11.192 ? 10.0.11.229  MySQL 179 Server Greeting proto=10 version=5.6.34-enterprise-commercial-advanced-log
5   0.228530  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797308067 TSecr=499217295
6   0.442516  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797308281 TSecr=499217295
7   0.871518  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797308710 TSecr=499217295
8   1.729521  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797309568 TSecr=499217295
9   3.445523  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797311284 TSecr=499217295
10   6.881535  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797314720 TSecr=499217295
11  10.025522  10.0.11.192 ? 10.0.11.229  TCP 66 3306 ? 43599 [FIN, ACK] Seq=114 Ack=1 Win=14848 Len=0 TSval=797317864 TSecr=499217295
12  13.745525  10.0.11.192 ? 10.0.11.229  TCP 179 [TCP Retransmission] 3306 ? 43599 [FIN, PSH, ACK] Seq=1 Ack=1 Win=14848 Len=113 TSval=797321584 TSecr=499217295
Edited by matrixzj

I am sorry it still did not help to me. I have no idea what is a relation between mysql and ssh. Please provide more information otherwise we will not be able to help. The ideal would to to provide reproducer which we can use in-house.

Please reopen if you can provide the information requested in the previous comment.

Metadata Update from @jhrozek:
- Issue close_status updated to: worksforme
- Issue status updated to: Closed (was: Open)

Hi, lslebodn

Got help from RedHat professional support.

We found that the root cause is that a higher IPQos was set via sss_ssh_knownhostsproxy.

NOTE:
rsync source: 192.168.0.40
rsync dest: 192.168.14.30

By default, rsync IPQos should be 0x8 (Throughput from ssh_config). Here is a snippets of tcpdump data without 'sss_ssh_knownhostsproxy'

04:24:11.676742 IP (tos 0x8, ttl 64, id 29266, offset 0, flags [DF], proto TCP (6), length 1624)
    192.168.0.40.36674 > 192.168.14.30.ssh: Flags [P.], cksum 0x95e1 (incorrect -> 0xa45a), seq                     
24464:26036, ack 2684, win 75, options [nop,nop,TS val 966951562 ecr 3865550466], length 1572
04:24:11.677069 IP (tos 0x8, ttl 64, id 18511, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.14.30.ssh > 192.168.0.40.36674: Flags [.], cksum 0x9ad4 (correct), ack 26036, win 83, 
options [nop,nop,TS val 3865550551 ecr 966951562], length 0
04:24:11.826789 IP (tos 0x8, ttl 64, id 29268, offset 0, flags [DF], proto TCP (6), length 1368)
    192.168.0.40.36674 > 192.168.14.30.ssh: Flags [P.], cksum 0x94e1 (incorrect -> 0x0c78), seq 
26036:27352, ack 2684, win 75, options [nop,nop,TS val 966951712 ecr 3865550551], length 1316
04:24:11.828897 IP (tos 0x8, ttl 64, id 18512, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.14.30.ssh > 192.168.0.40.36674: Flags [.], cksum 0x9483 (correct), ack 27352, win 83, 
options [nop,nop,TS val 3865550702 ecr 966951712], length 0
04:24:11.951713 IP (tos 0x8, ttl 64, id 29269, offset 0, flags [DF], proto TCP (6), length 1368)
    192.168.0.40.36674 > 192.168.14.30.ssh: Flags [P.], cksum 0x94e1 (incorrect -> 0xf4c6), seq 
27352:28668, ack 2684, win 75, options [nop,nop,TS val 966951837 ecr 3865550702], length 1316

With 'sss_ssh_knownhostsproxy', IPQos is set to '0x00'

04:20:06.470137 IP (tos 0x0, ttl 64, id 42776, offset 0, flags [DF], proto TCP (6), length 1368)
    192.168.0.40.36668 > 192.168.14.30.ssh: Flags [P.], cksum 0x94e1 (incorrect -> 0xfd1c), seq 
32152:33468, ack 2960, win 75, options [nop,nop,TS val 966706356 ecr 3865305219], length 1316
04:20:06.509803 IP (tos 0x8, ttl 64, id 545, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.14.30.ssh > 192.168.0.40.36668: Flags [.], cksum 0xefd2 (correct), ack 33468, win 83, 
options [nop,nop,TS val 3865305384 ecr 966706356], length 0
04:20:06.595168 IP (tos 0x0, ttl 64, id 42777, offset 0, flags [DF], proto TCP (6), length 1624)
    192.168.0.40.36668 > 192.168.14.30.ssh: Flags [P.], cksum 0x95e1 (incorrect -> 0x331f), seq 
33468:35040, ack 2960, win 75, options [nop,nop,TS val 966706481 ecr 3865305384], length 1572

mysql traffic will be always with IPQos 0x8.

According to default qdisc policy 'qfifo_past', traffic with TOS value 0x8 will be put in band2, and traffice with IPQos '0x00' will be put in band1. As long as there are packets waiting in band 0, band 1 won't be processed. Same goes for band 1 and band 2. Since there is always packets in 'band 1' , packets in 'band 2' has to wait more.

Metadata Update from @matrixzj:
- Issue status updated to: Open (was: Closed)

sss_ssh_knownhostsproxy does not set any IPQos and is executed before any other ssh connection. We cannot guess which IPQos will be used by following connection.

I do not think we can do anything with that. Maybe something could be fixed on openssh side.
Edited by lslebodn

Yes, the sssd helper programs are just dummies that sshd calls into..

I agree that more investigation should be done with openssh..

Metadata Update from @jhrozek:
- Issue close_status updated to: worksforme
- Issue status updated to: Closed (was: Open)

SSSD is moving from Pagure to Github. This means that new issues and pull requests
will be accepted only in SSSD's github repository.

This issue has been cloned to Github and is available here:
- https://github.com/SSSD/sssd/issues/4427

If you want to receive further updates on the issue, please navigate to the github issue
and click on subscribe button.

Thank you for understanding. We apologize for all inconvenience.
Metadata
None
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.