SSH Tunnel - key rejected when DPI-SSH is enabled
As part of a hybrid render farm network running in our office and on AWS, a service that is part of the render farm manager (AWS Thinkbox Deadline) creates an SSH tunnel to an EC2 instance that servers as the gateway to the render nodes, ensuring communication with the on-prem database, license server port forwarding etc.
We have configured the IAM user for SonicWall and the EC2 instance's public IP address is automatically added to an Address Object, and port 22 is enabled. However, when the service checks the connection with the server, the encryption key gets rejected.
To debug, we tried connecting to the same server using PuTTY and a ,ppk version of the key. The key also got rejected.
So we tried to turn off the DPI-SSH option in the SonicWall configuration, and lo and behold, both PuTTY and the tunnel service of our render farm connected properly!
FYI, when the service tries to check the connection, it asks the server for the supported keys, and saves a UserKnownHostsFile .keys file with the reply.
We noticed that when we run through our old firewall or through the SonicWall with DPI-SSH off, the .keys file contains several lines like this:
# XXX.XXX.XXX.XXX:22 SSH-2.0-OpenSSH_7.4
Also, the file contains the 3 supported key types listed in the sshd_config file on the server.
The sshd_config looks like this:
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
However, when DPI-SSH is on, the .keys file contains lines like this:
# XXX.XXX.XXX.XXX:22 SSH-2.0-libssh_0.9.0
and it lists only the ssh_rsa key. I assume it is using a different config file.
Looks like SonicWall ends up causing the one or the other implementation installed on the AMI to be used by the tunnel depending on the state of the DPI-SSH option. libssh v0.9.0 is ancient (June 2019) and is probably part of the base Amazon Linux 2 OS, so I would understand if SonicWall is preventing it from communicating.
Please note that we have no control over the AMI used to launch that server, as it is controlled by AWS Thinkbox. However, if there is anything that has to be reconfigured there, we could ask AWS Thinkbox to modify their AMI to avoid this in the future (I have opened a support ticket with them about this issue).
We would like to find out how to have both DPI-SSH enabled and communicate with the OpenSSH_7.4 implementation to be fully security compliant.
If any of this rings a bell, please let us know!
Answers
I should have RTFM first, that Newbie badge I got for my first post is quite deserved :)
"To effectively inspect an encrypted message, such as SSH, the payload must be decrypted first. DPI-SSH works as a man-in-the-middle (MITM) or a packet proxy. Any preset end-to-end communication is broken, and pre-shared keys cannot be used."
"Putty uses GSSAPI. This option is for SSH2 only, which provides stronger encrypted authentication. It stores a local token or secret in the local client and server for the first time communication. It exchanges messages and operations before DPI-SSH starts, however, so DPI-SSH has no knowledge about what was exchanged before, including he GSSAPI token. DPI-SSH fails with the GSSAPI option enabled."