Reduced Throughput with DPI-SSL
Jason_Faiferlick
Newbie ✭
I am noticing a significant drop in throughput on the TZ series when DPI-SSL is turned on, and this seems to only happen recently. For example, on a TZ 400 with a 100 Mbps WAN, speed tests show about 50 Mbps. If I turn it off, it jumps to 90+Mbps.
The TZ 400 is rated for 180 Mbps for DPI-SSL throughput. All other security services are turned on.
Wondering if others have noticed this, or if this is an issue with the 6.5.4 firmware.
Category: Entry Level Firewalls
0
Comments
Hey @Jason_Faiferlick
Let me do a little bit of research here and see if anyone else has seen anything similar.
Hi @Jason_Faiferlick,
the TZ 400 is a 4 core appliance. I'am not 100% certain that with the latest SonicOS it is still the case, but the mentioned performance values on the datasheet showing the overall-performance. A single download (flow) is bound to one core, which means that the maximum speed to be expected is datasheet-value divided by 4 for a TZ 400.
You should be able to verify this by starting multiple simultaneous downloads and see what the combined throughput is.
--Michael@BWC
Hey @Jason_Faiferlick
I just had a support case where I was told that there is a problem within the memory and certificate management on version 6.5.4.x which have impact on things like DPI-SSL.
Me by myself had a problem with the throughput when GAV is active. After upgrading to 6.5.4.4 the issue seems to be fixed. So if you're not running 6.5.4.4 you may try this.
Hi @Jason_Faiferlick
SonicWall DPI-SSL numbers on the datasheet are with DPI-SSL and ONLY IPS enabled. Gateway Anti-Virus is a big hit to throughput so once you enable that with DPI-SSL your throughput speeds are cut down by about 30-40% depending on the traffic.
SonicWall did annotate this on one of their datasheets but I have not seen it on all of the datasheets that they have published.
I have done my own personal throughput testing with all measure of features enabled and disabled and there are differences between having certain features on or off.
You can see a difference in speeds by just disabling GAV under DPI-SSL, rebooting and then do a speedtest and you can see a difference.
Thank you
Ben Davis
Western NRG
Hello@Jason_Faiferlick ,
Thanks for raising this question. There can be multiple reasons based on different scenarios for this issue to be occurring. We will have to understand the following:
This issue can be Software based/ Config based etc. We can expect the decrease in download with DPI SSL as we need to decrypt the data and encrypt again and during this process, there is write and read to OpenSSL. So we can say a total of 4 copies with decrypt+encrypt.
Issues similar to this have been reported to backend.
Issues Id's: 219430 and 221940
Knowledge Management Senior Analyst at SonicWall.
Hi all,
I can 2nd what @WNRG_BenDavis mentioned, having DPI-SSL and GAV enabled drops down the speed big times, in my case from 95 to around 60 Mbit in downstream. Upstream was even worse, from 40 Mbit to around 5 Mbit.
But it seems that uploads are causing more trouble, because with Security Services disabled and just DPI-SSL enabled, downloads are using CPU Cores 2-4 for around 40-70%, but uploads causing 100% load on CPU Cores 2-4 still stuck at around 5 Mbit.
My VDSL line is synced (and able to deliver) with 100/40 Mbit.
It seems that my old knowledge is no longer current and the load is spread over all CPU cores (except Core 0).
All tests done on a TZ 400 with SonicOS 6.5.4.5.
--Michael@BWC
Just interested to know if you see handshake errors under DPI-SSL "Show Connection Failures".
There are multiple issues with the 6.5.4.4-5 firmware by the way. Especially around DPI-SSL. You may be able to get a hotfix from support.
@Halon5 For this I would advise actually moving up to 6.5.4.5-53n--HF222651 to fix more of the DPI-SSL issues. The 6.5.4.5 release has most of the memory fixes and DPI-SSL lock up issues resolved in that version. I am suggesting the HF222651 as that is the fix for the DHCP deletion issue when changing management settings on an interface.
Thank you
Ben Davis
Western NRG
Hi all,
I thought I give 6.5.4.6 Beta a try and enable DPI-SSL again on my TZ 400 at home, but it's not useable. The performance is going down dramatically and standard websites are not coming up, or need at least multiple refreshes in the browser to show correctly.
I'am really asking myself, who really is using this important feature in production? If I would need to count my customers having DPI-SSL enabled, I wouldn't need count that much.
--Michael@BWC
Hello @BWC,
The performance issue observed on SonicWall firewalls with DPI-SSL enabled is going on for a long time now. But the good part is, the problem was bug reported and the respective team has worked on the fix. I did a bit of research and happy to share my findings here.
As per the bug report, we have certain patched firmware available that can take care of the DPI-SSL performance issues. The issue is noticed especially when CFS is enabled under DPI-SSL in the SonicWall firewall. According to the bug report, firmware version 6.5.4.6-73n contains the fix for this issue. I presume the beta firmware that you are using might be on version 6.5.4.6-70n. If yes, then you can continue validating the DPI-SSL performance issue on the firmware 6.5.4.6-73n version once its is available online. In case if you are looking for an immediate fix, I suggest to get the hotfix (patched firmware fix) based on the firewall running firmware version and apply.
I hope this clarifies.
Please update here for more questions and further clarifications.
Have a good day!!!
Regards
Saravanan V
Technical Support Advisor - Premier Services
Professional Services
Hi @Saravanan1990_V
yes I'am running the publicly available -70n release, turning of CFS indeed makes a difference, at least that's my first impression.
I'll keep an eye open to get -73n or higher when available, disabling CFS isn't a real option.
Thanks for sharing and best regards.
--Michael@BWC
Hi @BWC,
Sounds good.
Thanks for your confirmation.
Have a good one!!!
Regards
Saravanan V
Technical Support Advisor - Premier Services
Professional Services
Hi @BWC , @Saravanan1990_V , @WNRG_BenDavis,
I concur with your findings and we have been spending some extensive time with support to try and resolve issues on 6.5.4.5 with numerous hotfixes.
Best idea is stick to 6.5.3 in production IMHO. To late for us. We have turned of CF in DPI-SSL across our customer base since last year and no end in site yet. :(
Hadn't struck the DHCP deletion issue... but now I know about it.
We are having issues around TCP packets out of order when CleanVPN on our work zone back via our DC with DPI-SSL turned ON. Intercepted websites fail to load. Its finally going to development finally by the sounds.
We wont run beta in production but will follow this thread. (I have tried the 6.5.4.6 but am also hearing "everywhere" that there is a lot of effort going into this to resolve performance issues). Hope they get it right.
I did some further testing and what I mentioned a couple posts ago is still valid, max. download speed drops from 95 to around 60 Mbps, CFS disabled, GAV enabled. Upload is a nightmare with DPI-SSL enabled, drops from 40 to around 5 Mbps.
Will wait for the next release.
--Michael@BWC
Firmware 6.5.4.6-79n just released :)
@Matthiman Yes we released firmware 6.5.4.6-79n after intensive tests - this firmware addresses a lot of DPI-SSL slowness issues as well as introducing many memory fixes and much more!
You may want to take a look at our release notes for fixed issues and known issues: http://software.sonicwall.com/Firmware/Documentation/232-005208-00_RevA_SonicOS_6.5.4.6_ReleaseNotes.pdf
Yes I've just read release notes, this sounds great but I hope there is no unknown bug, I'll deploy it this week-end
The 6.5.4.5-79n seems to work better than the Beta -70n release, my first speedtests showing the following results:
TZ 400, VDSL 100/40 Mbps
with DPI-SSL enabled: 60/2-22 Mbps, 100% CPU Core 2-4 usage on download, 10-20% on upload
with DPI-SSL disabled: 92/36 Mbps, 10-30% CPU Core 2-4 usage on download and upload
It's all single flow tests, but I guess at 100% cpu usage I cannot squeeze more out of the box with multiple flows.
Datasheet says 180 Mbps, yeah, I don't think so.
Hopefully it will be more stable and not slow down like before, time will tell.
--Michael@BWC
The datasheet reports 180mbps but the test is conducted in a isolated environment using IMIX and multiflow, in a real scenario with other traffic running you're looking at lower values.
Also if the max connection (with DPISSL disabled) is 92Mbps, we do expect a drop in performance (due to the decryption/re-encryption) of 30% to 50% of the max speed so in your case you're looking at a 35% drop which is the average.
Based on our test, 6.5.4.6-79n has improved both stability and performance so keep an eye and let us know!
Hi @fmadia
will do further testing on other appliances as well. Rule of thumb in the past for me was, Datasheet values divided by the number of cores the appliance is having. In my case I don't think that multiple flows will squeeze more out of the already 100% occupied cores 2-4 (mot just one) when doing the speedtest.
Will keep this updated when tested on more potent appliances.
--Michael@BWC
Do let me know the outcome of your tests, I can help if needed.
I agree that the values on the datasheet are not reachable in a production environment and it's also linked to the number of cores. A TZ400 will easily reach 100% when too much traffic goes through the DPI-SSL so even a multiflow test won't help, a bigger box should give more accurate results!
For any question feel free to reach out!
Francesco
Hey Fellas,
Our speeds testing is in line with @BWC , the claimed speed is nowhere near what the datasheet's report.
We also sell circuits here and many of our customers are running a fibre 1Gb/500Mb connection. Not too many VDSL's left expect for redundancy. We are finding that the TZ300's are simply no longer viable. The newer firmware has just trashed them performance wise.
We still have serious problems with downstream SonicWALL's and DPI-SSL via Clean VPN Tunnel ALL. We have too turn off DPI-SSL on the downstream box since our upstream box at our DC also runs DPI-SSL .
-Steph.
Thanks for providing your feedback. Are you using the latest 6.5.4.6-79n or 6.5.4.5-53n?
About TZ300, the max throughput after DPI-SSL is 50Mbps (IMIX in isolated environment). As we keep adding more security features and checks, some small devices may also become slower over time but this is to guarantee the maximum layers of protection.
Only bigger TZs or NSAs will be able to achieve speeds beyond 100Mbps unfortunately.
Regarding the issue with the DPI-SSL in a VPN Tunnel All: are you saying that you have 2 SonicWalls (one downstream and one upstream) and you'd like to enable DPI-SSL on both of them?
If you share more details we may be able to help here.
Hi @FMADIA,
I was just reading the release notes again. Looks like engineering have created a DTS.
As far as small units go...
The trouble we have is the that many of the sites we manage we clean vpn / tunnel all back to a Head Office (common scenario).
The cost around the larger units / per head doesn't work. Many retail stores we manage only have one or two people in them. Performance is still expected.
e.g. We have one customer, a diamond merchant, they are always complaining about speed. Just a man and wife team. They actually have a small server rack and run some virtualized machines for their website and an old MS-SBS server. It has a TZ300 which once upon a time worked reasonably. - Hopefully this latest update will help.
Perhaps Sonicwall should reintroduce the 10 / 25 / Open licensing model of old. At least you could then buy a larger TZ but contain the ongoing licensing costs?
Some thoughts at least.
-Steph.
The issue is in the Known Issues in the releases notes now for 6.5.4.6-79n
Yes the issue is exactly the one described in the release notes and engineering is currently working to fix that. I may be able to take a look at Gen6-1160 and check the current status next week.
Regarding the licensing option, I'm not sure I can help, I believe our Sales dept can help there to find the best option.
I'd suggest you updating your firewall to the latest release but do follow also this KB article to optimize the configuration of the device: https://www.sonicwall.com/support/knowledge-base/troubleshooting-network-throughput-latency-and-bandwidth-issues-with-a-sonicwall-utm/170504563958424/
If the issue persist, you may want to open a Support ticket so we can help further.
I did some more digging. Having the Security Services set to "Performance Optimized" will boost the download speed from around 60 to 92 Mbps and CPU consumption on Core 2-4 from freaking 100 % down to around 60-70 %. But the upload is still bad, from the possible 35 Mbps only around 20-22 Mbps are left with DPI-SSL enabled.
Why is there such an impact on upload speeds, CPU consumption on Core 2-4 is 60-70 %, nearly the same as for the download right now. I don't wanna lose nearly 50% of my upload speed.
--Michael@BWC
Hey @BWC , @fmadia ,
Yep. We turned ON "Performance Optimized" only on our TZ300 / W's a very long time ago. There was a point where the newer firmware trashed them (I cant quite remember exactly when offhand). it helps somewhat.
Have you tried turning off DPI completely and doing a test? The results are quite surprising!
SET TO DPI DISABLED
Above an option for those that don't want to replace hardware.. go down the Capture Client route and drop CGSS/AGSS (DPI-SSL) on the unit.
What do you think @BWC ? (I really love your work by the way).
---Steph.
Hi @Halon5
disabling DPI seems to be the last resort, but I'am under the same impression that perimeter security (or should I say inspection) is getting more and more less important or losing its grip, because most threats can be handled only on the endpoint anyways.
I actually don't like that route, but I guess starting with certificate-pinning, TLSv1.3 etc. man-in-the-middle is getting more difficult and maybe impossible in the future. And without DPI-SSL on the firewall, what will be left to inspect?
Hopefully all the effort SNWL is putting into the Unified Policy Engine will be worth it, time will tell.
--Michael@BWC