100% Data Plane CPU load on TZ470 's in the last days
SimonT
Newbie ✭
Hi,
since 2 days we've noticed some strange behaviours of TZ470 with firmware 7.0.1.5145 .
They get nearly blocked by a high CPU load on the data plane, even when not a lot of traffic is passing the sonicwall. (20Mbits / 93 active connections)
A reboot fixes the issue.
Happened already on 3 devices of the same model/firmware. On different customer installations, with nothing in common.
Called Sonicwall Support today, but until now, there is no known issue and they cannot help without TSR done over SSH/Console.
Is anyone else experiencing similar problems?
Regards
Simon
Category: Entry Level Firewalls
0
Comments
Is there a reason why you can't get the TSR with SSH? What's the busiest process [support would surely have checked this anyway]?
The problem is, that the device is very unresponsive when this happens. Enabling SSH, connection etc. can be challenging there.
All firewalls are on remote locations.
In the last cases, businesses where completely blocked and my colleague needed a fix asap.
But maybe next time when it happens on an other FW, we have a bit more time and can get out some more informations.
IME SSH is far more responsive under load than the web interface ever is. There is no problem leaving SSH enabled permanently if you have the proper ACLs in place, same as with HTTPS.
I have experienced it on my own TZ 370, as well as on a couple of our customers' TZ 270 the only "solution" was as you mentioned yourself to restart the firewall, I will waiting for new firmware to be released.
I do have a backup of their config, before upgrade but some of them are on remote site which i cant access.
https://www.sonicwall.com/support/knowledge-base/how-do-i-safely-perform-a-firewall-firmware-downgrade/190506060217755/
Hey @MartinMP , for what firmware do you wait? 7.1.1. is already released, but since that has some completely new features, even the support did not suggest an update as first solution.
We have now also TZ270 afected.. But very random. Until now 4.
Do you have SNMP enabled on them? (that's what support asked me)
Regards
Simon
ok wait do i understand this correct:
the tz series is having serious issues with cpu load (TZx70) with firmware 7.0.x which occurs randomly? And there is no other solution as rebooting the firmware?
for the latest firmaware 7.1.x the sonicwall support says: we dont recommend to install it on TZx70 series?
If this is the case: What do you do as workaround/solution for your infrastructure?
Only the firmware 7.0.1-5145 of the TZx70 is affected. After the reboot of the device it didn't occur again on the same device..
So it's not a magior issue, but it is anoying. For now there there affected TZ270,TZ370,TZ470.
The upgrade to 7.1.1. was not suggested as first step of resolution, since that provides new functionalities what by nature could lead to other problems.
At the end it will be a bug in a special constellation, that we seem to have more often in our installations, and others not. Since no one else raised a ticket at sonicwall for that as it seems..
I have seen this on a TZ470 twice in the last 2 months with firmware 7.1.1-7040.
Contacted support and they just gave me a generic article about data plane mgmt cpu at 100%.
This issue has been mentioned on reddit multiple times.
Mine was with the process tRemoteBackupd process.
From a reddit user these 2 changes were a possible fix:
1. On this page /sonicui/7/m/mgmt/settings/diag turn off the option "Collect Top Memory Usage Report" ( REQUIRES A REBOOT )
2. On this page /sonicui/7/m/mgmt/diagnostics/report turn off the option "Periodic secure diagnostic reporting for support purposes"
I made these changes just over a week ago and it hasn't happened since, but it could take a month for it to happen again since that is the amount of time between logins for me last time.
Maybe Sonicwall support team aren't allowed access to Reddit :D
Hi Jack,
devices with the 7.1.1-740 have that known bug that you mentioned. We had that also. But in that case, the Management Plane is on 100%..
In my case, it is a 7.0.1-5145 Firmware and the Data Plane is on 100%. So it is a different issue. Until now, a reboot fixed the issue and it didn't come back on the same device.
does anyone know where to find this in the gui??
1. On this page /sonicui/7/m/mgmt/settings/diag turn off the option "Collect Top Memory Usage Report" ( REQUIRES A REBOOT )
2. On this page /sonicui/7/m/mgmt/diagnostics/report turn off the option "Periodic secure diagnostic reporting for support purposes"
thanks
T
never mind... i am a little slow today ;)
T
I've also have this on at least 5 firewalls, all TZ270 and TZ370 models, all running SonicOS 7.0.1.0-5145 R5175.
Gave me a 200 mile round trip yesterday, so following with interest
Sorry to hera That @NickEbstarLtd
But I think there is no hope that this ever gets resolved since the also closed our ticket, since we where not able to provide Logs of that blocked devices and after a reboot everything is gone.
Lucky for us, it did not happen again after these 4 devices..
Regards
Simon
I am seeing this on TZ 270 with firmware 7.1.1-7047. "tremotebackupd" is sitting at 89% cpu. I ran a backup and it worked fine, but the process stayed at 89%.
And is anyone seeing this with 7.0.1-5151-R5624, which was released March 12, 2024?