6.5.4.7-83n - any stability issues known?
BWC
Cybersecurity Overlord ✭✭✭
Hi guys,
like @Larry pointed out in another thread, being on the bleeding edge can be messy, but did anybody else is experiencing stability issues with the latest 6.5.4.7-83n? On my TZ 400 at home the appliance is restarting by itself, hope not my wife and two daugthers pushing the box to the limits :)
TSR showed it's restarted by the watchdog.
10/03/2020 12:15:05.928 System Startup F:6.5.4.7-83n T:base S:6.2.3.5 R: 5.6.0.11 XXXXXXX p: 12431; b: 12400 10/03/2020 12:32:42.944 System Startup F:6.5.4.7-83n T:base S:6.2.3.5 R: 5.6.0.11 XXXXXXX p: 12431; b: 12400
Can it be related to this? Because this is when the appliance restarts? Found no further crash information in the TSR.
=======Misc : Flight Data Recorder======= start fdr dump UTC 10/03/2020 10:16:39.512 tZeroTouch: Connection broken - read error 0x36 returned -1 UTC 10/03/2020 10:34:08.528 tZeroTouch: Connection broken - read error 0x36 returned -1 end fdr dump
Appreciate any thoughts as always.
Happy German Unity Day (3rd of October) to all, even you're not german :)
--Michael@BWC
Category: Entry Level Firewalls
Tagged:
1
Comments
You can probably forget about this, because it seems to be related to this:
I deleted the scheduled task for the failing firmware update, but it's still executing, WTF? Can't find my delete action in the console log of CSC-MA for yesterday, gave it another shot, fingers crossed for tomorrow.
--Michael@BWC
Apparently, SW doesn't "pre-load" certain information into C-ATP:
Like the release notes for the 83n update...
(Can't make this stuff up!)
Hey @BWC @BW57 ,
It fixes some pretty nasty stuff that 79n introduced on NSA's around port shielding.
Best, Steph.
Hi @Halon5
ja I saw that in the Release Notes, gladly I did not do any port shielding on NSA, TZ only.
Have to check if DPI-SSL acts more reliable now.
No further problems so far but time will tell.
--Michael@BWC
Hey @BWC ,
We just put a SWS14-48FPOE switch into an NSA2650 customer with the intention of daisy chaining a 24FPOE.
As soon as we started the port shielding on the switch it trashed the NSA2650 requiring a complete reset.
Just saying... We are opening a case.
Steph.
Brand new TZ350 is boot looping with 6.5.4.7-83n ...watchdog crashes just going through the web interface on initial configuration from factory default... This is after an import from TZ100 failed to run without random reboots (10 watchdog and it hard locks into recovery mode as well which is fun)
Case is open but not sure if it's a DOA buggy unit or something with this firmware at this point just figured I'd start searching for issues.
Censored watchdog logs:
10/14/2020 01:03:48.624 System Startup F:6.5.4.4-44n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19300; b: 19300
10/13/2020 17:13:48.272 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 17:22:02.464 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 17:24:53.144Reboot due to task suspension10/13/2020 17:24:53.144Task Trace tIdpDownloadSignatures:
tIdpDownlo> 80814dd8 417ce7f0 105 STOP0 0 8439f330 ad0003 0
Stack trace of tIdpDownloadSignatures:
Up: 0 d, 0:3:21
10/13/2020 17:26:06.464 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 17:35:57.720Reboot due to task suspension10/13/2020 17:35:57.720Task Trace tsfTaskDB:
tsfTaskDB 8068d160 342c6500 51 STOP+I 0 8439f320 ad0003 0
Stack trace of tsfTaskDB:
Up: 0 d, 0:10:21
10/13/2020 17:37:12.272 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 17:47:59.112Reboot due to task suspension10/13/2020 17:47:59.112Task Trace tAsyncLogProc:
tAsyncLogP> 80b54128 2cb9a5e8 65 STOP 0 843a0490 ad0003 0
Stack trace of tAsyncLogProc:
Up: 0 d, 0:11:18
10/13/2020 17:49:13.272 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R:`5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 18:12:12.176 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 18:14:07.256Reboot due to task suspension10/13/2020 18:14:07.256Task Trace tIdpDownloadSignatures:
tIdpDownlo> 80814dd8 417855a0 105 STOP 33b29400 8439f350 ad0003 0
Stack trace of tIdpDownloadSignatures:
Up: 0 d, 0:2:26
10/13/2020 18:15:22.176 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 18:17:44.512Reboot due to task suspension10/13/2020 18:17:44.512Task Trace tIdpDownloadSignatures:
tIdpDownlo> 80814dd8 417c6d90 105 STOP 6000200 843a0070 ad0003 0
Stack trace of tIdpDownloadSignatures:
Up: 0 d, 0:2:54
10/13/2020 18:18:58.176 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 18:21:49.784Reboot due to task suspension10/13/2020 18:21:49.784Task Trace tsfGenTask:
tsfGenTask 806303e8 2aaf6010 60 STOP 0 8439f320 ad0003 0
Stack trace of tsfGenTask:
Up: 0 d, 0:3:22
10/13/2020 18:23:04.176 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/13/2020 18:27:23.352 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/14/2020 12:30:24.352 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
--FACTORY RESET CLEAN CONFIG JUST WAN AND LAN AND DHCP NOTHING ELSE--
10/15/2020 02:59:54.336 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/15/2020 11:39:45.080 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300
10/15/2020 11:51:55.112Reboot due to task suspension10/15/2020 11:51:55.112Task Trace tWebMain05:
tWebMain05 80fabba0 2fa1ddc0 50 STOP 0 8439f2f0 ad0003 0
Stack trace of tWebMain05:
Up: 0 d, 0:12:45
10/15/2020 11:53:07.880 System Startup F:6.5.4.7-83n T:base S:6.2.3.12 R: 5.6.2.2 000000000000 p: 19331; b: 19300 #Blade_0_HYPERVISOR_END
Hi @xpair - Please give us the case number so we can make sure it gets the proper attention.
More importantly, why haven't we received a security notification from SonicWall regarding what update this FIXES?
@BWC New to Sonicwall but I can confrim that the two NSA 3650 Units we have in HA which were recently were updated to 6.5.4.7-83n were very unstable and rebooting every 2-3 minutes.
Had to roll them back to 6.5.4.6-79n and they are stable.
Will wait till this issue is resolved before trying to update again.
Any more good or bad experiences with 6.5.4.7-83n? I'd like to update quickly due to the recent vulnerability, but wouldn't want to go through rolling back if they aren't stable. Perhaps I'll just update my home unit first and see how it goes.
Hi @SonicAdmin80
I took the same approach, TZ 400 at home is running OK so far, updated only customer appliances which are actively using SSL-VPN. But no HA deployments, @B4zza raised some concerns.
--Michael@BWC
@BWC Yea, I'll try it on my TZ350 first as well. Almost all customers are using SSL-VPN so I'd like to start updating them, but will probably wait for the next scheduled downtime.
I have one TZ600 HA pair which is still running 6.2.7.5-36n because many of the newer ones have had known HA issues. But I guess I have to bite the bullet soon as looks like there isn't going to be an update for that firmware line.
Hey @BWC ,
We have it on a couple of NSA3600's. They are pretty vanilla through... One has some SonicWAVE's on it.
Will be starting to deploy on some TZ300's and TZ400's in the next day or two.
Best, S.
I plan to update our NSA 2600 and 3600 Cluster with the firmware.
But dont want to exchange security for instability.
Anyone having issues with the latest firmware? I dont wanna end in reboot loops or reboot in the wild
Hey @BWC ,
Rolled up to 83n across our TZ300 / TZ300W / TZ400 / TZ400W compliment of around ~30x overnight via GMS 9.2.
No calls yet ... :) So seems to be working I guess.
Steph.
Hi Guys,
I have to dig out this old thread again, but it's still current considering the fact that there is no new firmware.
I'am struggling with one deployment of a TZ 400 at the moment, that the appliance is crashing occasionally, resulting in being completely unresponsive. It cannot be accessed from WAN or LAN, even a ping did not get answered.
It might be load related, because it only happened when a few Zoom meetings, IMAP syncs and some uploads took place. We only have two VDSL lines attached via PPPoE, Bandwidth 50/10 Mbps for around 15 Users.
I already do a permanent logging via serial console, but cannot see any crash report. The TSR does not contain any information either.
Is anyone else experiencing something like this and was in touch with Support about a new firmware which addresses this form of hangs?
--Michael@BWC
I ran into an issue managing Gen6 devices (on 6.5.4.7-83n) with NSM. Just browsing the config in NSM would cause the devices to reboot. Support gave me a hotfix based on the same FW version. Worked on one TZ300, didn't work on another.
I have dozen or so firewalls on this version, not NSM managed, that are stable. None in a ISP situation like yours though.
@BWC - I would gather the standard data set and open a case. the Log Monitor exports are really important.
Please generate:
1) a tech support report from the SonicWALL's System > Diagnostic screen, including all checkboxes except Sensitive Keys,
2) an exported settings file from SonicWALL's System > Settings screen,
3) tracelogs (Last, All + Current) from the SonicWALL's Internal Settings screen (via diag.html)
4) an exported TXT and CSV log from the Log > Log Monitor screen
5) any console log data
Zip them up, then attach to the case itself.
Hi @John_Lasersohn
totally agree on that and I probably end up in a supoort call anyways.
It's a tricky one to catch for sure and these will be my next steps:
I'll update here if and how this issue got resolved.
--Michael@BWC
Hey @BWC ,
Please accept my apologies up front. :)
Have you tried to completely rebuild the firewall? (Manually)
Best, Steph.
Hi @Halon5
rebuilding will be last resort, because I have the feeling that my Attorney (aka Customer in that case) don't like the idea to pay for that.
--Michael@BWC
@BWC , Just we had an NSA2600 with that problem.. went thorugh all the calls, console cables, notebooks connected.. (you know the drill) , even after an RMA it happened again - but it wasn't every other day as it had been as I recall.
Then we upgraded the client to NSA2650 and rebuilt the firewall.. it didn't come back.
just saying.. it might actually be easier if you can... ( i hate the idea of having to do that all the same )..