SMA GeoIP - not only for remote access
BWC
Cybersecurity Overlord ✭✭✭
Hi,
while investigating some ongoing issues on the SMA (500v) it seems it might be related to a suspicion I had in the past about the usage of GeoIP blocking. My GeoIP Blocking Status went from Active to Offline today which raised some concerns. It was back to Active right after reboot, accessing to smabgdata.global.sonicwall.com and geoipdata.global.sonicwall.com was always possible.
Along with most of the other Countries, I usually block the United States of America via GeoIP because I don't expect any remote access from it. But it seems that GeoIP is blocked on iptables level and not just mod_geoip for restricting access to the underlying httpd.
This does not have to be problem, but it seems it interferes with GeoIP, Botnet or License updates.
I would think that GeoIP blocking makes only sense on the iptables INPUT chain for new connections initiated from the Internet, but it may affect related packets on the FORWARD chain as well, which is a show stopper.
Let's check what the syslog shows:
Jan 30 11:15:09 xx.xx.xx.xx kernel: DROP_BY_IPTABLES c=1003 IN=eth0 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx SRC=204.212.170.212 DST=xx.xx.xx.xx LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=443 DPT=54990 WINDOW=8192 RES=0x00 ACK URGP=0 time="2021-01-30 11:15:09" vp_time="2021-01-30 10:15:09 UTC"
The log on the SMA is giving me mixed signals about Allowing/Blocking connections.
It might be a surprise to some people, but blocking connections from the USofA is a legit measure of risk reduction.
It might be related with this as well:
Is this already addressed in some form? As a countercheck I'll (against my better knowledge) allow the USofA via GeoIP.
--Michael@BWC
Comments
Hi,
well the countercheck by removing the United States of America from GeoIP blocklist did no make any difference.
After around 9 hours of runtime the Protection Status switch from Active (online) to Active (Offline mode), it was around the same time local logging to the Appliance stopped working.
The syslog still shows every hour "Geo IP Regions Database is up-to-date" but Last Check stuck at Jan 31st 20:05:18, local logging stopped at 20:35.
Editing the GeoIP Policy (adding US again) results in an Error Message: "Error: can't make new policy effective".
This only started after setting the Appliance to factory settings and created from scratch. No errors on the VMware console though, so I guess the VM is good.
--Michael@BWC
Hi,
which Firmware version are you using?
We are also using GeoIP Filter and blocking some counties including the US but it is a SMA200.
GeoIP-Blokcing is working without any issues. We are on Firmware 10.2.0.3-24sv
Hi @Xronos
I'am running 10.2.0.3 as well and before the Factory Reset I did not experienced this odd behavior.
I'll have to grab a TSR when the problem occurs again.
I downloaded a TSR after reboot and log files showing some weird timestamp with date of tomorrow before jumping back to today, like in temp.db.log
[Tue Feb 2 02:40:25 2021] phonehome 1388: dbhGetInt: Can't fetch value: unknown error sql:SELECT value FROM Options WHERE key = 'windows'
Will update after next event.
--Michael@BWC
Hi all,
the reason seems not to be related to GeoIP blocking it all.
The geoBotD.log in the TSR reveals that the Disk storage gets filled up.
Mon Feb 1 17:32:18 2021 Error Message: Geo log receiver: failed to write log message, reason : No space left on device,
It's 20 GB Disk assigned to the SMA, which is the default for the OVA deployment.
Any clue what is going on? Because of the lack of shell access I cannot check what's eating up the space.
If this is not fixable the one and only solution seems to be deploying a new instance and importing the settings, which is annoying but not a big deal.
--Michael@BWC
Hi all,
in case someone faces the same problem, I ended up in re-deploying the SMA because I wasn't able to figure out what caused the lack of free disk space. Hopefully this resolves it for good.
BTW, I was generous and gave the SMA a whopping 48 GB of disk space, but it seems it's hard wired to just use 20 GB out of it.
--Michael@BWC
Hi all,
in my ongoing effort to track down weird stuff I can say with somewhat confidence that GeoIP is messing things up when US gets blocked. While examining the iptables ruleset on the SMA, all incoming packets from SRC addresses listed in the ipset table
denyIpsetwill be dropped.Some of the members on that table are unfortunately Addresses from SNWL:
This Blockage will prevent all kind of reply-packets for License-Validation, GeoIP DB Updates, they will be dropped. This cause silently all kind of licensing issues.
The solution is probably pretty simple. I assume that all kind of license checks, updates and phonehome etc. are initiated on the SMA and therefore outbound (OUTPUT chain). The reply packets are recieved on the INPUT chain.
Just add one of the following and we should be good to go, IMHO, both commands got accepted and added to the rule set:
Hopefully some PM is reading this, because tackling this with support wouldn't be fun.
--Michael@BWC
Hi all,
because @Micah or @Chris did not replied to my request I did some further digging in 10.2.0.6.
I somewhat oversaw the ipset defalutAllowIpset (love the TYPO :) ) and a bunch of SNWL related IP addresses are allowed for ANY incoming connection (INPUT chain). I find this a bit intrusive, because there is no need for SNWL to access the SMA from the outside, but who am I to judge. Gladly sshd is not started per default, which would make the unknown root password look a bit backdoorian, does not count for local console access though.
The list holds the local configured DNS resolvers and couple of addresses on Amazon AWS etc, but also these:
Are these entries newly added in 10.2.0.6 because this would be an explaination why the 204.212.170.21 got blocked above?
204.212.170.144 is the lm2.sonicwall.com, but KB article mentions that 204.212.170.143 (licensemanger.sonicwall.com) should be available as well, which is not part of the defalutAllowIpset (sorry, had to type it again, the TYPO though 😂). Neither is wsdl.mysonicwall.com 204.212.170.212.
My suggestion with the permit of related/established connections still seems to be the better option, -A INPUT should be replaced with -I INPUT 1 for that matter. 😎
Love to hear from you.
Best from Würzburg.
--Michael@BWC
Hi all,
is really noone having these issues? This really makes me doubt myself. I opened Ticket #43674616 to get the bottom of this anyways.
While doing some reasearch on the SMA it can be easily verified. Having USA blocked via GeoIP Filter immediately puts any host on the related ipset list denyIpset, when a packet is entering the SMA, even reply packets (License Information Request, etc.).
I tested this with a ping from the SMA:
This simple command could resolve the whole dilemma and probably reduce some load on the ipfilter at the same time:
--Michael@BWC
@BWC You have a good point Michael. I agree that GeoIP blocking the US should not render the SMA unusable.
At a minimum the system should white list the necessary back end sources that are required to keep the SMA 500v operational.
I've asked Imnan to open an engineering ticket to get the engineering team to resolve this problem.
Hi @Simon thanks for speeding this up, I provided Imnan the requested TSRs already, added one from my "modified" SMA as well.
The ipset in question looks like this at the moment, which is unfortunate, because it holds licensemanager.sonicwall.com :)
I would definitely go for the established/related approach, because whitelisting is way to static, IMHO.
--Michael@BWC
Hi,
just to keep this alive, a current Support Ticket suggested to whitelist 204.212.170.143 in the ipset and I've got a private build for that. But 10.2.1.0 puts another IP in the mix. I don't rooted the 10.2.1.0 put I'am quite sure that it ended on denyIpset as well.
postDeviceStatistics failed: LicenseManager failed to connect host: soniclicense.global.sonicwall.com(204.212.170.68:443)
It's so frustrating and it seems that Engineering is not aware of a Stateful Packet Filter with Connection Tracking or they just don't trust the 9-10 year old Linux Kernel 😡
--Michael@BWC
Hi,
well, another 6 months gone without any progress, 10.2.1.3 (which got pulled) is still struggling when US gets blocked via GeoIP. I provided a solution, but noone care. I can't understand why anyone in their right mind believes that filling a static ipset list can be a viable solution.
--Michael@BWC