Firewall system-generated traffic does not flow over NAT'ed VPNs
As far as I can tell, system-generated DNS lookups when using Proxy DNS and Split DNS, when the target for the Split DNS lookup is on the other side of an IPSec VPN tunnel, and the local subnet is being NAT'ed, does not follow the same NAT rules as the normal traffic. As such, I cannot get Proxy DNS with Split DNS working on any of my remote firewalls that are across a NAT'ed IPSec tunnel.
On any of my firewalls with regular, un-NAT'ed tunnels, the Proxy DNS queries the servers specified in Split DNS as expected. A DNS request from a client IP on the LAN hits the LAN IP of the SonicWall. The SonicWall sends its own DNS query, from its X0 LAN IP, to the DNS server on the other side of the VPN tunnel. Packet Monitor confirms this. ("Monitor Firewall Generated Packets" is enabled in Packet Monitor. Ingress interface shows "X0*(s)", Egress interface shows "--".)
However, on the firewalls where the VPN tunnel is being NAT'ed on the local subnet, then the SonicWall does not follow the same NAT rules as the other traffic. With "Monitor Firewall Generated Packets" DISABLED, Packet Monitor shows the request sent to the specified Split DNS destination server, from the X1 WAN IP of the firewall, without any NAT applied. (Ingress interface shows "--", and Egress interface shows "X1*(s)".) With the "Monitor Firewall Generated Packets" ENABLED, the firewall shows two packets sent to the same destination server. The 1st is from the X0 LAN IP of the firewall, un-NAT'ed. (Ingress interface shows "X0*(s)" and Egress interface shows "X1".) The 2nd is from the X1 (WAN) IP of the firewall. (Ingress interface shows "--" and Egress interface shows "X1*(s)".)
I find it especially strange to see that extra 2nd packet on the "NAT" firewalls, coming from my WAN IP address. It seems as though it wants to apply the default WAN masquerading NAT rule to the packets. It also seems to want to follow my default auto-generated WAN route policy, where traffic going to 0.0.0.0/0 goes out the WAN's gateway, instead of following whatever route gets dynamically created in the background when a tunnel is active. I don't have any special routing custom added to any firewalls myself that would bypass something.
I've tried using NAT directly in the VPN tunnel setup, and I've tried doing NAT via rules instead, and both cause the same issue. It should be noted though, that all regular traffic flows properly from clients behind the firewall.
This does not seem to be restricted to Proxy DNS -- ALL system-generated traffic from the firewall seems to do this as well. For example, I tried to generate pings from the GUI and Packet Monitor those, and they had the same result. However, my main need at this time is to get Proxy DNS working across the tunnels, so I'm willing to do a workaround for just that if needed. I considered trying to manually create a route, but as these are not currently "interface" tunnels, the SonicWall would not let me create a route myself. (No interface to point the route to.)
Answers
This is a known problem that firewall generated traffic does not get NAT'd over a site to site VPN tunnel. AFAIK there is no workaround.