Microsoft throttling connections

SonicAdmin80

Are other people that deploy Email Security seeing problems when sending email to Exchange Online? Every few months Microsoft starts throttling our ES IP addresses. Now it has happened twice in one week. Email Security queue shows "status 4xx, retry later" for dozens of messages. After a few hours when the retry timeout passes, users get non-delivery notifications.

I've battled with Microsoft support for months about this issue. They sometimes seem to be able to do something that releases the throttling on those IPs and the messages get delivered. But it always comes back sooner or later. Most of the first level support agents don't even understand what the issue is and start asking if we have connectors created on the tenant, which is irrelevant because the problem is sending to other Exchange Online tenants regardless of what the sending email platform is.

Is anyone else seeing this and have you been able to get to a solution with Microsoft?

David W

Yes, we also see this occur on our own Hosted Email Security as well.

Microsoft has been aware of the issue occurring on and off for a few months now.

If you start to see issues here on down detector it can at least give you an idea of the time they started as well.

https://downdetector.com/status/office-365/?fb_comment_id=1606152319610006_1224160817672730

As far as getting any real solution I'm afraid that answer can only really come from Microsoft but that may be hard to get.

SonicAdmin80

Good to hear we are not the only one. Also looking at comments on downdetector it looks to be a wider issue and not just a few IPs so good to know it’s not our fault. Some reassurance and a place to point the finger when users ask about the delays and NDR’s.

Microsoft support hasn’t once admitted that it could be a wider issue on their end and only try generic, tenant specific fixes.

SonicAdmin80

Once again our multi-tenant Email Security was throttled by Microsoft EOP. This time the Microsoft support tech was able to tell me that I probably shouldn't route all tenants through so few IP addresses and that I should spread the load to more addresses to avoid being throttled.

Why I wasn't told this over a year ago I will never know, but now I at least have something to try to avoid the problem. Support wasn't able to give any actual limits on how much traffic is allowed per IP address. They suggested one IP address per tenant, which seems quite wasteful.

It also creates a slight problem because in Exchange Online it isn't possible to set a TCP port for the outbound connector. So I'd need a unique public IP address with port 25 open for the outbound connector, then NAT that to a unique private IP address on the inside network where Email Security is listening on just for that connector. Then route outgoing connections from that private IP address via a unique public IP.

So this requires much more public and private addresses and can get quite cumbersome to manage with many tenants. Perhaps outgoing many-to-many NAT rule would work, but then I wouldn't have one unique IP per tenant but a generic load balancing.

I could also avoid the whole problem by not routing outgoing email through Email Security at all, but I kind of like the flood prevention functions which have actually saved the day a few times when a user's account was hacked and used to send spam.