Thursday, September 1, 2016

EWS: Throttling policies on bulk operations

Exchange Web Services (EWS) Managed API is a fantastic API that lets you do a whole lot of interesting things from C# code that allows the program to interact with Exchange. EWS has bulk operations, such as CreateItems, that let's a program send up a bundle of items from C# code to be created by the Exchange server.

The downside to this is that the Exchange server has quite a few throttling policies that it imposes on groups or individual accounts. This has the effect of squandering efforts to perform large bulk operations when it exceeds a particular throttling policy. This is because the Exchange throttling policies are like a cop pulling over speeding drivers, they don't let one driver run everyone off the road, rather a connection has to stay within the limits defined by the throttling policies. A grand overview of EWS throttling policies in Exchange 2013 server are outlined here.

This blog post attempts to outline what Exchange 2013 throttling policies might be run into using EWS to perform bulk Mail operation for creating mail items (i.e., just the CreateItems call  to create emails and not dealing with Calendar, Contact, or Meeting items) and how to appropriately change them. Reading up on EWS best practices is highly encouraged so that ramping down throttling policies is not used as an improvement for faulty code. Then, when its decided that its still necessary to fiddle with the throttling polices, definitely read up on the best practices for EWS throttling. The most important point to remember might be that Exchange throttling policies effect not only EWS, but also client connection to the server for other programs like Outlook and ActiveSync. If you're not sure whether Exchange is throttling your program you can check the EWS logs on the Client Access server (CAS) to investigate whether operations are being throttled.

This MSDN article is the most helpful to get started, as it outlines the specific throttling policies that effect EWS in Exchange 2013. The policies below are pulled from that list, with the italicized description from MSDN, followed by an explanation as to why they might come into play using the batch request CreateItems:

  • EwsCutoffBalance
    • MSDN - "Defines the resource consumption limits for EWS user before that user is completely blocked from performing operations on a specific component."
    • The CreateItems call consumes resources on the Client Access server (CAS). When this limit is exceed the server may return errors like "the operation has timed out". From researching EwsCutoffBalance policy further, it seems to be defined by the total number of milliseconds an EWS connection is consuming a specific resource. In other words, if an EWS operation takes longer than the EwsCutoffBalance (in ms), then the operation will time out. 
  • EwsMaxBurst
    • MSDN - "Defines the amount of time that an EWS user can consume an elevated amount of resources before being throttled. This is measured in milliseconds. This value is set separately for each component."
    • This throttling policy is a little bit easier to understand from its name and description. EwsMaxBurst is expressed in milliseconds. This should be less than EwsRechargeRate.
  • EwsRechargeRate
    • MSDN - "Defines the rate at which an EWS user's budget is recharged (budget grows by) during the budget time."
    • This throttling policy is also expressed in milliseconds. When a resource is being throttled by say the EwsMaxBurst or EwsCutoffBalance policy, it will start to recharge at the rate defined by this policy. Presumably, the throttling policy will be "recharged" in the time defined by EwsRechargeRate, but more research and testing needs to be done to be sure.
  • EwsMaxConcurrency
    • MSDN - "
    • Defines the number of concurrent open connections that a specific user can have against an Exchange server that is using EWS at one time. The default value for Exchange 2013 and Exchange Online is 27. 
    • This policy applies to all operations except for streaming notifications.
    • "
    • This policy is worth noting because there may be concurrent connections going on while one is making a call to the CreateItems bulk operation.

The first three policies above are all measured in milliseconds, and should be ordered in this way:
EwsMaxBurst < EwsRechargeRate < EwsCutoffBalance

This has been an intermediate look at what EWS throttling policies should be considered for changing when using the bulk operation CreateItems.