Aggregator rate limiting

Website name- www.customer.comSolution- Akamai DSA and WAF

The Problem• A method to limit the number of requests hitting the origin

servers emanating from a bunch of Aggregators (around 50 IPs)• These aggregators search and grab all of the content that is

needed to be summarized from the customer’s website (like a crawler)

• The aggregator traffic need to be blocked based on an absolute limit (say whenever it reaches 10000 total requests), something has to be done on Akamai to prevent further requests from hitting their origin

• We do not know when the traffic goes down the slope. Even in that case, Akamai has to intelligently start allowing traffic coming from those IPs

Proposed solution 1Sampling percentage logic in the delivery config - Setting 50% sampling rate for traffic from those Aggregator IPs. This will send 50% of the requests to their origin and send the rest to a Net storage location (failover page)

Challenge – This will be sending 50% traffic based on the number of requests at a given point of time. However, the customer wanted us to limit based on an absolute number (Eg: 10000 requests)For eg - there are 10 requests, 5 requests will be sent to Netstorage.

Proposed Solution 2• WAF rate limit capabilities - The minimum threshold we can define is ~120

requests over a 2 minute span or 5 requests in 5 seconds burst rate (calculated per edge server that an Aggregator / source IP would connect to)

• Challenge - Say the traffic spikes up during the 2 minute time window and WAF sets the variable. The next time WAF runs the 'rate limiting' logic would be after 10 more minutes which essentially means that for the next 10 minutes, the WAF custom variable will always be set regardless of the rate of traffic.

• Eg: A use case where the traffic spikes up during the rate limit computation and immediately slopes down will be a classic example of this challenge for the aforementioned scenario. We needed some methodology to give some leeway for the Aggregators so that they could ramp up by slowing down the traffic.

Proposed Solution 3• WAF rate limiting + Sampling percentage - To combine WAF limiting

capabilities and the sampling percentage logic in order to hit an optimal rate of traffic to be blocked/allowed at any point of time

• How? The WAF sets the rate limit variable and the request goes through a sample percentage logic (90-10) in the deliver config. In this way, at any given point of time in the 10 minute time-span we can always try to hit a value which is the optimal number of requests that will be sent to the origin. The percentage split can be changed according to how things pan out based on observing the rate of traffic after the traffic spikes up and hits the lower points in the graph

WAF rate limit policy

Delivery config match