Browser Challenge in Bot Manager has infinite fast refresh loop and fills logs with each refresh: edgio support says this is normal behavior, I think it is a bug

Hello,

I’ve noticed several issues with the Bot Manager. One such problem arises when a user agent is spoofed and detected as spoofed bot(a tactic I’ve seen normal users employ to try to bypass paywalls). In Edgio v7, if the Browser Challenge is used, such spoofing can trigger an infinite loop of rapid refreshes for the client. Additionally, during this endless refresh loop, the Edgio logs are inundated with each request. The requests and log events are recorded about 15 times per second in my tests. I believe this is a bug, although Edgio support claims otherwise.

I’ve attached a screenshot for reference. The logs are overwhelmed with each request. This situation is problematic for multiple reasons. The combination of the browser endlessly refreshing at a high speed and the consequent flooding of Edgio logs seems indicative of a bug. Notably, the refreshes and log entries occur about 15 times per second.

I hope someone from Edgio takes notice of this and reconsiders whether such behavior is indeed “normal.”

Here is a screen video capture of the issue occurring. You can observe the event ID changing rapidly due to refreshes like crazy, with each even id being logged:

edgio_browser_challenge

Here is logs showing 15 events logged per second because of this infinite rapid refresh behavior:

These rapid refreshes and log events will go on infinitely until the browser is closed.

Thank you.

I don’t feel this is normal either. However, I was not able to reproduce myself by spoofing my user agent. I did file a ticket for this and would like to add steps to reproduce if you could provide them. Thank you.

Reproduction steps:
Setup “Security” → “Bot Manager” with all “Known Bots” as all.
Setup “Security” → “Bot Manager” with all “Spoofed Bots” “Rule Action” as “Block”

For testing purposes installed

extension into chrome

Add the following user agent into the extension and select it
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

Visit related edg.io v7 domain with the above settings.

I was able to reproduce it using that UA. What I am gathering from the dev team is this is expected behavior when spoofing a bot UA in a browser that will be executing JS. The true bot will likely not be executing the code causing the refresh. And even if it were, the log is being hit with entries is beneficial to know the excessive amount of requests that would otherwise be direct traffic to the CDN.

@tristan.lee

I have seen real users do this to bypass paywalls(with Google bot user agent, but edgio v7 didn’t detect Google bot properly in the past so the test was with bing), so this is a valid false positive behavior.

Even if refresh and logging should occur, I would argue it should not occur 15 times per second.

Regarding “know the excessive amount of requests that would otherwise be direct traffic to the CDN”, it would be one normally, this causes it to be 15 times per second.

@tristan.lee

Just an additional note.
The reason for 15 times per second refreshes and log entries being made at that fast speed is only due to edgio block page making the user refresh so fast. The user or bad bot will be automatically refresh as fast as possible, even if they didn’t have any malicious intent this is basically making each client DoS the server.

If edgio accidently makes a mistake and detects a large number of users accidently incorrectly or there is a large amount of browsers with problematic user agent visiting a site, the current edgio v7 implementation is basically guaranteeing that the requests will be amplified and be similar to a DDoS attack on edgio infrastruction. As edgio is actually logging every single request, this will potentially cause a massive amount of load on edgio infrastruction.

Thank you for pointing that out. I have added this additional detail to the ticket.