I’ve noticed several issues with the Bot Manager. One such problem arises when a user agent is spoofed and detected as spoofed bot(a tactic I’ve seen normal users employ to try to bypass paywalls). In Edgio v7, if the Browser Challenge is used, such spoofing can trigger an infinite loop of rapid refreshes for the client. Additionally, during this endless refresh loop, the Edgio logs are inundated with each request. The requests and log events are recorded about 15 times per second in my tests. I believe this is a bug, although Edgio support claims otherwise.
I’ve attached a screenshot for reference. The logs are overwhelmed with each request. This situation is problematic for multiple reasons. The combination of the browser endlessly refreshing at a high speed and the consequent flooding of Edgio logs seems indicative of a bug. Notably, the refreshes and log entries occur about 15 times per second.
I hope someone from Edgio takes notice of this and reconsiders whether such behavior is indeed “normal.”
Here is a screen video capture of the issue occurring. You can observe the event ID changing rapidly due to refreshes like crazy, with each even id being logged:
Here is logs showing 15 events logged per second because of this infinite rapid refresh behavior:
I don’t feel this is normal either. However, I was not able to reproduce myself by spoofing my user agent. I did file a ticket for this and would like to add steps to reproduce if you could provide them. Thank you.
Reproduction steps:
Setup “Security” → “Bot Manager” with all “Known Bots” as all.
Setup “Security” → “Bot Manager” with all “Spoofed Bots” “Rule Action” as “Block”
For testing purposes installed
extension into chrome
Add the following user agent into the extension and select it
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Visit related edg.io v7 domain with the above settings.
I was able to reproduce it using that UA. What I am gathering from the dev team is this is expected behavior when spoofing a bot UA in a browser that will be executing JS. The true bot will likely not be executing the code causing the refresh. And even if it were, the log is being hit with entries is beneficial to know the excessive amount of requests that would otherwise be direct traffic to the CDN.
I have seen real users do this to bypass paywalls(with Google bot user agent, but edgio v7 didn’t detect Google bot properly in the past so the test was with bing), so this is a valid false positive behavior.
Even if refresh and logging should occur, I would argue it should not occur 15 times per second.
Regarding “know the excessive amount of requests that would otherwise be direct traffic to the CDN”, it would be one normally, this causes it to be 15 times per second.
Just an additional note.
The reason for 15 times per second refreshes and log entries being made at that fast speed is only due to edgio block page making the user refresh so fast. The user or bad bot will be automatically refresh as fast as possible, even if they didn’t have any malicious intent this is basically making each client DoS the server.
If edgio accidently makes a mistake and detects a large number of users accidently incorrectly or there is a large amount of browsers with problematic user agent visiting a site, the current edgio v7 implementation is basically guaranteeing that the requests will be amplified and be similar to a DDoS attack on edgio infrastruction. As edgio is actually logging every single request, this will potentially cause a massive amount of load on edgio infrastruction.
This behavior, coupled with the flooding of Edgio logs, indicates a potential bug that needs attention. It’s essential for the Edgio team to address this issue promptly to ensure the smooth functioning of the platform and prevent disruptions caused by such abnormal behavior.
As Tristan previously mentioned, this issue is due to the method through which the bot is being spoofed. Spoofed bot traffic should not execute the code that causes this refresh behavior.
But often it does, and I can reproduce this.
Chrome and headless chrome based bots are VERY common and in such cases will run the javascript and cause an infinite redirect loop. Also normal users sometimes spoof user agent headers with Chrome plugins.
To tell the truth, I have no idea why any security company would think this is normal.
