Edgio v7 Bot manager "Known Bots" problems and "Exceptions" problem

Product Feedback
1

Hello,

We noticed the list for “Known Bots” seems VERY old and in our real traffic is missing a lot of common bots.
It seems to need a large update to include common recent bots.

Also “Exceptions” feature doesn’t include condition statements.
So adding a new bot as “Exceptions” is VERY dangerous because there is no way to limit it to known ASN or IP ranges or better than reverse DNS lookup(which is officially recommended to identify common bots like google, bing, yandex bots etcetera)

References)

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26
https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.htm

Also for “Known Bots” and “Spoofed Bots” if one is turned on it need the other, but both of them have no ignore for “Rule Action” and the minimum setting is “Alerts” which fills the edgio logs with often unnecessary logs.
“Ignore” should be made available “Rule Action”, especially since the settings rely on each other.

Additionally in my tests,
Bot Manager | Edgio Documentation documentation regarding
“The Spoofed Bots section does not apply to the 200+ known bots defined within the other category.”
is incorrect.
If you test with a spoofed bing bot using
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
It is only detected when “other” is turned on in “Known Bots”.
So the documentation or behavior should be fixed.
On the other hand, bing bot should really not be in “other” but a seperate “Known Bots” as it is a major bot. This shows the Known Bots list is very old and needs updating.

2

There are some responsible bots that post their IP blocks and reverse DNS lookup patterns.

Here are two examples:

I wish there were more available. I had written a “FakeBots” module, but recently discovered that many webpages with instructions that I was using are no longer avialable.

3

Also can confirm the most common bot in the world which is Google bot was missing from known bots in v7 platform.
It was added today after opening a ticket with edg.io

Edg.io please consider updating the Known bots to reflect recent bots and also allow users to properly add additional known bots.

1 Like
4

Hi. Thank you for your feedback.

  • WRT to coverage for known bots, we update our known bot coverage every day.
  • I have passed on your suggestion to add conditions to exceptions.
  • Bing bot is covered as a known bot under the msn category.