Rate Limit Site Scan#366

I am getting many “Site XXXX is DOWN”, “Site XXXX is UP”, “GATEWAY is DOWN” etc messages - almost always with a 504 status.

This status is NOT a “server down” - it is a RATE LIMITER - your agent/bot is htting sites TOO FAST. This is causing a LOT of spurious alerts (I’m getting these 4 and 5 times a day).

While I understand you do not want your servers to spend too much time scanning user sites, the solution is NOT to hit sites faster - it is to INTERLEAVE fetches between DIFFERENT sites while rate-limiting fetches to any INDIVIDUAL site. This uses your server time efficiently while NOT harming user servers.

If you do not rate limit your scans, you risk YOUR SERVERS being flagged as in violation, and getting blocked.

Tracy Hall
SirlinHallJewelry.com
NoRosesJewelry.com