Can I automatically block a client that generates many 404 and other 4xx errors (directory scanning)?

Short answer

Yes. In Bot/DDoS protection, the Source IP Error Protection block of L7 DDoS Protection counts the 4xx error responses that a single source address receives in a time window and applies the Limit Action when the Error Limit is exceeded. The defaults are 50 errors in 1 minute and Show Captcha; Quarantine can also be selected for a harder enforcement. 5xx errors are not counted, and the measurement is an absolute count, not a rate. The block is on by default, but since L7 DDoS Protection starts in Monitor mode, the enforcement starts when the protection is set to Active. The block works only on HTTP vServices and has no hard ceiling.

Problem: Scanners Produce Many Errors with Few Requests

Tools that scan directories and files, and bots that look for an admin panel or a backup file, try addresses that do not exist on the site one after another. Since their requests per second can stay low, they are not caught by request limits; on the other hand, most of the responses they receive are 404 and similar 4xx errors. To catch this behavior, it is necessary to count not the volume but the failed responses that the source receives.

Solution

Source IP Error Protection

• How it works Each source address carries its own error counter. When the number of 4xx responses it receives within the Time Window exceeds the Error Limit, only that source is met with the Limit Action. Server-originated 5xx errors do not enter the counter; a backend failure is not read as an attack.
• Where it is openedBot/DDoS Management → L7 DDoS Protection card → pencil icon of the Source IP Error Protection block
• Requirement Bot/DDoS protection being on for the vService; for enforcement, L7 DDoS Protection being in Active mode
• vService type Only HTTP (on L7 TCP vServices, no error counting is done)
• Tier A single threshold; there is no hard ceiling, and for a hard enforcement Quarantine is selected directly as the Limit Action

Step-by-Step Configuration

1

Adım 1 Open the Bot/DDoS Management tab

The Left Monitor Panel is opened, the vService is clicked in the Bot/DDoS Protected vServices tab, and the Bot/DDoS Management tab is opened.

2

Adım 2 Open the edit window of the block

The pencil icon at the top right of the Source IP Error Protection block in the body of the L7 DDoS Protection card is clicked; the Edit - Source IP Error Protection window opens (Edit screen).

3

Adım 3 Verify that the block is on

The Source IP Error Protection switch at the top must be on (default: on).

4

Adım 4 Enter the window and the error limit

The Time Window is entered as a number and a unit (default 1 minute; at least 1 second, at most 1 hour). In the Error Limit field, the number of 4xx responses that a single address can receive within the window is written as an integer (default 50). The block is triggered when the counter exceeds this value.

5

Adım 5 Select the Limit Action

Show Captcha (default), Silent Drop, Block, Tarpit, Redirect, Show Content or Quarantine is selected. Show Captcha allows a human user to continue; to stop the scanner for a period of time, Quarantine is selected, and the source is met with the duration and action on the Quarantine card (default 30 minutes, Silent Drop). The Response Status Code (default 403) is entered for Block and Tarpit, the redirect address and code for Redirect, and the content page and status code for Show Content.

6

Adım 6 Save

The sentence at the bottom of the window is read (if traffic from any single IP exceeds <limit> errors in <window>, <action>), and the Edit button is clicked. There is no separate deployment step.

7

Adım 7 Set the protection to Active

In the window opened with the Edit link in the header of the L7 DDoS Protection card, the DDoS Protection field is set to Active and saved with the Edit button. Since this field moves all the L7 DDoS blocks to enforcement together, the limits of the other blocks are also reviewed.

Verification

1Error load per sourceTR7 side

In the vService's Live Stats tab, the Peak Err/IP (10s) card and the Err (10s) column in the Live Traffic table are checked.

ExpectedThe scanning source stands out in the Err (10s) column

The cards measure in a fixed ten-second window; since the default block window is 1 minute, the values are read proportionally.

2Event Logs rowTR7 side

In the vService's Event Logs tab, the row of the source is opened; DDoS Rule and Trigger (value / limit) are read on the L7 DDoS tab of the preview.

ExpectedDDoS Rule: Source IP Error Rate · Action: CAPTCHA shown (in Monitor: Would show CAPTCHA · Monitor)

If Quarantine was selected as the Limit Action, the → Quarantined (DDoS limit) suffix is added to the end of the Action text.

Common Mistakes

• Lowering the limit without taking legitimate errors into account. Broken links, missing images or the 401/403 responses of an application whose session has dropped are also counted as 4xx. The limit is narrowed after it has been verified with Live Stats and the Would… rows in Monitor mode.
• Not exempting one's own scanning and monitoring tools. Security scanners and monitoring systems deliberately produce many 4xx errors; their addresses are exempted with DDoS Whitelist Entries.
• Assuming that 5xx errors are also counted. Server-originated 5xx responses do not enter the counter of this block; backend failures do not penalize the source.
• Expecting a rate. The threshold is not a percentage but an absolute count; a client that sends many requests but receives few errors is not caught by this block.
• Looking for the block on an L7 TCP vService. Error counting is done only on HTTP vServices; this block is not present on L7 TCP.
• Assuming that search engine bots are exempt on their own. The Trust Verified Bots setting of L7 DDoS Protection is off by default; when it is turned on, verified bots are not caught by the limit of this block.

How it is done in the documentation

• Source IP Error Protection · What Is Source IP Error Protection?
• Source IP Error Protection · The Source IP Error Protection Edit Screen
• L7 DDoS Protection · In Which Order Are the Blocks Evaluated?
• Quarantine · What Is Quarantine?

Related questions

• How do I detect bots; can I allow good bots (Googlebot etc.) and block bad ones?
• Can I do rate limiting (X requests per second per IP)?
• I am blocking legitimate users by mistake; how do I find out?
• Can I limit the number of connections a single IP address can open (Source IP Connection Protection)?