How do I detect bots; can I allow good bots (Googlebot etc.) and block bad ones?

Short answer

Yes. The Bot Type condition recognizes the User-Agent header of the request by categories (e.g. Malicious bots, Search engines); the unwanted categories are bound to a Block, Quarantine or Advanced CAPTCHA action. Since the User-Agent can be spoofed, real search engine bots are separated with the Verified Bot group of the Network Type Group condition; this group looks not at the User-Agent but at the network type data of the source address. If Bot/DDoS protection is on, Bot Protection scores clients between 0 and 100, and the Trust Verified Bots option (on by default) exempts verified bots from the action rules.

Problem: Bad Bots Are Mixed Up with Good Bots

While content collection software (scrapers), vulnerability scanners and brute-force tools are to be blocked, search engine bots must continue to access the site. Bad bots often introduce themselves as Googlebot; a rule that looks only at the User-Agent header cannot tell a tool imitating Googlebot apart from the real search engine.

Solution: Four Methods

The methods are alternatives to each other; on a vService with Bot/DDoS protection on, Method 1 and Method 2 can also be used together.

Your situation	Suitable method
Blocking by bot category and separating fake search engine bots	Method 1: Bot Type + Network Type Group + Block
A score based on behavior and infrastructure; Bot/DDoS protection is licensed	Method 2: Bot/DDoS → Bot Protection
AGS-enabled vService; an AGS policy by score	Method 3: AGS → Bot Score policy
Setting up Methods 1 and 2 from the command line	Method 4: WEB-CLI

Method 1: Bot Type + Network Type Group + Block Recommended

• How it worksBot Type checks the bot category of the User-Agent header, and Network Type Group = Verified Bot checks whether the source address belongs to verified bot networks. Bad categories are blocked directly, and "looking like a search engine but not verified" requests are blocked with a second condition group.
• Where it is openedLoad Balancer → Conditions · Load Balancer → Actions → Block
• Requirement None
• vService type HTTP (the Bot Type condition and the Block action are used only on HTTP vServices)

Step-by-Step Configuration

1

Adım 1 Open the condition form

The condition form is opened with Load Balancer → Conditions → Add (how to add).

2

Adım 2 Define the bad bot condition

Condition Name = kotu-bot, Criteria = Bot Type is selected; in the Bot Type list, Malicious bots and Security checker (Nmap Scripting Engine, sqlmap/, TLSProbe etc.) are checked. If any of the categories selected in a condition matches, the condition is met.

3

Adım 3 Save and move on to the next condition

The condition is saved with Add & Repeat; the form stays open for the next condition.

4

Adım 4 Define the search engine condition

Condition Name = arama-motoru-ua, Criteria = Bot Type, and in the Bot Type list Search Engines (Googlebot, Yahoo, bingbot etc.) is checked.

5

Adım 5 Save and move on to the next condition

The condition is saved with Add & Repeat.

6

Adım 6 Define the Verified Bot condition

Condition Name = dogrulanmis-bot, Criteria = Network Type Group, and in the Network type group name list Verified Bot is selected.

7

Adım 7 Save the condition

The condition is saved with Add and the form closes.

8

Adım 8 Open the action form

The action form is opened with Load Balancer → Actions → Add.

9

Adım 9 Define the Block action

Action Name = bot-engelle, Action Type = Block is selected; Execution Level is left at the default On Request, Deny Type at the default Standard and Status Code at the default 403.

10

Adım 10 Bind the conditions

In the Conditions field, kotu-bot is selected in the first group. A second group is added with +; in this group, arama-motoru-ua and the negated form of the dogrulanmis-bot condition (!dogrulanmis-bot) are combined with and. The operator between the groups is left as or. Result: requests that are "a bad bot or (looking like a search engine and not coming from a verified bot network)" are blocked (How do I combine multiple conditions with AND / OR? · Can I use a condition in negated form ("if not")?).

11

Adım 11 Save the action

The action is saved with Add.

12

Adım 12 Edit the vService

On the Load Balancer → Load Balancer vService List screen, the HTTP vService is selected and the edit form is opened with Edit.

13

Adım 13 Add the action to the vService

In the Details → Actions field, bot-engelle is selected. Unless the action is added to the vService, it does not work.

14

Adım 14 Save

The form is saved with Save; the rule takes effect the moment it is saved. In a cluster, the change is transferred to the peer device automatically by default; if Manual Sync Mode is on, it is sent with Synchronize.

Verification

1Fake GooglebotClient side

Command: curl -s -o /dev/null -w '%{http_code}' -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' https://www.example.com/

Expected403

Since the request is sent from an address that does not belong to Google, it does not fall into the Verified Bot group.

2Real search engine botTR7 side

On the HTTP Logs screen of the vService, the response codes of the requests coming from search engine addresses are checked.

Expected200

If Hit Counter is turned on in the Block action, the number of blocked requests is also monitored in the Action Hits tab of the vService dashboard.

Method 2: Bot/DDoS → Bot Protection

• How it works A bot score between 0 and 100 is calculated from client behavior, browser signals and infrastructure factors; Action Rules compares the score with thresholds and applies an action. On HTTP, the default rules apply Block (403) to a score of 90 and above, and Show captcha to 75 and above.
• Where it is opened Left Monitor Panel → Bot/DDoS Protected vServices → vService → Bot/DDoS Management → Bot Protection card
• Requirement Bot/DDoS protection license; L7 DDoS & Threat Protection on in the vService
• vService type HTTP · L7 TCP (on L7 TCP, Bot Protection comes Off by default; instead of challenge and blocking, silent drop and quarantine are used)

Step-by-Step Configuration

1

Adım 1 Turn on Bot/DDoS protection (if needed)

If the protection is not on in the vService, the vService is edited, the L7 DDoS & Threat Protection box under Details is checked and saved with Save (two ways). When the protection is turned on, IP Reputation Protection starts in Active mode and can block some requests immediately (starting modes).

2

Adım 2 Open the Bot Protection card

In the Left Monitor Panel → Bot/DDoS Protected vServices → vService → Bot/DDoS Management tab, the Edit link of the Bot Protection card is clicked.

3

Adım 3 Set the mode to Active

In the Bot Protection field, Active is selected (the default on HTTP is Monitor; in Monitor, scores are only recorded and no action is applied). Trust Verified Bots is left on; the verification is done by looking not at the User-Agent but at the address range the bot comes from.

4

Adım 4 Save

The window is saved with the Edit button; there is no separate deployment step. If necessary, the score thresholds and the actions are edited separately from the Action Rules box of the card.

Verification

1Event recordTR7 side

The Bot/DDoS Event Logs → Bot tab is opened.

ExpectedClient score and the applied action

Method 3: AGS → Bot Score Policy

• How it works The AGS policy compares the bot score calculated on the vService with the Bot Score Threshold; it applies the policy's action (e.g. Captcha or Deny) to the client that exceeds the threshold.
• Where it is openedAGS → Protection Policies → Bot Protection Policies
• Requirement AGS license and AGS on in the vService. If Bot Protection is on in the vService, the score is calculated with its settings; if it is not on, the score is calculated automatically in Monitor mode for the AGS-enabled vService.
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Enable AGS (if needed)

If AGS is not on in the vService, it is turned on (two ways); since the default flow denies users when AGS is turned on, the flow is prepared first: How do I attach WAF, L7 DDoS & Threat Protection and AGS to a vService?

2

Adım 2 Open the policy form

The policy form is opened with AGS → Protection Policies → Bot Protection Policies → Add.

3

Adım 3 Define the policy

Bot Protection Policy Name is entered, Trigger Type = Bot Score is selected; Bot score threshold (default 60) and Action (e.g. Captcha or Deny) are set.

4

Adım 4 Save the policy

The policy is saved with Add. The built-in Bot Score profiles can also be used (built-in profiles).

5

Adım 5 Open the Protection section

In the Left Monitor Panel → AGS-Enabled vServices → vService → AGS Management tab, the Edit link of the Protection section is clicked.

6

Adım 6 Select the policy

In the Edit / Protection window that opens, the policy is selected in the Bot Protection Policies field.

7

Adım 7 Save

Accept is clicked; the policy is listed in the Bot Protection box in the AGS Management tab.

Verification

1Protection sectionTR7 side

The Bot Protection box of the Protection section in the AGS Management tab is checked.

ExpectedThe policy name and its sentence are listed

Method 4: WEB-CLI

• How it works The conditions and the Block action in Method 1 are set up with lb condition / lb action, and the Bot Protection in Method 2 with lb threat-protection.
• Where it is opened TR7 WEB-CLI → lb condition · lb action · lb service · lb threat-protection
• Requirement WEB-CLI access; Bot/DDoS protection license for Bot Protection
• vService type HTTP

Step-by-Step Configuration

WEB-CLI runs each line as a separate command; commands are written on a single line.

1

Adım 1 Create the conditions

The criteria that recognize the client: bad = Malicious bots, securityChecker = Security checker, search = Search engines, verified_bot = Verified Bot.

lb condition add criteria bot --botType.0 bad --botType.1 securityChecker --name kotu-bot
lb condition add criteria bot --botType.0 search --name arama-motoru-ua
lb condition add criteria nettype_group --nettypeGroup.0 verified_bot --name dogrulanmis-bot

2

Adım 2 Get the condition IDs

The --cond field of the action requires the ID of the condition, not its name. lb condition add gives no output when it succeeds; the IDs are seen in the id column of the list below (they start with lbConditions-).

lb condition list

3

Adım 3 Create the Block action

The --cond expression is written in quotes: or / and combine, ! negates, and at most one level of parentheses is used (blocking actions).

lb action add type deny name bot-engelle --cond "<bad-bot-id> or (<search-engine-ua-id> and !<verified-bot-id>)"

4

Adım 4 See the existing actions

The rules.0, rules.1… lines show the actions bound to the vService.

lb service show <vService>

5

Adım 5 Add the action to the vService

The new action is written to the first empty index, that is, to the number of existing actions. In the example, since the vService has two actions (rules.0, rules.1), --rules.2 is used; writing to a filled index replaces the action there.

lb service update id <vService> --rules.2 bot-engelle

6

Adım 6 Set Bot Protection to Active (if there is a Bot/DDoS license)

If the protection is off, it is turned on first; then Bot Protection is set to Active (true) (bot protection).

lb service update id <vService> --threatProtection true
lb threat-protection update id <vService> --botProtection true --botTrustVerifiedBots true

Verification

1ConfigurationTR7 side

Command: lb service show <vService>

Expectedbot-engelle on the rules.2 line

The blocking itself is tested with the curl attempt in Method 1.

Common Mistakes

• Relying only on the User-Agent. The Bot Type condition looks at the User-Agent header, and this header can be spoofed (security note); the Verified Bot group is used to verify search engines.
• Blocking all bot categories. The Monitoring, Feed and Social media categories also cover monitoring services and link previews; only the unwanted categories are selected.
• Leaving Bot Protection in Monitor. In Monitor mode, scores are only recorded and no action is applied.
• Writing the condition name in the --cond field. In WEB-CLI, --cond expects the condition ID; if the name is written, the action is bound to a condition that does not exist.
• Overlooking that verification is needed instead of a block. To show a CAPTCHA instead of a block to suspicious clients: Can I verify humans with a CAPTCHA?

How it is done in the documentation

• Bot Type · Bot Type
• Network Type Group · Network Type Group Name
• Bot Protection · Trust Verified Bots
• AGS Bot Protection Policies · Bot Score

Related questions

• Can I verify humans with a CAPTCHA?
• How do I recognize and handle Tor exit nodes, VPN/proxy addresses and data center (cloud/hosting) IPs separately?
• Can I do rate limiting (X requests per second per IP)?