How do I keep attack traffic out of the analysis (country, blacklist, bot, status code, source IP filters)?

Short answer

The filters in the Advanced section of the New Analyze window filter the learning log lines that will enter the analysis, and their values at the first opening are already filtering. Country Filter: Use Selected, Local Networks and Turkey; it is widened according to the user base. Skip Blacklist IPs: eliminates the individual addresses in the blacklist; the blocks in the blacklist do not match. Bot Filter: Eliminate Selected, all the bot categories; while it is in the Use Selected or Eliminate Selected mode, requests that carry no User-Agent are also eliminated. Status Code Filter: Use Selected, the codes below 400. Source IP Filter: an IPv4 or IPv6 address and block is written, and Use Selected or Eliminate Selected is selected. Host Filter: an exact name or a pattern with a wildcard character is written. Only Domains and Only Browsers are checked. The reasons for the eliminated lines are seen in the Filter Reasons breakdown, in the detail window while the analysis runs and in the result summary after it ends (the details of the running analysis). After the analysis ends, in the Advanced Settings window on the Analyze Page Types screen, the pages that come from a small number of sources are taken out of the tree with the Min Requests, Min Visitors, Min Ips and Country Limit thresholds. Since the filters cannot catch every attack request, before applying, the paths that come out in the analysis result are reviewed; only the paths for which a WAF rule will be created should remain in the list, and it is recommended that unknown or suspicious paths be excluded from the analysis with Exclude from Analysis.

Problem: Attack Traffic Gets Mixed into the Analysis

The learning log also contains crawler bots, scanning tools and attack attempts. If these enter the analysis, pages that real users never use and limits wider than necessary are added to the rule set; when Blocking mode is switched to, these pages stay open to the attacker.

Solution: Two Methods

The methods are used together: the first filters the lines that will enter the analysis, the second filters the page tree that comes out of the analysis.

Your situation	Suitable method
I am going to start the analysis; I want to select which traffic is read	Method 1: New Analyze filters
The analysis has ended; I want to take the pages that come from a single source out of the tree	Method 2: Analyze Page Types thresholds

Method 1: New Analyze Filters Recommended

• How it worksLearning Log lines are filtered by country, blacklist, bot, response code, source address and host before they enter the analysis.
• Where it is opened Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → mode card → Analysis & Learning → Advanced
• Requirement Modify permission in the WAF area · a learning log in the analysis period
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the analysis window

In the Left Monitor Panel → WAF-Protected vServices → vService → WAF Management tab, the Analysis & Learning link on the mode card of the host group is clicked; the window titled New Analyze for followed by the vService name opens (How do I generate rules from past traffic with Analysis & Learning?). The Advanced section comes open.

2

Adım 2 Set the country filter

The Country Filter mode is left at Use Selected and the countries that are served are added (Local Networks are the internal network addresses). In the Use Selected mode, addresses whose country cannot be determined are eliminated; in the Eliminate Selected mode, they enter the analysis.

3

Adım 3 Set the client filters

Skip Blacklist IPs and Only Browsers are left checked; Bot Filter is left in the Eliminate Selected mode with all the categories selected. If mobile application or API traffic is being learned, Only Browsers is removed (What are the "Only Domains" and "Only Browsers" filters for?).

4

Adım 4 Set the response code and host filters

Status Code Filter is left in the Use Selected mode with the codes below 400; thus the scan requests that receive 4xx and 5xx responses are eliminated. Only Domains is left checked. If specific domain names are to be learned, Host Filter is set to Use Selected and one name or a pattern such as dev* is written per line; the exact name match is case-sensitive.

5

Adım 5 Filter out source addresses (if needed)

For the traffic of test tools that scan or of known attacker addresses, Source IP Filter is set to Eliminate Selected and an address with a block length is entered on every line. If only the traffic of a specific team is to be learned, Use Selected is used.

6

Adım 6 Start the analysis

Start Analyzing is clicked. The filter selections (except Host Filter) are remembered per user and per vService.

Verification

1Filter ReasonsTR7 side

While the analysis runs, the progress bar on the mode card is clicked, and after it ends, the New badge is clicked; in the window that opens, General Filters, Filter Reasons and Parsed Logs / Used Logs are read.

ExpectedThe counts of the eliminated lines per filter

If Used Logs is very low, the filters may be eliminating legitimate traffic as well.

Method 2: Analyze Page Types Thresholds

• How it works The pages that do not come from the specified number of requests, visitors, IPs or countries are taken out of the analysis tree and do not produce a page type.
• Where it is openedAnalyze Page Types → Advanced Settings → Filter
• Requirement Modify permission in the WAF area · a completed analysis
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the result window of the analysis

When the analysis ends, the New badge that appears on the mode card is clicked.

2

Adım 2 Open the Analyze Page Types screen

In the window that opens, Edit is clicked; the Analyze Page Types tab opens. The same screen is also opened by selecting a row in the Analyze list opened with the house icon on the mode card and clicking Edit.

3

Adım 3 Open the Advanced Settings window

Advanced Settings in the toolbar is clicked.

4

Adım 4 Write the thresholds

In the Filter group, Min Requests, Min Visitors (IP and User-Agent pair), Min Ips and Country Limit are filled in; the fields take at least 2, and no threshold is applied for a field left empty. While Min Ips is filled in, Min Visitors is not evaluated (Why are there Min Requests, Min Visitors, Min Ips, Country Limit filters?).

5

Adım 5 Save

Accept is clicked; the change is written to the analysis record immediately and Current Page Count is recalculated.

6

Adım 6 Review the remaining paths

The paths that remain in the tree after the thresholds are reviewed one by one; only the paths for which a WAF rule will be created should remain in the list. Unknown or suspicious paths are selected and excluded from the analysis by clicking Exclude from Analysis; these addresses are added to the Excluded Paths list and produce no rule. The result is applied after this (Can I review and edit the analysis result before applying it (Analyze Page Types screen)?).

Verification

1CounterTR7 side

The Current Page Count value is examined (it also appears as a column in the Analyze list).

ExpectedA lower number than before the thresholds

Common Mistakes

• Leaving the country filter as it is at the first opening. Only Local Networks and Turkey are selected; the pages of users who come from other countries are not learned and may be blocked in Blocking.
• Leaving Only Browsers checked for mobile or API traffic. The traffic of clients that are not browsers does not enter the analysis.
• Thinking that the blacklist filter also eliminates the blocks. Skip Blacklist IPs matches only the individual addresses in the blacklist.
• Setting Bot Filter to No Filter and thinking that the requests without a User-Agent are eliminated. Requests without a User-Agent are eliminated only in the Use Selected and Eliminate Selected modes.
• Relying on the filters and not reviewing the path list. Suspicious requests that pass the filters may also produce page types; before applying, unknown paths are excluded with Exclude from Analysis.
• Not reviewing the filters when few pages are produced. Narrow filters eliminate legitimate traffic as well (The analysis produced very few page types / none at all; which filters should I check?).

How it is done in the documentation

• Analysis & Learning · Country Filter
• Analysis & Learning · Bot Filter
• Analysis & Learning · Status Code Filter
• Analysis & Learning · Advanced Settings

Related questions

• Why are there Min Requests, Min Visitors, Min Ips, Country Limit filters?
• The analysis produced very few page types / none at all; which filters should I check?
• What are the "Only Domains" and "Only Browsers" filters for?