The analysis produced very few page types / none at all; which filters should I check?

Short answer

First, how many lines the analysis read and how many of them it used is examined: the Parsed Logs, Used logs, Filter reasons and Static requests fields in the details window of the analysis show where the loss is. If Parsed logs is zero, there is no learning log in the selected period; the learning log is written only from the requests that come to a host group in Learning Mode and from the requests that come with the Conditional Learning identity. If the lines are eliminated in the filters, the defaults are reviewed: Country Filter takes only Local networks and Turkey; Only Domains leaves out the Host headers that contain a port or an uppercase letter, Only Browsers the clients that are not browsers, Status Code Filter the responses of 400 and above, and Bot Filter all the bot categories and the requests that carry no User-Agent. If no page type comes out although the lines are used, Rule Creating Mode is decisive: Page independent produces no page type at all; Data based does not make a page type of the requests that carry no query, no body and no method other than GET, HEAD, OPTIONS, and counts them as Static requests. On the result screen, the Min requests, Min visitors, Min Ips and Country limit thresholds under Advanced Settings and the Excluded paths and Dynamic paths lists further reduce the number of pages.

Problem: The Analysis Suggested Fewer Page Types Than Expected

Analysis & Learning was completed, but the result stayed far below the real number of pages of the site or suggested no page type at all; instead of the result opening, the "No logs found in analysis. Would you like to delete the analysis?" warning may also be seen. The analysis reads the learning log by passing it through several eliminations, and every elimination makes the result smaller; which one is in effect is read from the details of the analysis (How do I generate rules from past traffic with Analysis & Learning?).

Which Records Does the Analysis Use?

1. Source: Only the learning log is read. A request is written to this log when it comes while the host group is in Learning mode or with the Conditional learning identity; the requests in Monitor mode and Blocking mode, the Event Logs records, the requests that match the Whitelist Condition and the requests counted as static content are not read. While Smart Sampling (learning log) is on, only some of the requests of the same shape are written, so the analysis sees the sampled set.
2. Period and source device: The records in the Analyze Date range are read; in a cluster setup, the log of the device selected in the Source field is read (by default, the device being worked on). The number of lines read appears in Parsed logs.
3. Host group: Every line is mapped to a host group according to the Host header; the lines of the host groups that are not selected in the Virtual Host Groups field are counted in Parsed logs but do not enter Used logs and Filter reasons.
4. Filters: A line passes through the filters in this order: Country filter, Only domains, Only browsers, Skip Blacklist IPs, Status Code Filter, Bot filter, Source IP Filter, Host filter. A line is written to the Filter Reasons breakdown with the name of the first filter it is caught by; for this reason, a bot request that comes from abroad is counted not in Bot filter but in Country filter.
5. Static requests: If Rule Creating Mode is not Comprehensive, the requests that carry no query and no body and whose method is GET, HEAD or OPTIONS are counted as Static Requests and do not produce a page type. These lines are included in Used Logs.
6. Becoming a page type: The remaining requests are placed in the path tree. In Page independent, no path becomes a page type. The thresholds and the Dynamic paths patterns on the result screen reduce the Analysis Page Count to the Current page count.

Possible Causes

Symptom	Cause	Fix
Parsed logs 0 or the "No logs found in analysis" warning	In the selected period, the host group was not in Learning mode and no traffic came with the Conditional learning identity; or, in a cluster, the device that does not receive the traffic was selected as Source	The period is moved to the range in which the learning was done; for a host group in Blocking mode, records are first collected with Conditional learning (Can I still keep learning certain traffic while in Blocking mode (Conditional Learning)?); Both Devices is selected in Source
Parsed logs high, Used logs low, Country Filter dominant in Filter reasons	Country Filter is Use selected with only Local networks and Turkey selected; or TR7 is behind a CDN and, since there is no Trusted Proxy, the clients appear to be from the CDN's country	The countries of the users are added or No filter is selected; for a CDN, The logs show the proxy / CDN IP instead of the client IP; what should I do?
Only Domains in Filter reasons	The Host header carries a port (www.example.com:8443), an uppercase letter or an IPv6 address; a Host that consists only of an IPv4 address counts as valid	The Only Domains check is removed
Only Browsers in Filter reasons	The clients are a mobile application, an API client or an integration; they are not recognized as a browser	The Only Browsers check is removed (What are the "Only Domains" and "Only Browsers" filters for?)
Skip Blacklist IPs in Filter reasons	The client addresses are in the IP Intelligence & Reputation blacklist as individual addresses	Unless necessary, it is left checked; learning rules from the traffic of blacklisted sources is not wanted
Status Code Filter in Filter reasons	The application returns 4xx/5xx to most requests, or redirects are not selected	In the Status Code Filter default, the codes below 400 are selected; the application errors are fixed
Bot Filter in Filter reasons	Bot Filter is Eliminate selected with all categories selected; while Only browsers is not checked, test tools (curl, Postman), monitoring clients and the requests that carry no User-Agent are eliminated in this filter	The category that must enter the analysis is removed from the selection
Source IP Filter or Host Filter in Filter reasons	Use selected was selected and the list was left empty, or the pattern is wrong	Source IP Filter and Host Filter are set to No filter or the list is corrected
Used logs high, Static Requests also high	Rule Creating Mode is Data based; the page accesses that carry no data do not produce page types	If every page is to receive a separate rule, Comprehensive is selected (What is the difference between the Rule Creating Mode options (Comprehensive, Data Based, Page Independent); which one should I choose?)
No page type at all, Used logs high	Rule Creating Mode is Page independent	Data based or Comprehensive is selected
Analysis Page Count high, Current Page Count low	The Min Requests, Min Visitors, Min Ips, Country Limit thresholds on the result screen, or broad Dynamic Paths and Excluded Paths patterns	The thresholds are cleared and the patterns are reviewed (Why are there Min Requests, Min Visitors, Min Ips, Country Limit filters?, What are Dynamic Paths, Excluded Paths and Dynamic Path Exclusions for?)
A host group has no result	That group is not selected in the Virtual Host Groups field, or the Host header of the traffic falls into another group	The group is selected; the host group patterns are reviewed

Where Is It Monitored?

Screen	Path	What it shows
The details window of the analysis (details of the running analysis)	WAF Management → the progress bar on the mode card	While the analysis is running, Parsed Logs, Used Logs, the filters used and Filter Reasons
The analysis preview (preview of the selected analysis)	WAF Management → the house icon at the beginning of the Analysis & Learning row on the mode card → analysis list → row; or the New badge in the row	All the parameters of the finished analysis, Filter Reasons, Static Requests, Analysis Page Count / Current Page Count
Analyze Page Types	Analysis preview → Edit	The path tree, the thresholds and the path lists
Learning Log Sampling	WAF-Protected vServices → vService → Dashboard → Learning	Whether the lines written to the learning log are flowing

How to Diagnose?

1. Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.
2. The house icon at the beginning of the Analysis & Learning row on the mode card of the host group is clicked; the analysis list opens. The row is clicked and the preview on the right is read. If the analysis is still running, the same fields are seen by clicking the progress bar on the mode card.
3. The Parsed Logs value is examined. If it is zero, there is no learning log in the period: the Analyze date range, the mode of the host group in that period and, in a cluster, the Source field are reviewed.
4. Used Logs is compared with Parsed logs. If the difference is large, the filter with the highest value is found in the Filter Reasons breakdown; because of the order of the filters, the breakdown shows the first filter the line was caught by.
5. If the Static Requests value makes up most of Used logs, Rule Creating Mode is Data based and the pages that carry no data have not produced page types; the Ignore Old Rules row in the preview also shows the selected Rule Creating Mode.
6. Analysis Page Count is compared with Current Page Count. If Current Page Count is clearly lower, the thresholds and the path lists on the result screen have eliminated pages.

Solution

Losses caused by the filters, the period and Rule Creating Mode require running the analysis again; losses caused by the thresholds and the path lists on the result screen, on the other hand, are corrected on the existing analysis.

Your situation	Suitable method
Parsed logs low, one filter dominant in Filter reasons, Static requests high or the wrong Rule Creating Mode	Method 1: Running the analysis again with corrected settings
Analysis Page Count high but Current Page Count low	Method 2: Correcting the Analyze Page Types thresholds and path lists
There is no learning log at all in the selected period; the host group is in Blocking mode	Can I still keep learning certain traffic while in Blocking mode (Conditional Learning)?

Method 1: Running the Analysis Again with Corrected Settings Recommended

• How it works The same learning log is read again with a period, filters and a Rule Creating Mode suited to the traffic of the site; a new analysis record is formed (creating a new analysis).
• Where it is opened WAF-Protected vServices → vService → WAF Management → mode card → Analysis & Learning
• Requirement Modify permission in the WAF area; a learning log in the period
• Scope The selected host groups; the analysis does not touch the live rule set, it is applied only with Apply

Step-by-Step Configuration

1

Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2

Adım 2 Open the analysis window

The Analysis & Learning text on the mode card of the host group is clicked; the window titled New Analyze for followed by the vService name opens. The window opens with the filter values of the analysis most recently started for this vService; Host filter, Ignore Old Rules and the host group selection return to their default values at every opening.

3

Adım 3 Select the period

In the Analyze Date field, the range in which the learning log exists is selected (default Last 3 month(s)); besides the ready ranges, Custom Duration and Custom Date Range can be used.

4

Adım 4 Select the source device (cluster)

In a cluster setup, Both Devices is selected in the Source field in the Advanced section; if a single device is selected, the traffic that passes through the peer device does not enter the analysis.

5

Adım 5 Select the relationship with the existing rules

The Ignore Old Rules switch comes with Create new rules at every opening and ignores the existing rule list. If the existing rules are to be preserved, the switch is brought to the Combine with old rules position (What is the difference between "Create New Rules" and "Combine with Old Rules" in the analysis?).

6

Adım 6 Select the rule creating mode

Comprehensive (Each page is learned), Data based (Pages with dynamic data are learned) or Page independent (Pages are not learned) is selected from the Rule Creating Mode list; the default is Data based. When the analysis is applied, the Rule Creation Policy in Learning methods is also updated to this value.

7

Adım 7 Set the filters

The filter that comes out dominant in Filter reasons is loosened: the countries of the users are added to Country Filter or No filter is selected (Local networks stays selected for local and private addresses); the Only Browsers check is removed for API and mobile traffic, and the Only Domains check for Hosts that carry a port; in Bot Filter, the categories that must enter the analysis are removed from the selection. The Status Code Filter default, which takes the codes below 400, is preserved; Skip Blacklist IPs is left checked.

8

Adım 8 Start the analysis

Start Analyzing is clicked; the window closes and the progress is followed from the bar on the mode card. When the analysis is completed, the New badge appears in the Analysis & Learning row; the result is reviewed with Edit and applied with Apply (applying the analysis result). Before Apply, the paths that come out of the analysis are reviewed; only the paths for which a WAF rule will be created must remain in the list. It is recommended to select unknown or suspicious paths in the tree and exclude them from the analysis with Exclude from Analysis; these paths are added to the Excluded paths list and do not produce rules. Since applying cannot be undone, a backup is taken beforehand with Export from the Page types screen.

Verification

1Analysis detailsTR7 side

The progress bar on the mode card or, when the analysis is finished, the New badge is clicked; Used logs and Filter reasons are read.

ExpectedUsed logs increased, the dominant filter in Filter reasons decreased

2Page countTR7 side

The Analysis Page Count / Current Page Count value in the analysis preview is examined.

ExpectedA higher value than the previous analysis

Method 2: Correcting the Analyze Page Types Thresholds and Path Lists

• How it works Without running the analysis again, the thresholds that narrow the result tree are cleared and the path lists are reviewed; the tree is recalculated with every change (Advanced Settings).
• Where it is opened Analysis preview → Edit → Analyze Page Types tab → Advanced Settings
• Requirement Modify permission in the WAF area; an analysis result that has not been applied
• Scope The selected analysis result; the live rule set does not change until Apply

Step-by-Step Configuration

1

Adım 1 Open the analysis

In WAF Management, the house icon at the beginning of the Analysis & Learning row on the mode card is clicked, the row is selected in the analysis list and Edit is clicked; the Analyze Page Types tab opens. If there is no record processed in the analysis for the related host group, the tab does not open and deleting the analysis is suggested.

2

Adım 2 Open the Advanced Settings window

Advanced Settings on the toolbar is clicked; the Advanced Settings window opens.

3

Adım 3 Clear the thresholds

Filter is selected from the groups at the top of the window; the Min Requests, Min Visitors, Min Ips and Country Limit fields are cleared or lowered (the smallest value is 2; for an empty field, no threshold is applied). Max Logs per Path and Don't Learn OWASP Exclusions are not changed.

4

Adım 4 Review the path lists

In the same window, the Excluded Paths and Dynamic Paths groups are opened; the paths that must not stay out of the analysis are deleted from Excluded paths, and the broad patterns that gather pages different from each other into a single page type are deleted from Dynamic paths. The paths that must stay separate are added to Dynamic Path Exclusions.

5

Adım 5 Confirm

Accept is clicked; the tree is recalculated and the change is written to the analysis record without a separate save step. Before applying, the paths that come out of the analysis are reviewed; only the paths for which a WAF rule will be created must remain in the list. It is recommended to select unknown or suspicious paths in the tree and exclude them from the analysis with Exclude from Analysis; these paths are added to the Excluded paths list and do not produce rules. The live rule set changes only with Apply; if Current Page Count exceeds 500, applying is rejected.

Verification

1Current Page CountTR7 side

Current Page Count is read in the summary window on the Analyze Page Types tab or in the analysis list.

ExpectedA value approaching the Analysis page count

Common Mistakes

• Analyzing the traffic of a host group in Blocking mode. The requests in Monitor mode and Blocking mode are not written to the learning log; without the Conditional learning identity, the analysis finds no lines in this period.
• Leaving the Country filter default on a site that serves abroad. The default takes only Local networks and Turkey; all of the traffic that comes from other countries stays out of the analysis.
• Thinking that Only domains eliminates only the requests that come with an IP. A Host that consists only of an IPv4 address counts as valid; the Host headers that contain a port number, an uppercase letter or an IPv6 address are eliminated.
• Analyzing traffic produced with test tools. With the default settings, the requests of curl, Postman and similar tools and the requests that carry no User-Agent are caught by Only browsers and, if this check is removed, by Bot filter; the test traffic is produced with a browser or the filters are set accordingly.
• Starting the analysis without reviewing Ignore old rules. The switch comes with Create New Rules at every opening; when the analysis is applied, the existing rule list is replaced with the analysis result.
• Selecting a single device as Source in a cluster. Every device reads only its own learning log; Both Devices is selected in Source.

How it is done in the documentation

• Analysis & Learning · The Details of the Running Analysis
• Analysis & Learning · Country Filter
• Analysis & Learning · Rule Creating Mode
• Analysis & Learning · Advanced Settings
• Analysis & Learning · The Two Counters

Related questions

• How do I keep attack traffic out of the analysis (country, blacklist, bot, status code, source IP filters)?
• What are the "Only Domains" and "Only Browsers" filters for?
• What is the difference between the Rule Creating Mode options (Comprehensive, Data Based, Page Independent); which one should I choose?
• Why are there Min Requests, Min Visitors, Min Ips, Country Limit filters?