How do I generate rules from past traffic with Analysis & Learning?

Short answer

On the WAF Management tab, the Analysis & Learning link on the mode card of the host group is clicked, and in the New Analyze window that opens, Analyze Date (the last 3 months at the first opening), the Ignore Old Rules switch under Advanced (Create New Rules / Combine with Old Rules), Rule Creating Mode (Data Based at the first opening) and the traffic filters are selected, and Start Analyzing is clicked. The analysis reads the learning log and produces a page type suggestion; when it ends, the New badge appears on the mode card. The result is examined and narrowed on the Analyze Page Types screen with Edit: the paths that come out in the analysis result are reviewed, only the paths for which a WAF rule will be created should remain in the list; it is recommended that unknown or suspicious paths be excluded from the analysis with Exclude from Analysis. Then it is written to the rule list of the host group with Apply. There must be a learning log in the selected period: the group must be in Learning mode or traffic must have been collected with Conditional Learning. Applying cannot be undone; if the existing rules are to be preserved, an Export is taken first. There is no analysis command in WEB-CLI.

Problem: The Rule Set Is Too Big to Be Written Manually

Writing the rules manually for the hundreds of paths and parameters of the application is not practical; the page types, the allowed methods and the measured limits need to be produced from the accumulated traffic. Analysis & Learning does this from the learning log; the records collected during the time spent in Learning mode are turned into rules.

Solution

The analysis is started per host group. To process more than one host group in a single analysis, Collective Analysis & Learning is used: Can I run an analysis for several host groups at once (Collective Analysis)?

Analysis & Learning

• How it works The learning log of the selected period is read and a page type tree is suggested; Apply reads the same period again and writes the measured values to the rule list of the host group.
• Where it is opened Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → the mode card of the host group → Analysis & Learning
• Requirement Modify permission in the WAF area · a learning log in the analysis period (the group in Learning mode or Conditional Learning on) · if the existing rules are to be preserved, an Export beforehand
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Back up the existing rule set (if needed)

Since applying cannot be undone, if the existing rules are to be preserved, an Export is taken on the Page Types screen of the host group (Can I export a host group's rule set as JSON and move it to another environment; which settings are not included in the file?).

2

Adım 2 Verify Argument Checks (if needed)

For the allowed parameter and header lists to be learned, Argument Checks must be on in the Checks card in the Inspection & Blocking section. The analysis uses the setting at the moment it is started (What do the Structure Checks and Argument Checks master switches do; what happens if I turn them off?).

3

Adım 3 Open the analysis window

In the Left Monitor Panel → WAF-Protected vServices → vService → WAF Management tab, the Analysis & Learning link on the mode card of the host group is clicked; the window titled New Analyze for followed by the vService name opens. The vService and Virtual Host Groups fields are locked to the selected group.

4

Adım 4 Select the period

In the Analyze Date field, the period to be read is selected. At the first opening, the last 3 months are selected; at the following openings, the value last used for that vService comes.

5

Adım 5 Set the advanced options

The Advanced section comes open:

Field	Default	Detail
Source (only in a cluster)	This device	Both Devices for the records of the two devices
Ignore Old Rules	Create New Rules (at every opening)	What is the difference between "Create New Rules" and "Combine with Old Rules" in the analysis?
Rule Creating Mode	Data Based (Pages with dynamic data are learned)	What is the difference between the Rule Creating Mode options (Comprehensive, Data Based, Page Independent); which one should I choose?
Country Filter	Use Selected: Local Networks, Turkey	If there are users abroad, their countries are added
Only Domains · Only Browsers · Skip Blacklist IPs	Checked	What are the "Only Domains" and "Only Browsers" filters for?
Status Code Filter	Use Selected: the codes below 400	-
Bot Filter	Eliminate Selected: all the bot categories	-
Source IP Filter · Host Filter	No Filter	How do I keep attack traffic out of the analysis (country, blacklist, bot, status code, source IP filters)?

The selections other than Ignore Old Rules and Host Filter are remembered per user and per vService.

6

Adım 6 Start the analysis

Start Analyzing is clicked. On the mode card, a progress bar that shows the percentage and the number of lines read appears; when the bar is clicked, the detail window (Task Type, Parsed Raw Data, Used Logs, Filter Reasons) and the Stop Process button open (Can I stop a running analysis; how many analyses can be kept at most?).

7

Adım 7 Open the result window

When the analysis ends, the New badge appears on the mode card; when the badge is clicked, a window that carries the Edit, Apply, Delete and Ignore buttons opens.

8

Adım 8 Open the Analyze Page Types screen

Edit is clicked; the Analyze Page Types tab opens. If no log has fallen to this host group in the analysis (Used Logs 0), the tab does not open; the question "No logs found in analysis. Would you like to delete the analysis?" comes.

9

Adım 9 Review the paths

The paths that come out in the analysis result are reviewed one by one in the tree; only the paths for which a WAF rule will be created should remain in the list. It is recommended that unknown or suspicious paths be excluded from the analysis manually: the rows are selected and Exclude from Analysis is clicked (it is also done by right-clicking a branch in the tree). The excluded addresses are added to the Excluded Paths list and produce no rule.

10

Adım 10 Narrow the tree

Similar addresses are gathered with a glob; Current Page Count must not exceed 500 (Can I review and edit the analysis result before applying it (Analyze Page Types screen)?).

11

Adım 11 Apply

Apply in the toolbar is clicked; the Are you sure? question opens.

12

Adım 12 Confirm

The Are you sure? question is confirmed with Yes. On the mode card, a progress bar with the Applying Rules task type appears; when applying ends, the "Rules applied" notification appears on the screen and the Applied badge stays on the mode card for 5 minutes. Writing the rules does not require a separate deployment step; in a cluster, the change goes to the peer device with the default auto-sync, and if Manual Sync Mode is on, it is sent manually (What happens when I apply an analysis result?).

Verification

1Page typesTR7 side

The page type count row on the mode card is clicked and the Page Types list of the host group is examined.

ExpectedPage types coming from the analysis

2Learning Methods cardTR7 side

The rule policy sentence on the Learning Methods card is examined; applying sets the policy to the mode selected in the analysis.

ExpectedRule policy is Data based — unlearned accesses carrying data are blocked

If another mode was selected in the analysis, the sentence carries the name of that mode.

Common Mistakes

• Analyzing a period with no learning log. If the group is in Monitor or Blocking mode and Conditional Learning is not on, no record is formed; Used Logs comes as 0.
• Not verifying the country filter. At the first opening, only Local Networks and Turkey are selected; the traffic of users who come from other countries does not enter the analysis.
• Losing the existing rules with Create New Rules. The switch is in the Create New Rules position at every opening and ignores the existing rule list; if it is to be preserved, Combine with Old Rules is selected.
• Editing the rules manually after the analysis and then applying. Applying takes as its basis the rule copy at the moment the analysis started; the manual edits in between are lost.
• Overlooking the Source field in a cluster. The default is only the learning log of this device; if the traffic is distributed to two devices, Both Devices is selected.
• Applying without reviewing the paths in the analysis result. Unknown or suspicious addresses mixed into the analysis are also written as page types; these paths are excluded from the analysis with Exclude from Analysis before applying.
• Taking the result directly to Blocking. First, the result is verified by watching it in Monitor mode (When should I move from Learning to Monitor and from Monitor to Blocking?).

How it is done in the documentation

• Analysis & Learning · Creating a New Analysis
• Analysis & Learning · Starting the Analysis
• Analysis & Learning · The Application of the Analysis Result
• Analysis & Learning · Scope and Limits

Related questions

• What should I do first after enabling the WAF?
• What happens when I apply an analysis result?
• Can I still keep learning certain traffic while in Blocking mode (Conditional Learning)?