New pages are blocked after I released a new version; what should I do?

Short answer

The rule set recognizes the pages of the old version; since the new paths match no page type, they fall to the Any Paths rule. If the rule set was set up with an analysis run with Create New Rules, Any paths is restrictive: if it was set up with Data based, JSON, XML and form bodies and query parameters are blocked in Any paths and the allowed methods start with GET, HEAD, OPTIONS; if it was set up with Comprehensive, no method is allowed in Any paths. On the existing pages too, new parameters, body fields or values that have grown longer are caught by the learned limits. The diagnosis is made in Event Logs with Learn Status = No and Path filtering; if Corresponding Rule on the Attacks tab is Any paths, the request is from a new page. For a small number of new pages, the records are selected and taught with Allow; if Rule Creation Policy is Data based or Comprehensive, a new page rule is created for the requests that carry data and for the methods that are not allowed (Learning Suggestion on the Attacks tab: New rule will be created). For extensive changes, the test team browses the new version with the Teacher identity, or browses it with the Conditional Learning identity and the analysis is run with Combine with Old Rules; during the transition, the host group can be switched to Monitor Mode. The strategy for managing the increase in false positives is explained on the False positives increased after an application update; what should I do? page.

Problem: The Pages of the New Version Are Blocked

After the new version of the application is put into production, new screens, new API endpoints or changed forms return the blocking page (default status code 418). The positive security model of the WAF recognizes the learned pages and fields; access that has not been learned is restricted in the Any paths rule according to how the rule set was set up (How is a request that matches no page type inspected?).

Possible Causes

Cause	What appears in Event Logs (Attacks tab)
The new path matches no page type; Any paths was set up with the Data based rule	Corresponding Rule Any paths; Block All JSON Contents, Block All XML Contents, Block All Form Contents, Allowed Query Parameters, Total Query Size, Query Key Length, Query Key Count, Query Value Length or Allowed Methods
The new path matches no page type; Any paths was set up with the Comprehensive rule	Corresponding Rule Any paths; Allowed Methods
The new path matches no page type; Any paths was set up with a Page independent analysis and the new path uses a method or body type not seen in the analysis	Corresponding Rule Any paths; Allowed Methods, Block All JSON Contents, Block All XML Contents or Block All Form Contents
The new path matches no page type; Any paths was set up again with Sync with Default WAF Rules while OWASP Protection Level was Aggressive	Corresponding Rule Any paths; Allowed Methods, Allowed Query Parameters, Block All JSON Contents, Block All XML Contents or Block All Form Contents
A new parameter, body field or header was added to an existing page	Allowed Query Parameters, Allowed JSON Keys, Allowed Form Fields, Allowed XML Keys or Allowed Headers; the new name in the description
The values or the body grew on an existing page	Query Value Length, JSON Value Length, Form Value Length, Raw Body Size
The page is called with a new method (e.g. PATCH)	Allowed Methods
The new version was released with a new domain name and the requests fall into another host group	Host group on the Request & Location tab
The value of a changed field does not match a learned value pattern	Query Value Pattern, JSON Value Pattern, Form Value Pattern

Definitive rules:

• Any paths is restricted in two ways. The first is applying an analysis run with Ignore Old Rules = Create New Rules: in Data based and Comprehensive, JSON, XML and form bodies and query parameters are blocked (in Comprehensive, in addition, no method is allowed); in Page independent, since all pages are learned in Any paths, Any paths allows only the methods and body types seen in the analysis. The second is Any paths being set up again with Sync with Default WAF Rules while OWASP Protection Level is Aggressive: only the GET, OPTIONS and HEAD methods are allowed; JSON, XML and form bodies (including file uploads) are blocked. If Argument Checks is enabled, every query parameter counts as a violation since the allowed parameter lists start empty, and the Host and User-Agent headers are also required. An analysis run with Combine with Old Rules does not restrict Any paths in this way (What is the difference between "Create New Rules" and "Combine with Old Rules" in the analysis?).
• Teaching from Event Logs works according to Rule Creation Policy: in Data based, Query, JSON, XML, Raw and Form violations, and in Comprehensive, path violations in addition to these, open a new exact page rule for a request that falls to Any paths; an Allowed Methods violation is written to a new or existing page rule in every policy other than Page independent and does not widen Any paths. Header learnings are written to Any paths in every case.
• When a value pattern violation is taught, the pattern is removed from the field and the field is no longer inspected with any pattern; to preserve the pattern, the rule is edited (How do I edit the matching rule from a log with "Edit Corresponding Rule"?).
• An Event Logs record is formed only in Monitor mode and Blocking mode; in a host group in Learning mode, since the rules are not applied, new pages are not blocked by the rules.

Where Is It Monitored?

Screen	Path	What it shows
Event Logs	WAF-Protected vServices → vService → Event Logs	The blocked requests; Learn status and Path on the filter strip
Event Logs preview · Attacks	Row → Attacks	Corresponding rule, Learning suggestion (New rule will be created / Existing rule will be extended)
WAF Blocked Requests	WAF-Protected vServices → vService → Dashboard	The increase in blocking after the moment of release
Page types (accessing the screen)	WAF Management → the N page types row on the mode card	The page types formed after teaching

How to Diagnose?

1. Left Monitor Panel → WAF-Protected vServices → vService → Event Logs is opened. Date Range is selected so as to cover the period after the moment of release and Save & Get Logs is clicked (server filter).
2. On the Learn Status tab of the filter strip, No (and Partly for partly taught records) is selected; the taught records leave the list.
3. On the Path tab, the paths of the new version are selected; if Structure Policies and Structure Limits are dominant on the Category tab, the source of the block is the positive model; if signature categories are dominant, it is a signature false positive (False positives increased after an application update; what should I do?).
4. A row is clicked; Corresponding Rule is read on the Attacks tab of the preview. If it is Any paths, the request has matched no page type; if it is a page rule, the existing page has changed.
5. Learning Suggestion is read on the same tab: New rule will be created shows that the teaching operation will open a new page rule, and Existing rule will be extended that the existing rule will be widened (Which rule changes in the background when I Allow / Teach (Learning Suggestion)?).
6. If a new domain name is used, Host Group is read on the Request & Location tab; if the request falls into an unexpected host group, a new host group is defined (How is a Host Group added; what goes into the Virtual Host Patterns field?).

Solution: Four Methods

The methods are selected according to the scope of the change; during the transition, Monitor mode can be used together with the others.

Your situation	Suitable method
A small number of new pages or fields; the records are in Event Logs	Method 1: Teaching from Event Logs
The test team can browse the new version in the live environment without disturbing Blocking mode	Method 2: Teaching with Teacher
An extensive change; many new pages and fields	Method 3: Conditional learning and a Combine with Old Rules analysis
Users must not be blocked while the teaching is going on	Method 4: Temporarily switching the host group to Monitor mode
The new version is on a new domain name	How is a Host Group added; what goes into the Virtual Host Patterns field?

Method 1: Teaching from Event Logs Recommended

• How it works The violations triggered by the selected records are taught to the rule set; if Rule Creation Policy is Data based or Comprehensive, a new page rule is opened for the requests that carry data and fall to Any paths (Allow / Teach).
• Where it is opened WAF-Protected vServices → vService → Event Logs
• Requirement Modify permission in the WAF area; the host group in Monitor mode or Blocking mode
• Scope The selected records; if there is an attack-based selection on the filter strip, only the signatures that match the selection

Step-by-Step Configuration

1

Adım 1 Open the Event Logs tab

Left Monitor Panel → WAF-Protected vServices → vService → Event Logs is opened.

2

Adım 2 Fetch the period

In the server filter, Date Range is selected so as to cover the period after the moment of release (default Last 7 day(s)) and Save & get logs is clicked.

3

Adım 3 Filter the untaught records

No is selected on the Learn Status tab of the filter strip.

4

Adım 4 Select the new paths

On the Path tab, the paths of the new version are selected.

5

Adım 5 Examine the suggestions

The Attacks tab is opened in the preview of a few rows; the Learning Suggestion column shows what will change. If there is a value pattern violation, that record is not taught; the rule is edited (How do I edit the matching rule from a log with "Edit Corresponding Rule"?).

6

Adım 6 Select the records

The boxes at the beginning of the rows to be taught are checked.

7

Adım 7 Teach

The teaching button on the toolbar is clicked. The name of the button changes according to the selection: Allow if only blocked records are selected, Teach if only monitored records or records passed by Teacher are selected, and Allow / Teach if both are selected together.

8

Adım 8 Confirm

The Are you sure? question is answered with Yes; the learning is applied without a separate deployment step, and the rows whose signatures have all been taught turn green. A wrong teaching can be reverted with Revert learning under the learning record on the Log tab of the WAF - WAAP Data Flow Panel on the right of the Dashboard or WAF Management tab (Can I revert a learned rule?). In a cluster setup, the rule change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

Verification

1Attacks columnTR7 side

The Attacks column of the taught rows in Event Logs is examined.

ExpectedThe first number is 0, the row is green

2Page typesTR7 side

In WAF Management, the N page types row on the mode card is clicked.

ExpectedPage types for the new paths

That the user is not blocked when repeating the same request is verified by no new record being formed in Event Logs.

Method 2: Teaching with Teacher

• How it works The requests that come with the Teacher identity are not blocked in any mode; the requests that produce violations are taught to the rules at the end of the selected period, and new page rules are opened according to Rule Creation Policy (Teacher).
• Where it is opened WAF-Protected vServices → vService → WAF Management → Learning & Reporting → Learning Methods card
• Requirement Modify permission in the WAF area; the IPv4 address of the test team or the TR7-LEARN cookie or header value; the host group in Monitor mode or Blocking mode
• Scope The whole vService; the requests that carry the Teacher identity

Step-by-Step Configuration

1

Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2

Adım 2 Open the Learning Methods window

The pencil icon of the Learning Methods card in the Learning & Reporting section is clicked; the Learning Methods window opens.

3

Adım 3 Turn on Teacher

The Teacher switch is turned on; the IPv4 addresses of the test team are written, separated by spaces, in the Teacher IP field, or the TR7-LEARN value is written in the Teacher cookie / Teacher header field.

4

Adım 4 Select the period

A period between 1 minutes and 1 hours is selected from the Teaching Frequency list; while None, the default value, is selected, the learning is not applied.

5

Adım 5 Save

Edit is clicked; the identity takes effect without a separate deployment step. In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

6

Adım 6 Browse the new version

The test team, carrying the identity, uses all the screens and workflows of the new version. Teacher learns only from the requests for which an Event Logs record is formed; the requests whose score does not reach the blocking threshold are not blocked anyway.

7

Adım 7 Turn off Teacher

When the teaching is finished, the Teacher switch is turned off in the same window and Edit is clicked; while the switch is off, the Teacher identity recognizes no request. Since every request that carries the identity passes without being blocked and loosens the rules, Teacher is not left on.

Verification

1Teacher historyTR7 side

On the Teacher Tracking tab, View → Teacher history is selected.

ExpectedLearning rows for the new paths

If Teacher is not recognized: Teacher traffic is not recognized; why?

Method 3: Conditional Learning and a Combine With Old Rules Analysis

• How it works The requests of the test team are written to the learning log with the Conditional learning identity without being blocked; Analysis & Learning produces new page types by combining these records with the existing rules (Analysis & Learning).
• Where it is opened WAF Management → Learning Methods card; mode card → Analysis & Learning
• Requirement Modify permission in the WAF area; the identity of the test team; since applying the analysis cannot be undone, an Export backup beforehand
• Scope The selected host groups

Step-by-Step Configuration

1

Adım 1 Open the Page Types screen

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened and the N page types row on the mode card of the host group is clicked.

2

Adım 2 Take a backup

Export on the Page types toolbar is clicked; all the rules of the host group are downloaded as a JSON file without a confirmation being asked (Can I export a host group's rule set as JSON and move it to another environment; which settings are not included in the file?).

3

Adım 3 Open the Learning Methods window

In WAF Management, the pencil icon of the Learning Methods card in the Learning & Reporting section is clicked.

4

Adım 4 Enter the Conditional Learning identity

The Conditional Learning switch is turned on; the IPv4 addresses of the test team are written, separated by spaces, in the Conditional Learning IP field, or the TR7-ANALYZE value is written in the cookie or header field.

5

Adım 5 Save

Edit is clicked. In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

6

Adım 6 Browse the new version

The test team, carrying the identity, uses all the screens of the new version with a browser. These requests are not blocked and do not fall into Event Logs; they are written to the learning log.

7

Adım 7 Open the analysis window

The Analysis & Learning text on the mode card is clicked; the window titled New Analyze for followed by the vService name opens.

8

Adım 8 Set up the analysis

In the Analyze Date field, the range in which the browsing was done is selected with Custom Date Range. The Ignore old rules switch is brought to the Combine with old rules position (it comes with Create New Rules at every opening). Rule Creating Mode is selected the same as the existing Rule Creation Policy; applying updates this policy to the value in the analysis. If the test team worked from abroad or with a non-browser tool, Country filter, Only browsers and Bot filter are loosened accordingly (The analysis produced very few page types / none at all; which filters should I check?).

9

Adım 9 Start the analysis

Start Analyzing is clicked; the progress is followed from the bar on the mode card.

10

Adım 10 Review the result

When the analysis is finished, the New badge in the Analysis & Learning row is clicked and the Analyze Page Types tab is opened with Edit; it is reviewed that the new paths are present in the tree and that the paths that carry IDs are gathered in a single pattern (What are Dynamic Paths, Excluded Paths and Dynamic Path Exclusions for?). Before applying, the paths that come out of the analysis are reviewed; only the paths for which a WAF rule will be created must remain in the list. It is recommended to select unknown or suspicious paths in the tree and exclude them from the analysis with Exclude from Analysis; these paths are added to the Excluded paths list and do not produce rules.

11

Adım 11 Apply

Apply is clicked and the Are you sure? question is answered with Yes. If Current Page Count exceeds 500, applying is rejected. Applying the analysis cannot be undone with Revert learning; if needed, the file from step 2 is loaded back with Page types → Import.

12

Adım 12 Remove the identity

In the Learning methods window, while the Conditional Learning switch is on, the identity fields are cleared and Edit is clicked. The switch is not turned off in this save: while the switch is off, the change in the fields is not saved, and the saved identity continues to match even while the switch is off.

13

Adım 13 Turn off Conditional Learning

The Learning methods window is opened again, the Conditional Learning switch is turned off and Edit is clicked.

Verification

1Page typesTR7 side

The N page types number on the mode card and the Page types list are examined.

ExpectedThe page types of the new paths are in the list

Method 4: Temporarily Switching the Host Group to Monitor Mode

• How it works In Monitor mode, the rule set is applied as it is and the requests that exceed the threshold are written to Event Logs with the "Not blocked — Monitor mode" decision, but are not blocked (Monitor Mode). During this time, teaching with Method 1 is continued.
• Where it is opened WAF-Protected vServices → vService → WAF Management → the mode card of the host group → Edit
• Requirement Modify permission in the WAF area
• Scope The selected host group

Step-by-Step Configuration

1

Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2

Adım 2 Open the host group window

Edit on the mode card of the host group that serves the new version is clicked; the host group edit window opens.

3

Adım 3 Select Monitor mode

Monitor Mode is selected in the WAF Mode field.

4

Adım 4 Save

The Edit button of the window is clicked; the mode takes effect without a separate deployment step. In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize. When the false positives are cleared, the group is returned to Blocking Mode with the same steps (How do I use Monitor mode to weed out false positives?).

Monitor mode does not block real attacks either

In Monitor mode, the attacks the rule catches are passed too; the duration is kept short.

Verification

1Mode cardTR7 side

The mode card in WAF Management is examined.

ExpectedMonitor mode · Attacks are only monitored, not blocked

Common Mistakes

• Switching the whole host group to Learning mode. In Learning mode, the rules are not applied and no Event Logs record is formed; for new pages, teaching or Teacher is enough (If the application changes after moving to Blocking, should I go back to Learning mode?).
• Running the update analysis with Create new rules. The existing rule list is ignored and Any paths is restricted again; for an update, Combine with Old Rules is selected.
• Not taking an Export before applying the analysis. Applying the analysis cannot be undone; the way to return to the previous state is the Export file.
• Teaching value pattern violations in bulk. Teaching removes the pattern from the field; to preserve the pattern, the rule is edited.
• Leaving the Teacher or Conditional learning identity on. The requests that carry the identity are not blocked; when the work is finished, Teacher is turned off, while the Conditional learning identity fields are cleared and saved while the switch is on (the saved identity matches even while the switch is off).
• Teaching the next versions in production as well. A WAF step is added to the release process: How should I include the WAF in my application release process?

How it is done in the documentation

• Learning Methods · Rule Creation Policy
• Analysis & Learning · Ignore Old Rules
• Event Logs · Allow / Teach
• Event Logs · Attacks Tab
• WAF Rules · Runtime: Rule Matching

Related questions

• False positives increased after an application update; what should I do?
• What is the difference between "Create New Rules" and "Combine with Old Rules" in the analysis?
• How is a request that matches no page type inspected?
• How should I include the WAF in my application release process?