FAQ
Smarter by Design.Simpler for Business
TR7Enterprise Application Delivery & Security Platform

English

Türkçe

English

Türkçe

Appearance

Advanced

Why does a request that received 418 not appear in Event Logs? ​

Short answer

The most likely cause is that, in a host group in Blocking Mode, the WAF scan of the request could not be completed within the 2-second time limit. The uninspected request is blocked so that it does not reach the Backend Service, and the usual blocking response (Blocking Status Code, default 418) is returned; however, since the scan did not reach the stage at which it writes a record, the request does not appear in Event Logs, in Global WAF Event Log or in the WAF Inspection Distribution panel. The trace remains in the Debug Mode Logs record (Security Action WAF · Attack blocked — scan timed out, Triggered rules 369999) and in the WAF information sent to the log server (wafTime 2000); Debug Mode does not keep records retroactively, it must be on at the time of the event. In groups in Learning and Monitor mode, this block does not occur. The time limit cannot be changed; the solution is to find the request whose scan takes long and to reduce the scanned body (Max. Body Size to Process(kB)). If the blocks affect the service and the request cannot be reduced, Fail-open in the Request Body Limits window is turned on; while it is on, these requests are passed without being scanned. Other possibilities: the record being hidden from the list with WAF Log Suppression Condition, the Date Range, Line Limit, Request Unique ID or Log Filter scope on the Event Logs screen leaving the record out, the request going to another vService, and the 418 coming from a layer other than the WAF.

Problem: There Is No Record of the Request That Received 418 ​

The user reports receiving 418 and the Request ID on the blocking page, but a search made with Request Unique ID in Event Logs returns no record. Since there is no record, the rule that caused the block cannot be seen and teaching with Allow / Teach cannot be done (My users are getting a "418" error; what is happening?).

Possible Causes ​

CauseHow to tell it apart
The WAF scan could not be completed within the 2-second time limit (only in Blocking Mode, while Fail-open is off; apart from one exception that can occur while the rules are being loaded)In Debug Mode Logs, Security Action = WAF · Attack blocked — scan timed out · 1 attacks; in the WAF box of the row, Triggered rules = 369999
The record matches the WAF Log Suppression ConditionThe Log Settings card in WAF Management has the "WAF logging is suppressed for requests matching:" line; in Debug Mode Logs, Security action is WAF · Attack blocked · … (not a timeout)
The Event Logs scope leaves the record outThe time range that appears in the title of the server filter tab does not cover the time of the event, Line Limit has been reached, or a Log Filter is defined
The request went to another vServiceThe same Request ID is found in Global WAF Event Log
The 418 does not come from the WAF (418 is selected in the Load Balancer Block action, or the Backend Service returns 418)There is no WAF box in Debug Mode Logs; in HTTP Logs, Backend Service is TR7 or the name of the server

Definitive rules:

  • An Event Logs record is written only at the end of the scan and is formed only in two cases: when the score reaches the blocking threshold, or when a signature in Monitor state is triggered. A request whose scan is cut off at the time limit does not reach this stage. In a host group in Learning Mode and in a request that comes with the Conditional Learning identity, no Event Logs record is formed at all.
  • The timeout block occurs only in a host group in Blocking mode and while Fail-open in the Request Body Limits window is off (default). In groups in Learning mode and Monitor Mode, in requests recognized by Teacher and requests that come with the Conditional learning identity, and while Fail-open is on, a request whose scan could not be completed is passed (the only exception for groups in Learning and Monitor mode: if the scan was cut off while the rules were being loaded and before the host group was determined, and the vService has a group in Blocking mode, the request is blocked); this request leaves no WAF trace anywhere, including Debug Mode Logs.
  • The 2-second duration is not a setting; it cannot be changed from the interface or from WEB-CLI. The duration counts only the scan itself; waiting for the body to arrive from the network does not count toward this duration (What happens to a request if the scan exceeds a certain time (2-second scan limit)?).
  • The requests that match the Whitelist Condition and the requests counted as static content do not enter the WAF at all, so they do not time out either.
  • A request that times out is not counted in any series (Whitelisted, Clean, Blocked, Monitor, Teacher, Learning) of the WAF Inspection Distribution panel.
  • A suppressed record continues to be written to the disk; it is not shown in the Event Logs and Global WAF Event Log lists.

Where Does the Trace Remain on a Timeout? ​

SurfaceRequest blocked by a timeout
Event Logs · Global WAF Event LogDoes not appear
WAF Inspection Distribution · WAF Blocked RequestsNot counted
Debug Mode Logs (if Debug mode is on at the time of the event)Security action = WAF · Attack blocked — scan timed out · 1 attacks; in the WAF box on the Summary tab, Verdict = Attack blocked — scan timed out, Attacks = 1, Triggered rules = 369999
HTTP Logs (if LOG is on in the vService)Status code 418 (or the selected Blocking Status Code)
Log server (CEF, TR7 JSON, TR7 WAF Log formats)In wafInfo, rule 369999, score 100, wafTime 2000: I see rule 369999 in Debug Mode Logs or SIEM records; what is it?

How to Diagnose? ​

  1. On the server filter tab of Event Logs, Date Range is widened to cover the time of the event, a defined Log Filter, if any, is removed with Remove Filter, the Request ID is written in the Request Unique ID field and Save & Get Logs is clicked (server filter). If no record comes out, the same search is made on the Configuration Menu → WAF → Global WAF Event Log screen (How do I find a specific request by its Request Unique ID?).
  2. Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened; if the Log Settings card in the Learning & Reporting section has the "WAF logging is suppressed for requests matching:" line, whether the condition covers the request is examined.
  3. If there is still no record, on the Load Balancer → Load Balancer vService List screen, Status = Debug mode is selected in the edit form of the vService and Save is clicked. Under heavy traffic, the recording is narrowed to the IP of the user with the condition field under Debug (conditional debug). Records are formed only for the requests that come after this moment. Turning on Debug mode requires modify permission in the Load Balancer area, and viewing Debug Mode Logs requires monitoring permission in the Load Balancer area; for a user who has only WAF permission, these screens do not open (Logs & Tracking).
  4. The user is asked to repeat the request.
  5. Left Monitor Panel → Load Balancer vServices → vService → Logs & Tracking → Debug Mode Logs is opened. On the Security action tab of the Filter strip, WAF · Timeout is selected, or the Security action column, which is hidden by default, is added from the Columns menu.
  6. The row is clicked; if Verdict = Attack blocked — scan timed out and Triggered Rules = 369999 are seen in the WAF box on the Summary tab, the request was blocked by a scan timeout. The Path value, the body size and the Content-Type header of the request are noted; since usual scans take milliseconds, a timeout usually points to a request whose scan takes unusually long (a very large or deep body, a large number of parameters).
  7. If there is no WAF box in the row, the block did not come from the WAF: Legitimate users are being blocked; how do I find which rule is blocking them?
  8. When the examination is finished, Status is set back to Enabled and saved.

Solution: Three Methods ​

The timeout record cannot be moved to Event Logs and the time limit cannot be changed. The permanent solution is to find the request whose scan takes long and to reduce the scanned body (Method 1); if this cannot be done and the blocks affect the service, Fail-open is turned on (Method 2). If a suppression condition hides the record, the condition is narrowed (Method 3). The methods address different causes; for a timeout, Method 1 is tried first.

Your situationSuitable path
The timeout occurs in requests with large bodies; the scanned body can be reduced across the vServiceMethod 1: Reducing the scanned body
The timeout occurs only in large requests (POST, JSON) at a specific endpointLarge POST requests are blocked or cut off; why?
The timeout occurs in file uploadsFile uploads are being blocked by the WAF; how do I fix it?
The timeout occurs only at a specific endpoint that receives requests from a trusted sourceCan I exempt certain IPs or requests from WAF inspection entirely (Whitelist Condition)?
The blocks affect the service and the request cannot be reducedMethod 2: Fail-open
The record is not listed because of WAF Log Suppression ConditionMethod 3: Narrowing WAF Log Suppression Condition
The time limit itself and its result according to the modeWhat happens to a request if the scan exceeds a certain time (2-second scan limit)?
  • How it works The scanned body is limited by the larger of the Max. Body Size to Process(kB) value and the Buffer Size(kB) value in the vService Limits profile. When the limit becomes smaller, large bodies are not parsed and not scanned; what happens to these bodies is determined by the Block Unparsed Contents setting on the Raw tab of the page type.
  • Where it is opened Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → Inspection & Blocking → Request Body Limits card
  • Requirement Modify permission in the WAF area. If Buffer size(kB) in the vService Limits profile is larger, the limit stays at that value; it is lowered in that profile as well (if the profile is also assigned to other vServices, they are affected too)
  • Scope The whole vService; all host groups and page types
  • Request whose body exceeds the limit If Block Unparsed Body is selected in the page type, it is scored with a Block Unparsed Contents violation and blocked in Blocking mode; this time, its record is formed in Event Logs. If the body is to pass without being scanned, Allow Unparsed Body is selected in that page type (Large POST requests are blocked or cut off; why?)

Step-by-Step Configuration ​

1
Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2
Adım 2 Open the Request Body Limits window

The pencil icon of the Request Body Limits card in the Inspection & Blocking section is clicked; the Request Body Limits window opens (Edit screen).

3
Adım 3 Lower the limit

A smaller value that covers the usual requests of the application that must be scanned is written in the Max. Body Size to Process(kB) field (1 – 262144; 4096 when WAF is turned on). The body size of the timed-out request noted in Debug Mode Logs must be larger than this value.

4
Adım 4 Save

Edit is clicked; the WAF configuration is regenerated and applied without a separate deployment step. If the new value changes the request buffer (the larger of the two values), the vService is restarted with the save; the interface does not show a separate warning for this, so the change is made in a maintenance window (How do large request bodies (file uploads) affect WAF performance?). In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

Verification ​

1Debug Mode LogsTR7 side

While the vService is in Debug mode, the user repeats the request; the Security action column is examined.

ExpectedNo WAF · Attack blocked — scan timed out row
A request whose body exceeds the new limit appears in Event Logs with a Block Unparsed Contents record in a page type in which Block Unparsed Body is selected.

Method 2: Fail-open ​

  • How it works In Blocking mode, instead of blocking a request whose scan cannot be completed within the time limit, it passes the request without scanning it. The requests whose scan is completed are inspected and blocked as before; since these requests are already passed in Learning and Monitor modes, it has no effect in these modes.
  • Where it is opened Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → Inspection & Blocking → Request Body Limits window → Fail-open (off by default)
  • Requirement Modify permission in the WAF area
  • Scope The whole vService; all the Blocking mode host groups in the vService
  • Caution While it is on, a request that deliberately prolongs the scan can bypass the WAF. It is turned on only if these blocks affect the service and Method 1 cannot be applied
  • WEB-CLIwaf options update poolId <vService> --failOpen true (Body processing); it is written to multiple vServices with waf options update-many

Step-by-Step Configuration ​

1
Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2
Adım 2 Open the Request Body Limits window

The pencil icon of the Request Body Limits card in the Inspection & Blocking section is clicked; the Request Body Limits window opens.

3
Adım 3 Turn on Fail-open

The Fail-open switch below the Max. Body Size to Process(kB), Max. Body Size to Log(kB), Wait for body for a time and Time to Wait Body fields is turned on. When the field is clicked, the interface shows the warning that explains the security effect of the option.

4
Adım 4 Save

Edit is clicked; the setting is applied without a separate deployment step. In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

Verification ​

1Request Body Limits windowTR7 side

The window is opened again with the pencil icon of the Request Body Limits card.

ExpectedFail-open on
The summary of the card does not show the state of Fail-open; the state is seen only in the edit window.
2Debug Mode LogsTR7 side

While the vService is in Debug mode, the user repeats the request; the row is opened.

ExpectedThe response comes from the Backend Service; no WAF · Attack blocked — scan timed out row
While Fail-open is on, a request whose scan cannot be completed leaves no WAF trace on any screen; for this reason, whether timeouts continue can be monitored only while Fail-open is off.

Method 3: Narrowing WAF Log Suppression Condition ​

  • How it works The suppression condition affects only the listing; the later records of the requests removed from the condition appear in Event Logs. Since the suppression is marked at the moment the record is written, the records written before the condition changed do not start appearing in the list.
  • Where it is opened Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → Learning & Reporting → Log Settings card
  • Requirement Modify permission in the WAF area
  • Scope The whole vService; all host groups

Step-by-Step Configuration ​

1
Adım 1 Open the WAF Management tab

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management is opened.

2
Adım 2 Open the Log Settings window

The pencil icon of the Log Settings card in the Learning & Reporting section is clicked; the Log Settings window opens (Log Settings Edit screen).

3
Adım 3 Narrow the condition

In the WAF Log Suppression Condition field, the condition that covers the requests to be examined is removed or narrowed so as to select only the traffic that must be suppressed (e.g. the health check path). The field accepts only HTTP conditions that belong to the request side (How do I stop WAF logs of Health Check or probe traffic (WAF Log Suppression Condition)?).

4
Adım 4 Save

Edit is clicked; the setting is applied without a separate deployment step. In a cluster setup, the change is also sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize.

Verification ​

1Event LogsTR7 side

The user repeats the request; a search is made in Event Logs with the new Request ID (Save & get logs).

ExpectedThe new request is listed
The "WAF logging is suppressed for requests matching:" line on the Log settings card shows the new form of the condition; if the condition has been removed entirely, the line disappears.

Common Mistakes ​

  • Switching the host group to Monitor or Learning mode to remove the timeout block. In these modes, the timeout block does not occur, but all the blocking of the WAF also stops. First, the scanned body is reduced; if needed, Fail-open, which removes only these blocks, is turned on.
  • Turning on Fail-open as the first solution. While Fail-open is on, these requests pass without being scanned and the timeouts appear on no screen; it is not turned on before the request whose scan takes long is found and reduced.
  • Looking for a setting that increases the time limit. The 2-second duration is not a setting; for heavy legitimate requests, the setting is Max. Body Size to Process(kB).
  • Turning on Debug mode after the event and looking for the old request. Debug mode records only the requests that come after it is turned on; the request is made to be repeated.
  • Searching for 369999 in Signature & Rule Library. This ID is not a signature; it is a marker that shows the scan could not be completed within the time limit.
  • Looking for a timeout count on the Dashboard. The requests that time out are not counted in any series of the WAF panels; the only on-screen trace is Debug Mode Logs.
  • Thinking that the suppression condition removes the request from blocking. The condition affects only the listing; the request is still inspected and, if necessary, blocked.
  • Forgetting to turn off Debug mode. Debug mode records every request in detail; when the examination is finished, Status is set back to Enabled (What does Debug Mode record; how long should I keep it on?).

How it is done in the documentation

Related questions