Smarter by Design.Simpler for Business
TR7Enterprise Application Delivery & Security Platform
What happens to a request if the scan exceeds a certain time (2-second scan limit)?
Short answer
The time reserved for the WAF scan of a request is 2 seconds; this time is not a setting and cannot be changed from the interface or from WEB-CLI (how it works). Ordinary scans take milliseconds. What happens to a request that cannot be completed within the time is determined by the mode of the host group: in a group in Blocking Mode, the request is blocked so that an uninspected request does not reach the Backend Service, and the client receives the ordinary blocking response (Blocking Status Code, default 418, and Blocking Page). In the groups in Learning and Monitor mode, and in the requests that come with a Teacher identity or a Conditional Learning identity, the request is passed. The only exception: if the scan is cut off while the WAF rules are being loaded and before it is determined which host group the request belongs to, as long as there is at least one group in Blocking Mode on the vService, the request (except for the Teacher and Conditional Learning identities) is blocked even if it comes to a group in Learning or Monitor mode (WAF Mode · Blocking Mode). While Fail-open (off by default) in the Request Body Limits window is on, these requests are passed without being scanned in Blocking mode as well; since an attacker who deliberately prolongs the scan can bypass WAF, it is turned on only if these blocks affect the service. The blocked request is not written to Event Logs; while Debug Mode is on, it is seen in Debug Mode Logs with the Attack blocked — scan timed out decision and Triggered Rules 369999, and in the WAF information sent to the log server with rule 369999 and a wafTime value of 2000. The solution is to find the request whose scan takes long and to reduce the scanned data (Max. Body Size to Process(kB), the body inspections of the page type); Fail-open is the last resort.
How Does It Work?
- Entering inspection: The request enters WAF inspection. The requests that are counted as static with Do Not WAF Static Contents and the requests that match Whitelist Condition do not enter inspection, so they are not caught by the time limit either.
- Waiting for the body: The body is received before the scan starts (Wait for Body for a Time). Waiting for the body to arrive from the network does not count toward the 2-second time.
- Scan: The path, query, header and body values are scanned against the signatures; JSON, XML and multipart bodies are parsed. The main factor that increases the time is the amount of scanned data: the number of parameters, the body size and the JSON or XML depth (How do large request bodies (file uploads) affect WAF performance?).
- If the time runs out: If the scan does not finish within 2 seconds, it is cut off and WAF cannot calculate a score for this request.
- Result: The fate of the request is determined by the table below. The blocked client receives the same response as an ordinary WAF block; the response contains no information about the timeout.
Result by Mode
| State of the request | Fail-open off (default) | Fail-open on |
|---|---|---|
| Host group in Blocking mode | Blocked: Blocking Status Code (default 418) and Blocking Page; if there is a Conditional Blocking Pages record whose condition matches, its page and code | Passed without being scanned |
| Host group in Monitor mode | Passed | Passed |
| Host group in Learning mode | Passed | Passed |
| Request that comes with a Teacher or Conditional Learning identity (in every mode) | Passed | Passed |
| The scan was cut off while the WAF rules were being loaded and before the host group was determined; there is at least one group in Blocking mode on the vService (in every mode, except for the Teacher and Conditional Learning identities) | Blocked | Passed without being scanned |
How Is the Blocked Request Recognized?
If the user receives the blocking page (default 418) but a search with the Request ID in Event Logs returns no record, the first cause to be examined is the scan time limit.
| Surface | Blocked request (Blocking mode, Fail-open off) | Passed request |
|---|---|---|
| Event Logs · Global WAF Event Log · WAF Report | No record | No record |
| WAF Inspection Distribution · WAF Blocked Requests · notifications | Not counted | Not counted |
| Learning record | - | Not written (in Learning mode as well) |
| Debug Mode Logs (if Debug mode is on at the time of the event) | Security Action = WAF · Attack blocked — scan timed out · 1 attacks (in the filter WAF · Timeout); in the WAF box on the Summary tab, Verdict = Attack blocked — scan timed out, Triggered Rules = 369999 | No WAF information |
| HTTP Logs | Status code = Blocking Status Code | Ordinary response |
| Log server (CEF, TR7 JSON, TR7 WAF Log) | In wafInfo, rule 369999, score 100, wafTime 2000 (fixed values, not a measurement) | No WAF information |
Diagnosis steps: Why does a request that received 418 not appear in Event Logs? · Content of the record: I see rule 369999 in Debug Mode Logs or SIEM records; what is it?
What to Do?
- Finding the request: In Debug Mode Logs, the Path, body size and Content-Type values of the Attack blocked — scan timed out rows are compared; a record that repeats on the same endpoint shows that the scan of the request on that endpoint takes long.
- Reducing the scanned data: The scanned body is limited across the vService to the larger of the Max. Body Size to Process(kB) value (4096 when WAF is turned on) and the Buffer Size(kB) value in the vService Limits profile. A body that exceeds this limit is not parsed and is not scanned; what happens to it is determined by the Block Unparsed Contents setting on the Raw tab of the page type (What happens to request bodies that exceed the Max. Body Size to Process(kB) limit or do not complete in time?). The limit is made smaller by lowering the larger of the two values; since, if the buffer changes, the vService is restarted together with the save (the interface does not show a separate warning for this), the change is made in a maintenance window (How do large request bodies (file uploads) affect WAF performance?). If only one endpoint is affected, first the unnecessary body inspections are turned off in that page type (Large POST requests are blocked or cut off; why?, File uploads are being blocked by the WAF; how do I fix it?).
- Excluding the trusted source from inspection: If the request comes only from a trusted source to a specific endpoint, it can be left out of inspection with Whitelist Condition (Can I exempt certain IPs or requests from WAF inspection entirely (Whitelist Condition)?).
- Fail-open only if needed: If the blocks affect the service and the request cannot be reduced, Fail-open is turned on (below).
Fail-open in TR7
| Where | Left Monitor Panel → WAF-Protected vServices → vService → WAF Management → the pencil icon of the Request Body Limits card in the Inspection & Blocking section → Fail-open switch → Edit. The switch is below the Max. Body Size to Process (kB), Max. Body Size to Log (kB), Wait for Body for a Time and Time to Wait Body fields |
| Default | Off |
| What it does | In Blocking mode, instead of blocking a request whose scan cannot be completed within the time limit, it passes the request without it being scanned. In Learning and Monitor modes, these requests are already passed; in these modes, it only removes the blocking of a scan that is cut off while the rules are being loaded and before the host group is determined |
| Scope | Across the vService; it is applied to all the Blocking mode host groups on the vService |
| Caution | While it is on, a request that deliberately prolongs the scan can bypass WAF. It is turned on only if these blocks affect the service and the request cannot be reduced |
| Seeing the state | The summary of the Request Body Limits card does not show Fail-open; whether it is on or not is seen in the edit window |
| WEB-CLI | waf options update poolId <vService> --failOpen true (false to turn it off); it is written to multiple vServices with waf options update-many (bulk update) |
| Permission | Modify permission in the WAF area |
| Cluster | Like the other WAF settings, it is sent to the peer device with the default auto-sync; if Manual Sync Mode is on, it is sent with Synchronize |
Common Misconceptions
- "The time limit is a setting; the problem is solved by increasing it." The time is 2 seconds and cannot be changed. For heavy legitimate requests, the setting is Max. Body Size to Process(kB); if needed, Fail-open is turned on.
- "In Learning or Monitor mode, the timeout blocks the request." In these modes, the request is passed; the timeout blocks only in Blocking mode while Fail-open is off. The only exception is the scan being cut off while the rules are being loaded and before the host group is determined: if there is a group in Blocking mode on the vService, the request is blocked even if it comes to a group in these modes.
- "Slow uploads time out." Waiting for the body to arrive from the network does not count toward the scan time.
- "Fail-open turns off all WAF inspection." It affects only the requests whose scan cannot be completed within the time limit; the other requests are inspected and blocked as before.
- "A
wafTimeof 2000 is a measured two-second scan." This value is the fixed marker of the timeout record; it is excluded from the duration calculations in the SIEM (How do I measure the time the WAF adds to a request; for which requests is it visible?).
How it is done in the documentation
