What is Smart Sampling; how does it affect the learning log?

Short answer

Smart Sampling (learning log) is the setting that thins out the learning log on a request shape basis: the first N samples of the same shape (method, host, path, query parameter names and value length ranges, body field names, header names, matching signatures, response code) are written in full, and the rest are sampled randomly. It is on by default; N is the Sampling: Full Copies per Shape field, its default is 50 and it takes a value between 5 and 10000. When the quota is filled, approximately one of every 20 requests is written with the default; as N grows, sampling becomes more frequent (most frequently one in 5, most sparsely one in 100). A shape seen for the first time is written in full; however, if a large number of new shapes are formed within one minute (e.g. a scan that produces parameter names), during that minute new shapes are also written at approximately one in 100. The shape memory is kept separately on each device, and a shape that is not seen for 21 days is forgotten. If it is turned off, every request within the learning scope is written in full (recommended only on low traffic) and the Learning tab on the Dashboard is not visible.

How Does It Work?

Field	Default	Effect
Smart Sampling (learning log)	Checked	While it is unchecked, every request is written; the Learning tab is not visible
Sampling: Full Copies per Shape	50 (5–10000)	Number of samples written in full per shape; after the quota, approximately one in every 1000/N requests is written (one in at least 5 and one in at most 100)

• The setting is vService-wide and is effective only on an HTTP vService. Since only host groups in Learning mode or requests that carry a Conditional Learning identity are written to the learning log, the setting is of use only in this situation.
• The shape carries not the values themselves but their length ranges: requests that come to the same page with different values of a similar length are the same shape. A different attack signature or a different response code creates a new shape.
• Shape counts are not reset when the configuration is refreshed or when sampling is turned off and on; each device keeps its own count.
• The ratio of written and skipped rows is seen in the Learning Log Sampling panel (Written, Skipped) on the Learning tab.
• Managing it in terms of disk and CPU: How are the disk and CPU load of the learning log kept under control (Smart Sampling)?

The Setting in TR7

Left Monitor Panel → WAF-Protected vServices → vService → WAF Management tab → in the Learning & Reporting section, the pencil icon of the Log Settings card is clicked; in the Log settings window, Smart Sampling (learning log) is checked, Sampling: Full Copies per Shape is entered and Edit is clicked. There is no separate deployment step. While sampling is on, the card shows the sentence "The learning log is smart-sampled — the first 50 of each request shape are written in full".

The same setting with WEB-CLI:

waf options update poolId <vService> --smartSampling true --smartSamplingCopies 50

Common Misconceptions

• "Sampling misses new pages." A new shape is written in full when it is first seen; new shapes are also thinned out only when a large number of new shapes are formed in one minute.
• "Turning off sampling provides better learning." Every repetition is written in full and disk consumption increases; the Learning tab is not visible either.
• "The shape memory is shared by the two devices in the cluster." Each device keeps its own shape memory.

How it is done in the documentation

• Log Settings · Smart Sampling (Learning Log)
• Log Settings · Sampling: Full Copies per Shape
• Dashboard · Learning Log Sampling
• WEB-CLI waf options · Rule Management and Sampling

Related questions

• How are the disk and CPU load of the learning log kept under control (Smart Sampling)?
• Which tabs does the WAF Dashboard have (Summary, Attack Analysis, Learning); why is the Learning tab not always visible?
• What is the learning log; where is it kept?