How do I find out from the logs why a request failed (termination state)?

Short answer

How the request ended is seen in the Termination State field, which in the interface is found only in the Debug Mode Logs source; this field is not present on the HTTP logs and TCP logs screens. After the HTTP vService is put into Debug Mode, the Termination state filter and column of the Debug Mode Logs source are used in the Logs & Tracking tab; the value is written as the stage and the reason (e.g. During request · TR7 intercepted), and Normal appears for a request without problems. The Retries filter shows the retries made to the Backend Service, and the Security Action column shows the security module that cut the request (Filter). On the log server, the same information is found in the tstate field of the TR7 JSON Format line, and in an L7 TCP vService also in the Standard Log Format line as a raw code.

Problem: The Reason for the Failed Request Is Not Visible

A user reports a 502, 503, 504 or a dropped connection, but the status code alone does not tell where and why the request ended. It must be known whether the request was cut by the client, whether the Backend Service responded late, or whether TR7 responded.

Solution: Two Methods

The two methods are alternatives to each other: Debug Mode Logs is used for per-request examination in the interface, and the log server for long-term and central examination.

Your situation	Suitable method
In the interface, per request (HTTP vService)	Method 1: Debug Mode Logs → Termination state
On the log server (HTTP or L7 TCP vService)	Method 2: Log server

Method 1: Debug Mode Logs Recommended

• How it works The debug record keeps the timing and termination state of each request; Termination state shows the stage and the reason together.
• Where it is opened Left Monitor Panel → vService → Logs & Tracking → Debug Mode Logs
• Requirement HTTP vService with Debug mode on
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the HTTP vService is right-clicked and Edit is selected.

2

Adım 2 Select Debug mode

Debug Mode is selected in the Status field, or Debug is checked in the Details panel. Under heavy traffic, only the problematic client or path is selected with the condition field under Debug (conditional debug).

3

Adım 3 Save

Save is clicked. Recording starts from the requests that arrive after this moment.

4

Adım 4 Reproduce the problem

The error is reproduced after Debug mode is turned on; earlier requests are not recorded.

5

Adım 5 Open the vService screen

In the Left Monitor Panel, Load Balancer vServices is selected from the category selector and the vService is clicked (accessing the screen).

6

Adım 6 Open the Logs & Tracking tab

The Logs & Tracking tab is clicked.

7

Adım 7 Select the Debug mode logs source

Debug Mode Logs is selected from the source selector; the records are fetched with the selection. The records are not refreshed automatically; Refresh is used for requests that arrive later (Refresh).

8

Adım 8 Filter

In the filter strip, values other than Normal are selected in the Termination State card; if needed, it is narrowed with Status Class (e.g. 5xx) and Retries.

9

Adım 9 Turn on the columns

With Columns, the Termination State and Security Action columns, hidden by default, are turned on.

10

Adım 10 Examine the record

The row is clicked; Termination State and Response Source are read in the Summary tab, and Performance Info in the Request and Response tabs of the preview that opens on the right.

11

Adım 11 Open the edit form again

When the examination is finished, the vService is opened again with Edit.

12

Adım 12 Turn off Debug mode

Status = Enabled is selected; Debug mode does not turn off on its own.

13

Adım 13 Save

Save is clicked.

Reading the values: The value first writes the stage, then the reason (Termination state values).

Stage	Reason
During request · While queued · While connecting to backend · During response headers · During data transfer · During final data · While tarpitted	Client aborted · Server aborted/refused · TR7 intercepted · Handled by TR7 · Resource exhausted · Internal error · Server down · Server shutdown · Admin killed · Client timeout · Server timeout

• TR7 intercepted or Handled by TR7: the request was answered by TR7 before going to the Backend Service (e.g. an action or a security module); the module that made the decision is in the Security Action column.
• During response headers · Server timeout: the Backend Service did not send the response headers in time; the Tr time in the Performance Info table of the Request tab is checked.
• While connecting to backend and a Retries value greater than zero: a connection to the Backend Service could not be established.
• Client aborted: the client cut the connection itself.

Verification

1RecordTR7 side

A 503 is produced by disabling all the Backend Services of a vService used for testing, and the record is opened.

ExpectedA Termination state value other than Normal

Method 2: Log Server

• How it works The log profile sends the request lines to the log server; the TR7 JSON Format line carries the termination state in the tstate field and the retries in the retries field as raw codes.
• Where it is openedLoad Balancer → Profiles → Logs
• Requirement Log server; LOG on in the vService
• vService type HTTP · L7 TCP

Step-by-Step Configuration

1

Adım 1 Open the log profile form

The profile form is opened with Load Balancer → Profiles → Logs → Add (how to add).

2

Adım 2 Enter the target and format

Name is entered; in the Log Addresses row, the protocol (UDP or TCP) and the address and port of the log server are entered. TR7 JSON Format is selected as the Line Format. In an L7 TCP vService, the Standard Log Format line also carries the termination state. In manual formats, the HTTP termination state and TCP termination state Smart Variables are used. Details: How do I send logs to a SIEM (Syslog)?

3

Adım 3 Save the profile

Add in the form is clicked.

4

Adım 4 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is right-clicked and Edit is selected.

5

Adım 5 Check LOG and select the profile

LOG is checked in the Details panel and the created profile is selected in the Log Profile field (how to add it to a vService).

6

Adım 6 Save

Save is clicked. In a cluster setup, with the default auto-sync, the change is also sent to the peer device; if Manual Sync Mode is on, it is sent manually with Synchronize.

Verification

1Log serverExternal system

A request is sent to the vService and the incoming line on the log server is checked.

Expectedtstate field filled (starts with "--" for a request without problems)

In an HTTP vService, tstate has four characters: the first character is the reason, the second character is the stage, and the last two are the state of the persistence cookie; if no persistence cookie is used, the value is "----" for a request without problems. In an L7 TCP vService, tstate is two characters that carry only the reason and the stage. For a request without problems, the first two characters are "--".

Common Mistakes

• Looking for Termination state on the HTTP logs or TCP logs screen. In the interface, this field exists only in Debug Mode Logs.
• Looking for Debug mode in an L7 TCP vService. Debug mode exists only in the HTTP vService; for L7 TCP, the log server method is used.
• Not finding an old record because of the line limit. Filters work only on the fetched records; the Line Limit in the toolbar (default 1000, at most 50000) is increased and Refresh is clicked.
• Sending the Apache format or CEF Format to the log server and expecting the termination state. In an HTTP vService, these formats do not carry the termination state.
• Not examining the Security action separately. TR7 intercepted shows that the request was answered by TR7; for the module that made the decision: I am blocking legitimate users by mistake; how do I find out?

How it is done in the documentation

• Debug Logs · Termination State Values
• Debug Logs · Summary Tab
• TR7 JSON · tstate (Connection Termination State)
• L7 TCP Standard · Termination State (%ts)

Related questions

• What does Debug Mode record; how long should I keep it on?
• The site is slow; how do I tell whether the problem is in TR7 or in the backend?
• I am blocking legitimate users by mistake; how do I find out?