I get 504 Gateway Timeout; which timeout should I check?

Short answer

A 504 generated by TR7 occurs when the Backend Service does not send the response headers within the Server Timeout duration; the value to check is the Server Timeout in the Timeouts profile assigned to the vService; if no profile is assigned, the default Timeouts profile is applied and the Server Timeout is 90 seconds. The Server Timeout is not the total processing time but the longest time the server stays silent without responding; reports, exports and heavy queries that are always cut off at the same second are the typical symptom. First, it is verified that the page is TR7's "Response timeout" page and that the Termination State in Debug Mode Logs is During response headers · Server timeout. The Timeout in the Health Check profile and the HTTP Request Timeout do not produce a 504; the Queue Timeout produces a 503, and the Connect Timeout produces an error at the connection stage.

Problem: Long-Running Requests Are Cut Off Halfway

Long operations such as a report, an export or a heavy query are always cut off with a 504 after about the same time. It must be determined whether the 504 is produced by TR7, by the application or by a device in front of TR7, and which timeout is responsible. The Timeouts profile exists only on HTTP and L7 TCP vServices and is applied to all the traffic of the vService.

Which Timeout?

Observation	Responsible	Where it is set
TR7's "Response timeout" page; Termination state During response headers · Server timeout; the duration is always close to the Server Timeout	Server Timeout (default 90 s)	Timeouts profile
408 and the "Request is timeout (before completed)" page; During request · Client timeout	HTTP Request Timeout (default 30 s); it is not a 504	Timeouts profile
503; While queued · Server timeout	Queue Timeout (default 60 s) and the Connection Limit value of the Backend Service	Timeouts profile · Backend Service
While connecting to backend · Server timeout	Connect Timeout (default 20 s) and Max. Failed Connection Retries	Timeouts · vService Limits profile
The 504 page is not TR7's; in the TR7 record, During response headers · Client aborted, or no request at all	The CDN, firewall or proxy in front of TR7	That device
504; Termination state Normal, Backend service = the name of the server	The own 504 of the application or of the proxy behind it	Application side
The WebSocket connection drops after a certain time, no HTTP code	Tunnel(Websocket) Timeout	My WebSocket connections drop after a certain time; what should I do?

Meaning of the values: What do the values in the Timeouts profile (HTTP Keepalive Timeout, HTTP Request Timeout, Connect, Server, Client, Queue, Tunnel(Websocket) Timeout, Client FIN, Server FIN) mean?

Where Is It Monitored?

Screen	Path	What it shows
HTTP Logs	Load Balancer vServices → vService → Logs & Tracking	Requests that received a 504, the Backend service and the path
Debug Mode Logs	In the same tab (while the vService is in Debug Mode)	The Termination state and the Performance Info in the Request tab
Backend Response Time Distribution	vService → Dashboard	Response times per Backend Service

Solution: Two Methods

The two methods are applied in order: first the source of the 504 and the responsible timeout are determined, then the Server Timeout is adjusted.

Your situation	Suitable method
Who produces the 504, and its duration	Method 1: Determining the source of the 504
Adjusting the Server Timeout	Method 2: Timeouts profile → Server Timeout

Method 1: Determining the Source of the 504 Recommended

• How it works The Termination State and the cut-off time show the source of the 504 and the responsible timeout.
• Where it is opened Client · vService edit form → Status = Debug Mode; records in Logs & Tracking → Debug Mode logs
• Requirement HTTP vService; under heavy traffic, only the problematic client is recorded with a condition selected in the Select Conditions field of the Debug group.
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Measure the code and the duration

The code and the duration are measured from the client; if the duration is close to the same value each time, a timeout is in effect:

curl -o /dev/null -s -w '%{http_code} %{time_total}\n' https://www.example.com/rapor

2

Adım 2 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

3

Adım 3 Select Debug Mode

Debug Mode is selected in the Status field (the same setting is the Debug box in the Details panel); in heavy traffic, the condition that selects the problematic client is selected from the Select conditions field in the Debug group (What does Debug Mode record; how long should I keep it on?).

4

Adım 4 Save

The Save button is clicked and the long operation is run again.

5

Adım 5 Decide according to the Termination state

On the Logs & Tracking → Debug Mode Logs screen, the Termination State value of the 504 record is matched with the Which timeout? table; a high Tr time in the Performance Info table of the Request tab shows that the time passed while waiting for the server (The site is slow; how do I tell whether the problem is in TR7 or in the backend?).

6

Adım 6 Reopen the vService edit form

When the examination is finished, on the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

7

Adım 7 Turn off Debug Mode

Enabled is selected in the Status field. Debug Mode does not turn off on its own.

8

Adım 8 Save

The Save button is clicked.

Verification

1RecordTR7 side

The Termination state value of the 504 record is checked.

ExpectedDuring response headers · Server Timeout

If the value is different, the responsible timeout is the row in the table; increasing the Server Timeout does not change the result.

Method 2: Timeouts Profile → Server Timeout

• How it works The Server Timeout comes from the Timeouts profile assigned to the vService and is applied to all the traffic of the vService. For the long operation, a separate profile is created and assigned only to the related vService.
• Where it is openedLoad Balancer → Profiles → Timeouts · vService form → Details → Timeout
• Requirement The real duration of the long operation, measured directly on the server.
• vService type HTTP · L7 TCP

Step-by-Step Configuration

1

Adım 1 Find the assigned profile

In the edit form of the vService, the Timeout field in the Details panel is checked; if no profile is selected, the default values (Server Timeout 90 s) are used. The Used in tab in the preview of the selected profile shows which other vServices use the profile; if a shared profile is increased, all of them are affected.

2

Adım 2 Click Add on the Timeouts screen

On the Load Balancer → Profiles → Timeouts screen, the Add button is clicked.

3

Adım 3 Enter the Server Timeout value

A name (e.g. rapor-uzun) is written in the Timeout Profile Name field; a value (in seconds, 0–60,000) greater than the longest time the operation stays silent on the server is entered in the Server Timeout field. The other fields can be left with their default values.

4

Adım 4 Save the profile

The Add button in the form is clicked.

5

Adım 5 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

6

Adım 6 Select the profile

In the Details panel, the Timeout box is checked and the new profile is selected (How to add to a vService). Details: Why is the request cut off during long operations (report, file upload); which timeout should I increase?

7

Adım 7 Save

The Save button is clicked. The change is applied automatically; in a cluster, it is also transferred to the peer device with the default automatic sync.

The timeout of the CDN, firewall or proxy in front of TR7 must not be shorter than TR7's Server Timeout value; if it is shorter, the client sees the 504 of that device.

Verification

1OperationClient side

The long operation is run again: curl -o /dev/null -s -w '%{http_code} %{time_total}\n' https://www.example.com/rapor

Expected200 and the real duration of the operation

Common Mistakes

• Increasing the Timeout in the health check profile. This value is only the time the health check waits for a response; it does not affect traffic.
• Thinking the Server Timeout is the total request time. The value is the longest time the server stays silent without responding.
• Increasing the HTTP request timeout. This value affects the 408, not the 504.
• Increasing the shared profile used by all the vServices. A separate profile is created for the long operation (Operational best practices).
• Overlooking the timeout of the device in front of TR7. If that device gives up sooner, it produces the 504.

How it is done in the documentation

• Timeouts · Server Timeout
• Timeouts · The Scope of the Timeout Values
• Timeouts · How to Add to a vService?

Related questions

• Why is the request cut off during long operations (report, file upload); which timeout should I increase?
• My WebSocket connections drop after a certain time; what should I do?
• How do I find out from the logs why a request failed (termination state)?