It works when the backend is accessed directly but not through TR7; why?

Short answer

In a direct test, the request goes from the client's network with its own Host header and connection method; through TR7, on the other hand, a new connection is opened to the Backend Service from TR7's route table, with SNAT on Default, using the Interface IP of the interface that leads to the server in that route table and is in the same subnet as the server (or the gateway), and with the port, SSL Service, SNI and Proxy options in the Backend Service settings. The difference lies in one of these points: the server or the application does not allow TR7's IP, the health check shows the server as down, the TLS, port or Proxy setting does not match the server, or an action changes the header or the path. First, the response code returned by TR7 and the connection chain and header differences in Debug Mode Logs are read; then the same request is repeated from WEB-CLI with the route table of the Backend Service selected.

Problem: The Same Application Behaves Differently over Two Paths

When the server is accessed directly, the application works; when it is accessed through TR7, an error, a wrong page or a connection problem occurs. In HTTP and L7 TCP vServices, TR7 terminates the client's connection and opens its own connection to the Backend Service; for this reason, the source address, the network path, the protocol and the headers differ between the two paths. Which of these the difference stems from must be found.

Causes and Solutions

Cause	How to recognize it	Solution
The server or the application does not allow TR7's address	sys-cmd curl from TR7 fails; Retries is greater than zero in Debug Mode Logs	TR7's SNAT and health check source addresses are allowed (What does "Default" in the SNAT field mean; can I choose a specific SNAT IP?, Which IPs should I allow on the server firewall for health checks?)
The health check profile does not match the server	Backend Service red, 503	The profile is corrected according to the server (The server is running, but TR7 shows it as down; why?)
Wrong route table	No access from TR7 to the server	The Route Table field of the Backend Service is corrected (Can my backends be on different networks / in different route tables?)
Port difference	Unexpected port in Connection info	The Address port or Use Frontend Port is corrected (Can the Backend Service port be different from the frontend port (sending a request received on 443 to 8080)?)
SSL service, SNI or certificate verification does not match	The connection is not established, wrong certificate or wrong site	The settings are corrected according to the server's actual protocol
Proxy protocol does not match	The server closes the connection or rejects the request	Proxy is turned on only if the server expects a PROXY header
Host header or action difference	Wrong site or 404; a changed header in the Request tab	The action or the Host is corrected (Why is my Action not triggering?)
Letter case of header names	An old application does not recognize the header	My backend is sensitive to the letter case of header names; does TR7 change them (Keep Header Names / HTTP 1.0 Header Compatibility)?
Large header or cookie	400 from TR7 and the "Invalid or too large request" page	What is Buffer size used for; when should I change it?
The application expects the real client IP or HTTPS information	IP restriction, redirect loop	How do I see the real client IP on the backend?

Where Is It Monitored?

Screen	Path	What it shows
Backend Services list	Load Balancer → Backend Services	The health status of the Backend Service
Debug Mode Logs · Summary Tab	vService → Logs & Tracking → Debug Mode Logs	Connection info chain, Termination state, Retries
Debug Mode Logs · Request Tab	The same record	The headers that TR7 added, deleted and changed
The server's access log	Backend Service side	Whether the request comes from TR7's address

Solution: Four Methods

The methods are applied in order: first the response returned by TR7 is read, then what TR7 sends to the server is seen, and then the same request is repeated from TR7 and the Backend Service settings are compared.

Your situation	Suitable method
Guidance by the code TR7 returns	Method 1: Response code
What TR7 sends to the server	Method 2: Debug Mode Logs
Testing the network path from TR7	Method 3: Testing from TR7 with WEB-CLI
Backend Service settings	Method 4: Backend Service settings

Method 1: Guidance by Response Code Recommended

• How it works The response returned by TR7 shows at which stage the difference occurs.
• Where it is opened Client · request through TR7
• Requirement No additional setting is required.
• vService type HTTP · L7 TCP

Step-by-Step Configuration

1

Adım 1 Determine the code

A response is obtained from the client through TR7 with curl -sv https://www.example.com/, and the next step is chosen according to the code:

TR7's response	Related page
503	Users see "503 Service Unavailable" / "No Available Server"; what should I do?
502	I get 502 Bad Gateway; what are the possible causes?
504	I get 504 Gateway Timeout; which timeout should I check?
403, 418 or a CAPTCHA page	Legitimate users are being blocked; how do I find which rule is blocking them?
Redirect loop	An infinite redirect loop occurred; why?
The session drops	My application keeps sending users back to the login screen (the session is lost); why?
http:// links on the page	The backend application generates http:// in absolute URLs; how do I fix it?
The connection is never established	The vService cannot be reached at all; what should I check step by step?

2

Adım 2 Check the health status

In the Load Balancer → Backend Services list, the health indicator of the Backend Service is checked (Health status indicator). If it is red, the most common reason for it to appear down while direct access works is that the health check profile does not match the server (The server is running, but TR7 shows it as down; why?).

Verification

1CodeClient side

The status line in the curl -sv output is checked.

Expected< HTTP/1.1 503 / 502 / 504 … (the related page is opened)

Method 2: Debug Mode Logs

• How it works The debug record shows to which Backend Service address TR7 forwarded the request and which headers it added, deleted or changed.
• Where it is opened vService edit form → Status = Debug mode; records in Logs & Tracking → Debug mode logs
• Requirement HTTP vService; under heavy traffic, only the problematic client is recorded with a condition selected in the Select Conditions field of the Debug group.
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

2

Adım 2 Select Debug mode

In the Status field, Debug Mode is selected (the same setting is the Debug box in the Details panel). Under heavy traffic, the condition that selects the problematic client is selected in the Select Conditions field of the Debug group (What does Debug Mode record; how long should I keep it on?).

3

Adım 3 Save

The Save button is clicked and the problem is reproduced through TR7.

4

Adım 4 Read the connection chain

On the Logs & Tracking → Debug Mode Logs screen, the relevant record is opened. The Connection Info chain (CLIENT IP→TR7 vService IP→Backend Service IP:Port) in the Summary tab shows to which IP and port the request went; if the chain ends at TR7, the server was never reached. The Termination State and Retries values are also read (How do I find out from the logs why a request failed (termination state)?).

5

Adım 5 Compare the headers

In the Request tab of the same record, the headers that TR7 added, deleted and changed are shown with separate icons; the effect of the Host header and of the path actions is seen here. If the direct test was done with the IP, the server may be serving a different virtual site for the Host header that TR7 forwards.

6

Adım 6 Reopen the vService edit form

When the examination is finished, on the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

7

Adım 7 Turn off Debug mode

Enabled is selected in the Status field. Debug mode does not turn off on its own.

8

Adım 8 Save

The Save button is clicked.

Verification

1RecordTR7 side

After the fix, the Debug record for the same request is opened.

ExpectedConnection info reaches the server, Termination state Normal, Retries 0

Method 3: Testing from TR7 with WEB-CLI

• How it works When the same request is sent from TR7's route table, the network path and the request content are separated from each other.
• Where it is opened WEB-CLI → sys-cmd route-table set · sys-cmd curl
• Requirement A user with User Type = Admin (the sys-cmd tools work only with this user type).
• vService type HTTP · L7 TCP

Step-by-Step Configuration

1

Adım 1 Select the Backend Service's route table

In WEB-CLI, the route table in which the tools will run is selected with the value in the Route Table field of the Backend Service; for the device itself, Host is written in the v-device field, and the route table name is taken from the Tab list:

sys-cmd route-table set v-device Host route-table <route-table>

The current selection is seen with sys-cmd route-table show.

2

Adım 2 Send the request from TR7

The request is sent so that the Host and SNI go correctly; --resolve directs the domain name to the server IP (curl · Things to Know):

sys-cmd curl -sv --resolve www.example.com:443:192.0.2.21 https://www.example.com/

If the server listens on plain HTTP, the Host header is given separately: sys-cmd curl -sv -H "Host: www.example.com" http://192.0.2.21:8080/.

3

Adım 3 Interpret the result

If it also works from TR7, the network path is sound and the difference is in the request content (Methods 2 and 4). If it does not work, the server's firewall or the route in the route table is examined. In the server's access log, it is checked that TR7's address appears and that the application or the server firewall allows this address (Why do my backend servers see all requests as coming from TR7's IP?).

Verification

1ResponseTR7 side

The status line in the sys-cmd curl output is checked.

Expected< HTTP/1.1 200 OK

Method 4: Backend Service Settings

• How it works The port, TLS, SNI, Proxy and route table settings must be the same as what the server expects.
• Where it is openedLoad Balancer → Backend Services → Edit
• Requirement No additional setting is required.
• vService type HTTP · L7 TCP

Step-by-Step Configuration

1

Adım 1 Open the Backend Service edit form

On the Load Balancer → Backend Services screen, the Backend Service is selected and the Edit button is clicked.

2

Adım 2 Check the port and the route table

The Address port must be the same as the port the server listens on; if Use Frontend Port is on, the port in Address is ignored and the port of the request coming to the vService is used (What does the "Use Frontend Port" option do?). Route Table must be the route table through which the server is reached.

3

Adım 3 Check the TLS settings

If the server listens on HTTPS, SSL Service must be on; if it listens on plain HTTP, it must be off (What does the SSL Service option do; what is needed to connect to the backend over an encrypted connection?). If the server expects SNI, SNI is entered (I need to send SNI when connecting to the backend; how do I set it up?); if Certificate Verification is Required, it is checked that the selected CA profile contains the CA that signed the server's certificate (Can I verify the backend certificate when connecting to the backend over HTTPS?).

4

Adım 4 Check the Proxy setting

Proxy is selected only if the server expects a PROXY protocol header; a server that does not expect it counts this header as an invalid request (What is the Proxy Protocol setting, and when should it be enabled?).

5

Adım 5 Save

The Edit button is clicked. All the vServices that use the Backend Service run with the new setting; the affected ones are seen in the Used in tab of the preview.

Verification

1ResponseClient side

The request is repeated through TR7: curl -sv https://www.example.com/

ExpectedThe same response as with direct access

Common Mistakes

• Doing the direct test from a source other than TR7's address and concluding that there is no network problem. The server may be rejecting only TR7's address; the test is repeated from TR7 with sys-cmd curl.
• Doing the test with the IP and overlooking the Host and SNI difference. The server may be serving a different virtual site.
• Assuming that the source address of the health check and of the data traffic is the same. The two are set separately in the Backend Service (SNAT and Health Check Source IP).
• Testing in WEB-CLI without selecting a route table. A server in an isolated route table appears unreachable from the default route table.
• Mistaking TR7's 400 response to large cookies or headers for an application error. The "Invalid or too large request" page indicates the buffer limit (What is Buffer size used for; when should I change it?).
• Asking for support without collecting information if the problem persists. What to add to the support ticket: Which information should I collect before contacting TR7 support?

How it is done in the documentation

• Debug Logs · Request Tab
• sys-cmd curl · Things to Know
• sys-cmd route-table · route-table set
• Backend Services · SNAT

Related questions

• Why do my backend servers see all requests as coming from TR7's IP?
• The server is running, but TR7 shows it as down; why?
• What does the SSL Service option do; what is needed to connect to the backend over an encrypted connection?