A backend shows as down; how do I diagnose it?

Short answer

First, it is verified that the Health indicator is really red (Error): yellow Preparing is temporary, gray HC Disabled or No Data shows that there is no check result, the dark gray power icon (Off) shows that no health check is assigned, and the ambulance icon in the name column shows Maintenance Mode. If it is red, the cause is written in the Health Check card in the Backend Service preview and in the error text in the Health Checks tab (e.g. TCP connection timeout). In a cluster, if it is red on only one device, the problem is in the network path of that device. Then, whether the server can be reached directly, whether the port, path, expected code and content in the health check profile match the real response of the server, and whether the source IP of the health check traffic is allowed in the server firewall are checked in order.

Problem: The Backend Service Shows as Down

In the Backend Services list, one or more Backend Services appear red, and no traffic goes to these servers. An unhealthy Backend Service is removed from traffic automatically; once the server recovers and passes Required Successes (default 3) consecutive successful checks, it is taken back into traffic automatically. It must be determined whether the server is really not running, whether the health check profile is set incorrectly, or whether the check traffic cannot reach the server.

Causes and Solutions

Error text	Possible cause	Solution
TCP connection refused	The port is closed or the service is not running	The service on the server is started or the port in the profile is corrected
TCP connection timeout · Request timeout error: …ms	The server is down, a firewall, a route, or the Timeout (default 1 s) is tighter than the response time	Method 2 and Method 3
Unexpected status code	The code returned by the server is not in the Expected Status Codes list (including 301 and 302)	How do I expect a specific status code (e.g. 200, 302) in an HTTP health check?
HTTP(s) content check failed	The response content does not meet the Content Check Mode expectation	The expected content is corrected according to the real response of the server
Certificate error: …	The server certificate cannot be verified	The certificate verification setting of the profile or the server certificate is corrected
No error, ambulance icon in the name column	The Backend Service is in Maintenance Mode	What does Maintenance Mode do?

Detailed cause table: The server is running, but TR7 shows it as down; why?

Where Is It Monitored?

Screen	Path	What it shows
Health status indicator	Load Balancer → Backend Services	OK, Error, Preparing, HC Disabled, No Data, Off; in a cluster, two halves, one per device
Health Checks tab	Backend Service preview	The check history and the error texts
Live Stats → Backend Services	vService → Live Stats	Last check, Failed checks, Times down, Total downtime, Last change in the name cell tooltip
Details · Status Logs	Load Balancer vServices → vService → right panel	<name> is not healthy lines and status transitions

Solution: Three Methods

The three methods are applied in order: first the indicator and the error text are read, then whether the problem is in the server or in the profile is separated, and last, whether the check traffic reaches the server is checked.

Your situation	Suitable method
Reading the indicator and the error text	Method 1: Indicator and error text
Server or profile	Method 2: Direct request
Does the check traffic reach the server	Method 3: Source IP and network path

Method 1: Indicator and Error Text Recommended

• How it works The color of the Health indicator shows the status, and the error text in the Health Checks tab shows the cause.
• Where it is openedLoad Balancer → Backend Services → Backend Service → preview → Health Checks
• Requirement A health check must be assigned to the Backend Service; history records are written while Enable HC Logs is on.
• vService type HTTP · L7 TCP · Network

Step-by-Step Configuration

1

Adım 1 Verify the indicator

The Health column in the Load Balancer → Backend Services list is checked. If it is yellow Preparing, the wait continues until the check result becomes final; gray HC Disabled or No Data and the dark gray power icon Off do not mean down; if there is an ambulance icon in the name column, the Backend Service is in Maintenance Mode (What do the health colors and icons in the Backend Service list mean?).

2

Adım 2 Select the Backend Service

The row of the Backend Service in the list is clicked; the status and the error text are read in the Health Check card in the preview that opens on the right. In a cluster, if the error texts of the two devices differ, both are shown separately.

3

Adım 3 Open the Health Checks tab

The Health Checks tab is opened in the preview; in the Only Change Logs view, the Error and OK transitions and the error texts are listed (How do I see the health check history (when a server went down/up)?). The error text is matched with the Causes and Solutions table.

4

Adım 4 Separate the cluster

If the indicator is split into two halves and only one half is red, the problem is in the path from that device to the server: the route table and the routes of that device and the firewall rules on the server side are checked (What does a status circle split in two mean in the vService list in a cluster?).

Verification

1IndicatorTR7 side

After the fix, the Backend Services list and the Health Checks tab are checked.

ExpectedHealth green (OK); Error → OK transition in Health Checks

The indicator turns green after Required successes (default 3) consecutive successful checks.

Method 2: Direct Request

• How it works When the request in the health check profile is produced manually, whether the problem is in the server or in the profile is separated.
• Where it is opened The same network as the server · WEB-CLI → sys-cmd curl
• Requirement The Path + Query, Expected status codes and Virtual Domain (Host/SNI) values of the profile.
• vService type HTTP · L7 TCP · Network

Step-by-Step Configuration

1

Adım 1 Read the profile values

On the Load Balancer → Profiles → Health Checks screen, the profile assigned to the Backend Service is selected; its type and its Path + Query, Expected Status Codes, Virtual Domain (Host/SNI) and Content Check Mode values are noted.

2

Adım 2 Produce the request manually

The same request is sent from the same network as the server:

curl -sv -H "Host: www.example.com" -w '%{http_code} %{time_total}\n' http://192.0.2.21:8080/health

If the same request is to be tried from TR7, the route table of the Backend Service is first selected in WEB-CLI (for the device itself, Host is written in the v-device field; the route table name is taken from the Tab list):

sys-cmd route-table set v-device Host route-table <route-table>
sys-cmd curl -sv -H "Host: www.example.com" http://192.0.2.21:8080/health

3

Adım 3 Separate according to the result

If the direct request also fails (refused, timeout or 5xx), the server is fixed; once it recovers, the Backend Service returns to traffic automatically. If the request succeeds, the profile is corrected according to the real response of the server (The server is running, but TR7 shows it as down; why?). If the response time is close to the Timeout value of the profile and the Backend Service goes down and up from time to time, the Timeout and Required Failures (default 2) are reviewed (The health check keeps going back and forth between up/down (flapping); what should I do?).

Verification

1ResponseServer side

The status code and the duration in the curl -sv output are checked.

ExpectedA code in the Expected status codes list, a duration shorter than the Timeout

Method 3: Source IP and Network Path

• How it works If the check requests do not reach the server, it appears down even if the server is healthy. The source address of the checks is determined according to the Health Check Source IP field of the Backend Service.
• Where it is opened The access log of the server · Backend Service form
• Requirement Access to the server logs.
• vService type HTTP · L7 TCP · Network

Step-by-Step Configuration

1

Adım 1 Check the server log

In the access log of the server, it is checked that the check requests arrive at the Period (default 1 s) interval of the profile, and their source address is checked. If they do not appear, TR7's check address is allowed on the server or on the firewall in between (Which IPs should I allow on the server firewall for health checks?, Which source IP are health check packets sent from; can I change it?).

2

Adım 2 Open the Backend Service edit form

On the Load Balancer → Backend Services screen, the Backend Service is selected and the Edit button is clicked.

3

Adım 3 Check the route table and the checked address

It is checked that the Route Table field is the route table through which the server is reached. If Use Different IP for Health Check is on, the check goes to the address and port entered here instead of the Address field of the server; it is verified that this address is up to date (Can I check a different IP/port of the server for the health check (Use Different IP for Health Check)?).

4

Adım 4 Save

If a change was made, the Edit button at the bottom of the form is clicked.

Verification

1LogServer side

The access log of the server is checked.

ExpectedCheck requests arrive regularly from TR7's address

Common Mistakes

• Thinking a gray or Off indicator means down. These states show that there is no check result or that no health check is assigned. A Backend Service with no health check assigned is always considered up and receives requests even if it is down (The server has failed, but TR7 still sends traffic to it; why?).
• Changing the profile constantly during Preparing. A few check results are waited for.
• Not questioning the profile because the page opens in the browser. The browser follows redirects and sends the correct Host; the health check may not send it.
• Thinking the source address of the health check and of the data traffic is the same. The two are set separately in the Backend Service (Health Check Source IP and SNAT).
• Thinking the problem of a single device in a cluster is general. The two halves of the indicator are read separately.
• Asking for support without collecting information if the problem persists. What to add to the support ticket: Which information should I collect before contacting TR7 support?

How it is done in the documentation

• Health Checks · Summary Text
• Backend Services · Health Status Indicator
• Health Checks profile · Timeout

Related questions

• The server is running, but TR7 shows it as down; why?
• How do I see the health check history (when a server went down/up)?
• The health check keeps going back and forth between up/down (flapping); what should I do?