The server is running, but TR7 shows it as down; why?

Short answer

The cause is that the profile settings do not match the real behavior of the server or that the check cannot reach the server: a wrong port or path, a code that is not in the Expected Status Codes list (including a redirect), a content check that does not match, a missing Virtual Domain (Host/SNI), a certificate that cannot be verified, a Timeout shorter than the response time; on the network side, the server firewall, a wrong Route Table, an old address in Use Different IP for Health Check or a port that expects the PROXY protocol. Which one it is is written in the error text in the Health Check card in the Backend Service preview and in the Health Checks tab.

How Does It Work?

TR7 counts a check as successful if every condition in the profile is met; if one of the conditions does not match, the check fails with an error text, and after Required Failures consecutive failures, the Backend Service becomes Error (red). The table below matches the error texts with their solutions.

Cause	Symptom / error text	Solution
Status code not in the list	Unexpected status code; the health endpoint returns 301/302	Adding the code to the Expected Status Codes list or writing the correct path (How do I expect a specific status code (e.g. 200, 302) in an HTTP health check?)
Content check does not match	HTTP(s) content check failed or JSONata check returned false	Correcting the searched text (upper/lower case) or the JSONata expression (How do I verify a specific text (e.g. "OK") in the response content?)
Host/SNI missing	The server returns 404 or the default site to the request that comes with the IP	Adding Virtual Domain (Host/SNI) to the profile (Can I specify a Host / SNI in a health check (Virtual domain)?)
Certificate cannot be verified	Certificate error	Registering the CA or reviewing the Verify Server Certificate setting (Can certificate verification be done in an HTTPS health check?)
Timeout short	Request timeout error; the server is slow under load	Raising the Timeout above the response time under load (below 120 s)
Wrong port / path	Connection refused or 404	Checking the Address, Use Different IP for Health Check and Path + Query fields
Firewall	Timeout; the check request does not reach the server	Allowing the source IP of the health check (Which IPs should I allow on the server firewall for health checks?)
Wrong route table	The server cannot be reached	Changing the Route Table field of the Backend Service to the table that reaches the server
PROXY protocol	The HTTP check fails, the server expects a PROXY header	Serving the health endpoint from a port that does not require PROXY
HTTP/2 selected	The server does not support HTTP/2	Setting the HTTP Version field to HTTP/1.1
Expect negative result on	A healthy server appears down	Turning off the option (What is "Expect Negative Result" used for?)
Preparing (yellow)	The profile was newly assigned or changed	Waiting for Required Successes consecutive successful checks

Diagnosis Steps

1. Health Check card: On the Load Balancer → Backend Services screen, the Backend Service is clicked; the last error is seen in the error strip of the Health Check card in the preview, and in HTTP checks, the last received code is seen in the Status Code row in the Details tab.
2. Health Checks tab: In the Health Checks tab of the same preview, when the error started and its text are seen.
3. Repeating the request from TR7: In WEB-CLI, the route table of the Backend Service is selected with sys-cmd route-table set (route-table set); then the same request is sent, for example sys-cmd curl -sv -H "Host: www.example.com" http://192.0.2.20:8080/health (sys-cmd curl).
4. Server side: With the access log or tcpdump -n host <TR7-IP>, whether the check requests arrive and the response given are looked at.
5. Cluster: If only one of the two halves of the Health indicator is red, the problem is the firewall rule applied to the source address of that device or the routing of that device (Health status indicator).

Common Misconceptions

• "It opens from the browser, so it is healthy." The browser follows redirects and sends the correct Host header; the health check does not follow redirects, and if Virtual domain is empty, it sends the server's address as the Host.
• "A gray indicator means down." Gray means no result (HC disabled / No data); the dark gray power icon shows that no profile is assigned.
• "If the port is open, the HTTP check also passes." The status code and the content must also be correct.

How it is done in the documentation

• Backend Services · Health Check Source IP
• Backend Services · Health Status Indicator
• Backend Services Dashboard · Health Check Card

Related questions

• Which source IP are health check packets sent from; can I change it?
• The health check keeps going back and forth between up/down (flapping); what should I do?
• A backend shows as down; how do I diagnose it?