The Health Check is "up", but users get errors; why?

Short answer

A Health Check tests only the address, port, path and expectation defined in its own profile; the user request, however, goes through a different path, with a different Host, and through TR7's action, limit and security layers. For this reason, who produced the error is separated first. If the Backend Service value in HTTP Logs is TR7, the request has not reached any Backend Service (no available server, queue, WAF or Bot/DDoS block); if a Backend Service name appears, the definitive distinction is made with the Termination State in Debug Mode Logs: if it is Normal, the Backend Service produced the error and the health check is not testing the actual function of the application; if it is Server timeout or TR7 intercepted, TR7 produced the 504 or 502.

Possible Causes

Cause	How to recognize it	Page to check
Superficial health check (port only, or a /health that returns 200 in every case)	Backend service = server name, status 5xx, Termination state Normal; the same request also fails when it is sent directly to the server	The server has failed, but TR7 still sends traffic to it; why? · What is the difference between a TCP port check and an HTTP check; which should I choose?
The health check tests a different Host or virtual site	The check goes with the IP address, the user with the domain name	Can I specify a Host / SNI in a health check (Virtual domain)?
A single server is faulty	In Live Stats, only the 5xx/s value of that server is high	The server is fixed or put into maintenance (What does Maintenance Mode do?)
The server is not checked at all	In Live Stats, the status is No Check	What does a Health Check do; what happens if I don't assign one?
No available server, or the queue time has expired	503, Backend service = TR7; Termination state While queued · Server timeout	Users see "503 Service Unavailable" / "No Available Server"; what should I do?
The Server Timeout has expired	504, Backend service = server name; Termination state During response headers · Server timeout	I get 504 Gateway Timeout; which timeout should I check?
An invalid response or a response with headers that are too large	502, Backend service = server name; Termination state During response headers · TR7 intercepted	I get 502 Bad Gateway; what are the possible causes?
A security layer block	403 or 418 (or the code selected in the rule); Backend service = TR7, Security Action is filled in Debug Mode Logs	Legitimate users are being blocked; how do I find which rule is blocking them?
The request goes to another conditional block	Errors on specific paths; the servers of that block are down or the block is empty	Where does traffic go if no condition matches?
The servers are in maintenance	In Live Stats, the status is Maintenance; the health check continues but the server does not receive traffic	What does Maintenance Mode do?
Decision delay	The error is seen in the few seconds after the failure	How many seconds does it take for a health check result to affect traffic?

Definitive rule: the Backend Service value in HTTP Logs is TR7 if no Backend Service was selected for the request, Cache if the response was served from the cache, and the name of that Backend Service if one was selected. For this reason, seeing a server name does not by itself show that the server produced the error; TR7 produces 504 when no response comes after it connects to the server and 502 when an invalid response comes, and the name of that server is still written in the record.

Where Is It Monitored?

Screen	Path	What it shows
HTTP Logs	Load Balancer vServices → vService → Logs & Tracking	The status code, Backend service and Response source value of the failed requests
Debug Mode Logs	In the same tab (while the vService is in Debug Mode)	Termination state, Retries, Security Action
Live Stats → Backend Services	vService → Live Stats	The status per server (Healthy, Unhealthy, Maintenance, No check) and 4xx/s, 5xx/s
Backend Response Code Distribution	vService → Dashboard	The distribution of the status code classes across the Backend Services

How to Diagnose?

1. The exact status code the user sees is obtained.
2. Left Monitor Panel → Load Balancer vServices → vService → Logs & Tracking → HTTP Logs is opened; in the Filter panel, 4xx or 5xx is selected from the Status Class card and the Backend Service card is checked (Filter). If there are no records, Log Profile is not on in the vService.
3. If the value is TR7, the request has not reached any Backend Service: if it is 503, Users see "503 Service Unavailable" / "No Available Server"; what should I do? is followed; if it is 403 or 418, Legitimate users are being blocked; how do I find which rule is blocking them? is followed.
4. If the value is a server name, the Status field of the HTTP vService is set to Debug Mode, the error is reproduced, and Termination state is read on the Debug Mode Logs screen (How do I find out from the logs why a request failed (termination state)?). If the value is Normal, the server produced the response; During response headers · Server timeout shows that the 504, and During response headers · TR7 intercepted that the 502, was produced by TR7.
5. If the server produced the response, whether the error is on a single server or spread across all of them is determined from the 5xx/s column in Live Stats or the Backend Response Code Distribution panel on the Dashboard. The same request is sent directly to the server, and the health check profile (Check Type, Path + Query, Virtual Domain (Host/SNI), Expected Status Codes, Content Check Mode) is compared with the real user request.
6. Whether the server that appears green is actually being checked is verified from its status in Live Stats (No Check, Maintenance).
7. When the investigation is finished, Status is set back to Enabled; Debug Mode does not turn off on its own.

Common Misconceptions

• "If the port is open, the application is working." A TCP port check tests only that the connection is established; it does not catch application errors. An HTTP or HTTPS check that tests the actual function of the application is used (What kind of /health endpoint should I design in my application for health checks?).
• "If there is a server name in the Backend service column, the server produced the error." TR7 produces 502 and 504, but the name of the server is still written in the record; the distinction is made with Termination state.
• "A 503 with Backend service = TR7 is a server failure." In this case, the request has not gone to any server: there is no available server, the request has waited in the queue, or the routing has not found a server.
• "If a server in maintenance appears green, it is receiving traffic." In Maintenance Mode, the health check continues but the server does not receive traffic.
• "The error page on the screen belongs to the application." If Show Content for Backend Service Errors is defined on the vService, the errors of the server are also shown with the custom page; the source is determined from the logs.

How it is done in the documentation

• Health Checks · HTTP
• HTTP Logs · Filter
• Debug Logs · Termination State Values
• Dashboard · Backend Response Code Distribution

Related questions

• The server has failed, but TR7 still sends traffic to it; why?
• A backend shows as down; how do I diagnose it?
• Users see "503 Service Unavailable" / "No Available Server"; what should I do?