The server has failed, but TR7 still sends traffic to it; why?

Short answer

The main cause is that no health check is assigned to the Backend Service; in this case, TR7 does not understand that the server has gone down, and a dark gray power icon (Off) appears in the Health column. The other causes: a superficial check such as TCP or Ping not catching an application failure, a few seconds passing for the decision to form, the health status not being reflected in the distribution on Network vServices with ANY - L3 or Port Range on, connections that remain open on the server, the check looking at another address with Use Different IP for Health Check, the OR logic in Scenario and an Expect Negative Result left on.

How Does It Work?

TR7 removes a server from traffic only when the health check of that server becomes Error; if there is no check, if it gives a healthy result or if the vService type does not use the health status, traffic continues to be sent.

Cause	How to recognize it	Solution
Health check not assigned	Dark gray power icon (Off) in the Health column, No Check in the Health Check Type column	Assigning a profile to the Backend Service (What does a Health Check do; what happens if I don't assign one?)
Superficial check	The type is TCP or Ping; the port is open while the application returns an error	Switching to HTTP/HTTPS and a content check (What is the difference between a TCP port check and an HTTP check; which should I choose?)
Decision time	Traffic is cut a few seconds after the failure	Expected behavior; the duration depends on Required failures, Period and Timeout (How many seconds does it take for a server to be considered "down", and how do I calculate it?)
Network ANY - L3 / Port Range	On the vService, Protocol ANY - L3 or Port Range on	On these vServices, the health status is not reflected in the distribution; if the health status is needed, TCP - L4 / UDP - L4 with Port Range off is used
Open connections	WebSocket or keep-alive connections on the old server	When the server becomes Error, new connections go to another server; open connections last until they close by themselves
Check looks at another address	Use Different IP for Health Check looks at another healthy endpoint	Directing the check address to an endpoint that reflects the real status of the server (Can I check a different IP/port of the server for the health check (Use Different IP for Health Check)?)
Health endpoint misleading	/health returns 200, but the actual function is broken	Designing the health endpoint so that it returns the real status of the application (What kind of /health endpoint should I design in my application for health checks?)
Scenario OR	If one of the sub-checks is successful, the result is healthy	Combining the required checks with AND (Can I assign more than one health check to a Backend Service?)
Expect Negative Result	The server is healthy even though the check fails	Turning off the option (What is "Expect Negative Result" used for?)
No health check on the backup server	When the main servers go down, traffic goes to a backup that is down	Also assigning a profile to the server marked Backup Service

Diagnosis Steps

1. Backend Services list: On the Load Balancer → Backend Services screen, the Backend Services to which no profile is assigned are found from the Health and Health Check Type columns; the Filter → Health card lists the assigned profiles.
2. HTTP Logs: Which Backend Service gave the faulty responses is seen from the Backend Service column on the vService's HTTP Logs screen.
3. Health Checks tab and Dashboard: The results in the Health Checks tab of the Backend Service and the Health Check State panel on the Dashboard are examined; if the status stayed OK during the failure, the check does not see the failure.

Common Misconceptions

• "TR7 notices a down server on its own." If no health check is assigned, it does not notice.
• "If the port is open, the application is running." A TCP check does not catch an application failure.
• "When the server becomes Error, open connections are also cut." Only new connections go to other servers; open connections continue.

How it is done in the documentation

• Health Checks · Traffic Routing
• Backend Services · Health Check
• Network vService · Health status and distribution

Related questions

• What does a Health Check do; what happens if I don't assign one?
• What is the difference between a TCP port check and an HTTP check; which should I choose?
• How many seconds does it take for a health check result to affect traffic?