Users see "503 Service Unavailable" / "No Available Server"; what should I do?

Short answer

A 503 shows that TR7 cannot find an available Backend Service to which it can forward the request. First, the Details tab of the vService preview is checked: No healthy backend services shows that all Backend Services are unhealthy or in maintenance, No backend services given shows that no Backend Service has been added, and in this second case the vService appears green (OK). If the Backend Services are green, the request may be falling into a Conditional Backend Services block whose servers are down, the Connection Limit may be filling up and the Queue Timeout being exceeded, or the request may be going to a down server with no health check assigned. The exact cause is read from the Termination State and Backend Service values in Debug Mode Logs.

Problem: Users See an Error Page

Users see the TR7 page with a large "503" and "No server was available to handle the request", or a custom error page defined for the vService. The 503 is generated by TR7: there is no Backend Service that receives traffic and can be selected for the request, or a connection to the Backend Service could not be established. It must be determined whether the problem occurs on all requests, only on specific paths or hosts, or at peak hours; the cause is understood from this pattern.

The decision is not made by looking at the page on the screen: if the Show Content for Traffic Manager Errors action has been added to the vService, TR7's 503 is shown with custom content. The source is always determined from the logs.

Causes and Solutions

Cause	How to recognize it	Solution
All Backend Services are unhealthy	No healthy backend services in Details, vService red (Error)	Health check diagnosis (A backend shows as down; how do I diagnose it?)
All Backend Services are in maintenance	<name> is under maintenance lines and No healthy backend services in Details	Maintenance Mode is turned off (What does Maintenance Mode do?)
No Backend Service has been added to the vService	vService green (OK), No backend services given in Details	A Backend Service is added to the Default Backend Services list
No condition matches and the Default list is empty	503 only on some requests; Backend Service = TR7 in HTTP logs	A Backend Service is added to the Default list (Where does traffic go if no condition matches?)
The servers of the matching conditional block are down	503 on a specific path or host; vService yellow (Warning)	The servers of the block are fixed; the block does not fall back to the Default list
Connection limit reached	Intermittent 503 at peak hours; Termination state While queued · Server timeout	The Connection limit or the number of servers is increased (What is Connection Limit; what happens to new requests when the limit is reached?)
No health check, server down	Status No Check in Live Stats; Retries greater than zero in Debug Mode Logs	A health check is assigned (The server has failed, but TR7 still sends traffic to it; why?)
Backend Service address in TR7's internal network	Taken out of service because they address TR7's own internal network in Details	The real server address is written in the Address field of the Backend Service
Intentional maintenance page	A Show Content action defined with the 503 status code on the vService	The action is removed from the vService (How do I publish a maintenance page?)

Where Is It Monitored?

Screen	Path	What it shows
Details	Load Balancer vServices → vService → right panel	The vService status and the Backend Service status messages
Status Logs	In the same panel	When and for what reason the status changed
Live Stats → Backend Services	vService → Live Stats	Healthy / Unhealthy / Maintenance / No check per Backend Service, queue and connection errors
HTTP Logs	vService → Logs & Tracking	The requests that received 503 and the Backend service value
Debug Mode Logs	vService → Logs & Tracking	Termination state and Retries

Solution: Three Methods

The three methods complement each other and are applied in order: first the status message is read, then the pattern of the 503s is identified, and if necessary the exact cause is found with Debug mode.

Your situation	Suitable method
First check: the status message of the vService	Method 1: Details tab
Which requests receive 503	Method 2: HTTP logs
The exact cause	Method 3: Debug Mode Logs → Termination state

Method 1: Details Tab Recommended

• How it works The Details tab of the vService preview shows the status of the vService and the status messages for each Backend Service.
• Where it is opened Left Monitor Panel → Load Balancer vServices → vService → right panel → Details
• Requirement No additional setting is required.
• vService type HTTP (the 503 status code is generated only on an HTTP vService)

Step-by-Step Configuration

1

Adım 1 Select the vService

In the Left Monitor Panel, the related vService is selected from the Load Balancer vServices list; the Details tab is opened in the right panel. The color strip below the tab shows the combined status of the vService: green OK, yellow Warning, red Error, gray Disabled (turned off manually or the license has stopped the traffic) (vService Status).

2

Adım 2 Read the status message

The next step depends on the message:

• No healthy backend services: all Backend Services in the vService are unhealthy or in maintenance; the vService appears red (Error). For health check diagnosis: A backend shows as down; how do I diagnose it?
• <name> is under maintenance: that Backend Service is in Maintenance Mode and does not receive traffic. A Backend Service in maintenance is not counted as working; if all of them are in maintenance, No healthy backend services also appears.
• No backend services given: no Backend Service has been added to the vService. The vService still appears green (OK), and 503 is returned for every request.
• Taken out of service because they address TR7's own internal network: the address of the Backend Service falls into TR7's internal network, and the Backend Service has been taken out of traffic; Address is changed to the real server address.
• <name> is not healthy: only that Backend Service is unhealthy; the vService appears yellow (Warning).

3

Adım 3 If the Backend Services are green, check their usage

The Used in tab of the Backend Service preview shows whether the Backend Service is actually used in this vService (How do I see which vServices use a Backend Service ("Used in")?). If the vService has Conditional Backend Services, the status of the servers of the block that the request falls into is also checked: when the condition matches, the request goes to that block; if the servers of the block are down, there is no fallback to the Default list and the request receives 503.

Verification

1StatusTR7 side

After the fix, the Details tab is checked again; the transition also appears in the Status Logs tab.

ExpectedThe error message disappears, the strip is green (OK)

2ClientClient side

The same address is requested again from the client: curl -sv https://www.example.com/ -o /dev/null

Expected< HTTP/1.1 200 OK

Method 2: HTTP Logs

• How it worksHTTP Logs shows the status code of each request and the Backend Service that served the request; if the request did not go to any Backend Service, TR7 is written as the Backend service value, and Cache if the response was served from the cache.
• Where it is opened Left Monitor Panel → Load Balancer vServices → vService → Logs & Tracking → HTTP Logs
• Requirement In the Details panel of the vService, the LOG box must be checked and a log profile must be selected (How are HTTP logs enabled).
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the HTTP logs screen

Left Monitor Panel → Load Balancer vServices → vService → Logs & Tracking → HTTP Logs is opened. If there are no records, logging is not on in the vService (What is the difference between HTTP logs and TCP logs?).

2

Adım 2 Filter the 503 records

In the Filter panel, 503 is selected from the Status Code card (Filter).

3

Adım 3 Check the Backend service value

The Backend Service value of the records is checked: if the value is TR7, the request did not go to any Backend Service (the list is empty, all of them are down, no condition matched, or the request timed out in the queue); if a Backend Service name appears, TR7 selected that server but could not establish a connection, or the request waited in the queue.

4

Adım 4 Identify the pattern

It is checked whether the 503s occur on all requests, cluster on a specific path or host, or cluster at peak hours. If they occur only on specific paths, the conditional routing is examined; if they occur at peak hours, the queue and the Connection limit are examined (How does the queue work; how long do requests wait at most?).

Verification

1RecordTR7 side

After the fix, the Status code 503 filter is applied again in HTTP logs.

ExpectedNo new 503 record is created

Method 3: Debug Mode Logs → Termination State

• How it worksTermination State shows at which stage and for which reason the request ended, in the "stage · reason" form; Retries shows how many times the connection attempts were repeated.
• Where it is opened vService edit form → Status = Debug mode; records in Logs & Tracking → Debug mode logs
• Requirement HTTP vService. Debug mode keeps a detailed record for each request; under heavy traffic, only the problematic client is recorded with a condition selected in the Select Conditions field in the Debug group.
• vService type HTTP

Step-by-Step Configuration

1

Adım 1 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

2

Adım 2 Select Debug mode

Debug Mode is selected in the Status field. The same setting can also be turned on with the Debug box in the Details panel; the Debug mode option appears only in the HTTP vService form (What does Debug Mode record; how long should I keep it on?).

3

Adım 3 Narrow the recording to the problematic traffic

Under heavy traffic, a condition that selects the IP of the problematic client or the related path is selected in the Select Conditions field in the Debug group. If no condition is selected, all HTTP requests coming to the vService are recorded.

4

Adım 4 Save

The Save button is clicked and the 503 is reproduced.

5

Adım 5 Read the Termination state value

On the Logs & Tracking → Debug Mode Logs screen, the Termination State and Retries values of the 503 record are checked:

Termination state	Meaning
While connecting to backend · Server aborted/refused	The server or a device in between refused the connection; if Backend Service = TR7, there was no server that could be selected
While connecting to backend · Server timeout	The Connect Timeout (default 20 s) expired: server down, firewall or route problem
While queued · Server timeout	The Connection limit is full; the request exceeded the Queue Timeout (default 60 s) in the queue

If Retries is 0, the connection was established on the first attempt; if it is greater than zero, connection attempts have failed.

6

Adım 6 Reopen the vService edit form

When the examination is finished, on the Load Balancer → Load Balancer vService List screen, the vService is selected and the Edit button is clicked.

7

Adım 7 Turn off Debug mode

Enabled is selected in the Status field. Debug mode does not turn off on its own.

8

Adım 8 Save

The Save button is clicked.

Verification

1RecordTR7 side

The 503 record is opened in Debug Mode Logs; the Connection info chain in the Summary tab shows where the request stopped.

ExpectedThe Termination state "stage · reason" value is filled

If the request never reached the Backend Service, the Connection info chain ends at TR7.

Common Mistakes

• Not checking the Backend Service list because the vService is green. A vService with no Backend Service added appears green (OK) and returns 503 for every request; No backend services given is written in Details.
• Overlooking maintenance mode. A Backend Service in Maintenance Mode does not receive traffic and is not counted as working.
• Not assigning a health check. Without a health check, the server is always considered up, and requests are also sent to a down server (What does a Health Check do; what happens if I don't assign one?).
• Assuming that the conditional block falls back to the Default list. A request whose condition matches goes to that block; if the servers of the block are down, it receives 503.
• Deciding on the source by looking at the custom error page. If a custom page is defined, the source is determined from the logs (Can I show a custom page for 503/502/504 errors generated by TR7 itself?).
• Looking for the Termination state in HTTP logs. This column exists only in Debug Mode Logs.
• Leaving Debug mode on. When the examination is finished, Status is set back to Enabled.
• Asking for support without collecting information if the problem persists. What to add to the support ticket: Which information should I collect before contacting TR7 support?

How it is done in the documentation

• HTTP (L7) vService · Default Backend Services
• Load Balancer vServices Dashboard · Panel Tabs
• Debug Logs · Termination State Values

Related questions

• A backend shows as down; how do I diagnose it?
• What do users see if all servers are down; can I show a custom page instead?
• How do I find out from the logs why a request failed (termination state)?