What does "Partially Healthy" mean on a Backend Group dashboard; how do I find which server in the group has a problem?

Short answer

Partially Healthy shows that the health check results of the Backend Group members differ from each other. The Health Check profile of the group is applied to each member separately, and the status of the group is calculated from the results of the members: if all members are in the same status, that status is shown; if they differ, Partially Healthy is shown. This does not always mean that a member has gone down; it is also seen while the result of a newly added member is not yet final (Preparing or No Data). The problematic member appears with its own color in the Backend Service Groups category of the Left Monitor Panel; when Error is selected in the Health filter, only the members that have gone down are listed. When it went down and the error text are read with the ip:port label of the member in the Health Check tab of the Dashboard and in the Health Checks tab of the preview.

Problem: The Status of the Group Does Not Tell Which Member Has a Problem

The colored dot on the Backend Group dashboard and in the lists is the aggregate status of the group. When the group turns yellow, part of the traffic still flows, but which server has a problem and what the problem is cannot be understood from this dot. The status of the group is calculated from the results of the members with the following rule:

Status of the members	Status of the group
All members OK	OK (green)
All members Error	Error (red)
All members Preparing	Preparing (yellow)
All members No Data	No Data (gray)
The results of the members differ (e.g. one OK, one Error; or one OK, one still Preparing or No Data)	Partially Healthy (yellow)
No members in the group	HC Disabled (gray)

If no Health Check profile is assigned to the group, the status is not calculated; a gray power icon appears on the dashboard instead of a colored dot. The group status is updated within a few seconds as the results of the members change. In a cluster setup, the dot has two halves: the left half is the status seen by this device, and the right half is the status seen by the peer device (Panel Header).

In terms of traffic, the group continues to serve: a member that fails the health check is taken out of traffic on its own, and the healthy members share the load; the member returns to traffic automatically when it reaches the Required Successes threshold in the profile (Health Check).

Solution: Four Methods

The methods complement each other: Method 1 and Method 4 show the problematic member at that moment, Method 2 the drops over time, and Method 3 gives the error text.

Your situation	Suitable method
I want to quickly see which member is down right now	Method 1: Left Monitor Panel → Backend Service Groups
I want to see when the member went down and how long it stayed down	Method 2: Dashboard → Health Check tab
I want to see why the member failed (the error text)	Method 3: Preview → Health Checks tab
I want to see all groups with their members in one table	Method 4: Backend Groups list → Show Backend Services

Method 1: Left Monitor Panel → Backend Service Groups Recommended

• How it works The members of the groups are listed under the group header with the ip:port label; the strip on the left of each member row carries the color of that member's own health check. Partially Healthy appears only in the group header, not in the member rows.
• Where it is openedLeft Monitor Panel → Backend Service Groups
• Requirement A Health Check profile assigned to the group; Load Balancer permission for the user
• Scope All Backend Groups

Step-by-Step Configuration

1

Adım 1 Open the category

Backend Service Groups is selected from the category selector at the top left of the Left Monitor Panel.

2

Adım 2 Look at the members of the group

The member rows under the header of the group with the yellow dot are examined; a row with a red strip is a member that has failed the health check, a row with a yellow strip is a member whose result is not yet final (Preparing), and a row with a gray strip is a member whose result could not be obtained (No Data). When the strip is hovered over, the name of the status appears.

3

Adım 3 List only the members that are down

In the status strip below the toolbar, the Error box is clicked; the status is added to the Health filter and the table shows only the members in the Error status. If there are many groups, the group is also selected in the Backend Service Groups tab of the Filter window.

Verification

1Member rowTR7 side

While the Error filter is on, the ip:port label of the rows remaining in the list is read.

ExpectedThe problematic member is listed with a red strip and its ip:port label

The boxes in the status strip are not affected by the search and the Health filter; the counts of all statuses remain visible.

Method 2: Dashboard → Health Check Tab

• How it works The Health Check State panel draws the result of each member in a separate band labeled ip:port, on a time axis, as Up / Down; the Health Check Time panel shows the check response time of the members.
• Where it is opened Left Monitor Panel → Backend Service Groups → group → Dashboard → Health Check tab
• Requirement A Health Check profile assigned to the group; Load Balancer permission for the user
• Scope A single Backend Group

Step-by-Step Configuration

1

Adım 1 Open the group's dashboard

In the Left Monitor Panel, in the Backend Service Groups category, the header of the group or one of its members is clicked; the Dashboard bearing the name of the group opens.

2

Adım 2 Switch to the Health Check tab

Health Check is selected from the view tabs of the dashboard.

3

Adım 3 Select the time range

The period in which the problem occurred is selected from the time range selector. In the Health Check State panel, the member whose band differs from the others is the problematic member; the percentage badge in the row shows for how much of the selected range the member stayed up (e.g. 50% Up).

Verification

1Status bandTR7 side

In the Health Check State panel, the bands of the members are compared.

ExpectedA Down slice in the band of the problematic member; the other members Up

The panel shows the member results; Partially Healthy does not appear in this panel as a separate value, it is read from the bands diverging.

Method 3: Preview → Health Checks Tab

• How it works The health check records of the group members are merged into a single timeline; if the group has two or more members, each row carries the ip:port label of the member the record belongs to and the result summary (the error text).
• Where it is opened Backend Group Dashboard → the preview panel on the right → Health Checks tab
• Requirement A Health Check profile assigned to the group; for records, Enable HC Logs on in the group (on by default)
• Scope A single Backend Group

Step-by-Step Configuration

1

Adım 1 Open the preview

The dashboard of the group is opened as in Method 2; if the Backend Group Preview on the right of the screen is collapsed, it is expanded from its handle.

2

Adım 2 Switch to the Health Checks tab

The Health Checks tab of the preview is clicked. On first opening, the list shows the last 24 hours, at most 50 records and only status changes (Only Change Logs).

3

Adım 3 Read the record

In the row whose status is Error, the ip:port label shows the problematic member and the result summary shows the reason. If the range or the record limit has been changed, the list is updated with Refresh; the records also open in a table with an Address column through the enlarge icon.

Verification

1Error recordTR7 side

The newest Error row in the Health Checks tab is read.

Expectedip:port · Error · error summary (e.g. timeout or unexpected status code)

The status values and the format of the summary text are described on the Health Checks page.

Method 4: Backend Groups List → Show Backend Services

• How it works When the Show Backend Services switch is turned on, each group row is split into as many sub-rows as there are members; the Health column shows each member's own indicator in a separate row.
• Where it is openedLoad Balancer → Backend Groups
• Requirement A Health Check profile assigned to the group
• Scope All Backend Groups

Step-by-Step Configuration

1

Adım 1 Open the Backend Groups list

The Load Balancer → Backend Groups steps are followed.

2

Adım 2 Show the members

The Show Backend Services switch above the table is turned on (off by default; its position is saved per user). In the sub-rows of the group, the Address value of the row whose Health indicator is red is the problematic member (Health Status Indicator).

Verification

1Sub-rowsTR7 side

The Health indicators in the sub-rows of the group are compared.

ExpectedThe group indicator is yellow; at least one of the members is a different color

Common Mistakes

• Assuming Partially Healthy is an outage. The group continues to serve; only the member that failed the health check is out of traffic. When all members go down, the group becomes Error.
• Assuming Partially Healthy is always a failure. Since the result of a member newly added to the group is No Data or Preparing until the first checks are completed, the group appears Partially Healthy for a short time; when the member reaches the Required Successes threshold, the group returns to OK.
• Looking for Partially Healthy in the member rows. This status is used only in the aggregate indicator of the group; OK, Error, Preparing or No Data appears in the member rows.
• Selecting Health Check State as a metric in the Left Monitor Panel. The Health Check State and Health Check Time metrics are not offered for group members in this category; the status of the member is read from the row strip (Backend Service Groups).
• Assuming the two halves in a cluster are a single status. If only one half of the dot is yellow, the problem is seen only in the check of that device (e.g. the source address of that device is not allowed in the server's firewall).

How it is done in the documentation

• Backend Groups Dashboard · Panel Header
• Backend Groups Dashboard · Health Check Tab
• Backend Groups Dashboard · Health Checks Tab
• Backend Service Groups · The Status Strip
• Backend Groups · Health Status Indicator

Related questions

• How do I see the health check history (when a server went down/up)?
• The server is running, but TR7 shows it as down; why?
• Which settings are shared at group level, and which are per server?