The health check keeps going back and forth between up/down (flapping); what should I do?

Short answer

First, the cause of the transitions (a timeout or an error code) is found in the Health Checks tab of the Backend Service. In the case of a timeout, the Timeout is raised above the response time of the server under load; by increasing Required Failures, momentary errors are tolerated, and by increasing Required Successes, the server is prevented from returning to traffic before it becomes stable. The health endpoint is lightened and the Period is not kept lower than necessary (Environments under load); the transition notifications can be reduced from the notification profile until the problem is solved.

Problem: The Server Keeps Entering and Leaving Traffic

When the health check stays at the threshold, the server repeatedly leaves traffic and comes back. At each transition, new requests shift to other servers, session persistence can break and notifications accumulate. The cause of this is response times that grow longer under load, short interruptions in the network or thresholds set too narrowly for the service.

Diagnosis first:

1. The Health Checks tab in the Backend Service preview is opened in Only Change Logs mode; the times of the transitions and the error type (Request timeout error or Unexpected status code) are looked at.
2. In the Health Check Time panel on the Dashboard, whether the check duration is approaching the Timeout is checked.
3. In a cluster, the two halves of the Health indicator are looked at: if only one device sees the fluctuation, the network path of that device is examined.
4. The response time of the server under load is measured (e.g. curl -o /dev/null -s -w '%{time_total}' http://192.0.2.20/health).

Solution: 4 Methods

Methods 1 and 2 eliminate the cause and can be applied together; Method 3 is the WEB-CLI equivalent of Method 1; Method 4 only reduces the notification noise.

Your situation	Suitable method
The errors are timeouts or occasional; the thresholds are narrow	Method 1: Adjusting the Timeout and thresholds
The health endpoint is heavy or slow under load	Method 2: Lightening the health endpoint
I update the profile from WEB-CLI	Method 3: WEB-CLI
The transition notifications create noise until the problem is solved	Method 4: Reducing the notifications

Method 1: Adjusting the Timeout and Thresholds Recommended

• How it works The Timeout covers slow but healthy responses; Required failures and Required successes filter short fluctuations: the server leaves only after Required failures consecutive failures and returns after Required successes consecutive successes.
• Where it is openedLoad Balancer → Profiles → Health Checks → profile → Edit
• Requirement Knowing on which Backend Services the profile is used (Used in tab)
• vService type All

Step-by-Step Configuration

1

Adım 1 Open the profile form

On the Load Balancer → Profiles → Health Checks screen, the profile is selected and Edit is clicked.

2

Adım 2 Increase the values

Timeout is raised above the normal response time under load (below 120 seconds), Required Failures for example to 3, and Required Successes for example to 5 (at most 10 in the form). If Period is below 1 second or puts load on the server, it is increased.

3

Adım 3 Save

The Edit button is clicked. All the Backend Services that use the profile return to Preparing and enter traffic after Required successes successful checks; for this reason, the change is made at a low-traffic time (If I change a health check profile, which Backend Services are affected?).

Verification

1StabilityHealth Checks tab

The Health Checks tab is opened in Only Change Logs mode; no new transition is seen throughout a heavy traffic period.

ExpectedNo new OK / Error transition

Method 2: Lightening the Health Endpoint

• How it works If the health endpoint slows down under load, the check falls into a timeout; a lightweight address that only returns that the application is running eliminates this fluctuation.
• Where it is opened Application side · profile → Path + Query
• Requirement A lightweight health endpoint on the application side (What kind of /health endpoint should I design in my application for health checks?)
• vService type All

Step-by-Step Configuration

1

Adım 1 Prepare the health endpoint

On the application side, instead of a heavy page or query, a lightweight address that only returns that the application is running is prepared; the dependency checks are cached.

2

Adım 2 Open the profile form

On the Load Balancer → Profiles → Health Checks screen, the profile is selected and Edit is clicked.

3

Adım 3 Change the address

The new address is written in the Path + Query field (e.g. /health).

4

Adım 4 Save

The Edit button is clicked.

Verification

1Check durationDashboard

In the Health Check Time panel, the check duration is seen to stay clearly below the Timeout.

ExpectedLow and stable check duration

Method 3: WEB-CLI

• How it works The fields in Method 1 are written with --timeout, --requiredFailure, --requiredSuccess and --testInterval; in WEB-CLI, the Required fields can be entered up to 100.
• Where it is opened WEB-CLI → lb hc update (lb hc · The Threshold Fields Found in Every Type)
• Requirement WEB-CLI access; the profile ID (lb hc list)
• vService type All

Step-by-Step Configuration

1

Adım 1 Write the thresholds

lb hc update id <profile-id> --timeout 3 --requiredFailure 3 --requiredSuccess 5

Verification

1RecordWEB-CLI

The profile record is displayed.

Commandlb hc show <profile-id>

Expectedtimeout 3, requiredFailure 3, requiredSuccess 5

Method 4: Reducing the Notifications

• How it works If Backend Service Status is turned off in the vService's notification profile, Backend Service status notifications are not produced; this does not solve the fluctuation, it only reduces the noise.
• Where it is openedLoad Balancer → Profiles → Notifications → profile → Backend Service Status
• Requirement A custom notification profile assigned to the vService (the built-in Default profile is not edited)
• vService type HTTP, L7 TCP

Step-by-Step Configuration

1

Adım 1 Open the profile form

On the Load Balancer → Profiles → Notifications screen, the profile used by the vService is selected and Edit is clicked.

2

Adım 2 Turn off the notification

The Backend Service Status option is turned off; the permanent solution is Methods 1 and 2.

3

Adım 3 Save

The Edit button is clicked; when the problem is resolved, the option is turned on again.

Verification

1NotificationsNotification Logs

On the Notification Logs screen, the Backend Service status notifications of this vService are no longer produced.

ExpectedNo new notification

Common Mistakes

• Expecting stability by lowering the Period. A more frequent check means more load; stability is provided by the thresholds.
• Increasing only Required failures. The return of the server must also be filtered; Required successes is also increased.
• Changing the profile at a busy hour. The Backend Services that use the profile do not receive traffic until Required successes successful checks.

How it is done in the documentation

• Health Checks · Recommended Values and Tuning Guide
• Health Checks · Health Status Transitions
• Health Checks · Environments Under Load

Related questions

• How should I set the Period, Timeout, Required Successes and Required Failures values?
• Can I see only status changes in the health check logs (Only Change Logs / All Logs)?
• The server is running, but TR7 shows it as down; why?