Health check requests are filling up my backend logs; what can I do?
Short answer
The check requests can be filtered or reduced on the server side. HTTP and HTTPS checks arrive with the tr7-health-check/2.0 User-Agent value and can be excluded from the logs on the web server based on this value; the health address can be kept on a separate path and the access log can be turned off for that path; a fixed source address can be selected with Health Check Source IP and excluded from the logs; the check can be directed to a separate port with Use Different IP for Health Check, or the Period can be increased. Turning off Enable HC Logs only stops the records on TR7; it does not reduce the requests sent to the server.
Problem: Server Logs Fill Up with Check Lines
With the default settings (Period 1 s), each Backend Service is checked every second; this means close to 86,400 requests per server per day, and since each device checks separately in a cluster, the number doubles. These lines bloat the access logs, make it harder to examine the real traffic and increase the log storage cost.
Solution: 5 Methods
Methods 1–4 separate the lines on the server side and can be used together; Method 5 reduces the number of requests.
| Your situation | Suitable method |
|---|---|
| I can filter by User-Agent on the web server | Method 1: Filtering by User-Agent |
| I can turn off the log for the health address | Method 2: Separate path and turning off the log |
| I want to filter by source address (including non-HTTP types) | Method 3: Fixed source IP |
| The server can publish health information on a separate port that is not logged | Method 4: Separate port |
| I want to reduce the number of requests | Method 5: Increasing the Period |
Method 1: Filtering by User-Agent Recommended
- How it works HTTP and HTTPS checks arrive with the
tr7-health-check/2.0User-Agent value; a condition that does not write the requests carrying this value to the access log is added to the log configuration of the web server. If a different value is wanted, theUser-Agentheader is changed with the HTTP Headers field in the profile. - Where it is opened The log configuration of the web server · if needed, profile → HTTP Headers
- Requirement The web server supporting conditional logging
- vService type All
Step-by-Step Configuration
On the web server, the requests whose User-Agent value starts with tr7-health-check are excluded from the access log; examples are given in the panel below.
The configuration of the web server is reloaded (e.g. nginx -s reload, apachectl graceful).
The examples do not write the requests whose User-Agent value starts with tr7-health-check to the access log; the log file path is replaced with the existing value on the server.
# in the http block
map $http_user_agent $loggable {
~^tr7-health-check 0;
default 1;
}
# in the server block
access_log /var/log/nginx/access.log combined if=$loggable;SetEnvIf User-Agent "^tr7-health-check" tr7hc
CustomLog "logs/access_log" combined env=!tr7hcOfficial documentation
In Nginx, the if= parameter of the access_log directive does not log the request when the condition is "0" or empty (ngx_http_log_module). In Apache, CustomLog … env=!name logs only the requests for which the environment variable is not defined (mod_log_config, mod_setenvif).
Verification
A health check line is searched for in the access log.
grep -c tr7-health-check /var/log/nginx/access.logNo new line is added after the reloadMethod 2: Separate Path and Turning Off the Log
- How it works The health check is made to a separate path (e.g.
/health), and on the web server the access log is turned off only for that path (e.g.location = /health { access_log off; }in Nginx). - Where it is opened Profile → Path + Query · the log configuration of the web server
- Requirement The health address being on a separate path (What kind of /health endpoint should I design in my application for health checks?)
- vService type All
Step-by-Step Configuration
On the Load Balancer → Profiles → Health Checks screen, the profile is selected and Edit is clicked.
The health path is entered in the Path + Query field (e.g. /health).
The Edit button is clicked.
On the web server, the access log is turned off for this path and the configuration is reloaded.
Verification
No new line for the health path is created in the access log; the status stays OK in the Backend Services list.
No /health line · OK (green)Method 3: Fixed Source IP
- How it works The checks come from the selected TR7 IP; the server excludes this address from the logs. It can also be used in non-HTTP types (TCP, LDAP, SMTP…).
- Where it is openedLoad Balancer → Backend Services → Edit → Details → Health Check Source IP → Device IP
- Requirement A TR7 IP in the Route table of the Backend Service
- vService type All
Step-by-Step Configuration
On the Load Balancer → Backend Services screen, the Backend Service is right-clicked and Edit is selected.
In the Health Check Source IP field in the Details section, Device IP is selected and the address is selected from the list that opens; for the details of the steps and the WEB-CLI equivalent: Which source IP are health check packets sent from; can I change it?. In a cluster, each device uses its own address in this record.
The Edit button is clicked.
In the log configuration of the web server, these source addresses are excluded (e.g. map $remote_addr $loggable in Nginx, SetEnvIf Remote_Addr in Apache).
Verification
No new line from the selected source address is created in the access log.
No line from the selected TR7 IPMethod 4: Separate Port
- How it works The check is directed to a port whose logging is configured separately or turned off; data traffic continues to go to the main port.
- Where it is openedLoad Balancer → Backend Services → Edit → Details → Use Different IP for Health Check → Different Address
- Requirement The server publishing health information on a separate port
- vService type All
Step-by-Step Configuration
On the Load Balancer → Backend Services screen, the Backend Service is right-clicked and Edit is selected.
In the Use Different IP for Health Check field in the Details section, Different Address is selected; the IP of the server and the health port are entered. For details: Can I check a different IP/port of the server for the health check (Use Different IP for Health Check)?.
The Edit button is clicked.
Verification
No check line is created in the access log of the main port; the status stays OK in the Backend Services list.
No check line in the main log · OK (green)Method 5: Increasing the Period
- How it works The number of log lines is lowered by reducing the check frequency; in return, a failure is noticed later (How many seconds does it take for a server to be considered "down", and how do I calculate it?).
- Where it is openedLoad Balancer → Profiles → Health Checks → profile → Edit → Period
- Requirement The longer detection time being acceptable
- vService type All
Step-by-Step Configuration
On the Load Balancer → Profiles → Health Checks screen, the profile is selected and Edit is clicked.
The Period is raised, for example, to 5 seconds; the number of requests drops to about one fifth. The value is kept below 120 seconds.
The Edit button is clicked; all the Backend Services that use the profile are affected.
Verification
The number of check lines per minute in the access log drops.
About 12 lines per minute (Period 5 s, single device)Common Mistakes
- Trying to solve it by turning off Enable HC Logs. This setting only stops the records on TR7; the requests sent to the server stay the same (What does Enable HC Logs do; should I turn it off?).
- Removing the health check completely. Server failures are not noticed and traffic is sent to the down servers.
- Switching from HTTP to TCP. A TCP check does not produce HTTP log lines, but it does not catch application failures either.
How it is done in the documentation
