How do I follow a specific user's problem in live traffic?
Short answer
The most direct way is the tcpdump tab on the vService screen; when the address of the user is written in the Client IP filter field and the capture is started, the packets of that client flow onto the screen live. For request-level detail in the HTTP vService, Debug Mode is turned on with a condition that matches only the IP of the user, and the Debug Mode Logs screen in Logs & Tracking is followed with Refresh. The Live Traffic table in Live Stats shows the busiest source IPs of the last 10 seconds, and the vService Sessions list shows the current open connections of the user. When the monitoring is finished, Debug mode is turned off and tcpdump is stopped.
Problem: The Problem of a Single User Cannot Be Reproduced
The problem occurs only for a specific user, and the traffic of that user needs to be monitored instantly while the user repeats the operation. Among the traffic of the other users, only the packets and requests of this client are to be seen. First, the IP address of the user as seen by TR7 is determined: if there is a CDN or a proxy in front of TR7, the source address in the packets belongs to that proxy.
Solution: Four Methods
The methods can be used together: tcpdump shows the packet flow, Debug mode shows the request detail, and Live Stats and vService sessions show whether the user is producing traffic at that moment.
| Your situation | Suitable method |
|---|---|
| Live flow at packet level (all vService types) | Method 1: tcpdump + Client IP Filter |
| Request-level detail (HTTP) | Method 2: Conditional Debug mode |
| Is the user producing traffic right now | Method 3: Live Stats → Live traffic |
| The open connections of the user | Method 4: vService sessions |
Method 1: tcpdump + Client IP Filter Recommended
- How it works The capture is taken on the frontend addresses of the vService and on the Backend Services; Client IP Filter keeps only the packets that carry the address of the user.
- Where it is opened Left Monitor Panel → Load Balancer vServices → vService → the tcpdump icon in the right panel header
- Requirement A running vService; the IP address of the user as seen by TR7
- vService type HTTP · L7 TCP · Network
Step-by-Step Configuration
In the Left Monitor Panel, Load Balancer vServices is selected from the category selector and the vService is clicked (accessing the screen).
The tcpdump icon in the header of the right panel, or tcpdump in the ⋮ menu to the right of the tab bar, is clicked; a closable tcpdump tab opens.
The Start button in the tab is clicked.
Client IP Filter is enabled and the address of the user is written; multiple addresses are separated with commas, and a CIDR block can also be written (e.g. 203.0.113.45, 192.0.2.0/24). If needed, Port Filter (e.g. 443) is also enabled. The Duration default is 30 seconds; for long monitoring, 5m or 10m is selected. In long monitoring, Max Packets (default 3000) under Advanced Settings is also increased.
The Accept button is clicked. While the user repeats the operation, the rows flow onto the screen; each row is labeled with the point where it was captured (vService or Backend service). The capture ends when the duration expires or with Stop; the pcap record and the other fields: Can I capture packets on TR7 and download them as a pcap (tcpdump)?
Client IP Filter is applied to all capture points. When the SNAT field of the Backend Service is left at the default (Default), the source of the packets going to the Backend Service is the address of TR7; these packets do not pass the IP filter of the user. If the Backend Service leg is also to be monitored, a second capture is started with Client IP Filter off, Frontends = Off, and narrowed with Port filter.
Verification
The rows and the status bar in the tcpdump tab are checked.
Rows labeled "vService" that carry the IP of the user · CapturingMethod 2: Conditional Debug Mode
- How it works Debug mode records only the requests that match the IP of the user; the records contain the timing, headers, body, Termination State and the security decision.
- Where it is opened vService edit form → Details → Debug → Select conditions
- Requirement A Source IP condition that selects the IP of the user
- vService type HTTP
Step-by-Step Configuration
If there is no condition that selects the IP of the user, the Add button is clicked on the Load Balancer → Conditions screen (how to add).
Criteria = Source IP and Source IP Type = Manual are selected, and the address of the user is written in the Source IP field (e.g. 203.0.113.45).
The Add button is clicked.
On the Load Balancer → Load Balancer vService List screen, the HTTP vService is selected and the Edit button is clicked.
Details is opened, the Debug box is checked, and the created condition is selected from the Select Conditions field in the Debug group.
The Save button is clicked.
Left Monitor Panel → vService → Logs & Tracking → Debug Mode Logs is opened; the list does not refresh on its own, and the Refresh button is clicked as the user performs operations. Interpreting the records: How do I find out from the logs why a request failed (termination state)?
When the monitoring is finished, the vService is selected on the Load Balancer → Load Balancer vService List screen and the Edit button is clicked.
In the Details panel, the Debug box is cleared (or Enabled is selected in the Status field). Debug mode does not turn off on its own (What does Debug Mode record; how long should I keep it on?).
The Save button is clicked.
Verification
After Refresh, the Client IP column of the Debug Mode Logs table is checked.
Only the requests that come from the IP of the userMethod 3: Live Stats → Live Traffic
- How it works The Live Traffic table shows at most 20 source IPs that stood out in the last 10 seconds, with their request, connection, error and traffic values.
- Where it is opened vService → Live Stats → Live Traffic
- Requirement The user producing traffic at that moment
- vService type HTTP · L7 TCP (this table does not exist in the Network vService)
Step-by-Step Configuration
In the Left Monitor Panel, Load Balancer vServices is selected from the category selector and the vService is clicked.
The Live Stats tab is clicked.
In the Live Traffic table, the address of the user is looked for in the Source IP column. Since the table shows only the busiest sources, a user producing low traffic may not appear in the list; in this case, Method 1 or Method 4 is used.
Verification
The Source IP column is checked.
The IP of the user and its request and connection valuesMethod 4: vService Sessions
- How it works The vService Sessions list fetches the currently open connections of the vService; it is narrowed to the user with the Source IP filter (Client IP in the Network vService).
- Where it is opened vService → Logs & Tracking → vService Sessions
- Requirement A running vService
- vService type HTTP · L7 TCP · Network
Step-by-Step Configuration
Left Monitor Panel → vService → Logs & Tracking → vService Sessions is opened.
In the Filter panel, the address of the user is selected from the Source IP card (the Client IP card in the Network vService) (Filter).
The list does not refresh on its own; the Refresh button is clicked for the current connections (Can I see which connections are currently open (vService Sessions)?).
Verification
After Refresh, the list is checked.
The open connections of the userCommon Mistakes
- Running tcpdump with the real IP of the user behind a CDN. The packets carry the address of the proxy; to search by the real address, the X-Forwarded-For filter in Debug Mode Logs is used (How do I find the requests of a specific client IP in the logs?).
- Expecting the packets of the Backend Service leg while Client IP filter is on. With the SNAT default, the source of these packets is the address of TR7; a separate capture is taken for that leg.
- Turning on Debug mode without a condition or forgetting it on. If there is no condition, all HTTP requests are recorded; Debug is turned off when the examination is finished.
- Treating not seeing the user in Live traffic as "no traffic". The table shows only the busiest 20 sources.
- Closing the tcpdump tab and assuming the capture continues. When the tab is closed, the capture stops.
How it is done in the documentation
