Can I send a copy of the traffic to a test environment (Traffic Mirroring)?

Short answer

Yes, with the Traffic Mirroring action on the HTTP vService. In the Mirror Target Service field of the action, the Backend Service in the test environment is selected, and the action is added to the Actions field of the vService; a copy of the requests goes to the test environment, the user always receives the response from the actual Backend Service, and the response to the copy is discarded. By adding a condition to the Conditions field, only certain requests are mirrored; by turning on Sampling, a percentage of the requests is mirrored. The Authorization and Cookie headers are removed from the copy by default (Headers removed from the copy).

Problem: The New Environment Must Be Tested with Real Traffic

A new version or infrastructure needs to be tested with real traffic to see the problems that synthetic tests cannot catch; users must not be affected by this test.

Solution

Traffic Mirroring Action

• How it works A copy of the request coming to the vService is sent to the Mirror Target Service; the user's request goes to the actual Backend Service in the normal way. If Conditions is empty, all requests are mirrored; if a condition has been added, only the requests that match the condition are mirrored; if Sampling is on, the selected percentage of them is mirrored.
• Where it is openedLoad Balancer → Actions → Traffic Mirroring · vService → Actions
• Requirement HTTP vService; a single Backend Service to which the copy is sent, with an IP address and port defined in the Address field; a Backend Group cannot be selected
• vService type HTTP
• Limit At most 8 Traffic Mirroring actions are added to a vService (How many Traffic Mirroring actions can I use on a vService at most?)

Step-by-Step Configuration

1

Adım 1 Add the target Backend Service (if needed)

If the server in the test environment is not yet defined as a Backend Service in TR7, the Backend Service Name and Address (IP address and port) are entered with Load Balancer → Backend Services → Add and saved with Add (How is a Backend Service added; which information is required?). No copy is sent to a Backend Service with Use Frontend Port selected; the port must be entered explicitly.

2

Adım 2 Create the condition (if only certain requests are to be mirrored)

If all requests are to be mirrored, this step is skipped. Otherwise, a condition is created with Load Balancer → Conditions → Add (how to add); for example, Criteria = Path, matching Start and the value /api/ are entered, and the condition is saved with Add.

3

Adım 3 Open the action add form

On the Load Balancer → Actions screen, the Add button is clicked.

4

Adım 4 Select the type and the target

Action Type = Traffic Mirroring is selected; in the Mirror Target Service (required) field, the Backend Service in the test environment is selected.

5

Adım 5 Review the copy settings

Defaults: Maximum Body Size (KB) 256, Shadow Response Timeout 2 s, Concurrent Copy Limit 500, Add X-Forwarded-For Header off. The Headers removed from the copy list contains Authorization, Cookie and Set-Cookie by default (at most 16 headers); other headers that the test environment must not see are added here.

6

Adım 6 Select the condition (optional)

If a condition was created in Step 2, it is selected in the Conditions field of the action. If Conditions is left empty, all requests coming to the vService are mirrored.

7

Adım 7 Turn on sampling (optional)

If only a percentage of the requests is to be mirrored, Sampling is turned on (off by default); Sampling Percentage (%) is entered (default 10, range 0–100). With Sampling Key in the Random (default) position, the decision is made independently for each request; in the Keyed position, the field is filled with the Client IP address variable, and the requests of the same client are either all mirrored together or not mirrored at all.

8

Adım 8 Save the action

The Add button at the bottom of the form is clicked.

9

Adım 9 Open the vService edit form

On the Load Balancer → Load Balancer vService List screen, the vService is selected and the edit form is opened.

10

Adım 10 Add the action to the vService

The created action is added in the Actions field of the Details panel; creating the action alone is not enough.

11

Adım 11 Save the vService

The Save button is clicked. In a cluster, the change is transferred to the peer device automatically with auto-sync (default); if Manual Sync Mode is on, it is sent with Synchronize.

Verification

1MirroringTR7 side

The Mirrored/s, Skipped/s and Dropped/s values in the Traffic Mirroring section of the Live Stats screen of the vService are checked; for the past flow, the Mirror Flow chart in the Traffic Mirroring tab of the Dashboard is checked (monitoring the mirroring). If Sampling is on, the skipped requests appear in Skipped/s.

ExpectedMirrored/s > 0

2Test environmentBackend Service side

The copy requests are checked in the access log of the server in the test environment.

ExpectedThe same path and parameters as the requests coming to the actual server; no Authorization and Cookie headers

Common Mistakes

• Forgetting to bind the action to the vService. The created action does not work unless it is added to the Actions field of the vService.
• Expecting to see the response of the test environment. The response to the copy does not return to the client and is discarded.
• Thinking that the body is also cleaned. Only the headers in the Headers removed from the copy list are removed; the request body is copied as it is. Requests whose body is larger than the Maximum Body Size (KB) value are not mirrored.
• Testing Keyed sampling from a single client. In key-based sampling, the same client is either always mirrored or never mirrored; the test is done from different clients or in the Random position.
• Mirroring requests with side effects. The test environment actually processes the copy requests; requests that touch external systems, such as payment or e-mail, are excluded from mirroring with a condition.
• Configuring the peer device separately in a cluster. While auto-sync (default) is on, the configuration is done on a single device and passes to the peer device on its own; Synchronize is needed only while Manual Sync Mode is on.

How it is done in the documentation

• Traffic Mirroring · How to Add?
• Traffic Mirroring · Sampling
• Traffic Mirroring · Headers Removed from the Copy
• Traffic Mirroring · Monitoring the Mirroring

Related questions

• How many Traffic Mirroring actions can I use on a vService at most?
• How do I do a canary or blue-green deployment?
• Can I send only a certain percentage of traffic to a new version?