Lab session

Fault Experiments with Toxiproxy

Inject and remove a controlled dependency delay with a self-cleaning Toxiproxy experiment.

Release Date Nov 9, 2026

Due Date Nov 13, 2026

Starter Code Fork repository →

Submission Link

Open assignment in Canvas→

Overview

In this lab, you learn to use Toxiproxy to inject a controlled delay between a small application and one of its dependencies. You will observe the system before, during, and after the fault, then encode that sequence as a repeatable experiment that always cleans up.

🎯 Learning Goals

After completing this lab, you should be able to:

• Explain where a Toxiproxy proxy and toxic sit in a request path
• Add, inspect, and remove a latency toxic with the Toxiproxy CLI
• Relate an injected network condition to an observed service response
• Complete and run a supplied self-cleaning fault experiment

Deliverables

By the end of the lab, your fork should complete the supplied self-cleaning Toxiproxy experiment so it distinguishes healthy, delayed, and recovered checkout behavior. You will submit the URL of the specific commit containing that experiment through Canvas.

Setup

Fork the starter repository to your own account and open it in the provided devcontainer. Do not push commits to the upstream course repository. Work entirely in your fork and run every command from the repository root.

The devcontainer starts a small Docker Compose stack:

• app exposes GET /checkout at http://localhost:3000
• pricing exposes GET /price and normally responds quickly
• toxiproxy sits between the app and pricing and exposes its administration API on port 8474

The app already calls pricing through a proxy named pricing. It returns a bounded HTTP 503 response when pricing remains unavailable past its request deadline. Only edit lab08/experiment.sh; the services, stack, and checker are supplied context.

Run the supplied checker before changing anything:

./scripts/check-lab

The initial check should report that the incomplete experiment cannot create its latency toxic. It should not report missing tools, missing files, or an unavailable stack. Restart a stopped stack with ./scripts/start-lab.

Tasks

Task 1: Observe a Controlled Delay

First establish the healthy baseline:

curl --fail-with-body \
  --write-out '\nhttp_code: %{http_code}\ntime_total: %{time_total}s\n' \
  http://localhost:3000/checkout

The request should return HTTP 200 quickly.

A proxy and a toxic have different jobs

A Toxiproxy proxy is a named TCP listener with an upstream destination. The app calls that listener, which normally forwards the connection to pricing unchanged. A toxic changes the bytes or timing on the proxied path while both services continue running. Toxiproxy operates on TCP streams, so it creates the network condition but does not choose the HTTP status or JSON returned by the app.

List the configured proxies and inspect the request path:

toxiproxy-cli list
toxiproxy-cli inspect pricing

Before injecting a fault, predict what checkout will return and how long it will take when pricing responses are delayed by three seconds.

Use toxiproxy-cli toxic add --help to add a downstream latency toxic with these properties:

• Proxy: pricing
• Toxic name: pricing_latency
• Toxic type: latency
• Latency: 3000 milliseconds

Inspect the proxy again to confirm that the toxic is active, then repeat the checkout request. Relate the observed status, body, and duration to the app's bounded dependency behavior.

Use toxiproxy-cli toxic remove --help to remove pricing_latency, inspect the proxy, and confirm that checkout returns to its healthy behavior.

Check your reasoning: What did Toxiproxy control?

Toxiproxy controlled the delay on the connection from pricing to the app. The app's own timeout and error handling converted that condition into a bounded HTTP response for its caller.

Task 2: Automate the Experiment

Open lab08/experiment.sh. The script already captures and checks the baseline, delayed, and recovered responses and registers cleanup traps for normal exit, assertion failure, and interruption.

Complete inject_latency and remove_latency with the Toxiproxy operations you used manually. Use the supplied proxy name, toxic name, and latency variable rather than embedding an alternate experiment. Do not remove the supplied assertions or cleanup trap.

Run the experiment:

./lab08/experiment.sh

It should inject the delay, observe the bounded failure, remove the toxic, observe recovery, and exit zero. Inspect the proxy afterward to confirm that pricing_latency is absent.

Cleanup is part of the experiment

A fault experiment can stop because an assertion fails or a person interrupts it. Cleanup therefore belongs in a trap that runs on every exit path, not only in the final successful step.

Run the complete checker:

./scripts/check-lab

The checker runs your experiment against the supplied stack and confirms that the toxic is absent afterward. It also runs the experiment against a control service that stays healthy during the injected delay; the experiment must reject that observation and still clean up.

Check your reasoning: Why use an always-healthy control?

A script that only runs commands or always exits zero does not establish that the fault changed behavior. The control proves that the experiment checks its observation as well as injecting and removing the condition.

Additional Resources

• Toxiproxy documentation: proxy configuration, toxic types, and CLI examples
• curl manual: exit codes and --write-out: recording response status and duration
• Bash trap: registering cleanup for script exit and signals

Submission

1. Commit lab08/experiment.sh, then push that commit to your fork.
2. Confirm that git status --short produces no output, then run ./scripts/check-lab one last time.
3. Submit the URL of that exact commit—not a branch or pull request—using the Canvas link below.

Open assignment in Canvas→

This lab is graded pass/fail.