Homework 4

Release With Evidence

Use information from the running system to understand a customer-facing change, deliver it safely, and decide what should ship.

Observability Progressive delivery Telemetry Release engineering

Release Date Oct 19, 2026

Due Date Nov 1, 2026

Weight 200 pts

Overview

Course staff will supply a customer-facing change to Tartan Tickets. This new version is the candidate. Your job is to try it with a limited set of simulated customers, watch what happens, and decide whether it should ship.

In the first week, measure how the current version behaves to establish a baseline for comparison. In the second, build controls for choosing who receives the candidate and use the results to promote, pause, or abort it.

🎯 Learning Goals

After completing this assignment, you should be able to:

• Use records and measurements to connect what customers experience to what the system does
• Compare release approaches and the risks each can control
• Build controls that limit who receives a new version and switch back when needed
• Adjust a rollout as you learn from the running system
• Explain a release decision and the uncertainty that remains

🚩 Checkpoints

Checkpoint	Points	Due	Last accepted
Checkpoint A: Observable Baseline	80 pts	Oct 25, 2026	Oct 29, 2026
Checkpoint B: Delivery and Final Decision	120 pts	Nov 1, 2026	Nov 5, 2026

Due and last-accepted times are 11:59 p.m. Eastern Time.

Late Days Carry Across Checkpoints

HW4A and HW4B use the same course-wide late-day balance. Each late day used for HW4A moves your HW4B due time one day later; staff will record the new due time in Canvas. HW4B's November 5 final submission deadline stays fixed. Staff will adjust deadlines without charging late days if a staff delay or course-tool outage prevents you from working.

Course Support

Staff will supply a baseline module in hw4/baseline/ that preserves your application's existing behavior. The PR will explain the behavior, interface, and how to connect it to your application. For Checkpoint A, connect and deploy it following those instructions.

For HW4B, staff will open a pull request (PR) on your repository that adds the candidate in hw4/candidate/. It may arrive while you are finishing HW4A; continue using the baseline until your HW4B plan and release controls are ready. Both modules use the same interface, so the code you wrote to connect the baseline will also work for the candidate. Do not create that directory yourself.

You choose the release controls and observability tools. Your system will now begin to receive simulated course traffic. Checkpoint A gives you time to learn how to observe it.

Requirements

Requirement 1: Understand the Current Behavior

For Checkpoint A, use simulated course traffic to collect evidence about the ticket purchasing journey. Save queries or views that show:

• What simulated customers experienced during their sessions.
• What your system did during those sessions.
• Whether records are missing or out of date.

Use metrics, logs, traces, or a combination; alerts and automatic rollback are optional. HW4A grades whether this behavior is observable, including failures; you do not need to diagnose or fix them. For Checkpoint B, also show which version each session received and what changed for customers or the system.

Start With One Traffic Run

Note when the run starts and ends, then check that your views show records from that time window. Find one customer's session and the system records for it. Save the queries so staff can view the same records.

Run your observability tools in production and commit their configuration, including saved queries and the check for missing data. Retain at least 30 days of records and set a limit on data size. Exclude secrets, personal data, and other sensitive data from your records. Give staff read-only access.

Requirement 2: Prepare the Release

Review and merge the candidate PR, then connect it to your application in the same way as the baseline version. You may inspect the supplied code, but keep both supplied modules unchanged. You may change the surrounding application and deployment.

Keep both versions available and make it possible to tell which one a session received. After merging, keep serving the baseline version to everyone until you are ready to try the candidate. Build controls that show who receives the candidate and let you change that or turn the candidate off without rebuilding the application. Record each change to who receives the candidate in your report's rollout log.

Before sending customer traffic to the candidate, commit hw4/PLAN.md to main using the template.

If you change your risk assumptions, traffic limits, or decision criteria, commit the updated plan before acting on it. Do not rewrite earlier commits of the plan.

Requirement 3: Try the Candidate and Decide

Use simulated course traffic to try both versions in production. Some simulated customers must actually receive the candidate's results; running it only in the background for comparison does not count.

Stay within the traffic limits in your plan and use what you observe to decide what to do next. You choose how many stages to run and how far to increase traffic. Stop candidate traffic if you reach a safety or correctness limit.

Compare the versions under similar traffic and starting conditions. Explain differences that matter to your chosen risk and what the data cannot yet tell you. Formal statistical tests and a fixed number of sessions are not required. Follow course instructions for resetting test data; preserve the business data needed for HW5.

When Evidence Is Inconclusive

Check that you know which version each session received and that the records are complete and up to date. Hold or reduce candidate traffic while you investigate.

Choose a final decision and leave the running system in the matching state:

Decision	What to leave running
Promote	Send all traffic to the candidate and keep a way to switch back.
Pause	Send all traffic to the baseline version. Explain what you would need to learn before trying the candidate again.
Abort	Send all traffic to the baseline version. Explain why the candidate should not ship.

Stopping Can Be a Successful Outcome

A justified pause or abort can earn full credit. Your grade depends on your controls, decisions, and reasoning, not on whether the candidate helps or harms customers. Sending more traffic earns no extra credit, and a good outcome does not excuse unsafe choices.

Documents and Evidence

Copy the templates below into hw4/PLAN.md and hw4/REPORT.md and replace the prompts. Short bullets are enough. Complete the report's Checkpoint A section for HW4A and its Checkpoint B section for HW4B.

Save screenshots in hw4/evidence/ and embed them in the report. Each screenshot must show its time window, with the query or link to the saved view that produced it underneath. Staff grade from the report and check a sample against your running system, so keep read-only access working through November 5 at 11:59 p.m. Eastern, or your approved later deadline.

Template: hw4/PLAN.md

# HW4 Plan

## Risk
- What is the most important thing that could go wrong?
- What records or measurements would reveal it?

## Release approach
- What approach will you use, and why does it fit that risk?
- What is one reasonable alternative? Compare how each limits traffic and what each lets you learn.

## Rollout
- Who will receive the candidate first, and how much traffic will you allow?
- What results would make you increase, hold, reduce, or stop candidate traffic?

## Switching back
- How will you switch back to the baseline version?
- What effects would remain afterward?

Template: hw4/REPORT.md

# HW4 Report

## Checkpoint A: Baseline

- Staff access: How staff can view your dashboards and saved queries.
- Baseline window: Start and end times, with time zone.
- Summary: What customers experienced and what the system did (2–3 sentences).

### Customer experience

![Customer experience](evidence/a-customers.png)

Query: Query text or link to the saved view.

### System behavior

![System behavior](evidence/a-system.png)

Query:

### One session, from customer to system records

![One session](evidence/a-session.png)

Query:

### Missing or out-of-date records

![Missing or out-of-date records](evidence/a-completeness.png)

Query:

## Checkpoint B: Release

### Rollout log

| Time (ET) | Change to who receives the candidate | Plan commit | Deployed commit | Why |
| --- | --- | --- | --- | --- |
| | | | | |

### Stage 1: Who received the candidate

Copy this section for each stage.

- Window: Start and end times, with time zone.

![Baseline and candidate compared](evidence/b-stage1.png)

Query:

- What you observed and decided (1–2 sentences):

### Final decision

- Decision: Promote, pause, or abort.
- Who now receives each version:

![Final state](evidence/b-final.png)

- Why the evidence supports this decision:
- How your plan changed, if at all:
- One remaining risk or uncertainty:

Add an ## HW4 entry to DISCLOSURE.md describing any AI tools used and how, or stating that you used none. Update it before HW4B to cover the complete assignment.

Staff grade the repository, running system, and evidence available at your latest accepted Canvas submission.

Submission and Deliverables

Finish the documents on main and check that CI passes. Deploy that commit through your CD pipeline to your registered production target, then submit its GitHub URL in Canvas. Use these checklists to check your work.

HW4A: Observable Baseline

Due Sunday, October 25 at 11:59 pm ET.

Last accepted Thursday, October 29 at 11:59 pm ET.

• Connect the supplied baseline code to your application and deploy it.
• Commit your observability configuration and saved queries, and give staff read-only access.
• Complete the Checkpoint A section of hw4/REPORT.md using the template.
• Add an HW4 entry to DISCLOSURE.md.
• Submit the GitHub commit URL in HW4A on Canvas.

Open HW4A in Canvas→

HW4B: Delivery and Final Decision

Due Sunday, November 1 at 11:59 pm ET.

Last accepted Thursday, November 5 at 11:59 pm ET.

• Commit hw4/PLAN.md to main using the template before sending customer traffic to the candidate. Commit any later changes to the plan before acting on them.
• Try the candidate with a limited set of simulated customers and decide whether to promote, pause, or abort it.
• Complete the Checkpoint B section of hw4/REPORT.md, including the rollout log.
• Deploy your submitted commit and keep staff access working through November 5, or your approved later deadline.
• Update the HW4 entry in DISCLOSURE.md to cover the complete assignment.
• Submit the GitHub commit URL in HW4B on Canvas.

Open HW4B in Canvas→

Grading

Each criterion allows partial credit. Full credit requires working features and evidence staff can check. Incomplete work can earn partial credit; missing work or claims without supporting evidence earn little or no credit.

The assignment is worth 200 points.

HW4A — Observable Baseline80 pts

• The supplied baseline code is connected to the production application. Your evidence shows what simulated customers experienced, what the system did, and one session traced from customer to system records.0–40 pts
• Your baseline measurements describe the time window and observed behavior. A saved query or check reveals missing or outdated data.0–25 pts
• Observability configuration and saved queries are committed. At least 30 days of records are retained within a size limit, records contain no sensitive data, and staff have read-only access.0–15 pts

HW4B — Delivery and Final Decision120 pts

• Before trying the candidate, you identify a risk, compare two reasonable release approaches, and explain your traffic limits and the evidence you will use.0–20 pts
• Both supplied versions remain unchanged and available. You can check, limit, change, and stop candidate traffic without rebuilding, and your rollout log records each change.0–40 pts
• You follow your plan or update it before changing course. Your traffic decisions follow from the evidence, and you stop when a safety or correctness limit is reached.0–35 pts
• Your evidence supports the final decision, and the running system matches it. You explain changes to the plan and what remains uncertain.0–20 pts
• Your plan, report, and AI disclosure are complete and follow the templates. CI passes on `main`, the submitted commit is running in production, and submitted evidence contains no sensitive data.0–5 pts