Privacy-aware observability

What you will be able to do

Telemetry is a copy of what happens to every customer’s ticket, so it can leak what customers wrote. Decide what goes into it, and remove private text before it leaves the service. Keep fewer traces, limit who can read them and for how long, and measure what each choice costs.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Logs, metrics and traces and Quality and costs.
• Data minimisation and retention from Minimize and retain deliberately in AI Security, Privacy and Responsible Design.

What you need on your computer

• Your project at the snapshot end-of-m03. To run the settings yourself, Docker for PostgreSQL and Valkey, as in the first lesson of this module.

Without Docker, read the author’s outputs below, and do the guided practice and the task at the end: they use only the saved telemetry.

The camera in the shop

A shop has a security camera at the entrance. It helps: when a parcel goes missing, the owner can see who came in. But if the camera also films the card machine at the till, every recording holds customers’ PIN codes. Now the recordings are a danger, and anyone who can watch them is a danger too.

So a careful shop points the camera away from the card machine. It keeps the recordings for 30 days and then deletes them. It locks the room with the screen, and only two people have the key. And it does not keep every minute of every camera in full quality; that would fill the disks.

Each of these choices has a name in telemetry. Pointing the camera away is minimisation: do not collect what you do not need. Covering a part of the picture is redaction. Deleting after 30 days is retention. The locked room is access control. Keeping only some recordings is sampling.

The problem: “log everything, just in case”

Tomás has a proposal: “Put every prompt and every answer in the traces, in full. When something breaks, we will have everything.”

The service can do that. TELEMETRY_CAPTURE_CONTENT=true adds the ticket’s subject and body, the messages sent to the model, the model’s output and the reply. The author ran it with redaction switched off (TELEMETRY_REDACT=false) and TELEMETRY_FILE=telemetry-raw/{service}.jsonl. The run sent five practice tickets, and one more ticket written by hand:

curl -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" \
  -d '{"customer_id": "C-0022", "subject": "Charged twice", "body": "I paid twice for one order. Please reply to sam.lee@example.com."}'

Then ops privacy scanned the two files. It counts what it finds and never prints the text:

python -m ops privacy telemetry-raw/ticket-api.jsonl telemetry-raw/ticket-worker.jsonl

ticket-api.jsonl, ticket-worker.jsonl: 6 tickets
signal     KB  KB per ticket
------  -----  -------------
log      10.1           1.68
metric   23.8           3.97
span    116.4          19.39
all     150.3          25.04
FOUND (counts only; the text is not printed):
  attribute ticket.subject: 6
  attribute ticket.body: 6
  attribute answer.text: 6
  attribute gen_ai.input.messages: 6
  attribute gen_ai.output.messages: 6
  e-mail address: 2

Every ticket’s words are now in the telemetry, five times over, and the customer’s email address is there twice. These files go to Grafana, where every visitor can look, into backups, and into a laptop’s copy for “a quick look”. The ticket database has rules for who reads it and when it is deleted. The telemetry would be a second copy of the same messages, without those rules.

And “just in case” rarely pays back. In this module, no answer about last week needed one word of a ticket. The outcome, the tokens, the finish reason and the time were enough.

The idea: decide before export, then limit the rest

Privacy-aware observability means telemetry that answers the operations questions with as little personal data as possible. It works in layers, from the code to the reader:

The privacy layers of the service's telemetry, from the code to the reader. Each layer has its own setting.

Text description of the diagram

The code is the first layer: by default it describes the ticket without its text. The sampler drops most traces, but never logs or metrics, so every ticket is still counted. Redaction runs inside the process, before any exporter, so the text never reaches a file or the network. After export, retention decides how long the data lives, and access control decides who can read it.

The first layer matters most: describe the input, do not copy it. The first lesson of this module showed the event ticket.received with ticket.message_chars (the length), ticket.language_guess and ticket.message_sha256. These describe the message well enough to measure drift in the next module, and to find repeated messages. None of them holds the words.

Redaction is the safety net, for text that slips in anyway. Before export, it removes every attribute that holds text people wrote, and it masks email addresses and card numbers in all other text. Here is the project’s redact() function on one set of attributes, run on its own:

>>> telemetry.redact({
...     "ticket.body": "I paid twice for one order. Please reply to sam.lee@example.com.",
...     "ticket.message_chars": 64,
...     "note": "customer sam.lee@example.com, card 4111 1111 1111 1111",
...     "team": "payment"})
{'ticket.message_chars': 64, 'note': 'customer [email], card [card]', 'team': 'payment'}

The body is gone: it is on the list of text attributes. The note is kept, but its email address and card number are masked. The length and the team stay, because they are not personal. Masking works by patterns, so it catches only what the patterns know: a name or a street address in free text gets through. That is why removing the text attributes comes first, and masking second.

Larkfield: three settings, measured

The author sent five more practice tickets and the same hand-written ticket with redaction switched on (TELEMETRY_CAPTURE_CONTENT=true, TELEMETRY_REDACT=true), into telemetry-redacted/:

ticket-api.jsonl, ticket-worker.jsonl: 6 tickets
signal    KB  KB per ticket
------  ----  -------------
log      7.4           1.23
metric  21.1           3.51
span    67.6          11.26
all     96.0          16.00
No text attribute, no e-mail address, no card number found.

Nothing found, and the spans are back to their normal size. The course team measured the same three settings on 200 tickets each, so that the fixed size of the metrics matters less:

Kilobytes of telemetry per ticket (API and worker, every trace kept, 200 practice tickets per setting), measured by the course team. Redaction removes exactly what capturing the text adds.

Text description of the chart

Three bars. Default, text off: 12.9 KB per ticket. Text kept without redaction: 21.6 KB, two-thirds more. Text kept with redaction: 12.9 KB, the same as the default. In the second setting, ops privacy found the subject, the body, the prompt, the model’s output and the reply in all 200 tickets. In the other two, it found nothing.

The third bar is the lesson. Capturing the text and then redacting it gives exactly the default, plus a risk. The text exists inside the process for a moment, and one wrong setting exports it. So Larkfield keeps the default, TELEMETRY_CAPTURE_CONTENT=false, and keeps redaction on as the safety net. When Priya needs the words of a failed ticket, she takes its ID from the telemetry. Then she reads the ticket in the ticket database, under its own access rules.

Sampling: fewer traces, every ticket still counted

Most of the size is spans: about 11 of the 12.9 KB per ticket. A busy service does not need the trace of every good ticket. Sampling keeps the trace of only a share of the tickets. TELEMETRY_SAMPLE=0.1 keeps about 1 in 10.

The decision is made once per trace, from the trace ID, at the start, so a ticket keeps all its spans or none. The worker’s span follows the API’s decision, because the job carries the traceparent. Logs and metrics are never sampled, so every SLI still counts every ticket.

The author sent 20 practice tickets with TELEMETRY_SAMPLE=0.1 and TELEMETRY_FILE=telemetry-10/{service}.jsonl, so that each setting had its own files:

python -m ops summary telemetry-10/ticket-api.jsonl telemetry-10/ticket-worker.jsonl

Records: 40 logs, 88 spans, 42 metric records, from +0.0 min to +0.4 min
Ticket requests: 20 (202: 20)
Tickets triaged: 20 (ok: 20)

All 20 tickets are counted, from 40 events. The 88 spans are 4 traces of 22 spans. So 4 tickets of 20 kept a trace, not 2: a share of 10% is a probability, not an exact count. Your number will differ. The size fell to 4.94 KB per ticket. The run of the first lesson, with every trace, had 18.58 KB per ticket, for only 5 tickets.

The course’s saved files show the same effect at scale. The normal week kept 10% of the traces: 34,412 records for 5,754 tickets, 20.4 MB, and 2.2 MB compressed. Last week’s replay kept every trace: 56,334 records for 2,400 tickets, 34.1 MB, and 2.6 MB compressed. Per ticket, that is about 3.5 KB against 14.2 KB.

Sampling saves work in the worker, but not in the API:

Jobs per second for one worker with a simulator that answers at once, median of three passes, measured by the course team. Sampling gives back most of what tracing every ticket costs.

Text description of the chart

Three bars. With telemetry off, the worker finished 238 jobs a second. With every trace, 201: 16% fewer. With 10% of the traces, 229: 4% fewer. The API told a different story. A ticket request took a median of 2.39 ms with telemetry off, 3.66 ms with every trace, and 3.75 ms with 10%. The API still makes its logs, its metrics and its own span for every request, so sampling did not reduce its cost.

Retention and access: how long, and who

Retention is how long telemetry is kept. The setting is TELEMETRY_RETENTION_DAYS=30: long enough to compare this week with last month, and to review an incident. ops prune deletes the older records of a file:

python -m ops prune telemetry/ticket-worker.jsonl --days 30

telemetry/ticket-worker.jsonl: kept 115 records, deleted 0 older than 30 days

Do not run prune on the saved course files in data/telemetry/. They were recorded long before you read this, so it would delete them all. In Grafana, each store has its own setting (Prometheus, Loki and Tempo), and observability/README.md names them. Mei owns telemetry storage, so she sets them and checks them.

Access control decides who can read telemetry. In compose.yaml, people who are not signed in to Grafana are Viewers: they can look at dashboards, not change them or the alert rules. The image’s own default is Admin, so the project changes it. On your computer, the admin account is admin with the password admin. Change it before anyone else can reach port 3000. Files need the same care. The project’s .gitignore already lists telemetry/: never put these files in a repository or a chat message.

When to sample and redact, and what it costs

Use these settings on every service that handles customers’ words. Keep the content capture off. Keep redaction on, even though the default code sends no text: it catches the mistakes of the next person who adds an attribute.

Sample traces when their volume becomes a cost: storage, network, or the worker’s time. Do not sample while the traffic is small, or while you debug: you need every trace then. Sampling has a real cost of its own. A rare, slow ticket may have no trace. In the normal week, only 570 of 5,754 tickets kept a trace. Larger teams use tail sampling: the collector waits until a trace ends, then keeps every trace with an error or a long duration. This project does not do that. It keeps the logs of every ticket, so you can still find and count a failed ticket.

Short retention costs history: after 30 days, the old tickets are gone. Keep the metrics longer than the traces if you need trends: they hold numbers, not tickets.

Common mistakes

Common mistake

No trace for the ticket you need

Symptom: with TELEMETRY_SAMPLE=0.1, the waterfall of a ticket prints:

No trace for ticket T-500018: it was not sampled (TELEMETRY_SAMPLE keeps only some tickets' traces). Its logs are there: python -m ops trace.

Cause: this ticket’s trace ID fell outside the 10% that the sampler keeps. In the author’s run, 16 of the 20 tickets had no trace.

Fix: use the ticket’s logs: ops trace from Artifact lineage reads them, with the outcome, tokens and cost. To debug one problem, set TELEMETRY_SAMPLE=1.0 for a while, and send the ticket again.

Common mistake

A hash is not an anonymous value

Symptom: someone suggests that the telemetry has no personal data, “only hashes”.

Cause: ticket.message_sha256 and customer.ref are pseudonyms: the same text, or the same customer, always gives the same value. Anyone who has a customer’s message can compute its hash and find it in the telemetry. Many privacy laws still treat such values as personal data.

Fix: treat the telemetry as personal data anyway: retention, access control and no copies outside the team. Ask your privacy lead what applies to you. The hashes are short on purpose (16 and 10 characters): enough to find repeats, and less useful as a key.

Knowledge check

Check your understanding

Tomás says: 'Log every prompt and answer in full, just in case.' Answer him with two measurements from this lesson.

Show the answer

With the text captured and no redaction, ops privacy found the subject, the body, the prompt, the model’s output and the reply in every ticket. It also found the customer’s email address twice. Every reader of Grafana and every copy of the files would hold customers’ messages, without the rules of the ticket database. It also made the telemetry two-thirds bigger: 21.6 KB per ticket instead of 12.9 KB. And it was not needed: outcomes, tokens, finish reasons and times answered every question about last week.

Check your understanding

Predict: you set TELEMETRY_SAMPLE=0.1. What happens to the accept and draft SLIs, to the waterfall of a random ticket, and to the API's time per request?

Show the answer

The SLIs do not change: they come from logs and metrics, which count every ticket. A random ticket has a trace with a probability of about 10%; for the others, ops waterfall prints “No trace … it was not sampled”. The API’s time per request stays about the same: 3.66 ms with every trace, 3.75 ms with 10%. It still writes its events, its metrics and its own span. The worker gains most of its speed back: 229 jobs a second instead of 201.

Check your understanding

Practical check: run ops privacy on your own telemetry files from the first lesson of this module. What does it print, and why?

Show the answer

python -m ops privacy telemetry/ticket-api.jsonl telemetry/ticket-worker.jsonl prints the size per signal and No text attribute, no e-mail address, no card number found. The default settings never add the text (TELEMETRY_CAPTURE_CONTENT=false), and redaction is on as a safety net (TELEMETRY_REDACT=true). In the author’s run of 5 tickets, the files held 18.58 KB per ticket, most of it spans. Your sizes will differ a little.

Guided practice: a ticket that waited 29 seconds

This module’s last skill is to find a slow, expensive step in a trace. First, follow a worked example with every step visible. The file is a practice run of a later module, with the same service.

1. Print the trace of the slowest sampled ticket:

   python -m ops waterfall data/telemetry/canary.jsonl.gz --slowest

   Trace 0193d235f0d3d4ec0dcc2ee84c1571bf, ticket T-501092 (the slowest of 2400 sampled): 33 spans, 29,239 ms from first to last
   span                          start ms        ms                                        cost US$
   POST /v1/tickets                     0      14.7  #
     (waiting in the queue)            15      15.5  #
     process ticket job                30  10,027.4  ############
       triage ERROR                    32  10,018.9  ############
         chat chat-small ERROR         32  10,018.3  ############
           POST ERROR                  33  10,015.6  ############
     process ticket job            11,691  10,029.9                ############
       triage ERROR                11,698  10,016.8                ############
         chat chat-small ERROR     11,698  10,015.3                ############
           POST ERROR              11,698  10,013.6                ############
     process ticket job            24,980   4,259.2                                #####
       triage                      24,984   4,251.0                                #####
         chat chat-small           24,984   4,248.0                                #####   0.000388
           POST                    24,985   4,245.2                                #####
       embed                       29,235       2.5                                     #
         cache get                 29,235       0.5                                     #
   (18 database queries hidden; --all shows them)
   Longest step: triage, 10,019 ms (34% of the trace)
   Costliest step: chat chat-small, US$0.000388 of US$0.000388: 1894 tokens in, 398 out (288 of them reasoning), finish stop

2. Count the attempts: three process ticket job spans, so the job ran three times.

3. Read the first two attempts: each chat chat-small ERROR lasts 10.0 s, the provider timeout. They have no cost and no tokens: no answer came back.

4. Read the gaps: attempt 2 starts at 11,691 ms, 1.6 s after attempt 1 ended. Attempt 3 starts at 24,980 ms, 3.3 s after attempt 2. These are the retry waits, and ops draws no bar for them.

5. Read the third attempt: a normal answer in 4.25 s, US$0.000388, finish stop.

6. Add it up: 20.0 s of timeouts, 4.9 s of retry waits, 4.3 s of real work. The bottleneck is the timeouts, 69% of the 29.2 s. The costliest step is the one call that worked.

ops says “Longest step: triage, 34%”. That is true for one span, but it misses the pattern: two failed attempts of the same step. Read the whole waterfall. The ticket’s practice message has several recordings, and the replay chose a recording longer than 10 s twice. This is the timeout problem of Quality and costs, and here it delayed an answer by 25 seconds.

Your turn: find the slow, expensive stage

Run python -m ops waterfall data/telemetry/shadow.jsonl.gz --slowest. This file is another practice run of a later module. Find the slowest stage and the most expensive one. Say what the trace proves and what it cannot tell you, and which signal would tell you the rest. Do not use the model’s reasoning: you cannot read it.

A worked solution

The output starts with Trace f265715a2074a667f3e7bc9a404b3fd5, ticket T-505297 (the slowest of 588 sampled): 26 spans, 67,073 ms from first to last and ends with:

Longest step: (waiting in the queue), 26,782 ms (40% of the trace)
Costliest step: chat chat-small, US$0.000388 of US$0.000388: 1894 tokens in, 398 out (288 of them reasoning), finish stop

Read it as in the guided practice:

Stage	Start (ms)	Time	Cost
API request	0	17 ms	–
Waiting in the queue	17	26.8 s	–
Attempt 1: chat chat-small ERROR	26,803	10.0 s (timeout)	none recorded
Gap before attempt 2 (retry wait, queue)	36,821	26.0 s	–
Attempt 2: chat chat-small	62,802	4.25 s, finish stop	US$0.000388
embed	67,060	8 ms (from the cache)	–

Slowest stage: the queue, 26.8 s, 40% of the trace. With the gap before the second attempt, also time in the queue, the ticket waited about 53 of its 67 seconds. Most expensive stage: the second triage call, the only call with a cost. The slow stage and the expensive stage are different here.

What the trace proves: the AI work was only about 14 s of 67 s, and only 4.25 s of it gave an answer. Most of the time, the ticket waited for a free worker, not for the provider.

What it cannot tell: why the queue was long. A trace shows one ticket. The queue gauges show all of them: in the file, app.queue.depth reported 49 jobs queued and 16 running when this ticket arrived. That is every slot of the two workers, 8 jobs each. In this practice run, most jobs also made a second triage call for a later module’s test, a shadow run. So each job held its slot longer.

What to do: the fix is capacity or less work per job (Amira), not the prompt.

Check your understanding

Completion check: explain the bottleneck of ticket T-503354 in last week's replay without needing the model's hidden reasoning. Use only what the trace shows.

Show the answer

python -m ops waterfall data/telemetry/release-full.jsonl.gz --ticket T-503354 shows a 9.2 s trace. The API took 17 ms, and the queue wait 165 ms. The triage call chat chat-small took 9,008 ms, 98% of the trace. It had all of the cost, US$0.000650. Its attributes say why: 1,000 tokens out, all of them reasoning, and finish length. The model reached the token limit before it wrote the answer, so the job is marked ERROR and has no embedding. The reasoning’s text is hidden. But its size and its time are in the span, and they are enough. The bottleneck is prompt v3 with this token limit, which is Priya’s area. The API, the queue, the database and the worker are not the cause.

Module recap

In this module, you made the ticket service observable. In the first lesson, you added OpenTelemetry: events with fixed names, metrics for every ticket, and one trace per ticket across the queue. In the second, you split outcomes, failure categories, tokens and cost by prompt version, and found where one ticket’s time and money went. In this lesson, you kept customers’ words out of all of it and measured what each choice costs.

Now the service can answer four questions for every ticket and every version. Did it get a valid answer? How long did it take? What did it cost? If it failed, how?

Recap

• Minimise first: describe a ticket with its length, language guess and hash, never its text. TELEMETRY_CAPTURE_CONTENT stays false.
• Redaction before export removes text attributes and masks email addresses and card numbers. Capturing text and then redacting it adds risk and nothing else.
• Sampling keeps the trace of a share of tickets; logs and metrics still count every one. It saved most of the worker’s tracing cost, and none of the API’s.
• Retention (30 days, ops prune) and access control (Grafana Viewers, no copies of the files) limit what is left.

Next module: the service’s inputs change. A share of French tickets rises, and the shop’s documents change. You test whether the inputs drifted, and whether that means the assistant got worse.

Next: Data and concept drift