Choose tools for a team
- Lesson 27
- 35 min
- Module 8: Tools: Langfuse, others and the run inspector
Where we are
In the last lesson, we sent the same telemetry to Grafana and to Langfuse, and we saw what each one does well. Other teams will now ask which tools they should choose. In this lesson, we compare the main tools on stated criteria, such as OpenTelemetry support, self-hosting, data residency, cost and lock-in, and we recommend a set for a team. We start with a bank account.
Lesson 27 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsChoose tools for a team
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Choose observability tools for a team on criteria that you write down first. Then you can explain the choice, and you know what it costs to change it later.
Before you startWhat to know and to have ready
What you need to know
- What Grafana and Langfuse each showed for the same run, from Send the same traces to Langfuse.
- The Collector’s exporters and pipelines, from The Collector.
What you need on your computer: nothing new. This lesson has no commands. You read, compare and write a recommendation.
About the facts. The vendor facts come from each vendor’s own documentation and pricing pages, read when the course was written. Products and prices change quickly in this field: check each vendor’s current pages before you decide. The course shows no screens of these products; it describes what their documentation says.
Choosing a bank account
Section titled “Choosing a bank account”When you choose a bank account, the advertisement talks about one feature: a free card, or a good app. But the questions that matter come later. What does it cost each month after the first year? In which country is my money kept? And if I leave, what must I do again? Every automatic payment and every saved payee must be set up again at the new bank.
So a careful person writes the questions down before reading the advertisements, and asks every bank the same ones.
The problem: a choice by demo
Section titled “The problem: a choice by demo”At Larkfield, three requests reach Kwame, who owns the platform and its bill, in one week:
- Priya wants Langfuse for her reviews, after the last lesson.
- Tomás saw a demo of LangSmith with his agent framework and wants it “for everything”.
- The privacy team asks one question: “Where will the customer data live?”
Each request is reasonable, and each one looks at one feature. Imagine that Kwame says yes to the demo. The demo used the vendor’s own SDK, not OpenTelemetry. A year later, the team wants to leave. Now the instrumentation must be written again. The dashboards, the review scores and a year of history stay behind. And the free plan that looked big in the demo ran out in the second month, when the traffic grew.
None of this was in the demo. It was in the pricing page, the licence and the region list.
The idea: stated criteria, then a matrix
Section titled “The idea: stated criteria, then a matrix”Write the criteria first: the questions that you ask every tool, in the same words. Then put the answers in a matrix, with one row per criterion and one column per tool. Some criteria are must-haves: a tool that fails one is out, whatever else it does well. The others are trade-offs that you compare.
The course uses six criteria. Each one comes from a problem that you met in this course:
- OpenTelemetry support. Can the tool take OTLP from our Collector, and does it understand the GenAI names? If yes, the tool is one exporter, as Langfuse was.
- Self-hosting and licence. Can we run it on our own servers, and under which licence?
- Evaluation features. Scores, model judges, datasets and review queues, for Priya’s sampled reviews.
- Data residency. In which regions can the data live?
- Cost. What does the free plan allow, what does it count, and what comes after it?
- Lock-in. What stays behind if we leave?
The last criterion is the one people forget. Lock-in means that leaving a tool costs more than staying, because of work and data that exist only inside it.
A tiny worked example: three must-haves
Section titled “A tiny worked example: three must-haves”Here is the method on a small case. Larkfield’s privacy team sets one rule for real customer data: it stays in Canada, or on Larkfield’s own servers. Kwame adds two more: the tool must take OTLP from the Collector, and the budget for a new tool this year is US$0. These three rules are assumptions of the course’s story, not legal advice.
| Tool | OTLP in | Data in Canada or on our servers | US$0 this year | Result |
|---|---|---|---|---|
| Grafana stack, self-hosted | yes | yes: our servers | yes: free software | in |
| Langfuse | yes | self-hosted only: its cloud is in the EU, US or Japan | self-hosted: yes | in, self-hosted |
| LangSmith | yes | self-hosting needs the Enterprise plan | no | out |
| Arize Phoenix | yes | yes: self-hosted | yes | in |
| Datadog | yes | no Canadian site in its list | free plan only | out |
| Honeycomb | yes | US or EU | free plan only | out |
Three tools are left, and each one is in for a different reason. Notice what the must-haves did: they removed the demo tool before anyone compared features. Notice also what they did not do. Grafana Cloud has a Canadian region, and so does Arize’s cloud. A rule written as “Canada” and not as “our servers” keeps those options open.
The matrix of the main tools
Section titled “The matrix of the main tools”Here are the six tools on the six criteria. Each cell is a short answer from the vendor’s documentation, colored by how well it meets the criterion.
Text description of the diagram
Read it by row, not by column. On OpenTelemetry, every tool takes OTLP now, so that row decides little. The rows that split the tools are self-hosting, evaluation, the free plan and data out. Grafana is strong where you run it yourself and weak on evaluation. Langfuse and Phoenix are strong on evaluation and can run on your servers. LangSmith, Datadog and Honeycomb are cloud services, so their regions and their plans decide.
What each row says, in more detail:
OpenTelemetry. All six take OTLP over HTTP, and Grafana, Phoenix and Honeycomb also over gRPC. All six read the GenAI names in some way. Langfuse typed our spans as generations and tools, as you saw. LangSmith maps gen_ai.* attributes, and Phoenix converts them into its own convention, OpenInference, when they arrive. Datadog needs its own header to route spans to its agent views. Grafana stores any attribute; its views made for AI agents were a preview when the course checked.
Self-hosting and licence. Grafana, Tempo and Loki are AGPL-3.0, and Prometheus is Apache-2.0. Langfuse’s core is MIT. A few enterprise features in its ee folders need a paid licence key. Phoenix is under the Elastic License 2.0, which is not an open-source licence. You can run it for yourself, but you cannot sell it as a hosted service. LangSmith runs on your servers only on its Enterprise plan. Datadog and Honeycomb are cloud only.
Evaluation. Langfuse, LangSmith, Phoenix and Datadog all have scores, model judges and datasets, and most have review queues. The open Grafana stack has none: Priya’s reviews stay events, as in Module 7.
Data residency. A cloud tool keeps data in the regions it offers, and some never let you move a project to another region. Check the list against your rule before you sign up, not after.
What the free plans count
Section titled “What the free plans count”Free plans look alike, but they count different things. Langfuse counts units: traces, observations and scores. LangSmith counts traces. Arize counts spans, Datadog counts only model-call spans, and Honeycomb counts every span as an event. So the same week fills each plan differently. Here is a month of the replayed week’s traffic, about 45 tickets a day, as a share of each free plan:
Text description of the chart
Five bars. Langfuse Hobby: 21.6% (10,804 units (traces, observations, scores) a month of 50,000). LangSmith Developer: 37.2% (1,860 base traces a month of 5,000). Arize AX Free: 34.7% (8,687 spans a month of 25,000). Datadog (LLM spans): 7.7% (3,077 LLM spans (model calls only) a month of 40,000). Honeycomb Free: 0% (8,687 events (each span is one) a month of 20,000,000). At today’s traffic, every free plan is enough. At ten times the traffic, three bars pass 100%: LangSmith 372%, Arize 347.5% and Langfuse 216.1%.
Read it with a growth plan in mind. The 45 tickets a day of this course’s data fit in every free plan. At 450 tickets a day, LangSmith’s 5,000 traces would last about eight days, and Langfuse’s 50,000 units about two weeks. Datadog counts only model calls, so its plan lasts longer than our span count suggests. Honeycomb’s 20 million events would last for years at this traffic. Note also the limits that are not volume. LangSmith’s free plan has one seat and Langfuse’s has two users. Each plan keeps data for a fixed time.
What it costs to change later
Section titled “What it costs to change later”Kwame’s last question is “what if we are wrong?”. The answer depends on where the work lives.
Text description of the diagram
On the left is what this course built in your own code and files. That is the spans with the GenAI names, the Collector’s configuration, the audit store and the price table. Changing a tool changes one exporter there. On the right is what a team builds inside a tool. That is dashboards, alert rules, scores, review queues, datasets and the history of old traces. That work must be done again, or exported first, when you leave.
So the cost of a change is mostly on the right. For Larkfield, moving from Langfuse Cloud to a self-hosted Langfuse, or to Phoenix, would mean:
- One exporter and its keys in
collector/otelcol-config-langfuse.yaml. Small. - The review scores. The course team sent them with Langfuse’s API, so the script must change. Medium.
- The history. The old traces stay in the old tool until its retention ends, unless you export them first. Langfuse documents an API and a bulk export; LangSmith’s bulk export is now on its Enterprise plan only for new customers. Check this row before you depend on a tool.
- People’s habits. Priya’s saved views and Grace’s links. Real, and hard to measure.
The instrumentation does not change at all. That is the reason this course recorded once, with OpenTelemetry, before it chose any tool.
Larkfield’s recommendation
Section titled “Larkfield’s recommendation”Kwame writes the team’s choice with its reasons:
- Grafana stays the system view, self-hosted: alerts, metrics, TraceQL, and every skill of this course. Free software on servers Larkfield already runs.
- Langfuse, for Priya’s reviews. Now on Langfuse Cloud with synthetic practice data only. For real data, self-hosted Langfuse in Canada, because the cloud regions do not meet the privacy rule. Phoenix is the second choice; its licence allows the internal use.
- The audit store stays separate, as Module 6 built it. No tool on the matrix is an audit trail.
- Not now: LangSmith, Datadog and Honeycomb. Each one fails the data rule or the budget this year. Check them again if the rules change.
- Cost to change later: one exporter, the score script and the review history. The instrumentation stays.
The recommendation fits in ten lines because the criteria were written first.
Common mistake
The free plan runs out in the middle of the month
Symptom: traces stop appearing in the cloud tool, or a bill arrives, although “we were far below the limit”.
Cause: the plan counts something that you did not count. A Langfuse unit is a trace, an observation or a score, not a run. Honeycomb counts each span as an event, and also its span events and links. A run of our workflow is about 6.4 spans, and two traces when a person approves.
Diagnose: count what the plan counts in your own data, over a known period. For the replayed week: 316 runs, 434 traces, 2,027 spans, 718 model-call spans and 60 scores. Multiply by your traffic’s growth, and compare with the limit.
Recover: sample before that destination in the Collector, as the Grafana pipeline does. But remember that Priya’s reviews need their runs. Or send only the traces that matter, or move up a plan on purpose. Write the plan’s counting rule next to the exporter in the configuration.
Your turn: recommend a tool set
Section titled “Your turn: recommend a tool set”A team asks you for a recommendation. Here is their situation:
- Four engineers and two support leads at an online shop in Germany, with an agent like Larkfield’s.
- Their rule: customer data stays in the European Union.
- Their budget: US$100 a month for tools. They have no servers of their own and no time to run any.
- The support leads must review and score about 50 model answers a week.
Write five to ten lines, as Kwame did. Say which tools, why each one, which tools are out and why, and what it costs to change later. Check each fact against the vendor’s current pages first. A worked answer is in the third knowledge check.
Knowledge check
Section titled “Knowledge check”Check your understanding
Every tool in the matrix takes OTLP. Why is 'OpenTelemetry support' still the first criterion, and which row splits the tools more?
Show the answer
It is the criterion that keeps the others cheap. If a tool takes OTLP and reads the GenAI names, adding it is one exporter, and leaving it is one exporter too. The instrumentation never changes. A tool that needs its own SDK for its main features costs a rewrite to leave. Today the row barely splits the tools, because all six take OTLP. The rows that split them are self-hosting and licence, evaluation, data residency and the free plan’s limits. For Larkfield, the data rule removed three tools before anyone compared features.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'LangSmith's free plan is enough: our week has only 434 traces.' Check his claim with the numbers, and say what else limits the free plan.
Show the answer
A month of the week is about 434 × 30/7 ≈ 1,860 traces, 37% of the free plan’s 5,000. So today he is right. At ten times the traffic, a month is about 18,600 traces, 372% of the plan: it would last about eight days. Other limits matter too. The free plan has one seat, so Priya and Grace cannot both use it. Base traces are kept 14 days. It is cloud only, in the regions it lists, which fails Larkfield’s data rule. And self-hosting and bulk export need the Enterprise plan. These figures are from the vendor’s pricing page when the course was written: check the current one.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your turn: what is a sound recommendation for the German shop (EU data, US$100 a month, no servers, 50 reviews a week)?
Show the answer
One sound answer: a cloud tool in an EU region, because they have no servers to run. For reviews and traces, Langfuse Cloud in the EU region. If their traffic is like Larkfield’s, a month is about 10,800 units, far below the Hobby plan’s 50,000. But Hobby allows only 2 users, and six people need the Core plan: US$29 a month when the course checked. LangSmith’s Plus plan, at US$39 a seat, would cost about US$234 a month for six people: over the budget. For the system view, Grafana Cloud in an EU region. Its free plan allows 3 active users, so plan for its Pro plan, with a US$19 monthly platform fee when the course checked. Together, about US$48 a month before any extra volume. Keep the audit trail in their own store, not in a tool. Out: self-hosted options (no servers), and any tool without an EU region. Cost to change later: one exporter in the Collector, their scores and dashboards, and the history until it expires. Other answers are fine if each fact is checked and each choice has a reason.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Write the criteria before you look at tools. Mark the must-haves: one failure removes a tool.
- The course’s six criteria: OpenTelemetry support, self-hosting and licence, evaluation, data residency, cost and lock-in.
- Every tool in the matrix takes OTLP now. Self-hosting, evaluation, regions and the free plan’s counting rule split them.
- Free plans count different things: units, traces, spans or model-call spans. Count them in your own data and multiply by your growth.
- Lock-in lives in what you build inside a tool: dashboards, scores, datasets and history. Recording once with OpenTelemetry keeps the rest cheap to move.
- Vendor facts change quickly: check the current pages before you decide.
You can now choose a tool set and explain what it costs to change. But some questions are still slow in every general tool, such as the story of one run for an auditor. Next, you tour a view built only for agent runs, and you read how one of its failure flags is computed.