Skip to content
Français

Choose tools for a team

  • Lesson 27
  • 35 min
  • Module 8: Tools: Langfuse, others and the run inspector

Where we are

In the last lesson, we sent the same telemetry to Grafana and to Langfuse, and we saw what each one does well. Other teams will now ask which tools they should choose. In this lesson, we compare the main tools on stated criteria, such as OpenTelemetry support, self-hosting, data residency, cost and lock-in, and we recommend a set for a team. We start with a bank account.

What you will be able to do

Choose observability tools for a team on criteria that you write down first. Then you can explain the choice, and you know what it costs to change it later.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer: nothing new. This lesson has no commands. You read, compare and write a recommendation.

About the facts. The vendor facts come from each vendor’s own documentation and pricing pages, read when the course was written. Products and prices change quickly in this field: check each vendor’s current pages before you decide. The course shows no screens of these products; it describes what their documentation says.

When you choose a bank account, the advertisement talks about one feature: a free card, or a good app. But the questions that matter come later. What does it cost each month after the first year? In which country is my money kept? And if I leave, what must I do again? Every automatic payment and every saved payee must be set up again at the new bank.

So a careful person writes the questions down before reading the advertisements, and asks every bank the same ones.

At Larkfield, three requests reach Kwame, who owns the platform and its bill, in one week:

  • Priya wants Langfuse for her reviews, after the last lesson.
  • Tomás saw a demo of LangSmith with his agent framework and wants it “for everything”.
  • The privacy team asks one question: “Where will the customer data live?”

Each request is reasonable, and each one looks at one feature. Imagine that Kwame says yes to the demo. The demo used the vendor’s own SDK, not OpenTelemetry. A year later, the team wants to leave. Now the instrumentation must be written again. The dashboards, the review scores and a year of history stay behind. And the free plan that looked big in the demo ran out in the second month, when the traffic grew.

None of this was in the demo. It was in the pricing page, the licence and the region list.

Write the criteria first: the questions that you ask every tool, in the same words. Then put the answers in a matrix, with one row per criterion and one column per tool. Some criteria are must-haves: a tool that fails one is out, whatever else it does well. The others are trade-offs that you compare.

The course uses six criteria. Each one comes from a problem that you met in this course:

  1. OpenTelemetry support. Can the tool take OTLP from our Collector, and does it understand the GenAI names? If yes, the tool is one exporter, as Langfuse was.
  2. Self-hosting and licence. Can we run it on our own servers, and under which licence?
  3. Evaluation features. Scores, model judges, datasets and review queues, for Priya’s sampled reviews.
  4. Data residency. In which regions can the data live?
  5. Cost. What does the free plan allow, what does it count, and what comes after it?
  6. Lock-in. What stays behind if we leave?

The last criterion is the one people forget. Lock-in means that leaving a tool costs more than staying, because of work and data that exist only inside it.

Here is the method on a small case. Larkfield’s privacy team sets one rule for real customer data: it stays in Canada, or on Larkfield’s own servers. Kwame adds two more: the tool must take OTLP from the Collector, and the budget for a new tool this year is US$0. These three rules are assumptions of the course’s story, not legal advice.

Tool OTLP in Data in Canada or on our servers US$0 this year Result
Grafana stack, self-hosted yes yes: our servers yes: free software in
Langfuse yes self-hosted only: its cloud is in the EU, US or Japan self-hosted: yes in, self-hosted
LangSmith yes self-hosting needs the Enterprise plan no out
Arize Phoenix yes yes: self-hosted yes in
Datadog yes no Canadian site in its list free plan only out
Honeycomb yes US or EU free plan only out

Three tools are left, and each one is in for a different reason. Notice what the must-haves did: they removed the demo tool before anyone compared features. Notice also what they did not do. Grafana Cloud has a Canadian region, and so does Arize’s cloud. A rule written as “Canada” and not as “our servers” keeps those options open.

Here are the six tools on the six criteria. Each cell is a short answer from the vendor’s documentation, colored by how well it meets the criterion.

A matrix with seven rows and six tool columns: Grafana stack, Langfuse, LangSmith, Arize Phoenix, Datadog, Honeycomb. OTLP in: Grafana yes, HTTP and gRPC (green); Langfuse yes, HTTP only (yellow); LangSmith yes, HTTP (yellow); Phoenix yes, HTTP and gRPC (green); Datadog yes, HTTP, own header (yellow); Honeycomb yes, HTTP and gRPC (green). Reads GenAI names: Grafana stores them, AI views in preview (yellow); Langfuse yes, types by operation; LangSmith yes, maps gen_ai.*; Phoenix yes, converts to its own; Datadog yes; Honeycomb yes, agent timeline (all green). Self-host and licence: Grafana yes, AGPL-3.0, Prometheus Apache (green); Langfuse yes, MIT core plus paid parts (green); LangSmith Enterprise plan only (red); Phoenix yes, Elastic License 2.0 (yellow); Datadog no, cloud only (red); Honeycomb no, cloud only (red). Evaluation: Grafana none in the open stack (red); Langfuse scores, judge, datasets, queues (green); LangSmith evals, datasets, queues (green); Phoenix judge, annotations, datasets (green); Datadog evaluators, queues (green); Honeycomb none found (red). Where data can live: Grafana your servers, many regions (green); Langfuse EU, US, Japan, or your servers (green); LangSmith US, EU, APAC, self-hosted on Enterprise (yellow); Phoenix your servers, cloud US, EU, CA (green); Datadog its sites US, EU, JP, AU, UK (yellow); Honeycomb US or EU (yellow). Free plan: Grafana software free, you run it (green); Langfuse 50k units, 30 days, 2 users (yellow); LangSmith 5k traces, 14 days, 1 seat (red); Phoenix self-host free, cloud 25k spans (green); Datadog 40k LLM spans, 15 days (yellow); Honeycomb 20M events, 60 days (green). Data out: Grafana open formats in your store (green); Langfuse API and bulk export (green); LangSmith bulk export Enterprise only (red); Phoenix datasets out, spans not found (yellow); Datadog export API, search (yellow); Honeycomb not checked (yellow). A key: green strong on this criterion, yellow partly or with limits, red weak or missing.
Six tools on six criteria, from each vendor's documentation and pricing page when the course was written. Products and prices change: check the current pages. 'Not found' and 'not checked' mean that the course team did not find it in the official documentation.
Text description of the diagram

Read it by row, not by column. On OpenTelemetry, every tool takes OTLP now, so that row decides little. The rows that split the tools are self-hosting, evaluation, the free plan and data out. Grafana is strong where you run it yourself and weak on evaluation. Langfuse and Phoenix are strong on evaluation and can run on your servers. LangSmith, Datadog and Honeycomb are cloud services, so their regions and their plans decide.

What each row says, in more detail:

OpenTelemetry. All six take OTLP over HTTP, and Grafana, Phoenix and Honeycomb also over gRPC. All six read the GenAI names in some way. Langfuse typed our spans as generations and tools, as you saw. LangSmith maps gen_ai.* attributes, and Phoenix converts them into its own convention, OpenInference, when they arrive. Datadog needs its own header to route spans to its agent views. Grafana stores any attribute; its views made for AI agents were a preview when the course checked.

Self-hosting and licence. Grafana, Tempo and Loki are AGPL-3.0, and Prometheus is Apache-2.0. Langfuse’s core is MIT. A few enterprise features in its ee folders need a paid licence key. Phoenix is under the Elastic License 2.0, which is not an open-source licence. You can run it for yourself, but you cannot sell it as a hosted service. LangSmith runs on your servers only on its Enterprise plan. Datadog and Honeycomb are cloud only.

Evaluation. Langfuse, LangSmith, Phoenix and Datadog all have scores, model judges and datasets, and most have review queues. The open Grafana stack has none: Priya’s reviews stay events, as in Module 7.

Data residency. A cloud tool keeps data in the regions it offers, and some never let you move a project to another region. Check the list against your rule before you sign up, not after.

Free plans look alike, but they count different things. Langfuse counts units: traces, observations and scores. LangSmith counts traces. Arize counts spans, Datadog counts only model-call spans, and Honeycomb counts every span as an event. So the same week fills each plan differently. Here is a month of the replayed week’s traffic, about 45 tickets a day, as a share of each free plan:

Constructed estimate: the week's measured counts (2,027 spans, 434 traces, 718 model-call spans, 2,521 Langfuse units) times 30/7 for a month, divided by each free plan's monthly limit from the vendor's pricing page. For a decision, not a bill.
Text description of the chart

Five bars. Langfuse Hobby: 21.6% (10,804 units (traces, observations, scores) a month of 50,000). LangSmith Developer: 37.2% (1,860 base traces a month of 5,000). Arize AX Free: 34.7% (8,687 spans a month of 25,000). Datadog (LLM spans): 7.7% (3,077 LLM spans (model calls only) a month of 40,000). Honeycomb Free: 0% (8,687 events (each span is one) a month of 20,000,000). At today’s traffic, every free plan is enough. At ten times the traffic, three bars pass 100%: LangSmith 372%, Arize 347.5% and Langfuse 216.1%.

Read it with a growth plan in mind. The 45 tickets a day of this course’s data fit in every free plan. At 450 tickets a day, LangSmith’s 5,000 traces would last about eight days, and Langfuse’s 50,000 units about two weeks. Datadog counts only model calls, so its plan lasts longer than our span count suggests. Honeycomb’s 20 million events would last for years at this traffic. Note also the limits that are not volume. LangSmith’s free plan has one seat and Langfuse’s has two users. Each plan keeps data for a fixed time.

Kwame’s last question is “what if we are wrong?”. The answer depends on where the work lives.

Two groups. On the left, in green, 'Cheap to move': instrumentation (OpenTelemetry spans, GenAI names) with an arrow to the Collector configuration (one exporter per tool), the audit store (your schema, your chain), and the price table (dated, in your code). On the right, in red, 'Costly to move': datasets and experiments; dashboards, saved queries and alert rules; scores, annotations and review queues; history of old traces past the plan's retention. An arrow labelled 'swap one exporter' goes from the Collector configuration to the right-hand group.
What moves with you when you change a tool, and what stays behind. The course kept the left side in open formats on purpose.
Text description of the diagram

On the left is what this course built in your own code and files. That is the spans with the GenAI names, the Collector’s configuration, the audit store and the price table. Changing a tool changes one exporter there. On the right is what a team builds inside a tool. That is dashboards, alert rules, scores, review queues, datasets and the history of old traces. That work must be done again, or exported first, when you leave.

So the cost of a change is mostly on the right. For Larkfield, moving from Langfuse Cloud to a self-hosted Langfuse, or to Phoenix, would mean:

  • One exporter and its keys in collector/otelcol-config-langfuse.yaml. Small.
  • The review scores. The course team sent them with Langfuse’s API, so the script must change. Medium.
  • The history. The old traces stay in the old tool until its retention ends, unless you export them first. Langfuse documents an API and a bulk export; LangSmith’s bulk export is now on its Enterprise plan only for new customers. Check this row before you depend on a tool.
  • People’s habits. Priya’s saved views and Grace’s links. Real, and hard to measure.

The instrumentation does not change at all. That is the reason this course recorded once, with OpenTelemetry, before it chose any tool.

Kwame writes the team’s choice with its reasons:

  • Grafana stays the system view, self-hosted: alerts, metrics, TraceQL, and every skill of this course. Free software on servers Larkfield already runs.
  • Langfuse, for Priya’s reviews. Now on Langfuse Cloud with synthetic practice data only. For real data, self-hosted Langfuse in Canada, because the cloud regions do not meet the privacy rule. Phoenix is the second choice; its licence allows the internal use.
  • The audit store stays separate, as Module 6 built it. No tool on the matrix is an audit trail.
  • Not now: LangSmith, Datadog and Honeycomb. Each one fails the data rule or the budget this year. Check them again if the rules change.
  • Cost to change later: one exporter, the score script and the review history. The instrumentation stays.

The recommendation fits in ten lines because the criteria were written first.

Common mistake

The free plan runs out in the middle of the month

Symptom: traces stop appearing in the cloud tool, or a bill arrives, although “we were far below the limit”.

Cause: the plan counts something that you did not count. A Langfuse unit is a trace, an observation or a score, not a run. Honeycomb counts each span as an event, and also its span events and links. A run of our workflow is about 6.4 spans, and two traces when a person approves.

Diagnose: count what the plan counts in your own data, over a known period. For the replayed week: 316 runs, 434 traces, 2,027 spans, 718 model-call spans and 60 scores. Multiply by your traffic’s growth, and compare with the limit.

Recover: sample before that destination in the Collector, as the Grafana pipeline does. But remember that Priya’s reviews need their runs. Or send only the traces that matter, or move up a plan on purpose. Write the plan’s counting rule next to the exporter in the configuration.

A team asks you for a recommendation. Here is their situation:

  • Four engineers and two support leads at an online shop in Germany, with an agent like Larkfield’s.
  • Their rule: customer data stays in the European Union.
  • Their budget: US$100 a month for tools. They have no servers of their own and no time to run any.
  • The support leads must review and score about 50 model answers a week.

Write five to ten lines, as Kwame did. Say which tools, why each one, which tools are out and why, and what it costs to change later. Check each fact against the vendor’s current pages first. A worked answer is in the third knowledge check.

Check your understanding

Every tool in the matrix takes OTLP. Why is 'OpenTelemetry support' still the first criterion, and which row splits the tools more?

Show the answer

It is the criterion that keeps the others cheap. If a tool takes OTLP and reads the GenAI names, adding it is one exporter, and leaving it is one exporter too. The instrumentation never changes. A tool that needs its own SDK for its main features costs a rewrite to leave. Today the row barely splits the tools, because all six take OTLP. The rows that split them are self-hosting and licence, evaluation, data residency and the free plan’s limits. For Larkfield, the data rule removed three tools before anyone compared features.

Check your understanding

Tomás says: 'LangSmith's free plan is enough: our week has only 434 traces.' Check his claim with the numbers, and say what else limits the free plan.

Show the answer

A month of the week is about 434 × 30/7 ≈ 1,860 traces, 37% of the free plan’s 5,000. So today he is right. At ten times the traffic, a month is about 18,600 traces, 372% of the plan: it would last about eight days. Other limits matter too. The free plan has one seat, so Priya and Grace cannot both use it. Base traces are kept 14 days. It is cloud only, in the regions it lists, which fails Larkfield’s data rule. And self-hosting and bulk export need the Enterprise plan. These figures are from the vendor’s pricing page when the course was written: check the current one.

Check your understanding

Your turn: what is a sound recommendation for the German shop (EU data, US$100 a month, no servers, 50 reviews a week)?

Show the answer

One sound answer: a cloud tool in an EU region, because they have no servers to run. For reviews and traces, Langfuse Cloud in the EU region. If their traffic is like Larkfield’s, a month is about 10,800 units, far below the Hobby plan’s 50,000. But Hobby allows only 2 users, and six people need the Core plan: US$29 a month when the course checked. LangSmith’s Plus plan, at US$39 a seat, would cost about US$234 a month for six people: over the budget. For the system view, Grafana Cloud in an EU region. Its free plan allows 3 active users, so plan for its Pro plan, with a US$19 monthly platform fee when the course checked. Together, about US$48 a month before any extra volume. Keep the audit trail in their own store, not in a tool. Out: self-hosted options (no servers), and any tool without an EU region. Cost to change later: one exporter in the Collector, their scores and dashboards, and the history until it expires. Other answers are fine if each fact is checked and each choice has a reason.

Recap

  • Write the criteria before you look at tools. Mark the must-haves: one failure removes a tool.
  • The course’s six criteria: OpenTelemetry support, self-hosting and licence, evaluation, data residency, cost and lock-in.
  • Every tool in the matrix takes OTLP now. Self-hosting, evaluation, regions and the free plan’s counting rule split them.
  • Free plans count different things: units, traces, spans or model-call spans. Count them in your own data and multiply by your growth.
  • Lock-in lives in what you build inside a tool: dashboards, scores, datasets and history. Recording once with OpenTelemetry keeps the rest cheap to move.
  • Vendor facts change quickly: check the current pages before you decide.

You can now choose a tool set and explain what it costs to change. But some questions are still slow in every general tool, such as the story of one run for an auditor. Next, you tour a view built only for agent runs, and you read how one of its failure flags is computed.

Next: What a purpose-built view adds