Imagine that your neighbour puts a note on the door of your building: “Looking for a handyman. Plumbing, electrical, tiling, carpentry, painting. 5+ years of experience. 5+ years with pipes.” You knock on the door and ask what the problem is. She shows you the kitchen. Water comes out under the sink, but only some days, and only after somebody uses the washing machine. Now you know the real job. It is one leak, and the hard part is to find out why it happens only sometimes.
A job description is often like that note. The team has a real problem. Then somebody writes the posting, adds every tool that the company has ever discussed, and asks for many years of everything. If you read only the list, you will ask the wrong question: “Do I have all these tools?” The better question is: “What is leaking, and do I know how to find out why?”
In this guide, we read one Data Engineer posting together. We look for the problem behind it, we notice what it leaves out, and we decide what to ask the team.
The posting below is for a company that does not exist. We built it from 10 real Data Engineer postings at energy companies, retailers, fintech companies and start-ups. We kept their real patterns: the clear parts, the vague parts, and the small problems that real postings have. Then we put it in one company, so that the whole guide can use the same team and the same data.
Tidewell Energy sells electricity to homes and businesses in British Columbia, Canada, and helps to balance the power grid. The head office is in Vancouver, and about 900 people work there. Every Tidewell customer has a smart meter: a small device at the house that measures how much electricity the house uses and sends the number over a network. Tidewell has about 1.2 million of them.
Each meter sends one reading every 15 minutes. That is 96 readings a day for each meter, so about 115 million readings a day in total (1.2 million × 96). Meters also send events, for example “the power went out” or “somebody opened the meter box”. These readings decide each customer’s bill. They also feed a team that forecasts how much electricity people will use tomorrow, and, since this summer, an AI assistant that explains bills to customers. A small Data Platform team of seven data engineers moves all of this data, cleans it and makes it ready for the people who use it.
Tidewell Energy powers more than a million homes and businesses across British Columbia. We are building the data foundation for a cleaner, smarter grid, and data is at the heart of everything we do.
We are looking for a passionate Data Engineer to join our Data Platform team. You will design, build and scale real-time and batch data pipelines that turn billions of smart-meter readings every month into trusted, AI-ready data for billing, demand forecasting and our new customer-facing AI products.
The expected base pay for this role is CAD 120,000 to 150,000 a year. The final offer depends on your experience and level. If you need an accommodation at any stage of the process, tell our recruiter.
Composite job description for a fictional company, built from 10 real Data Engineer postings.
Before you compare the posting with your CV, answer three questions about it. Write your answers down. It takes about ten minutes. Then compare them with ours below.
What problem does the team have? Use one sentence, in plain words, with no tool names.
Who uses the data that this person moves? And what happens to those people when the data is wrong?
Count the tools in the “What you bring” list. How many of them do you think one team of seven uses every day? Which ones would you ask about?
Our answers
1. The problem. Tidewell receives a very large number of small readings from meters, and it needs to turn them into numbers that people can trust: the right bill, a complete history, the right answer to a customer. The posting says “trusted” once and “data quality” once. Everything else in it is about tools and speed. That is a small sign that trust is the real problem, and that it is not solved yet.
2. The users, and the cost of a mistake. There are three groups, and a mistake hurts each one in a different way:
Billing. If a reading is counted twice or missed, a real customer gets a wrong bill. People notice wrong bills, they call, and some of them complain to the regulator.
The demand-forecast team. If the history has holes, their model learns from wrong data, and the forecasts get worse. Nobody notices for weeks.
The customer-service AI assistant. It reads a customer’s usage to explain a high bill. If the data is old or wrong, it explains the bill wrongly to the customer. If the access rules are wrong, it could show one customer’s data to another customer. That is a privacy problem, not only a quality problem.
3. The tools. The list has ten tools. Several of them do the same job. Snowflake and BigQuery are two cloud data warehouses, and Databricks can also be one. Airflow and Dagster are two tools that schedule and run pipelines. Spark Structured Streaming and Flink are two engines for streaming. A team of seven usually picks one of each kind and stays with it. Our guess: the team really uses Kafka, Databricks with Spark, Airflow and dbt, and the rest came from other postings or from old plans. You find out by asking. We come back to this on the next page.
If your answers are different, that is fine. What matters is the habit: start from the data and the people who use it, not from the list of tools. In the interviews, the conversation is about the problem. Tools come up only as ways to solve it.
On the next page, we read the posting line by line and say honestly what each part means. We look at what is required and what is only a wish. We also look at what the posting does not say: who fixes the pipeline at 5 a.m., and who pays for the warehouse. After that, we look at the five interviews in Tidewell’s process. Then we follow five different candidates through the same pipeline design interview.
