Read it honestly

On the previous page, you found the problem behind the posting: about 115 million meter readings a day, and three groups of people who need numbers they can trust. Now we read the posting line by line, as an experienced data engineer would. We sort every line into one of three groups:

• Really required: the team cannot hire someone without it, and the interviews will test it.
• A wish: the team would like it, but the job does not depend on it. Long lists feel safer to the people who write postings, so wishes collect there.
• Missing: the posting does not say it, but the job clearly needs it. These lines are often the most important, because the interviews ask about them and many candidates do not prepare.

Our reading is a careful guess, not a fact. So we also list the questions that turn each guess into an answer.

The level and the pay

“5+ years” appears three times in “What you bring”. Real postings do this often. It usually means that somebody copied lines from older postings, not that the team counted carefully.

The strangest line is “5+ years of experience building real-time streaming pipelines at scale”. Most data work is batch: a job runs on a schedule, for example every night, and processes what arrived since the last run. Streaming means that data is processed continuously, seconds after it arrives. Very few data engineers have five years of streaming work, so this line is a wish.

Now look at the pay and the duties: CAD 120,000 to 150,000 in Vancouver, “build and maintain”, “contribute to best practices”. That is an intermediate data engineer: someone who owns pipelines and fixes them without step-by-step help. Words such as “mentor” or “set technical direction”, which senior postings use, are not here. So the years say “senior”, and the pay says “intermediate”. Ask the recruiter: “What level is this role, and is there flexibility for a more senior candidate?” The answer changes the title and the pay that you can ask for.

The range is in the posting because of a law. In British Columbia, since 1 November 2023, the Pay Transparency Act requires every publicly advertised job posting to include the expected pay or pay range. The same law says that an employer must not ask what you were paid in past jobs. So if someone asks “What do you earn now?”, you can answer with the range that you expect for this role. Other provinces and countries have different rules.

What is really required

Under the long lists, the job needs five things. The interviews will spend most of their time on them.

1. SQL, and thinking about the shape of the data. SQL was in all 10 real postings that we studied. The posting adds “data modelling and data warehousing concepts” in one short line, but this line is bigger than it looks. Data modelling is the decision about how tables are shaped: what one row means (the grain), which tables hold facts like readings, and which hold descriptions like accounts and meters. A good model makes the right answer easy to get. A bad model makes the wrong answer easy.

2. Python and Spark for a lot of data. 115 million rows a day is too much for one laptop, but it is a normal amount for Spark, a tool that splits work across many computers. You need to know what makes a Spark job slow or expensive, for example reading all the history when you need only three days, and how you would find out.

3. Pipelines that you can run again safely. The posting does not say this, but it is the most important part of the job. A pipeline is a chain of steps that moves data from its source to the people who use it. Pipelines fail, and then somebody runs the job again. Does the second run give the same result as one good run, or does it add the same data twice? A job that you can run again with the same result is called idempotent. When a bill depends on the output, this property is not a nice extra.

4. Late and messy data. Meters lose their network connection. When the connection comes back, they send the old readings. So a reading from Monday can arrive on Wednesday. A meter can also be replaced, the clocks change twice a year, and a source system can change its file format. The posting says “ensure data quality across the platform”, one vague line. Expect the interviews to ask what you would check, where, and what happens when a check fails.

5. Working with the people who use the data. “Partner with billing, data science and customer-service teams” sounds like a standard line. At Tidewell, it is real work. The billing team decides what a correct bill is, and the forecast team knows which gaps hurt their model. You must explain a data problem to them without jargon, and listen when they explain a billing rule to you.

What is probably a wish

Most of the tool list. Ten tools in one line, and several do the same job. Our guess for Tidewell: Kafka, Databricks with Spark, Airflow and dbt are the daily tools. Snowflake, BigQuery, Flink, Dagster and Fivetran are probably not. Do not learn ten tools before the interview. Learn the ideas under them: batch and streaming, scheduling, transformation, storage. Then ask the recruiter which tools the team really uses.

Streaming as the main work. The posting says “real-time” twice and names three streaming tools. But think about the users. A bill is calculated once a month, and a forecast learns from months of history. Neither needs data within seconds. The AI assistant needs recent data, but for a bill question “recent” probably means “today”. Our guess: readings arrive through Kafka, one streaming job writes them to storage, and most work after that is batch. If someone asks “Does billing need streaming?”, a candidate who says “no, and here is why” shows good judgement.

Vector databases. “Experience with vector databases and embedding pipelines” is in “Nice to have”. A vector database stores lists of numbers that represent the meaning of text, so that an AI system can find documents with a similar meaning. Not one of the 10 real postings that we studied asked for it. Tidewell’s assistant needs a customer’s usage: numbers in tables, found by account number, not text found by meaning. So this line probably came from an article about AI trends, not from the team’s work. Spend your preparation time elsewhere.

The degree. “A degree in Computer Science, Engineering or a related field” is in the required list, but nothing in the duties depends on it. Many good data engineers come from BI, operations or other fields.

“AI fluency”. Seven of the 10 real postings asked for some kind of AI skill, so this line is normal in 2026. It does not mean that the interviews allow AI tools. The team wants a person who uses AI and still checks every line, because a wrong join in a billing table is expensive. Be ready to say how you check code that an assistant wrote.

What is missing, and matters

“AI-ready data” without a definition. The posting says “AI-ready” twice and never explains it. In one real posting, it meant one shared set of metric definitions, so that AI tools and people calculate the same numbers. For Tidewell’s assistant, our guess is that it means:

• Correct: no double readings, and no missing days without a clear mark.
• Recent: today’s usage, not last week’s.
• Clearly described: the assistant and the people who build it know what each column means.
• Protected: the assistant can read only the data of the customer it is talking to.

The last point is the most serious one. Electricity usage is personal information. It shows when a family is at home, when they sleep, and when they are away on holiday. Expect a question about who can see which data.

On-call. The posting does not mention it, although five of the 10 real postings did. A billing pipeline that fails at night must be fixed before the bills go out. Ask the recruiter: “Is there an on-call rotation? How often, and what time do the most important jobs run?” The answer affects your life more than most lines in the posting.

Cost. Not a word about it. But with 115 million new rows a day, a job that reads all the history every night, instead of only the recent days, becomes very expensive. Somebody pays for that. Ask: “Who owns the cost of the platform? Has it changed this year?” If the answer is “nobody” or a long pause, you have learned something useful.

Who can change the sources. Meter data comes through the meter vendor’s system, and account data from a CRM (customer relationship management) system. If their owners change a format without warning, the data team finds out when a pipeline breaks. Ask: “When a source system changes its format, how does the team find out?”

How the interviews run. None of the 10 real postings described the process, and neither does this one. Ask the recruiter in the first call.

Words that describe the culture

Some words in a posting are about the culture: “passionate”, “strong sense of ownership”. They can mean trust, and work that people really use. They can also mean that a small team owns too much, every night of the week. So ask a neutral question: “What happened the last time a pipeline failed before billing ran?” A healthy team answers easily, with a real example, and says what it changed.

Who should apply

Nobody matches every line of this posting, and nobody expects it. Use the five requirements above, not the full list:

• A BI or analytics developer with strong SQL and modelling, who has moved work to dbt or a cloud warehouse: apply. Modelling is your strength, and many interviews weigh it most. Prepare Spark and Kafka basics honestly.
• A backend engineer who has built reliable services: apply if you have worked with data at volume. Your instinct for safe re-runs is valuable. Prepare analytical modelling: grain, star schemas, history tables.
• One to three years in a modern, managed stack: this posting is probably a little above your level. Apply if you can explain what happens under your tools, for example how late data and re-runs are handled.
• Many years in data, with older tools, and a move to the cloud behind you: apply, and ask about the level. Your experience with incidents and with numbers that must match to the cent is what this team is missing. Learn the current names for ideas that you already know.
• Years in meter operations, billing or another part of a utility, and you have learned to build: your domain knowledge is worth more to this team than “nice to have” suggests. This exact role may be too big a step today, but a conversation can lead to a better-matched one. Be honest about the gap in scale.

In Part 3, you follow five candidates, one for each of these situations. Each one goes through the same interview.

Turn the posting into a preparation plan

Connect each requirement to a probable question, and to the evidence that you can show.

From the posting	What they will probably ask	Evidence that you need
SQL and data modelling	“What is one row in this table? How do you keep the history of which meter belongs to which account?”	A model that you designed, and a reason for its grain
Python and Spark at volume	“Your job is twice as slow as yesterday. What do you look at first?”	A job that you made faster or cheaper, with numbers
Safe re-runs	“What happens if your job runs twice?”	A pipeline that you made idempotent, and how you tested it
Late and messy data	“A reading arrives three days late. Where does it go?”	A story about late, duplicated or broken data, and what you changed
Work with data users	“Tell me about a time a user said your numbers were wrong.”	A real disagreement, and how you found the truth
On-call (missing)	“Tell me about your worst incident.”	A clear incident story: what broke, how you found it, what you changed
Cost (missing)	“This query reads five years of data every night. What would you change?”	One example where you reduced cost, even a small one
Protected data (missing)	“The AI assistant reads usage per account. What could go wrong?”	Clear rules: who can see what, and how you enforce it

If you have no evidence for a row yet, that row is your preparation. Our lesson Prepare for applications shows the same method for any posting.

Questions to ask the team

An interview goes in both directions (our advice). These questions turn our guesses into facts. Ask each one where it fits best.

• Recruiter: “Which tools does the team use every day?” “What level is this role?” “Is there an on-call rotation, and how often?” “How many interviews are there, and what does each one cover?”
• SQL and Python round: “Is this close to the SQL that the team writes every day?” “How does the team review SQL before it goes to production?”
• Pipeline design round: “What was the worst data incident this year, and what changed after it?” “How does the team handle readings that arrive late?” “Who owns the cost of the platform?”
• Data modelling round: “How do you keep the history of accounts and meters?” “Is there an agreement with the teams that own the source systems about format changes?”
• Values round, with billing: “When billing and the data team disagree about a number, how is it decided?” “What does a correct bill mean for your team, in practice?”

A clear answer with an example is a good sign. A vague answer to a simple question is also information.

Your turn: read a posting that you want

Find a real Data Engineer posting for a role that you want. Then do four things:

1. Write the problem behind the posting in one sentence, with no tool names.
2. Count the tools. Group them by what they do (storage, processing, scheduling, transformation, loading). In each group, guess which one the team really uses.
3. Sort the other lines into really required, a wish, and missing. Check especially for on-call, cost, late data and re-runs.
4. Make the preparation table for the main requirements, and write one question for each interview stage.

If you cannot decide whether a line is required or a wish, make it a question for the recruiter.

What happens next

On the next page, we look at the five interviews in Tidewell’s process: who runs each one, what each one checks, and how a serious interviewer moves from what you say to the edge of what you know.