The big-tech engineer

Think of a surgeon who did one kind of operation for eight years, thousands of times, in a very large hospital. Nobody does it better. Now a small clinic asks her to join. There, she will also talk to patients, plan the day, and sometimes answer the phone. She is still a great surgeon. The question is whether she wants that kind of day.

Many backend candidates come from very large technology companies. They worked on one internal service at a scale that a company like Casewell will never see. They know consistency, replication, back-pressure and retries better than almost anyone in the room. But often their customers were other engineers, inside the same company. They never had to ask a lawyer what she needs, or decide something when nobody can give them a clear specification.

Arash is this candidate. His interview shows two things. First, how to disagree with a senior interviewer about a technical claim, politely and with reasons. Second, how an interview can help you, not only the company, decide that a job is not right for you.

AK

The candidate

Arash Karimi

Senior engineer, distributed storage, 8 years

• 8 years at a large cloud company
• Go and C++
• Distributed systems
• Vancouver

• Senior Software Engineer, Halden Cloud (large cloud provider), 2018 to now
• Led the redesign of the metadata layer for internal object storage, about 1 million requests per second at peak
• Designed per-tenant admission control and quotas for about 3,000 internal tenants, using weighted fair queueing
• Wrote the team’s retry-budget rules after a retry storm; adopted by four other storage teams
• Incident commander for more than 20 incidents
• MSc Computer Science, 2018
• Go, C++, Python for tooling, Raft, Kubernetes

What Rafael sees before the call

Rafael reads the CV next to the job description, and he is impressed. Per-tenant fairness and isolation are exactly Casewell’s problems, and Arash has solved them at a scale a thousand times larger. The Python is only for tooling, but Rafael does not worry about that. A strong engineer learns FastAPI in a few weeks.

What worries him is the other half of the posting. It says “product engineer”. At Casewell, an engineer talks to customer success every week, sometimes to a customer, and often decides things with no clear specification. The CV has no line about a product, a customer or a user.

One phrase also gets a mark: “Led the redesign”. Rafael wants to know which part Arash led.

He wants to find out:

• How deep is the distributed-systems knowledge, and can Arash explain it simply?
• What does Arash do when the question is about a person, not a system?
• How does he behave when Rafael says something that is not quite right?

Terms on this page

• Tenant: one customer organisation in a shared system. At Casewell, each law firm is a tenant.
• Row-level security (RLS): a Postgres feature that adds a filter to every query on a table, for example “only rows of the current tenant”.
• Retry storm: many clients retry failed requests at the same time, so the extra load keeps the failing system down.
• Retry budget: a limit on retries, for example “retries can be at most 10% of all requests”.
• Weighted fair queueing: a way to share capacity between users in proportion to a weight, so that one big user cannot take everything.
• Ethical wall: a rule inside a law firm that stops some lawyers from seeing some matters, to avoid a conflict of interest.
• pgvector: a Postgres extension that stores embeddings and finds the most similar ones.

The interview

Minutes 0 to 5: the opening

Rafael(interviewer):

Hi Arash. Rafael, staff engineer. Thank you for coming. I see Vancouver behind you, no?

Arash(candidate):

Yes, it’s raining, as usual. Nice to meet you.

Rafael(interviewer):

The plan is simple. You tell me about a system you built, I ask many questions. Then one real problem from Casewell. Then your questions. No AI tools in this one. We had a round with AI already. This one is just talking.

Arash(candidate):

That’s fine. I like talking about systems more than coding live, honestly.

Minutes 5 to 25: the storage service, then deeper

Rafael(interviewer):

So. Your system.

Arash(candidate):

Okay. So at Halden, I work on the metadata layer for internal object storage. When any team in the company stores a file, the bytes go to one system, and the information about the file goes to ours. Name, size, where the pieces are, who can read it.

The problem we had was noisy tenants. One team runs a big batch job, and everybody else gets slow. What I did is the admission control. Each tenant gets a share, a weighted share, and when we are busy, we reject the requests above the share early, with a clear error, instead of letting everyone get slow.

Rafael(interviewer):

Your CV says you led the redesign. All of it?

Arash(candidate):

No. Good question. The redesign had three parts. I led admission control. Two other people led the storage engine and the migration. “Led the redesign” is too big, yes.

Why this works

Arash answers the direct question in one sentence and does not defend the CV. He knows exactly which part was his, and that makes the rest of his story easy to trust.

Rafael(interviewer):

How does a request find its share? Where is that decision?

Arash(candidate):

In the front layer, before the request touches storage. Each tenant has a queue in memory on each server. We take from the queues with deficit round robin, with weights. If a tenant’s queue is too long, we answer “slow down” right away.

Rafael(interviewer):

“Slow down” is an error. What do the clients do with it?

Arash(candidate):

That is the incident, actually.

Two years ago, one cell got slow after a deploy. Our clients retried. But the clients of our clients also retried, and their clients too. Three layers, three attempts each. So one user request could become 27 requests at the bottom. The cell never recovered on its own. We had to block traffic by hand.

Rafael(interviewer):

How long?

Arash(candidate):

About two hours of bad latency, maybe a bit more. I was the incident commander.

Rafael(interviewer):

And what changed after?

Arash(candidate):

Three rules. Retry in one layer only, the one closest to the problem. Every client has a retry budget, ten percent of its normal requests. And always random wait times, jitter, so they don’t come back together.

And then the “slow down” answer says how long to wait. That was… sorry, I lost my point. What was the question?

Rafael(interviewer):

What changed after the incident. You said three rules.

Arash(candidate):

Right. And then we tested it. We made one cell slow on purpose, in a test environment with real traffic patterns, and checked that the load at the bottom stayed under about 1.2 times the normal. Before the rules, it went to six or seven times.

How the interviewer goes deeper

Rafael always comes back to “how did you know it worked?” Arash has an answer with a measurement, from a test that broke the system on purpose. At this level of experience, that is what a strong answer looks like.

Rafael(interviewer):

Very good. Different question. Who were your customers?

Arash(candidate):

Other teams. Engineers. They opened tickets, or they wrote in our channel.

Rafael(interviewer):

Did you ever talk to someone outside the company? An end user?

Arash(candidate):

No. Never, actually. Hm. I didn’t think about it until now.

Minutes 25 to 40: the Casewell problem

Rafael(interviewer):

Now our real problem. At the start of a deal, a law firm uploads up to 2,000 contracts at once. Today we process them inside the upload request; it times out, people upload again, we get duplicates. Last month one big upload slowed down every other customer for hours. And — I will be honest with you — we had an incident where one customer saw text from another customer’s document in a chat answer. It was a cache bug, fixed now. How would you redesign document processing and the chat so these things cannot happen?

Arash(candidate):

Okay. Before the design, some numbers. How long is one document to process? And the model provider, how does it limit you? Requests, or tokens?

Rafael(interviewer):

About 20 seconds for a normal contract. The provider limits both, tokens per minute and requests per minute. Tokens is the one we hit.

Arash(candidate):

Then tokens is the real resource, not workers. Good to know.

So: the upload only stores the file and answers 202 with a batch ID. Each file gets a job. Duplicates, by a hash of the bytes, unique per tenant. Then I would not share capacity by queue position. I would share tokens. Each tenant gets a weighted share of the token budget, and a cap on jobs running at one time. The scheduler takes from tenants in turn, the same way as my admission control.

Rafael(interviewer):

So the firm with 1,800 files…

Arash(candidate):

Gets its share, maybe more when nobody else is busy. But a small firm that uploads three files still starts in seconds. Their files don’t wait behind 1,800.

Why this works

This is Arash’s strongest area, and it shows. He asks for the real limit first, finds that it is tokens and not workers, and designs the fairness around it. It is the same idea as his storage work, moved to a new resource.

Rafael(interviewer):

Okay, I believe you on the scheduler. Let me ask a different kind of question. A firm uploads 2,000 contracts. 37 fail: scanned images with no text, two with passwords. What does the lawyer see? And when?

Arash(candidate):

Each file has a status. At the end, a list of the failures, with an error code for each one. Like “no text layer”, “encrypted”.

Rafael(interviewer):

Hmm. The lawyer is closing a deal on Friday. It is Wednesday night. Does she want error codes?

Arash(candidate):

Pause. No. Probably not.

Honestly, I don’t know what she wants. In my job, the person who read the error was always an engineer. I would want to ask her. Or ask your customer success team. My guess is she wants to know early, not at the end. And what she can do about it. “Ask the other side for the Word version.” Something like that.

The edge

This is the first edge, and it is not technical. Arash notices that his answer is for an engineer, says “I don’t know what she wants”, and then makes a reasonable guess that starts from her situation. He also names who to ask. That is a good way to behave at an edge you have never seen before.

Rafael(interviewer):

And duplicates. Same contract, but one copy has the signature page and one does not. Is it a duplicate?

Arash(candidate):

The hash will say no, the bytes are different. Whether it is the “same contract” for a lawyer… that is a product decision. I can build either one. I can’t decide it.

Where this goes wrong

“I can build either one” is true, but at a small company it is often not enough. Here, nobody hands you a specification. You go and find out: you talk to customer success, you look at real uploads, you propose a rule and check it with two customers. An engineer who waits for the decision can wait for weeks.

Minutes 40 to 50: isolation, and a disagreement

Rafael(interviewer):

Okay. Now the incident. The cache key was the question text only, not the tenant. After the fix, we also checked that every tenant table has row-level security in Postgres. So…

He thinks aloud for a moment.

So I think now we are safe. With RLS, the database protects us.

Arash(candidate):

Can I disagree with that a little?

Rafael(interviewer):

Please.

Arash(candidate):

RLS is good. I would keep it. But it protects queries that go through Postgres, with the right tenant set on the connection. The incident was in a cache, right? In Redis? RLS never sees Redis.

Rafael(interviewer):

Yes. The cache is in Redis.

Arash(candidate):

So I would list everything that holds tenant data and is not a normal query. The cache. The vector search: today it’s pgvector in the same Postgres, so RLS applies. But if one day it moves to a separate service, it doesn’t. Background jobs: what database role do the workers use? If it is the table owner, RLS is skipped, unless the table forces it. If it is a role with “bypass RLS”, the policy does nothing.

Rafael(interviewer):

Hmm. The workers use… I would need to check. I think the migration role.

Arash(candidate):

Then that’s worth checking tomorrow. And two more. The logs: if you log prompts, there is contract text from every tenant in one place, and support can search it. And the prompt itself. If anything ever puts passages from two requests into one model call, maybe to save cost, RLS can’t help there either.

Rafael(interviewer):

Pause. You are right. And it is a little embarrassing, because the incident was exactly this. It was outside the database. I said “the database protects us” like I forgot my own incident.

Arash(candidate):

I think everyone says it. RLS feels like a guarantee. It’s a guarantee for one layer.

Disagreeing well

Rafael believed what he said; it was not a test. Arash asks before he disagrees, agrees with the true part first (“RLS is good, I would keep it”), and then gives specific places where the claim is not true. He ends with an action (“worth checking tomorrow”), not a victory. When Rafael admits the mistake, Arash makes it easy for him. See our advice on disagreeing with reasons.

Rafael(interviewer):

So how would you design it?

Arash(candidate):

The tenant is part of every request, from the first line. One object, passed everywhere: to the database session, to the cache function, to the vector query, to the logger. A cache call without a tenant should not compile, or should fail in tests.

And tests that attack it. Two tenants with similar documents, and for each layer, try to read the other one. Run them on every change.

Rafael(interviewer):

Okay. Inside one firm, we also have ethical walls. How does your design handle them?

Arash(candidate):

Sorry, I don’t know that term. Ethical walls?

Rafael(interviewer):

Inside one law firm, a lawyer working for one client must not see the documents of another matter. If there is a conflict of interest. So the same tenant, different permissions.

Arash(candidate):

Oh. Then the tenant is not the right unit. It’s the matter. Or the permission set of the user. That changes the cache key, the vector filter, everything.

Where do the walls come from? Does the firm manage them in your product, or in another system?

Rafael(interviewer):

Both. Some firms sync them from their document system every night. Some set them in Casewell by hand.

Arash(candidate):

Then there’s a delay. A wall added at 3 p.m. arrives at night. Is that acceptable for a law firm? I really don’t know. I would need to learn how firms think about this before I design it.

The second edge

Arash meets a domain concept he has never heard of. He asks what it means, moves his design to the right unit at once, and then finds a real question: the delay in the nightly sync. He does not pretend to know what a law firm accepts.

A stronger answer

With a little preparation about the domain, Arash could raise this himself, before Rafael asks: “In a law firm, is the tenant really the unit of access? Or are there rules inside a firm too?” Ten minutes of reading about how law firms handle confidentiality would give him that question. For a role in a specialised product, learn the basic words of the customer’s work before the interview.

Rafael also asks about the vector search. Arash says that RLS works with pgvector, but he would check the query plan. An approximate index finds the nearest neighbours first, and then the policy removes rows of other tenants. A small tenant may get very few results. “Safe, but bad answers.” Rafael says yes, they saw this, and turned on iterative index scans, a pgvector feature since version 0.8: the index keeps searching until it finds enough rows that pass the filter, or reaches a set limit. Arash did not know about that feature, and he writes it down.

Minutes 50 to 60: Arash's questions, and the close

Arash(candidate):

You use Celery today. Is there a plan to move?

Rafael(interviewer):

Yes, to a durable workflow tool. Long jobs that survive a restart. Honestly… nobody owns that migration yet.

Arash(candidate):

Oh, that is interesting. That’s the part I would enjoy.

Second question. What is a normal week like? How much is that kind of work, and how much is product?

Rafael(interviewer):

Mostly product. Wait, let me be fair. Maybe 70% product: features, customer issues, small fixes the same day. Infrastructure work happens between them. You talk to customer success every week. Sometimes a customer joins a call.

Arash(candidate):

Okay. Pause. Okay, thanks. That’s useful.

What the interviewer is checking

Rafael corrects himself in the middle of the answer (“wait, let me be fair”) because “mostly product” was not precise enough. Arash’s pause is not a bad sign on its own. He is learning something about the job that the posting does not say, and he is thinking about it. See an interview is a conversation in both directions.

Rafael(interviewer):

We are at time. Next is system design with Anjali, and a conversation with Marguerite, our head of customer success. Ben will write to you. And thank you for the RLS part. I will check the worker role tomorrow.

Arash(candidate):

Please tell me what you find. Thanks, Rafael.

After the call: Rafael’s notes

Rafael writes his notes against the job description:

Distributed systems, queues and fairness: strong evidence. The best per-tenant fairness answer I have heard. He designed around tokens, not workers.

Reliability and incidents: strong evidence. Retry storm, retry budgets, and a test that proved the fix.

Security and tenant isolation: strong evidence. He corrected my claim about RLS, politely and correctly. The worker role needs checking.

Python and FastAPI: some evidence. Tooling only. Not a concern.

LLM features: some evidence. Understood token limits fast. No experience with streaming or prompt injection.

Product engineer mindset: concern. Never worked with end users. When the question was about the lawyer, he was honest but had no instinct yet. The legal domain is new to him.

Recommendation: yes, move to system design and values. Marguerite, please talk about the real week here, including on-call.

The next morning, Rafael checked. The background workers did use a role that bypassed RLS. He opened a ticket to give them their own role, and he sent Arash a short thank-you through Ben.

What happened next

Arash’s system design with Anjali went very well. She wrote that his scheduler design was better than the one in the team’s own draft.

Then he had the values conversation with Marguerite Hall. She told him about a real week. A firm called at 8 p.m. on a Thursday, because a deal was closing the next day and 200 files were stuck. The on-call engineer fixed it that night. On-call is one week in eight, and the posting does not say so. Small fixes ship the same day. Engineers sit in customer calls.

Arash asked good questions, and he listened. Two days later, he wrote to Nora and withdrew. He wrote that the people were excellent and the problems were real. But in the interviews, the parts that made him happy were the scheduler and the isolation layers. The parts about the lawyer’s screen and the Friday deadline made him tired just thinking about them. He wanted to stay in infrastructure, and he did not want to take a job and leave after six months.

Nora thanked him and said that she agreed it was a good decision. Rafael asked if he could use Arash’s list of “layers outside RLS” in the team’s incident review. Arash said yes.

This was not a failure for anyone. Arash learned what he wants, before he changed jobs. Casewell avoided hiring someone who would leave soon. Withdrawing from a process is a normal, professional thing to do, if you do it early and politely.

What to take from this

• Disagree with a specific list, not a feeling. Arash named each layer that RLS does not cover. That made his point impossible to miss and easy to check.
• Agree with the true part first, then end with an action. “RLS is good, I would keep it” and “worth checking tomorrow” kept the conversation friendly.
• Deep skill moves to new problems. Arash’s admission control became a token scheduler in a few minutes. If you come from a big company, show how your idea fits the new resource.
• “Product engineer” is a real requirement. Ask what it means for the week. Then decide honestly if you want that week.
• An interview helps you decide too. Arash’s questions found facts the posting did not say, and those facts changed his decision. See an interview is a conversation in both directions.

If your profile is like Arash’s

Your systems knowledge is a real head start (see Your experience is a head start). The work is to show that you can also work with a product and its users, or to decide that you do not want to.

1. Decide what kind of week you want before you apply. Write down the work that made you happiest in the last two years. If it was all infrastructure, look for platform or infrastructure roles at product companies. They exist, and they fit you better.
2. Find one story where you worked with a user. Even an internal one. A team that needed something unclear, and how you found out what they really needed. If you have none, say so honestly, and ask in the interview how the team works with customers.
3. Learn the basic words of the customer’s domain. For legal software: matter, privilege, ethical wall, document management system. Thirty minutes of reading changes the questions you can ask.
4. Practise explaining one deep idea simply. Retry storms, fair queueing, consistency. Say it in two minutes to a friend who is not an engineer. Our lesson Explain decisions shows how.
5. Fix your “led” lines. Say which part you led. It is still impressive, and it survives the first follow-up question. See Portfolio anatomy.
6. Learn the stack that the posting names. For this role: Python, FastAPI and Postgres row-level security, including how roles can bypass it. A weekend with a small project is enough to talk about it with confidence.