Hiring an AI engineer for the first time can feel hard. A title like "ML engineer" can mean ten different things, and the work spans data, modeling, infrastructure, and product.
This is a short list of questions you can use in a 30-minute screening call. Each question is paired with what a strong answer looks like.
1. "Tell me about a project most like ours. What did you build, and what didn't work?"
You are listening for a real project, not a recap of a course. A good answer names the goal, the data, the tools, the trade-offs they made, and at least one mistake they corrected.
If the candidate cannot describe a failure, they probably have not shipped much.
2. "How do you decide whether to use a hosted model API, an open model, or a fine-tuned model?"
A strong answer covers cost, latency, privacy, output quality, and team capacity to operate the system. They should be comfortable saying "start with a hosted API, measure, and only move to a fine-tuned model when the data shows we need to."
A weak answer is dogmatic — "always fine-tune" or "always use GPT-4" — without context.
3. "What evaluation method do you use?"
You want to hear about offline evaluation (a test set with labels), human review (a sampling process), and online metrics (what changes after the system is live).
If they only mention vibes ("we just looked at the outputs"), that is a yellow flag for production work.
4. "Where will the data live?"
This is a privacy and compliance question disguised as a technical one. You want them to ask about your jurisdiction, who can see the data, and whether any of it is regulated.
If they are happy to send all your data to a third-party API without checking, treat that as a problem.
5. "How will we know when the system is broken in production?"
Listen for logging, monitoring, alert thresholds, and a way to compare current behavior to a baseline. Models drift; data shifts; APIs change. The candidate should have a routine for catching this.
6. "What does hand-off look like at the end of this engagement?"
A senior engineer treats hand-off as part of the work, not an afterthought.
Strong answers include:
- A README that lets a new engineer run the system locally.
- A short evaluation report and the test data.
- A list of known limitations.
- A live walk-through.
If the candidate seems uncomfortable with hand-off, they may be optimizing for being kept around.
7. "Where are you most likely to underestimate the work?"
This is a humility check. Every engineer underestimates something — data cleaning, eval, deployment, edge cases. A candidate who can name their soft spot is more reliable than one who claims they always estimate accurately.
8. "If we asked you to start tomorrow, what would your first three steps be?"
You are looking for a small, sane plan. Strong patterns:
- Read the data and write a one-page summary of what is in it.
- Build the smallest end-to-end pipeline that works, even badly.
- Set up the evaluation harness before iterating.
That sequence catches problems early.
Red flags
- Confident answers to vague questions, lacking specifics.
- "I can do anything in AI."
- Promising a target accuracy before seeing the data.
- Refusing to share code samples, demos, or references.
- Pushing for a fixed-price, long-term contract before a small paid trial.
Green flags
- Asks more questions than they answer in the first 10 minutes.
- Suggests a small paid pilot before committing to the whole scope.
- Sends a follow-up note with a sharper outline of the work.
- Calls out something risky in your brief.
A note on rates
Rates vary widely by region, niche, and experience. Treat the conversation as a fit-and-clarity check, not a price negotiation. Once you trust the engineer, the price discussion is short.