Machine learning engineers sit between research and production. They have to train models that work, serve them fast and cheaply, and notice when they quietly get worse. These machine learning engineer interview questions test all three, with a large section on LLM systems because that's where much of the 2026 work is. They're for ML leads and hiring managers who need a structured round, and for candidates who want to see what strong answers include. Use the sections that fit the role. A team serving its own models needs different depth than one building on hosted APIs.
What a strong machine learning engineer interview checks
- Production ML judgment: spotting training-serving skew, leakage and metric mismatch.
- Training skills: debugging unstable training and fitting large models into memory.
- Serving and LLM systems: latency, cost, retrieval quality and safe tool use.
- Evaluation: building test sets and metrics you can trust before and after launch.
- MLOps: versioning, reproducibility, monitoring and gradual rollouts.
- Pragmatism: picking the simplest approach that meets the need.
Machine learning engineer interview questions: models in production
1. What is training-serving skew, and how do you prevent it?
What good looks like: the model sees different data in production than in training. Causes include separate feature code for training and serving, features computed with data that isn't available live, and different preprocessing. Prevention: share one feature code path or a feature store, and log the features actually used at serving time so future training data matches.
2. A new model beats the old one offline, but the online A/B test shows no lift. Why might that be?
What good looks like: the offline metric doesn't track the business goal, leakage made offline results look better, the live data distribution differs, the new model added latency that hurt users, or feedback loops trained the old model's behavior into the data. They'd check each with specific analysis.
3. What does a feature store solve?
What good looks like: consistent features across training and serving, low-latency lookup for online inference, and point-in-time correct joins, so each training example only uses feature values that existed at that moment. Without point-in-time joins you leak future data into training.
4. When would you use batch inference instead of real-time inference?
What good looks like: batch when predictions can be computed ahead of time, like nightly recommendations or risk scores, since it's cheaper and simpler. Real-time when the prediction depends on fresh context, such as a fraud check during payment. Many systems mix both.
Training and fine-tuning
5. Training loss suddenly becomes NaN. How do you debug it?
What good looks like: check the data for NaN or infinite values, lower the learning rate, look for log of zero or division by zero in the loss, add gradient clipping for exploding gradients, and check for fp16 overflow (bf16 or loss scaling helps). They'd log gradient norms to find where it starts.
6. Your model runs out of GPU memory during training. What are your options?
What good looks like: a smaller batch with gradient accumulation to keep the effective batch size, mixed precision, activation checkpointing to trade compute for memory, shorter sequences, and sharding the model and optimizer state across GPUs with FSDP or ZeRO. They know optimizer state often takes more memory than the weights.
7. Explain LoRA and when you'd use it.
What good looks like: LoRA freezes the base model and trains small low-rank matrices whose product is added to selected weight matrices. That cuts trainable parameters and memory sharply, and adapters are small enough to swap per task. QLoRA does the same over a 4-bit quantized base model. Use it when full fine-tuning is too costly.
8. For an LLM feature, how do you choose between prompting, RAG and fine-tuning?
What good looks like: start with prompting and a good eval set. Add retrieval when answers depend on private or changing knowledge, or need citations. Fine-tune when you need consistent format, tone or behavior, or want a smaller, cheaper model to match a larger one. They note fine-tuning is a poor way to add facts that change often.
Red flag: fine-tuning first, before trying prompts or measuring anything.
9. How do you make a training run reproducible?
What good looks like: version the code, the data snapshot and the config, pin library versions, set random seeds, and log everything to an experiment tracker. They know some GPU operations are nondeterministic, so exact repeat results may need deterministic settings at some speed cost.
Serving and LLM systems
10. How do you lower latency and cost when serving an LLM?
What good looks like: the KV cache to avoid recomputing attention for past tokens, continuous batching in a serving engine such as vLLM, quantization (8-bit, 4-bit or FP8), speculative decoding, prompt caching for repeated prefixes, and smaller or distilled models where quality allows. They separate time to first token from throughput, and stream tokens to improve perceived speed.
11. Users say your RAG assistant gives wrong answers. How do you find the cause?
What good looks like: split retrieval from generation. First check whether the right passage was retrieved at all (recall at k on a labeled set). If not, look at chunking, the embedding model, hybrid keyword plus vector search, and a reranker. If retrieval is fine, look at the prompt, context length and whether the model is told to answer only from sources.
12. How do you evaluate an LLM feature before launch?
What good looks like: a test set drawn from real cases, with clear pass criteria. Automated checks where possible, an LLM judge only after checking it agrees with human labels, and human review of a sample. The eval set runs on every prompt or model change, like a regression test suite. After launch, online metrics and user feedback.
13. Your agent can read email and call tools. How do you defend against prompt injection?
What good looks like: treat any retrieved or user-supplied text as untrusted data, never as instructions. Give tools the least privilege needed, require human confirmation for actions with side effects, validate tool arguments and outputs, and log everything. They say plainly that no prompt alone fully solves injection, so limits on what the agent can do matter most.
14. Design a recommendation system that responds in under 100 ms.
What good looks like: two stages. Fast candidate retrieval (approximate nearest neighbor search over embeddings, plus simple rules) narrows millions of items to hundreds, then a ranking model scores those using features from an online store. They set a latency budget for each step, cache popular results, and have a fallback when a component times out.
MLOps and monitoring
15. What do you monitor for a model in production?
What good looks like: service health (latency, errors, throughput), input data drift, the distribution of predictions, model quality once labels arrive, performance on key user segments, and the business metric the model exists to move. Alerts go to an owner, with a runbook.
16. How do you roll out a new model version safely?
What good looks like: register and version the model, run it in shadow mode against live traffic, then a small canary or A/B test with guardrail metrics, and keep a one-step rollback to the previous version. For LLMs, they rerun the eval suite before every rollout.
Behavioral (STAR)
17. Tell me about a model that failed in production.
What good looks like: what went wrong, how it was detected (ideally by monitoring), what they did to limit harm, and the process change after. Strong candidates explain the root cause, not just the symptom.
18. Describe a time you chose a simpler model or approach over a complex one.
What good looks like: they compared options against the real requirement, showed the simpler one was good enough with data, and saved time or cost. This shows judgment that rarely appears in a resume.
Red flag: every project used the newest, biggest model available.
19. Tell me about explaining an uncertain ML timeline to a product team.
What good looks like: they broke the work into stages with checkpoints, set a baseline early, shared the risks plainly, and agreed what "good enough to ship" meant before starting.
A simple scoring rubric
Running this interview with AI
ML roles attract many applicants with similar resumes, which makes a consistent first screen valuable. NTRVSTA's exam library includes AI/ML topics and Python, and the interviewer adapts difficulty to each answer, so a candidate who handles training-serving skew well moves on to a harder serving or RAG question. System-design prompts like the 100 ms recommender use an interactive whiteboard, and a hint system shows how someone works when they get stuck.
The hiring team gets a score breakdown and video snippets of key moments to review before the deep technical rounds. Read how AI interviews are scored and our AI interview software comparison. Related question sets: data scientist, data engineer, Python and system design.
Questions
What is the difference between a machine learning engineer and a data scientist interview?
A data scientist loop leans on statistics, experiments and analysis. A machine learning engineer loop leans on building, serving and monitoring models as software: latency, memory, pipelines, testing and rollouts. Most MLE loops also include a regular coding round.
Should I ask candidates to derive formulas like backpropagation by hand?
Rarely. For research roles it can make sense. For most production MLE roles, asking them to explain why training diverged or why a model is slow in serving tells you far more about the work they will actually do.
Is experience only with LLM APIs enough for an ML engineer role?
It depends on the job. If the role builds features on hosted models, strong evaluation, retrieval and reliability skills may be exactly right. If it trains or serves its own models, you need evidence of training, GPU memory and deployment work too. Write that down before the first interview.
Want to see an AI interviewer take an ML candidate from fundamentals to system design? Schedule a 20-minute demo, or try a sample interview on your own.
