Senior/Staff Evaluation Data Analyst (Agentic Search)

<div class="content-intro"><p><strong>About Nebius:</strong></p> <p>Nebius is leading a new era in cloud infrastructure for the global AI economy. We are building a full-stack AI cloud platform that supports developers and enterprises from data and model training through to production deployment, without the cost and complexity of building large in-house AI/ML infrastructure.</p> <p>Built by engineers, for engineers. From large-scale GPU orchestration to inference optimization, we own the hard problems across compute, storage, networking and applied AI.</p> <p>Listed on Nasdaq (NBIS) and headquartered in Amsterdam, we have a global footprint with R&amp;D hubs across Europe, the UK, North America and Israel. Our team of 1,500+ includes hundreds of engineers with deep expertise across hardware, software and AI R&amp;D.</p></div><ul class="d9579f95 _005fca58"> <li> <p>In a rapidly evolving world, trust in AI depends on agents being grounded in fresh, verified real-world data. Search is the foundation that makes this possible.<br>We’re looking for an experienced Data Analyst who has built evaluation systems for complex technology products and is excited to take end-to-end ownership of quality in search built for AI agents.<br>This role combines hands-on engineering, analytical judgment, and mathematical rigor, with the scope and influence to shape both technical decisions and product direction. We move at a startup pace, with plenty of open questions and difficult problems to solve.</p> <p><strong><u>In this position, you will:</u></strong></p> <p>• Own the quality measurement and improvement cycle end-to-end: identify user needs, define success, build evaluations, diagnose failures, and work with engineers and product managers to improve outcomes..<br>• Design, build, and evolve evaluation systems, including datasets, benchmarks, metrics, and automated pipelines that support development and launch decisions..<br>• Evaluate response quality across multiple dimensions, including relevance, completeness, freshness, factual accuracy, and usefulness to downstream agents..<br>• Investigate quality regressions and metric anomalies, uncover root causes, and identify the improvements with the greatest potential impact..<br>• Help prioritize the product and engineering roadmap, making quality trade-offs explicit and assessing whether changes deliver meaningful improvements for users..</p> <p><strong><u>We expect you to have:</u></strong></p> <p>• Substantial hands-on experience designing and building evaluation or quality measurement systems for complex technology products, such as search, AI agents, voice technologies, autonomous driving, recommendation systems, advertising, or other similar complex systems..<