AI Evaluation Engineer

This position is at Aurora Jobs ApS

The selection process will be fully managed by Aurora Jobs ApS.

--

**Sede:** San Francisco, CA - Full-time, On-site **Salario:** $130K - $220K **Equity:** $60K - $120K options/year **Sponsorizzazione Visto:** Disponibile caso per caso **About Artificial Analysis** Artificial Analysis is the leading independent AI benchmarking and insights company. They help engineers, enterprises, investors, media, and policymakers understand AI capabilities and make critical decisions about AI strategy. Their benchmarks are trusted by hundreds of thousands of users and are the go-to reference for leading AI labs including OpenAI, Google, Meta, NVIDIA, and Anthropic. Their work is also cited by the Wall Street Journal, Bloomberg, Financial Times, and The Economist. Their benchmarks do not just measure frontier AI — they actively shape it. The company was founded in 2024, has raised $3.5M, and has a team of 30+ employees across San Francisco, Sydney, and Melbourne, with plans to double by mid-year. They are backed by Nat Friedman, Daniel Gross, Andrew Ng, Adam D’Angelo, Clem Delangue, and other leading figures in AI. **About Aurora** Aurora helps top engineers discover opportunities at some of the most ambitious startups worldwide. We work closely with companies to identify exceptional talent and match them with roles where they can have real impact. We are currently helping Artificial Analysis expand their team. **Position: Member of Technical Staff** **What This Role Actually Is** This role is best described as an AI Evaluation Engineer / Technical Generalist. It is not a traditional software engineering role and not a pure ML research position. It combines AI benchmarking, technical product work, strategic analysis, customer-facing collaboration, and startup execution. **About the Role** Artificial Analysis is hiring Members of Technical Staff to design the evaluations that define how AI is measured, produce analysis that shapes how companies and the broader market understand AI, and work directly with the leading AI labs and enterprise customers who rely on their insights. You will develop new benchmarking methodologies, manage relationships with some of the most important AI labs and enterprise customers in the world, and help drive the product direction of the platform. The success bar for this role is becoming a world expert in modern AI technologies. This is a unique combination of product, research, technical, and client-facing work suited to highly driven technical generalists who thrive on breadth and ownership. You will work directly with the founders and across the full team, with outsized impact on both the company and product. **What You’ll Do** * Design, structure, and execute projects evaluating AI systems and technologies. Develop new benchmarking methodologies and datasets that improve evaluation capabilities. * Build reports, strategic analysis, and data visualizations that help enterprises shape their AI strategy. * Collaborate with leading AI companies to benchmark agentic AI applications, foundation models, and hardware systems. * Identify opportunities to improve the benchmarking platform and work with developers to ship those improvements. * Operate in an AI-native workflow using cutting-edge AI tools to create leverage and maintain speed. * Contribute to company strategy and lead major initiatives that support the company’s goal of becoming the leading AI benchmarking platform. **Requirements** * 2–10 years of experience in MBB consulting or technical roles such as TPM, ML Engineer, or similar. * Strong analytical ability, technical fluency in modern AI systems, excellent communication skills, ownership mindset, and comfort operating in a fast-moving startup environment. **Tech Stack** * Python **Why Join** This is a rare chance to get a front-row seat to frontier AI and evaluate cutting-edge systems before they are publicly released. You’ll work directly with the world’s top AI labs while helping define what best-in-class AI performance actually means. The company is financially strong, fast-growing, highly connected, and offers strong ownership and visibility from day one.

--

[#J-ONE] [#J-MIN]

[#J-INTERNAL]