SDET - AI Evaluation Specialist - AI Trainer
About the job
Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Code-Data Eval Author — Software Test Engineer / SDET
Type: Contract
Compensation: $30–$100/hour
Location: Remote
Commitment: 30+ hours/week
Role Responsibilities
- Design verifiers and correctness rubrics for coding tasks to ensure AI agent code functionality.
- Enumerate edge cases and build adversarial test cases for comprehensive agent/model evaluation.
- Grade agent trajectories and improve test/rubric quality through detailed review.
- Work independently and asynchronously to meet deadlines while enhancing AI model performance .
- Collaborate with subject matter experts to ensure test consistency and relevance.
Qualifications
Must-Have
- 5+ years as an SDET / software test engineer at a real product organization.
- Proficiency in writing code and tests using automation frameworks like pytest , Playwright , Cypress .
- Experience with CI/CD processes; SDET preferred over manual-only QA.
- Clear written communication skills.
Preferred
- Familiarity with AI tools and evaluations.
Interview Process
- Mercor Technical Screen : Paid $200 for completing all three steps.
- Live Code Review Session .
- Domain Expert Interview.
Compensation & Legal
- Hourly contractor .
- Paid weekly via Stripe Connect .
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check:
- For any help or support, reach out to: [email protected]
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.
#hiringmercorRecommended Jobs
Engineering Technician IV
Saalex is seeking an Engineering Technician IV in Ridgecrest, CA. Saalex Solutions, a division of Saalex Corp., is an Engineering and Information Technology Services company with a focus on Test R…
DoD Skillbridge Internship for Transitioning Military Service Members
SciTec, Inc. is a dynamic small business with the mission to deliver advanced sensor data processing technologies and scientific instrumentation capabilities in support of National Security and Defen…
Climate Strategy Consultant, Global Business Solutions
Every day, we work together for what matters – bold, swift, and equitable climate action. ABOUT THE ROLE Join our team at 3Degrees, a global leader in climate solutions. As a Climate Strategy…
Director of Business Development - Channel Partner Sales Reef Discovery | eDiscovery Technology Platform
About Reef Discovery Do you want to work for an innovative leader in the eDiscovery technology space? Are you passionate about helping law firms, corporate legal departments, and service providers f…
Kinaxis Senior Consultant
Deloitte's US Supply Chain & Network Operations team helps clients transform supply chain planning through technology, analytics, and operational insight. As a Kinaxis Senior Consultant, you will supp…
Institutional Analyst - Finance & Student Funding
Stanbridge University is seeking an Institutional Analyst – Finance & Student Funding to support data-driven decision making across finance, enrollment, financial aid, and campus operations. This is …
Rep, Mobile Examiner
Under the direction of the Branch Manager or Field Leader, the Mobile Examiner's primary responsibility is to provide coverage in the field ensuring that mobile exams are completed accurately and on …
Veterinarian
Discover Your Future in Coastal Camarillo- $150K–$185K + $30K Sign-On Bonus | 4-Day Week | No On-Call | Small Animal. Since 1965, Pleasant Valley Animal Hospital has proudly served Camarillo with co…
Vice President, Resource Development & Advancement
Vice President, Resource Development & Advancement Help Shape the Future of Community Development in the East Bay EBALDC is seeking a dynamic, relationship-driven fundraising leader to serve as…
Warehouse - Lead
Looking for a new opportunity? Excited to join a growing team? Join us and contribute to a company that values quality, safety, and continuous improvement. At DP World, we work toward continuous grow…