Job Opportunities API

About the Company CodeNinja is a global technology and AI firm with 10+ years in the industry, helping enterprises modernize systems and embed AI into their workflows. We work across enterprise software, financial services, defense, health, logistics, and more, with 240+ clients across 15 countries. We partner with organizations to deliver AI infrastructure, dedicated engineering teams, and forward-deployed engineers who work directly inside client environments to own outcomes. About the role We are hiring an Senior AI QA Automation Engineer an experienced quality engineer who owns feature-level test strategy and automation for AI-enabled products . In this role, you will design and maintain automated tests and AI evaluation suites , integrate quality signals into CI/CD pipelines , and partner with Scrum teams to prevent defects and deliver reliable AI capabilities. Key Responsibilities Own test strategy and execution for assigned AI features across unit , API , UI , and end-to-end layers, aligned to Definition of Done and release criteria . Create and maintain automated AI evaluation suites (including golden datasets , scoring , and regression baselines ) covering key user journeys and failure modes. Validate prompt , retrieval , and orchestration changes using repeatable test harnesses and defined acceptance thresholds . Build and extend automation frameworks , reusable utilities, and improve test stability , speed, and signal-to-noise . Integrate automated tests and AI evaluations into CI/CD , triage pipeline failures, and reduce flaky tests . Partner with engineers to improve testability (contracts, logging, observability) and drive shift-left quality practices. Analyze defects and AI output issues, perform root cause analysis , and implement regression coverage to prevent recurrence. Provide guidance to junior SDETs on automation , test design , and AI validation patterns; communicate quality risks to stakeholders. Support production readiness by validating guardrails , monitoring signals, and performance requirements for AI features. Requirements Bachelor’s degree in Computer Science/Engineering (or equivalent practical experience). 4 + years of experience in SDET / QA automation / quality engineering with strong hands-on automation and test design skills. Proficiency in at least one programming language (e.g., Python , C# , Java ) and SQL , with the ability to debug services and tests. Experience building and maintaining automated tests (API/UI and/or data/ETL ) and working in CI/CD pipelines . Experience testing AI-enabled systems or demonstrated applied AI fluency (prompt/RAG changes, evaluation basics, common failure modes). Experience working in Agile/Scrum teams and collaborating effectively with Engineering and Product. Preferred: Experience with LLM evaluation techniques (golden sets, rubric-based grading, automated scoring, review workflows). Preferred: Familiarity with AI safety and guardrail testing (grounding, PII handling, injection risks). Preferred: Experience with cloud tooling , containers , observability , and release automation . Why Join Us Build quality systems for real-world AI-enabled products , including LLM evaluations and reliability guardrails. Own feature-level quality end-to-end: from test strategy to CI/CD quality gates to production readiness . Work with strong cross-functional teams across Engineering , Product , and AI stakeholders in an Agile environment. Grow in modern quality engineering practices: observability , automation frameworks , shift-left testing , and AI evaluation patterns. Disclaimer: CodeNinja is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All aspects of employment including the decision to hire, prom