Role Overview
We are looking for an experienced QA Manager to lead our quality engineering function, with a strong focus on test automation, evaluation of AI/LLM-based systems, and team leadership. The ideal candidate combines deep hands-on technical expertise with proven experience managing and mentoring QA teams.
Key Details
Position: QA Manager – Automation Total Experience: 10 to 17 years Team Management Experience: Minimum 2 years (leading QA/automation teams) Employment Type: Full-timeRoles & Responsibilities
Lead, mentor, and grow a team of QA/automation engineers, driving their technical and professional development. Define and own the overall test automation strategy, framework architecture, and quality roadmap for the product. Design, build, and maintain scalable automation frameworks using Playwright for web application testing. Drive evaluation (eval) strategies for AI/LLM-based features — designing test harnesses, golden datasets, scoring rubrics, and regression suites for model outputs. Partner with data science/ML engineering teams to define quality benchmarks for LLM-powered and AI-driven systems, including hallucination checks, output consistency, and prompt regression testing. Write and review robust, maintainable automation code in Java, integrating with CI/CD pipelines. Establish QA processes, best practices, and metrics (coverage, defect leakage, automation ROI) across the team. Collaborate closely with Product, Engineering, and Data Science stakeholders to define acceptance criteria and quality gates, including for AI-driven features. Own test planning, risk assessment, release sign-off, and reporting to leadership. Stay current with emerging tools and techniques in AI/LLM testing and evaluation methodologies.Required Skills & Experience
10–17 years of total QA/testing experience, including automation-focused roles. Minimum 2 years of experience managing/leading a QA or automation team. Strong hands-on experience with Playwright for automated UI/end-to-end testing. Solid Java programming skills for building and maintaining test frameworks. Demonstrated experience testing AI/LLM-based systems — prompt testing, model output validation, regression testing for generative AI features. Practical knowledge of eval skills — designing evaluation frameworks, benchmarks, scoring criteria, and golden datasets for AI systems. Working knowledge of LLM concepts (prompting, context windows, hallucinations, RAG, fine-tuning basics) sufficient to design meaningful test/eval strategies. Experience with CI/CD integration for automated test suites (Jenkins, Git Hub Actions, or similar). Strong understanding of QA methodologies: functional, regression, integration, performance, and exploratory testing. Excellent communication and stakeholder management skills.Good to Have
Experience with additional automation tools (Selenium, Cypress, etc.). Exposure to API testing tools (Postman, REST Assured). Familiarity with cloud platforms (AWS/Azure/GCP) and containerized test environments. Prior experience in a product-based or Saa S company testing AI-powered features.Qualifications
Bachelor's/Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).This listing is sourced from a third-party job board. Applying will redirect you to the original posting.