Quality Assurance Manager

Role Overview

We are looking for an experienced QA Manager to lead our quality engineering function, with a strong focus on test automation, evaluation of AI/LLM-based systems, and team leadership . The ideal candidate combines deep hands-on technical expertise with proven experience managing and mentoring QA teams.

Key Details

• Position: QA Manager – Automation
• Total Experience: 10 to 17 years
• Team Management Experience: Minimum 2 years (leading QA/automation teams)
• Employment Type: Full-time

Roles & Responsibilities

• Lead, mentor, and grow a team of QA/automation engineers, driving their technical and professional development.
• Define and own the overall test automation strategy, framework architecture, and quality roadmap for the product.
• Design, build, and maintain scalable automation frameworks using Playwright for web application testing.
• Drive evaluation (eval) strategies for AI/LLM-based features — designing test harnesses, golden datasets, scoring rubrics, and regression suites for model outputs.
• Partner with data science/ML engineering teams to define quality benchmarks for LLM-powered and AI-driven systems , including hallucination checks, output consistency, and prompt regression testing.
• Write and review robust, maintainable automation code in Java , integrating with CI/CD pipelines.
• Establish QA processes, best practices, and metrics (coverage, defect leakage, automation ROI) across the team.
• Collaborate closely with Product, Engineering, and Data Science stakeholders to define acceptance criteria and quality gates, including for AI-driven features.
• Own test planning, risk assessment, release sign-off, and reporting to leadership.
• Stay current with emerging tools and techniques in AI/LLM testing and evaluation methodologies.

Required Skills & Experience

• 10–17 years of total QA/testing experience, including automation-focused roles.
• Minimum 2 years of experience managing/leading a QA or automation team.
• Strong hands-on experience with Playwright for automated UI/end-to-end testing.
• Solid Java programming skills for building and maintaining test frameworks.
• Demonstrated experience testing AI/LLM-based systems — prompt testing, model output validation, regression testing for generative AI features.
• Practical knowledge of eval skills — designing evaluation frameworks, benchmarks, scoring criteria, and golden datasets for AI systems.
• Working knowledge of LLM concepts (prompting, context windows, hallucinations, RAG, fine-tuning basics) sufficient to design meaningful test/eval strategies.
• Experience with CI/CD integration for automated test suites (Jenkins, GitHub Actions, or similar).
• Strong understanding of QA methodologies: functional, regression, integration, performance, and exploratory testing.
• Excellent communication and stakeholder management skills.

Good to Have

• Experience with additional automation tools (Selenium, Cypress, etc.).
• Exposure to API testing tools (Postman, REST Assured).
• Familiarity with cloud platforms (AWS/Azure/GCP) and containerized test environments.
• Prior experience in a product-based or SaaS company testing AI-powered features.

Qualifications

• Bachelor's/Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).