AI Test Engineer

STATERA is looking for an AI Test Engineer to automate the testing of an AI-powered customer support and product troubleshooting solution. The role combines software test automation with hands‑on evaluation of LLMs, RAG pipelines, and conversational agents.

Responsibilities

• Build automated evaluations for RAG retrieval accuracy, context relevance, semantic search precision, and response quality.
• Measure faithfulness, answer relevance, hallucination rates, and guardrail compliance.
• Test multi‑turn troubleshooting flows across product issues such as connectivity failures, battery degradation, and performance faults.
• Simulate complex customer interactions to verify diagnostic state navigation and resolution paths.
• Prevent regressions when product documentation, firmware specifications, or telemetry models change.
• Automate API and end‑to‑end tests across the AI agent, customer support workflows, and warranty‑related integrations.
• Run load and scalability tests on conversational APIs.
• Develop adversarial tests covering prompt injection, out‑of‑scope requests, and hallucination risks.
• Integrate AI evaluation and regression suites into Azure DevOps CI/CD pipelines .

Required experience

• 4+ years in QA automation or SDET roles, including 1–2 years testing LLMs, chatbots, or agentic AI systems.
• Advanced proficiency in Python (including pytest and asyncio) and/or TypeScript or C#.
• Strong API testing and mocking experience with REST/GraphQL, WebSockets, Postman, WireMock, or custom mock servers.
• Experience evaluating RAG systems, semantic search, and vector database integrations.
• Ability to design reliable automated tests for multi‑turn conversations and AI behavior.
• Experience with CI/CD and Azure DevOps.

Valuable additional experience

• IoT telemetry and MQTT.
• Embedded diagnostics or consumer electronics support.
• Mocking device telemetry and testing AI‑driven troubleshooting workflows.

Location: Dubai, UAE — on-site.

#J-18808-Ljbffr