We are seeking an experienced Engineer, AI – AI Evaluation & Model Risk Lead to lead how AI models are evaluated, cleared, monitored, and documented for enterprise use. This role owns model-risk clearance, ongoing model evaluation, model cards, evaluation harnesses, and test suites to ensure AI models are measured, documented, and current.
The ideal candidate will have hands-on experience with machine learning models, AI tools, Python, cloud computing, data analysis, data modeling, model tuning, prompt engineering, and model evaluation . You will collaborate with Safety Standards, Cyber, Responsible AI, and cross-functional teams to support the safe and effective adoption of AI solutions.
Key Responsibilities
- Own initial model-risk clearance and approved-use ratings that gate AI models into use.
- Own ongoing model evaluation and re-clearance following model version changes.
- Evaluate models against live traffic and identify model drift.
- Author and maintain model cards documenting what was tested, approved use cases, and identified risks.
- Build and run AI evaluation harnesses and test suites aligned with established Safety Standards.
- Collaborate with Cyber teams responsible for security red-teaming and Responsible AI teams.
- Collaborate with cross-functional teams to ensure effective integration and functionality of AI systems.
- Develop, test, debug, document, and communicate application and component development activities.
- Validate technical results with user representatives and support overall solution integration.
- Select appropriate technical solutions by reusing, improving, reconfiguring, or creating components.
- Optimize application development for efficiency, cost, and quality .
- Review and create unit test cases, scenarios, test plans, and execution results .
- Contribute to HLD, LLD, SAD, architecture, application features, business components, and data models .
- Perform defect root-cause analysis and mitigation and identify proactive quality improvements.
- Support release processes, configuration management, documentation, project delivery, and technical presentations.
- Interface with customers and development teams to clarify requirements and provide technical guidance.
Required Qualifications
- 2+ years of professional experience in relevant AI, machine learning, software development, or data-focused roles.
- Bachelor's degree with 3 years of related work experience , OR advanced degree with 1 year of related work experience , OR equivalent combination of education and experience.
- Degree in Computer Science, Engineering, Artificial Intelligence, Data Science, or related field preferred.
- 2–4+ years of experience developing and deploying machine learning models , including fine-tuning and prompt engineering.
- 2–4+ years of experience with AI tools and software development using Python or R .
- 2–4+ years of experience collaborating with cross-functional teams to integrate AI solutions into business workflows.
- Strong knowledge of Cloud Computing .
- Strong experience in Data Analysis, Data Management, and Data Modeling .
- Knowledge of Model Tuning and Prompt Engineering .
- Strong understanding of Model Evaluation and AI model-risk concepts.
- Experience creating and maintaining Model Cards & Documentation .
- Understanding of Model Risk & Clearance processes.
- Strong analytical and problem-solving abilities with the ability to work under pressure and manage multiple tasks.
Preferred Qualifications
- Experience with AI model-risk evaluation and clearance .
- Experience building and maintaining AI evaluation harnesses and test suites .
- Experience evaluating models against production/live traffic and identifying model drift.
- Experience with model cards, model documentation, and AI governance practices .
- Experience with machine learning fine-tuning and prompt engineering .
- Preferred certifications include:
- Certified Analytics Professional (CAP)
- Machine Learning Certification
- Certified Data Scientist (CDS)
ABOUT BRICKRED SYSTEMS
BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.