Azure Databricks & Agentic AI Architect

Role : Azure Databricks & Agentic AI Architect

Location: Chicago, IL(Hybrid- 3 days/week in office)

Contract role

Looking for 16-20 Years minimum experience

Mandatory skills

· Azure, DataBricks, Agentic AI, ETL, SQL, Data Engineering

Role Summary:

We are seeking a visionary Azure Databricks & Agentic AI Architect to design and implement next-generation AI-powered data platforms. This role combines deep expertise in Azure Databricks, Lakehouse Architecture, Data Engineering, and Generative AI to build intelligent, autonomous, and self-optimizing data ecosystems.

The ideal candidate will lead the adoption of Agentic AI within Data Engineering and AI-DLC, enabling autonomous data ingestion, transformation, quality management, lineage discovery, observability, optimization, testing, and governance.

Key Responsibilities

Agentic Data Engineering Leadership

· Design and implement AI-powered Data Engineering platforms leveraging Azure Databricks and Lakehouse architecture.

· Define autonomous workflows using AI Agents for:

· Data ingestion

· Data mapping

· Schema evolution

· Data quality remediation

· Metadata Enrichment

· Pipeline optimization

· Root cause analysis

· Establish frameworks for Human-in-the-Loop (HITL) decision-making and governance.

AI-Driven Data Lifecycle (AI-DLC)

· Lead architecture for AI-enabled Data Development Lifecycle across:

· Requirement analysis

· Data modeling

· Pipeline generation

· Automated testing

· Code review

· Documentation

· Deployment

· Monitoring

· Implement AI copilots to accelerate developer productivity.

· Enable automated lineage creation and intelligent impact analysis.

Lakehouse & Data Platform Architecture

· Design scalable Lakehouse platforms using:

· Azure Databricks

· Delta Lake

· Unity Catalog

· ADLS Gen2

· Databricks Workflows

· Delta Live Tables

Enterprise GenAI Integration

· Architect RAG-based solutions using enterprise data assets.

· Design agent orchestration frameworks using:

· Azure OpenAI

· LangGraph

· Semantic Kernel

· AutoGen

· MCP-enabled architectures

· Build domain-specific AI agents supporting Data Engineering and Analytics teams.

AI Governance & Responsible AI

· Define guardrails for enterprise GenAI adoption.

· Implement:

· Prompt governance

· Observability

· Cost monitoring

· Auditability

· Explainability

· Security controls

· Establish governance models for autonomous AI agents.

AI-Powered Platform Optimization

· Design self-healing data pipelines.

· Implement AI-driven:

· Incident triage

· Failure prediction

· Capacity planning

· Cost optimization

· SLA monitoring

· Enable intelligent workload placement and model routing.

Technical Skills

· Data Platform

· Azure Databricks

· Delta Lake

· Unity Catalog

· Azure Data Factory

AI & Agentic Frameworks

· Azure OpenAI

· Knowledge Graph

· RAG Architecture

· LangChain

· LangGraph

· MCP Protocol

· Vector Databases

· AI Agent Orchestration

Data Engineering

· PySpark

· Spark SQL

· Python

· SQL

· ELT/ETL Modernization

DevOps & AI-DLC

· Azure DevOps

· GitHub Actions

· CI/CD

· MLOps

· LLMOps

· Evaluation Frameworks

· AI Testing Frameworks

Leadership Expectations

· Drive AI-First Data Engineering transformation.

· Define enterprise patterns, accelerators, and reusable AI agents.

· Mentor architects, data engineers, and AI engineers.

· Lead executive conversations on AI adoption, ROI, and transformation roadmaps.