Senior Data Engineer (Databricks, Lakehouse & AI-BI)

Title: Senior Data Engineer (Databricks Lakehouse & AI-BI)

Location: ABU DHABI

Exp: 6 Years

Job Description:

Key Responsibilities

1. 1. Pipeline Orchestration: Design and build robust data ingestion and orchestration pipelines with Databricks (Lakeflow Jobs/Workflows) as the primary platform and Azure Data Factory supporting legacy/external needs taking full ownership of pipeline reliability and the transition toward Databricks-native orchestration.
2. 2. Lakehouse Data Engineering: Build and maintain scalable Delta Lake tables under Unity Catalog schema design access controls lineage tracking and data quality across bronze/silver/gold layers.
3. 3. Lakeflow Pipeline Development: Build declarative ETL pipelines using Databricks Lakeflow incorporating CDC incremental processing and data quality validations.
4. 4. AI/BI & Genie Enablement: Design and maintain semantic layers metrics and certified datasets to support Genie spaces and AI/BI dashboards enabling natural-language querying and self-service analytics.
5. 5. SharePoint/External Source Integration: Configure and maintain Lakeflow Connect (or similar) integrations for ingesting SharePoint and other external data sources into the Lakehouse with minimal manual effort.
6. 6. Governance & Best Practices: Apply Unity Catalog governance controls access policies data masking audit logging and uphold strong engineering practices (CI/CD version control testing).

Must-Have Skills

• Strong hands-on Databricks expertise PySpark/SQL Workflows/Jobs orchestration Delta Lake Unity Catalog
• Working knowledge of Azure Data Factory
• Experience with Lakeflow (Declarative Pipelines/Connect) including CDC and data quality validations
• Exposure to Databricks AI/BI or Genie (semantic layers certified datasets)
• Familiarity with CI/CD and version control (Git Databricks Repos/Asset Bundles)
• Demonstrated ability to own deliverables fully and resolve issues independently
• Strong communication skills for working with business/analytics stakeholders

Required Experience:

Senior IC

#J-18808-Ljbffr