Job Summary
We are looking for an experienced AWS Databricks Data Engineer to design, develop, administer, and optimize data platforms built on Databricks and AWS. The ideal candidate will have strong hands-on experience with Databricks, Apache Spark, AWS cloud services, data pipelines, Unity Catalog, security, infrastructure automation, and cost governance.
The candidate will be responsible for building and supporting scalable data engineering solutions while maintaining security, reliability, performance, and operational standards.
Key Responsibilities
Design, develop, and maintain scalable data engineering solutions using Databricks on AWS.
Develop and optimize data pipelines using Apache Spark, PySpark, SQL, and Delta Lake.
Build ETL/ELT pipelines for batch and near-real-time data processing.
Work with AWS services including S3, IAM, Glue, Lambda, CloudWatch, KMS, and VPC.
Administer and support Databricks workspaces, clusters, jobs, notebooks, compute policies, and access controls.
Implement and manage Unity Catalog for centralized data governance, access control, lineage, auditing, and data discovery.
Configure catalogs, schemas, external locations, storage credentials, permissions, and workspace access.
Implement secure integration between Databricks and AWS S3 using appropriate IAM roles and policies.
Automate Databricks and AWS infrastructure using Terraform / Infrastructure as Code.
Monitor and optimize Databricks cluster performance, workloads, and resource utilization.
Implement Databricks job scheduling, monitoring, alerting, and troubleshooting.
Establish and enforce data security, governance, and compliance standards.
Optimize cloud and Databricks costs through cluster policies, autoscaling, job optimization, and resource monitoring.
Support CI/CD implementation for Databricks notebooks, jobs, workflows, and code.
Work with DevOps teams to integrate Databricks deployments with Git, CI/CD pipelines, and Terraform.
Troubleshoot production data pipeline failures, performance issues, and infrastructure problems.
Collaborate with data architects, application teams, DevOps, security, and business stakeholders.
Required Skills
12+ years of experience in Data Engineering / Big Data / Cloud Data Platforms.
Strong hands-on experience with Databricks on AWS.
Strong knowledge of Apache Spark and PySpark.
Advanced SQL skills.
Experience with Delta Lake / Delta tables.
Strong experience with AWS S3 and IAM.
Experience with Databricks Unity Catalog and data governance.
Experience with Databricks administration and workspace management.
Experience with Terraform / Infrastructure as Code.
Experience with Git and CI/CD.
Strong understanding of data pipeline architecture, ETL/ELT, and data warehousing.
Experience with monitoring, troubleshooting, and performance optimization.
Preferred Skills
AWS Glue
AWS Lambda
AWS CloudWatch
AWS KMS
AWS VPC
Databricks Workflows
Databricks Jobs
Databricks SQL
Delta Live Tables / Lakeflow
MLflow
Kafka or other streaming technologies
Python
Scala
Airflow
Azure DevOps / GitHub Actions / Jenkins
Docker / Kubernetes
Data quality and observability tools
This listing is sourced from a third-party job board. Applying will redirect you to the original posting.
General Dynamics Information Technology, Inc.
McKinsey
Agoda Company Pte. Ltd.