AWS Data Engineer

Job Title: Data Engineer - Databricks (AWS) | PySpark | Delta Lake

Location : Bengaluru, Noida, Gandhi Nagar, Pune, Chennai

Experience : 4+ Years

Job Summary

We are looking for a talented Data Engineer with strong expertise in Databricks on AWS, PySpark, Delta Lake, and modern data engineering practices . The ideal candidate will be responsible for building scalable data pipelines, implementing data governance frameworks, optimizing platform performance, and supporting enterprise analytics initiatives within the Healthcare/Life Sciences domain.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks on AWS.
• Build and support batch and streaming data ingestion frameworks.
• Implement data quality, governance, lineage, and security controls using Unity Catalog and Databricks best practices.
• Develop and optimize ETL/ELT processes using PySpark and SQL.
• Design and implement Medallion Architecture (Bronze, Silver, Gold) data solutions.
• Build and maintain CDC-based ingestion pipelines using AWS and Databricks services.
• Optimize Spark performance, cluster utilization, and platform costs.
• Implement and support Delta Lake features to ensure data reliability and governance.
• Manage Structured Streaming, Auto Loader, and real-time data processing requirements.
• Develop CI/CD processes using Git, Databricks Asset Bundles, and deployment automation tools.
• Monitor production environments, troubleshoot issues, and ensure SLA compliance.
• Collaborate with business stakeholders, analysts, and engineering teams to deliver high-quality data solutions.
• Mentor junior engineers and contribute to data engineering best practices.

Required Skills & Qualifications

Technical Skills

• 4+ years of experience in Data Engineering.
• 4+ years of hands-on experience with Databricks on AWS.
• Strong expertise in PySpark and Spark optimization techniques.
• Experience with Delta Lake and Lakehouse architecture.
• Strong SQL programming skills.
• Experience with Unity Catalog and Databricks governance frameworks.
• Knowledge of dimensional data modeling and Medallion Architecture.
• Hands-on experience with:
• Structured Streaming
• Auto Loader
• Change Data Capture (CDC)
• Experience with AWS services:
• Amazon S3
• IAM
• AWS DMS
• Kinesis and/or MSK
• AWS Glue
• Experience with Git, CI/CD pipelines, and Databricks Asset Bundles.
• Experience supporting production environments, performance tuning, and cost optimization.

Domain Experience

• Healthcare or Life Sciences domain experience preferred.
• Understanding of data governance, analytics, privacy, and regulatory compliance requirements.

Education

• Bachelor's or Master's Degree in Computer Science, Information Technology, Engineering, or related field.
• Flexible for candidates with strong relevant experience.

Certifications (Preferred)

• Databricks Certified Data Engineer Associate/Professional
• AWS Certified Data Engineer
• AWS Certified Solutions Architect
• Microsoft Azure Certified Developer
• Google Professional Cloud Developer