Data Engineer – AWS & Databricks
Role Overview
The Client Data Engineer will be responsible for the design, development, validation, optimization, and ongoing ownership of data solutions primarily using AWS and Databricks . The role will work closely with internal teams, business SMEs, architects, governance teams, Power BI /analytics teams, and implementation partners.
The ideal candidate should have 5+ years of experience in data engineering, ETL/ELT, and data integration , with strong hands-on expertise in Databricks, AWS, SQL, Python, PySpark, and Spark.
Key Responsibilities
- Design, develop, and support AWS and Databricks data pipelines using SQL, Python, PySpark, and Spark.
- Develop and maintain Bronze, Silver, and Gold data layers using Databricks and Delta Lake.
- Work with source-system data, business rules, mappings, and integration requirements.
- Review and validate data solutions delivered by implementation partners.
- Perform data profiling, reconciliation, testing, and troubleshooting.
- Support data quality, metadata, lineage, and Unity Catalog activities.
- Optimize Databricks pipelines and queries for performance, scalability, and cost efficiency.
- Participate in code reviews, CI/CD, deployment, and production support.
- Maintain technical documentation and support knowledge transfer to internal teams.
- Collaborate closely with BI and analytics teams on downstream reporting and data requirements.
- Support integration with Tableau and Power BI for reporting and analytics use cases.
Required Skills
- 5+ years of experience in data engineering, ETL/ELT, or data integration.
- Strong hands-on experience with Databricks .
- Strong experience with AWS data services and cloud-based data platforms .
- Strong SQL and Python skills.
- Strong experience with PySpark / Apache Spark.
- Experience designing and implementing data pipelines and data transformation frameworks.
- Strong understanding of data integration, transformation, reconciliation, and data quality.
- Strong analytical, troubleshooting, and communication skills.
Preferred Skills
- Databricks, Delta Lake, and Unity Catalog .
- AWS Databricks / Databricks on AWS experience.
- Medallion/Lakehouse architecture.
- Experience with Tableau reporting and Power BI .
- Git, CI/CD, and DevOps practices.
- Experience with data governance, metadata, lineage, and data quality frameworks.
- Databricks certification is a plus.