Tasks
About the Role
We are an emerging AI-native product-driven agile start-up under Abu Dhabi government AND we are seeking a motivated and technically versatile Data Engineer to join our team. You will play a key role in delivering data platforms pipelines and ML enablement within a Databricks on Azure environment.
As part of a stream-aligned delivery team youll work closely with Data Scientists Architects and Product Managers to build scalable high-quality data solutions for clients. Youll be empowered by a collaborative environment that values continuous learning Agile best practices and technical excellence.
Ideal candidates have strong hands-on experience in Databricks Python ADF and are comfortable in fast-paced client-facing consulting engagements.
Skills and Experience requirements
1. Technical
- Databricks (or similar) e.g. Notebooks (Python SQL) Delta Lake job scheduling clusters and workspace management Unity Catalog access control awareness
- Cloud data engineering ideally Azure including storage (e.g. ADLS S3 ADLS) compute and secrets management
- Development languages such as Python SQL C# javascript etc. especially data ingestion cleaning and transformation
- ETL / ELT including structured logging error handling reprocessing strategies APIs flat files databases message queues event streaming event sourcing etc.
- Automated testing (ideally TDD) pairing/mobbing. Trunk Based Development Continuous Deployment and Infrastructure-as-Code (Terraform)
- Git and CI/CD for notebooks data pipelines and deployments
2. Integration & Data Handling
- Experienced in delivering platforms for clients including file transfer APIS (REST etc.) SQL/NoSQL/graph databases JSON CSV XML Parquet etc
- Data validation and profiling - assess incoming data quality. Cope with schema drift deduplication and reconciliation
- Testing and monitoring pipelines: Unit tests for transformations data checks and pipeline observability
3. Working Style
- Comfortable leveraging the best of lean agile and waterfall approaches. Can contribute to planning estimation and documentation but also collaborative daily re-prioritisation
- Able to explain technical decisions to teammates or clients
- Documents decisions and keeps stakeholders informed
- Comfortable seeking support from other teams for Product Databricks Data architecture
- Happy to collaborate with Data Science team on complex subsystems
Requirements
Nice-to-haves
- MLflow or light MLOps experience (for the data science touchpoints)
- Dbt / dagster / airflow or similar transformation tools
- Understanding of security and compliance (esp. around client data)
- Past experience in consulting or client-facing roles
Candidate Requirements
- 58 years (minimum 34 years hands-on with cloud/data engineering 12 years in Databricks/Azure and team/project leadership exposure)
- Bachelors degree in Computer Science Data Engineering Software Engineering Information Systems Data Engineering
Job Type: Full-time
Benefits
Visa Insurance Yearly Flight Ticket Bonus scheme relocation logistics covered
Interviewing process consists of 2 or 3 technical/behavioral interviews
Key Skills Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala
Employment Type : Full Time
Experience: years
Vacancy: 1
Yearly Salary Salary: 8000 - 8000