Senior Data Engineer – Data Pipelines & Cloud Databases
Location: Pune | Experience: 4+ years | Type: Full-Time
Role Overview
Design, build, manage enterprise data pipelines on Azure and Databricks. Own schema design, API development, and data infrastructure for analytics and intelligence products.
Key Responsibilities
Data Pipeline Development
Implement robust, scalable data pipelines using Microsoft Azure and Databricks stack Build reusable data pipeline components and frameworks Design and optimize data workflows for performance and reliability Monitor pipeline health and implement automated alertingDatabase Architecture
Design relational and non-relational database schemas Mongo DB: schema design, aggregation pipelines, indexing, sharding, replica sets, performance tuning Azure Cosmos DB: multi-model access, partitioning, consistency levels, throughput management (RU/s), global distribution Azure Cosmos DB Gremlin API: graph data modeling, traversals, vertex/edge design, relationship analyticsAPI Development
Build RESTful APIs using Fast API with async endpoint design Implement Pydantic models, middleware, dependency injection, background tasks Generate and maintain auto-generated Open API/Swagger documentation Configure Azure API Management (APIM) for API gateway and securityProject & Stakeholder Management
Collect progress updates from squads regularly Consolidate updates into weekly project status reports Develop L3-level project plans with task details, milestones, dependencies Participate in early-stage design and feature definition Communicate complex data insights to non-technical stakeholdersCollaboration & Integration
Work across multiple engineering teams on prototype integration Support integration of proven prototypes into core intelligence products Strong team collaboration and cross-functional communication Knowledge sharing and documentationRequired Experience:
Data Engineering & Databases
4+ years data engineering or data platform experience Relational and non-relational database expertise Mongo DB: advanced schema design, aggregation, indexing, sharding, performance optimization Azure Cosmos DB: multi-model APIs, partitioning, consistency, throughput management Graph databases: Cosmos DB Gremlin API, graph modeling, traversals, relationship analyticsAPI & Backend Development
Fast API proficiency: async endpoints, Pydantic models, middleware, dependency injection Background tasks and job scheduling RESTful API design and best practices Open API/Swagger documentationCloud Platform
Azure fundamentals and hands-on experience Azure Data Factory or Databricks for ETL/ELT Azure Cosmos DB multi-region setup Azure API Management (APIM) configurationData Pipeline Skills
ETL/ELT pipeline design and development Data quality validation and monitoring Schema design for analytics and reporting Performance optimization and scalabilityPreferred Experience
Databricks Delta Lake experience Azure Synapse Analytics Python for data engineering Spark SQL optimization Real-time data streaming Data governance and metadata management Agile/Scrum development model CI/CD pipeline experienceThis listing is sourced from a third-party job board. Applying will redirect you to the original posting.