Data Engineer

*No C2C solicitation will be allowed*

Pay: $70 per hour

Type: Onsite Wilmington, DE, McLean, VA, Richmond, VA or NYC hybrid

Duration: 4 months

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Java, Scala, and Apache Spark .
• Build and support multi-stage AWS data pipelines using S3, AWS Glue, Step Functions, and Lambda .
• Develop and maintain event-driven data architectures using Amazon EventBridge and S3 event notifications .
• Design pipeline workflows that appropriately handle event-triggered reruns, retries, failures, and recovery scenarios .
• Configure and manage AWS Step Functions state machines , including scheduling, orchestration, and synchronization of downstream data processing jobs.
• Develop and optimize AWS Glue jobs , including configuration, resource allocation, and scaling for large-volume data workloads.
• Work with modern data formats and technologies such as Apache Iceberg , including troubleshooting platform and version-specific limitations.
• Implement data quality and validation components to ensure data security, integrity, and consistency across pipelines.
• Identify and address schema drift, data changes, and backfill requirements within data pipelines.
• Troubleshoot pipeline failures, performance issues, and data inconsistencies across distributed AWS environments.
• Collaborate with data engineers, architects, and other technical stakeholders to develop reliable and maintainable data solutions.
• Contribute to technical design decisions, documentation, testing, and continuous improvement of the data platform.

Qualifications:

• Strong hands-on experience with Java and/or Scala .
• Strong experience with Apache Spark and distributed data processing.
• Hands-on experience building AWS data pipelines using:
• Amazon S3
• AWS Glue
• AWS Step Functions
• AWS Lambda
• Experience with event-driven architectures , particularly EventBridge rules and S3 event notifications .
• Understanding of Step Functions state machines, scheduling, orchestration, and downstream job synchronization .
• Experience configuring and scaling AWS Glue jobs .
• Strong understanding of data quality, schema validation, schema drift, and backfill strategies .
• Experience troubleshooting and supporting production data pipelines.

Preferred Qualifications

• Experience with Apache Iceberg , including Glue/Iceberg integration and troubleshooting.
• Experience designing resilient pipelines with rerun, retry, and failure-recovery mechanisms .
• Experience implementing data validation or security-focused data quality components.
• Experience working with large-scale cloud data platforms and distributed processing environments.
• Strong problem-solving skills and ability to independently research and resolve technical limitations.