Senior Big Data Engineer

Our client is seeking an experienced Big Data Engineer to design, develop, and support large-scale data pipelines and transformation processes. The ideal candidate will have strong SQL, Python, ETL/ELT, and data engineering experience, along with the ability to communicate technical concepts clearly to clients, stakeholders, and cross-functional teams.

Java experience is helpful for supporting existing systems and integrations but is not the primary focus of this position.

Responsibilities

• Design, build, maintain, and optimize ETL/ELT pipelines that move and transform large-scale data across source and target systems.

• Write and optimize complex SQL queries, stored procedures, data models, and validation processes.

• Develop data transformation, integration, and automation solutions primarily using Python.

• Use Java as needed to support existing applications, services, and system integrations.

• Parse, transform, and validate XML-based data formats.

• Work within Linux and shell environments to support scripting, scheduled jobs, cron processing, log reviews, and troubleshooting.

• Implement data validation, error handling, monitoring, and traceability controls.

• Troubleshoot pipeline failures, data discrepancies, performance issues, and integration defects.

• Collaborate with client, security, infrastructure, data engineering, and delivery teams.

• Communicate technical issues, risks, and recommendations clearly to technical and nontechnical stakeholders.

• Participate in Agile planning, status reporting, documentation, issue tracking, and cross-team coordination.

Required Qualifications

• Bachelor’s degree from an accredited college or university or equivalent professional experience.

• Five or more years of professional experience in data engineering, database development, ETL/ELT development, or a related data integration role.

• Strong hands-on SQL experience, including complex queries, performance tuning, stored procedures, and data modeling.

• Experience designing, building, optimizing, and supporting large-scale data pipelines.

• Strong Python programming experience supporting data transformation, integration, automation, or pipeline operations.

• Experience parsing, transforming, and validating XML-based data.

• Experience working in Linux and shell environments, including scripting, scheduled jobs, cron, and log troubleshooting.

• Experience implementing data validation, error handling, pipeline monitoring, and production troubleshooting.

• Strong written and verbal communication skills.

• Ability to explain technical concepts clearly to clients and stakeholders.

• Ability to collaborate across security, infrastructure, engineering, and delivery teams.

• Self-motivated and proactive, with a positive and collaborative approach.

• U.S. citizenship is required.

• Must be willing and able to undergo a background investigation for a Public Trust suitability determination.

Preferred Qualifications

• Experience using Python as the primary language for data pipeline automation, transformation, monitoring, or production troubleshooting.

• Experience supporting Java-based applications, services, or integrations.

• Working knowledge of AWS data services, including S3, Glue, Lambda, RDS, or Redshift.

• Exposure to XBRL or other structured financial, regulatory, or compliance-driven data formats.

• Experience supporting federal, government, or compliance-driven data environments.

• Experience with Oracle, PostgreSQL, Redshift, or other relational database platforms.

• Familiarity with reporting, analytics, or dashboarding tools such as Power BI or MicroStrategy