Senior Data Engineer with Python and Databricks
EndavaJob Description
📋 Description Work together with data scientists and analysts to understand data needs and build data pipelines Design, implement, and maintain data pipelines for ingestion, processing, and transformation using Create and maintain data solutions in Azure using Data Factory, Synapse, Databricks/Spark Implement data validation and cleansing to ensure data quality, integrity and dependability Improve scalability, efficiency, and cost-effectiveness of data pipelines Monitor and resolve data pipeline problems ensuring data consistency and availability 🎯 Requirements Proficient in Python and SQL Knowledge of data design principles (Kimball, Star Schema), data warehousing concepts and data Strong experience with Relational and ideally NoSQL databases Strong understanding of ETL/ELT processes Exposure to data monitoring and observability tools Experience with Event-Driven Architecture 🎁 Benefits Hybrid work and flexible working hours