Medior Data Engineer

Function

We’re looking for a skilled consultant to join our client’s dynamic team, specializing in data engineering. Your expertise in Databricks and its integration with AWS will be pivotal in developing and maintaining complex data transformation pipelines. You will play a critical role in managing data ingestion processes and ensuring the robustness and security of data flows across various systems.

Responsibilities

You will be responsible for:

• Designing, developing, and maintaining scalable data pipelines.
• Managing data ingestion processes from multiple sources (including XML ingestion and Kafka).
• Implementing and optimizing ETL (Extract, Transform, Load) processes for efficient data transformation and integration.
• Ensuring data quality, integrity, and security throughout all data pipelines.
• Monitoring and troubleshooting data pipeline performance, carrying out necessary maintenance and updates.
• Documenting data pipeline processes, technical specifications, and best practices.
• Facilitating knowledge transfer to the data engineering team, preparing them to take over platform responsibilities in the future.

Profile of the Ideal Candidate

The successful candidate will have:

• Expertise in Databricks.
• Proficiency in Python and SQL.
• Experience with DATAIKU and SAS, with a preference for familiarity with energy market processes.
• Fluency in English, along with either French or Dutch, is essential. Understanding the other language is a plus.

If you have a passion for data engineering and a proven track record of utilizing Databricks to create efficient data solutions, we encourage you to apply and contribute to our client’s innovative projects.