At Capgemini Engineering, the world leader in engineering services, we bring together a global team of engineers, scientists, and architects to help the world’s most innovative companies unleash their potential. From autonomous cars to life-saving robots, our digital and software technology experts think outside the box as they provide unique R&D and engineering services across all industries. Join us for a career full of opportunities. Where you can make a difference. Where no two days are the same.

Data Engineer

Job Description:
As a Data Engineer with expertise in PySpark, Databricks, and Microsoft Azure, you will be responsible for designing, developing, and maintaining robust and scalable data pipelines and processing systems. You will work closely with data scientists, analysts, and other stakeholders to ensure our data solutions are efficient, reliable, and scalable.

Responsibilities:
Design, develop, and optimize ETL pipelines using PySpark and Databricks to process large-scale data on the Azure cloud platform.
• Implement data ingestion processes from various data sources into Azure Data Lake and Azure SQL Data Warehouse.
• Develop and maintain data models, data schemas, and data transformation logic tailored for Azure.
• Collaborate with data scientists and analysts to understand data requirements and deliver high-quality datasets.
• Ensure data quality and integrity through robust testing, validation, and monitoring procedures.
• Optimize and tune PySpark jobs for performance and scalability within the Azure and Databricks environments.
• Implement data governance and security best practices in Azure.
• Monitor and troubleshoot data pipelines to ensure timely and reliable data delivery.
• Document data engineering processes, workflows, and best practices specific to Azure and Databricks.

Requirements:
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with a strong focus on PySpark and Databricks. .
• Strong experience with Azure data services, including Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, and Azure Databricks.
• Strong SQL skills and experience with relational databases (e.g., MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB, Cassandra).
• Experience with big data technologies such as Hadoop, Spark, Hive, and Kafka.
• Strong understanding of data architecture, data modeling, and data integration techniques.
• Familiarity with Azure DevOps, version control systems (e.g., Git), and CI/CD pipelines.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills. Preferred Qualifications:
• Experience with Delta Lake on Azure Databricks.
• Knowledge of data visualization tools (e.g., Power BI, Tableau).
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes).
• Understanding of machine learning concepts and experience working with data scientists

Capgemini is a global business and technology transformation partner, helping organizations to accelerate their dual transition to a digital and sustainable world, while creating tangible impact for enterprises and society. It is a responsible and diverse group of 340,000 team members in more than 50 countries. With its strong over 55-year heritage, Capgemini is trusted by its clients to unlock the value of technology to address the entire breadth of their business needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fuelled by its market leading capabilities in AI, cloud and data, combined with its deep industry expertise and partner ecosystem. The Group reported 2023 global revenues of €22.5 billion.