Responsibilities:

• Design, develop, and implement data processing pipelines using Spark in Databricks notebooks.
• Optimize data workflows for performance, scalability, and reliability.
• Collaborate with data scientists and analysts to understand data requirements and deliver actionable insights.
• Ensure compliance with data governance and security standards.
• Incorporate core data management competencies including data governance, data security, and data quality.
• Work cross-functionally to understand data opportunities and build pipelines and dictionaries for potential data sources.
• Ensure data quality through automating testing and auditing.
• Experience in CI/CD pipelines using Jenkins.
• Manage code repositories and version control using Bitbucket.

Skills:

• Proficiency in Databricks and basic AWS.
• Familiarity with data science, business intelligence, and data analytics.
• Knowledge of data integration, data warehousing, modeling, and presentation tactics and concepts.
• Proficiency in Power BI, including data visualization, relational databases, and communication.
• Experience with data transformation and ETL processes.
• Strong grasp of data modeling principles such as Dimensional modeling and data normalization principles.
• Proficiency in Python and SQL.
• Experience with Jenkins for continuous integration and continuous delivery.
• Familiarity with Bitbucket for code hosting and collaboration2.

Experience:

• Recommended experience: 2+ years of hands-on experience performing the data engineering tasks.