Job Description
KPI Partners is seeking a highly skilled and experienced Sr Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with expertise in Python Pandas, Databricks, Azure Data Factory (ADF), and Dev Ops practices.
Key Responsibilities:
- Design, develop, and implement robust data pipelines using Python Pandas and Databricks to ingest, process, and analyze large datasets.
- Collaborate with cross-functional teams to gather requirements and ensure data solutions meet business needs.
- Utilize Azure Data Factory to orchestrate data workflows and integrate data from various sources.
- Implement best practices for data governance, quality, and security throughout the data lifecycle.
- Develop and maintain automated deployment pipelines using Dev Ops tools to streamline the development process.
- Monitor and optimize data processes for performance and reliability.
- Troubleshoot and resolve data-rel...