Azure Data Engineer
KeySkills
Company Name
Job Description
Develop and optimize data solutions using Databricks SQL and Delta Lake to enhance data accessibility and performance.
-
Design and implement scalable workflows using Databricks Workflows for end-to-end data pipeline orchestration.
-
Utilize PySpark to process, clean, and analyze large datasets while ensuring high data accuracy.
-
Troubleshoot and resolve complex data engineering issues within the Databricks ecosystem.
-
Integrate data from multiple structured and unstructured sources into the Databricks platform.
-
Implement and maintain best practices in data governance, security, and compliance.
-
Identify opportunities for process automation and performance optimization.
-
Collaborate with stakeholders to gather requirements and deliver data solutions on time.
-
Monitor and maintain data pipeline performance for reliability and scalability.
-
Document workflows, technical specifications, and processes for knowledge sharing.
-
Participate in code reviews and maintain high code quality standards.
-
Stay updated with the latest innovations in Databricks and cloud technologies to drive continuous improvement.
-
Support data-driven decision-making by generating insights and recommendations.