Data Engineer
KeySkills
-
1.GCP Services: Dataflow,
BigQuery,
Pub/Sub,
Cloud Storage,
Bigtable,
Dataproc
2.Data Engineering: ETL/ELT pipeline development,
data ingestion,
streaming & batch processing
3.Infrastructure as Code (IaC): Terraform,
Cloud Deployment Manager
4.Programming & Scripting: Python,
SQL,
Java/Scala (for Dataflow or Spark jobs)
5.Data Modeling & Warehousing: Star/Snowflake schema design,
partitioning,
clustering,
query optimization
6.Monitoring & Troubleshooting: Stackdriver/Cloud Monitoring,
logging,
alerting,
debugging pipelines
7.Data Quality & Validation: Unit tests,
integration tests,
data validation frameworks
8.Version Control & Collaboration: Git,
GitHub/GitLab,
CI/CD pipelines
9.Soft Skills: Problem-solving,
documentation,
knowledge sharing,
collaboration with cross-functional teams,
Company Name
Job Description
- Job Summary :
We are looking for a skilled GCP Data Engineer to design, implement, and maintain scalable data solutions on Google Cloud Platform. The ideal candidate will have hands-on experience with GCP services, data pipelines, and cloud infrastructure management.
Responsibilities :
-
Design, develop, and maintain data pipelines using GCP services such as Dataflow, BigQuery, and Pub/Sub.
-
Implement and manage data warehouse solutions using BigQuery, ensuring secure and efficient storage.
-
Use Pub/Sub for real-time data ingestion and streaming analytics.
-
Provision and manage GCP infrastructure using Terraform, following best practices in Infrastructure as Code (IaC).
-
Optimize data storage and retrieval processes to enhance performance and reduce costs.
-
Monitor and troubleshoot data pipeline issues to ensure high availability and reliability of data services.
-
Ensure data quality and integrity through robust testing and validation processes.
-
Stay updated with the latest GCP features and best practices, integrating them into existing workflows.
-
Document data workflows, infrastructure setups, and processes for future reference and knowledge sharing.
-