10000 Real Time Data Engineer jobs near me

Role: Data Engineer - GCP

Job Summary

• We are seeking a highly skilled Data Engineer with deep expertise in Google Cloud Platform (GCP) and modern data architecture. The ideal candidate will have hands-on experience designing scalable data pipelines, implementing Medallion Architecture, and building robust enterprise-grade data solutions.
• This role requires strong technical proficiency in BigQuery, PySpark, Dataflow, and Airflow, along with a solid understanding of cloud data governance, performance optimization, and CI/CD practices.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines on GCP
• Implement and manage Medallion Architecture (Bronze, Silver, Gold layers) for data processing
• Build high-performance data transformations using Python and PySpark
• Develop and optimize complex SQL queries for analytical workloads
• Work extensively with BigQuery for large-scale data processing and performance tuning
• Develop and deploy pipelines using Cloud Dataflow
• Orchestrate workflows using Cloud Composer (Apache Airflow)
• Manage data storage and lifecycle using Google Cloud Storage (GCS)
• Implement version control and CI/CD pipelines using Git-based tools
• Ensure data security, governance, and access control using GCP IAM
• Optimize data solutions for performance, scalability, reliability, and cost-efficiency

Required Skills & Experience

• Strong hands-on experience with Google Cloud Platform (GCP)
• Expertise in BigQuery (partitioning, clustering, query optimization)
• Proven experience implementing Medallion Data Architecture
• Hands- on Experience with Java Development
• Strong programming skills in Python and PySpark
• Advanced proficiency in SQL (complex joins, window functions, performance tuning)
• Hands-on experience with Cloud Dataflow
• Experience with Cloud Composer (Airflow) for orchestration
• Experience working with Google Cloud Storage (GCS)
• Knowledge of version control systems (Git) and CI/CD practices
• Strong understanding of GCP IAM and cloud security best practices

Experience:

• Google Cloud Platform (GCP): 6 years (Required)
• Medallion Architecture: 6 years (Required)
• Apache Beam / Java Development : 5 years (Required)

Work Location: Remote