Data Engineer
- India
Posted 2 days ago
About the role
Project Role : Data Engineer
Project Role Description : Design, develop and maintain data solutions for data generation, collection, and processing. Create data pipelines, ensure data quality, and implement ETL (extract, transform and load) processes to migrate and deploy data across systems.
Must have skills : Data Engineering
Good to have skills : NA
Minimum 5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary:
As a Data Engineer, a typical day involves designing, developing, and maintaining comprehensive data solutions that support the generation, collection, and processing of data. This role requires creating efficient data pipelines and ensuring the seamless migration and deployment of data across various systems. The position demands continuous attention to data quality and the implementation of processes that facilitate smooth data flow and accessibility, contributing to the overall data infrastructure and operational success.
Roles & Responsibilities:
- Expected to be an SME, collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Lead the design and implementation of scalable data architectures to support business needs.
- Coordinate cross-functional efforts to optimize data workflows and improve system performance.
- Mentor junior team members to foster skill development and knowledge sharing.
- This role offers the opportunity to work with modern cloud platforms, contribute to enterprise data governance practices, and engage with emerging AI and GenAI technologies.
- Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL. Build and manage ETL/ELT pipelines for batch and large-scale data processing. Implement and maintain Medallion Architecture as a core data structuring standard. Develop data transformation frameworks, optimize Spark workloads for performance and cost efficiency, and implement data quality checks and monitoring. Collaborate cross-functionally to translate business requirements into scalable solutions, troubleshoot production pipelines, and partner with data architects to deliver enterprise-grade data solutions.
Professional & Technical Skills:
- Databricks Python PySpark SQL Medallion Architecture (Bronze / Silver / Gold)
- 4–6 years of production Data Engineering experience. Strong hands-on expertise across all mandatory skills. Deep understanding of ETL/ELT design patterns and large-scale data processing.
Good-to-Have Skills
- Cloud: Microsoft Azure (ADF, ADLS, Synapse) GCP (BigQuery, GCS, Dataflow)
- Governance & Architecture: Data Governance Delta Lake Lakehouse Architecture Data Lineage Metadata Management
- Bonus / Plus Skills
- AI Generative AI (GenAI) Large Language Models (LLMs) Agentic AI Canonical Data Models AI–Data Platform Integration
- Preferred Profile
- Strong analytical and problem-solving skills. Ability to design and deliver maintainable, scalable data solutions independently. Excellent communication skills for engaging both technical and business stakeholders. Eagerness to learn and work with emerging technologies across Data Engineering, AI, and GenAI.
- Must To Have Skills: Proficiency in Data Engineering.
- Experience with building and managing data pipelines and ETL processes.
- Strong knowledge of data storage solutions and database management.
- Familiarity with cloud-based data platforms and distributed computing frameworks.
- Ability to ensure data quality through validation and monitoring techniques.
- Competence in scripting and programming languages relevant to data processing.
Additional Information:
- The candidate should have minimum 5 years of experience in Data Engineering.
- This position is based at our Hyderabad office.
- A 15 years full time education is required.
15 years full time education