Data Engineer – QA
- India
Posted yesterday
About the role
Data Engineer – QA
Experience: 5–6 Years
Employment Type: C2C
Location: Remote
Working Hours: 11:00 AM – 9:00 PM
Job Summary
We are looking for an experienced Data Engineer – QA with 5–6 years of experience in data validation, reconciliation, data testing, test automation, and large-scale data processing.
The ideal candidate should have strong hands-on experience with Python, SQL, PySpark, Azure Data Factory, Azure Synapse Analytics, Azure Storage, Azure Cosmos DB, AWS S3, and Power BI validation.
The role involves requirement analysis, business rule validation, source-to-target verification, automated validation framework development, data pipeline testing, production support, defect investigation, and stakeholder collaboration.
Key Responsibilities
QA & Solution Analysis
- Analyze requirements and clarify business/data requirements.
- Validate business rules and expected data processing logic.
- Prepare test plans and execute data validation scenarios.
- Investigate defects and perform Root Cause Analysis (RCA).
- Collaborate with stakeholders and delivery teams in an Agile environment.
- Perform comprehensive data validation and reconciliation.
Data Validation & Reconciliation
- Verify source-to-target mappings.
- Perform record count and data completeness analysis.
- Validate schemas, layouts, and column sequences.
- Verify invalid records and exception handling.
- Perform Production vs. Staging comparison.
- Validate source, intermediate, and output datasets.
- Analyze exception, rejected, and invalid records.
Automation & Development
- Develop automation utilities using Python.
- Design and maintain automated data validation frameworks.
- Develop SQL queries for data analysis and validation.
- Develop PySpark notebooks for large-scale data processing and validation.
- Develop and execute test pipelines using Azure Data Factory.
- Build automated validation and reporting utilities.
Cloud & Data Platforms
- Work with Azure Data Factory (ADF).
- Develop and validate Azure Synapse Pipelines and Notebooks.
- Work with Azure Storage Accounts and Containers.
- Validate data stored in Azure Cosmos DB.
- Work with Azure Privileged Identity Management (PIM) where applicable.
- Validate AWS S3 storage and data processing.
- Perform Azure-to-AWS file transfer validation.
- Support job monitoring and production support activities.
Metrics, Reporting & Analytics
- Extract and verify source metrics.
- Validate metrics stored in SQL Server databases.
- Validate Power BI dashboards and reports.
- Perform reconciliation across File → Database → Power BI reporting layers.
- Verify data consistency between source systems and reporting outputs.
Data Processing & Quality
- Process and validate large-scale datasets.
- Work with multiple file formats, including:
- CSV
- Delimited files
- Fixed-width files
- Excel
- Validate source, intermediate, and output datasets.
- Analyze exception, reject, and invalid data.
Required Technical Skills
Programming: Python, SQL, PySpark
Azure: Azure Data Factory, Azure Synapse Analytics, Azure Storage, Azure Cosmos DB, Azure PIM
AWS: AWS S3
Database: SQL Server, SSMS
Analytics: Power BI, Excel
Testing: Data Validation, Data Reconciliation, Data Quality Testing, Pipeline Testing, Test Automation
Other: Source-to-Target Mapping, Production Monitoring & Support, Rally
AI/Automation: Microsoft Copilot
AI & Automation Enablement
- Utilize Microsoft Copilot for productivity and automation where applicable.
- Identify opportunities to improve QA and data validation processes through automation.
- Develop reusable validation and reporting utilities to improve testing efficiency.
Candidate Profile
- 5–6 years of experience in Data Engineering QA / Data Validation / Data Testing.
- Strong hands-on experience in automated data validation and reconciliation.
- Strong understanding of data pipelines and data processing workflows.
- Excellent analytical and problem-solving skills.
- Ability to analyze complex datasets and identify data quality issues.
- Strong defect investigation and RCA capabilities.
- Good communication and stakeholder management skills.
- Ability to work effectively in an Agile environment.
- Strong attention to detail while validating large-scale and multi-format datasets.
Pay: ₹70,000.00 - ₹90,000.00 per month
Experience:
- data validation: 5 years (Required)
Work Location: Remote