Data Engineer – QA

Data Engineer – QA

Experience: 5–6 Years

Employment Type: C2C

Location: Remote

Working Hours: 11:00 AM – 9:00 PM

Job Summary

We are looking for an experienced Data Engineer – QA with 5–6 years of experience in data validation, reconciliation, data testing, test automation, and large-scale data processing.

The ideal candidate should have strong hands-on experience with Python, SQL, PySpark, Azure Data Factory, Azure Synapse Analytics, Azure Storage, Azure Cosmos DB, AWS S3, and Power BI validation.

The role involves requirement analysis, business rule validation, source-to-target verification, automated validation framework development, data pipeline testing, production support, defect investigation, and stakeholder collaboration.

Key Responsibilities

QA & Solution Analysis

- Analyze requirements and clarify business/data requirements.

- Validate business rules and expected data processing logic.

- Prepare test plans and execute data validation scenarios.

- Investigate defects and perform Root Cause Analysis (RCA).

- Collaborate with stakeholders and delivery teams in an Agile environment.

- Perform comprehensive data validation and reconciliation.

Data Validation & Reconciliation

- Verify source-to-target mappings.

- Perform record count and data completeness analysis.

- Validate schemas, layouts, and column sequences.

- Verify invalid records and exception handling.

- Perform Production vs. Staging comparison.

- Validate source, intermediate, and output datasets.

- Analyze exception, rejected, and invalid records.

Automation & Development

- Develop automation utilities using Python.

- Design and maintain automated data validation frameworks.

- Develop SQL queries for data analysis and validation.

- Develop PySpark notebooks for large-scale data processing and validation.

- Develop and execute test pipelines using Azure Data Factory.

- Build automated validation and reporting utilities.

Cloud & Data Platforms

- Work with Azure Data Factory (ADF).

- Develop and validate Azure Synapse Pipelines and Notebooks.

- Work with Azure Storage Accounts and Containers.

- Validate data stored in Azure Cosmos DB.

- Work with Azure Privileged Identity Management (PIM) where applicable.

- Validate AWS S3 storage and data processing.

- Perform Azure-to-AWS file transfer validation.

- Support job monitoring and production support activities.

Metrics, Reporting & Analytics

- Extract and verify source metrics.

- Validate metrics stored in SQL Server databases.

- Validate Power BI dashboards and reports.

- Perform reconciliation across File → Database → Power BI reporting layers.

- Verify data consistency between source systems and reporting outputs.

Data Processing & Quality

- Process and validate large-scale datasets.

- Work with multiple file formats, including:

- CSV

- Delimited files

- Fixed-width files

- Excel

- Validate source, intermediate, and output datasets.

- Analyze exception, reject, and invalid data.

Required Technical Skills

Programming: Python, SQL, PySpark

Azure: Azure Data Factory, Azure Synapse Analytics, Azure Storage, Azure Cosmos DB, Azure PIM

AWS: AWS S3

Database: SQL Server, SSMS

Analytics: Power BI, Excel

Testing: Data Validation, Data Reconciliation, Data Quality Testing, Pipeline Testing, Test Automation

Other: Source-to-Target Mapping, Production Monitoring & Support, Rally

AI/Automation: Microsoft Copilot

AI & Automation Enablement

- Utilize Microsoft Copilot for productivity and automation where applicable.

- Identify opportunities to improve QA and data validation processes through automation.

- Develop reusable validation and reporting utilities to improve testing efficiency.

Candidate Profile

- 5–6 years of experience in Data Engineering QA / Data Validation / Data Testing.

- Strong hands-on experience in automated data validation and reconciliation.

- Strong understanding of data pipelines and data processing workflows.

- Excellent analytical and problem-solving skills.

- Ability to analyze complex datasets and identify data quality issues.

- Strong defect investigation and RCA capabilities.

- Good communication and stakeholder management skills.

- Ability to work effectively in an Agile environment.

- Strong attention to detail while validating large-scale and multi-format datasets.

Pay: ₹70,000.00 - ₹90,000.00 per month

Experience:

• data validation: 5 years (Required)

Work Location: Remote