20,393 Senior Data Engineer jobs in India

Role Overview

We are seeking an experienced Lead Data Engineer to join our Data Engineering team at Paytm, India's leading digital payments and financial services platform. This is a critical role responsible for designing, building, and maintaining large-scale, real-time data streams that process billions of transactions and user interactions daily. Data accuracy and stream reliability are essential to our operations, as data quality issues can result in financial losses and impact customer trust.

As a Lead Data Engineer at Paytm, you will be responsible for building robust data systems that support India's largest digital payments ecosystem. You'll architect and implement reliable, real-time data streaming solutions where precision and data correctness are fundamental requirements . Your work will directly support millions of users across merchant payments, peer-to-peer transfers, bill payments, and financial services, where data accuracy is crucial for maintaining customer confidence and operational excellence.

This role requires expertise in designing fault-tolerant, scalable data architectures that maintain high uptime standards while processing peak transaction loads during festivals and high-traffic events. We place the highest priority on data quality and system reliability, as our customers depend on accurate, timely information for their financial decisions. You'll collaborate with cross-functional teams including data scientists, product managers, and risk engineers to deliver data solutions that enable real-time fraud detection, personalized recommendations, credit scoring, and regulatory compliance reporting.

Key technical challenges include maintaining data consistency across distributed systems with demanding performance requirements, implementing comprehensive data quality frameworks with real-time validation, optimizing query performance on large datasets, and ensuring complete data lineage and governance across multiple business domains. At Paytm, reliable data streams are fundamental to our operations and our commitment to protecting customers' financial security and maintaining India's digital payments infrastructure.

Key Responsibilities

Data Stream Architecture & Development Design and implement reliable, scalable data streams handling high-volume transaction data with strong data integrity controlsBuild real-time processing systems using modern data engineering frameworks (Java/Python stack) with excellent performance characteristicsDevelop robust data ingestion systems from multiple sources with built-in redundancy and monitoring capabilitiesImplement comprehensive data quality frameworks, ensuring the 4 C's: Completeness, Consistency, Conformity, and Correctness - ensuring data reliability that supports sound business decisionsDesign automated data validation, profiling, and quality monitoring systems with proactive alerting capabilities Infrastructure & Platform Management Manage and optimize distributed data processing platforms with high availability requirements to ensure consistent service deliveryDesign data lake and data warehouse architectures with appropriate partitioning and indexing strategies for optimal query performanceImplement CI/CD processes for data engineering workflows with comprehensive testing and reliable deployment proceduresEnsure high availability and disaster recovery for critical data systems to maintain business continuity

Performance & Optimization Monitor and optimize streaming performance with focus on latency reduction and operational efficiencyImplement efficient data storage strategies including compression, partitioning, and lifecycle management with cost considerationsTroubleshoot and resolve complex data streaming issues in production environments with effective response protocolsConduct proactive capacity planning and performance tuning to support business growth and data volume increases

Collaboration & Leadership Work closely with data scientists, analysts, and product teams to understand important data requirements and service level expectationsMentor junior data engineers with emphasis on data quality best practices and customer-focused approachParticipate in architectural reviews and help establish data engineering standards that prioritize reliability and accuracyDocument technical designs, processes, and operational procedures with focus on maintainability and knowledge sharing

Required Qualifications

Experience & Education Bachelor's or Master's degree in Computer Science, Engineering, or related technical field

7+ years (Senior) of hands-on data engineering experience

Proven experience with large-scale data processing systems (preferably in fintech/payments domain)

Experience building and maintaining production data streams processing TB/PB scale data with strong performance and reliability standards

Technical Skills & RequirementsProgramming Languages:

Expert-level proficiency in both Python and Java; experience with Scala preferred

Big Data Technologies: Apache Spark (PySpark, Spark SQL, Spark with Java), Apache Kafka, Apache Airflow

Cloud Platforms: AWS (EMR, Glue, Redshift, S3, Lambda) or equivalent Azure/GCP services

Databases: Strong SQL skills, experience with both relational (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Cassandra, Redis)

Data Quality Management: Deep understanding of the 4 C's framework - Completeness, Consistency, Conformity, and Correctness

Data Governance: Experience with data lineage tracking, metadata management, and data cataloging

Data Formats & Protocols: Parquet, Avro, JSON, REST APIs, GraphQL Containerization & DevOps: Docker, Kubernetes, Git, GitLab/GitHub with CI/CD pipeline experience

Monitoring & Observability: Experience with Prometheus, Grafana, or similar monitoring tools

Data Modeling: Dimensional modeling, data vault, or similar methodologies

Streaming Technologies: Apache Flink, Kinesis, or Pulsar experience is a plus

Infrastructure as Code: Terraform, CloudFormation (preferred)

Java-specific: Spring Boot, Maven/Gradle, JUnit for building robust data services

Preferred Qualifications

Domain Expertise

Previous experience in fintech, payments, or banking industry with solid understanding of regulatory compliance and financial data requirementsUnderstanding of financial data standards, PCI DSS compliance, and data privacy regulations where compliance is essential for business operationsExperience with real-time fraud detection or risk management systems where data accuracy is crucial for customer protection

Advanced Technical Skills (Preferred)

Experience building automated data quality frameworks covering all 4 C's dimensionsKnowledge of machine learning stream orchestration (MLflow, Kubeflow)Familiarity with data mesh or federated data architecture patternsExperience with change data capture (CDC) tools and techniques

Leadership & Soft Skills Strong problem-solving abilities with experience debugging complex distributed systems in production environmentsExcellent communication skills with ability to explain technical concepts to diverse stakeholders while highlighting business valueExperience mentoring team members and leading technical initiatives with focus on building a quality-oriented cultureProven track record of delivering projects successfully in dynamic, fast-paced financial technology environments

What We Offer

Opportunity to work with cutting-edge technology at scaleCompetitive salary and equity compensation

Comprehensive health and wellness benefits

Professional development opportunities and conference attendanceFlexible working arrangements

Chance to impact millions of users across India's digital payments ecosystem

Application Process

Interested candidates should submit:

Updated resume highlighting relevant data engineering experience with emphasis on real-time systems and data quality

Portfolio or GitHub profile showcasing data engineering projects, particularly those involving high-throughput streaming systems

Cover letter explaining interest in fintech/payments domain and understanding of data criticality in financial services

References from previous technical managers or senior colleagues who can attest to your data quality standards

PI1255d80c7d

LOOKING FOR IMMEDIATE JOINERS OR 15 DAYS NOTICE PERIODS AND THIS IS WORK FROM HOME OPPORTUNITY

Position: Senior Data Engineer / Data Engineer

Desired Experience: 3-8 years

Salary: Best-in-industry

You will act as a key member of the Data consulting team, working directly with the partners and senior

stakeholders of the clients designing and implementing big data and analytics solutions. Communication

and organisation skills are keys for this position, along with a problem-solution attitude.

What is in it for you:

Opportunity to work with a world class team of business consultants and engineers solving some of

the most complex business problems by applying data and analytics techniques

Fast track career growth in a highly entrepreneurial work environment

Best-in-industry renumeration package

Essential Technical Skills:

Technical expertise with emerging Big Data technologies, such as: Python, Spark, Hadoop, Clojure,

Git, SQL and Databricks; and visualization tools: Tableau and PowerBI

Experience with cloud, container and micro service infrastructures

Experience working with divergent data sets that meet the requirements of the Data Science and

Data Analytics teams

Hands-on experience with data modelling, query techniques and complexity analysis

Desirable Skills:

Experience/Knowledge of working in an agile environment and experience with agile

methodologies such as Scrum

Experience of working with development teams and product owners to understand their

requirement

Certifications on any of the above areas will be preferred.

Your duties will include:

Develop data solutions within a Big Data Azure and/or other cloud environments

Working with divergent data sets that meet the requirements of the Data Science and Data Analytics

teams

Build and design Data Architectures using Azure Data factory, Databricks, Data lake, Synapse

Liaising with CTO, Product Owners and other Operations teams to deliver engineering roadmaps

showing key items such as upgrades, technical refreshes and new versions

Perform data mapping activities to describe source data, target data and the high-level or

detailed transformations that need to occur;

Assist Data Analyst team in developing KPIs and reporting in tools viz. Power BI, Tableau

Data Integration, Transformation, Modelling

Maintaining all relevant documentation and knowledge bases

Research and suggest new database products, services and protocols

Essential Personal Traits:

You should be able to work independently and communicate effectively with remote teams.

Timely communication/escalation of issues/dependencies to higher management.

Curiosity to learn and apply emerging technologies to solve business problems

Position Summary:

The BIA Data Engineer Designs, implements, and maintains complex data engineering solutions in the Business Intelligence and Analytics team

Responsible for design, development, implementation, testing, documentation, and support of analytical and data solutions/projects requiring data aggregation/data pipelines/ETL/ELT from multiple sources into an efficient reporting mechanism, database/data warehouse using appropriate tools like Informatica, Azure Data Factory, SSIS. This includes interacting with business to gather requirements, analysis, and creation of functional and technical specs, testing, training, escalation, and follow-up.

Support of the applications would include resolving issues reported by users. Issues could be caused by bugs in the application or user errors or programming errors. Resolution process will include, but not limited to, investigate known bugs on software vendor support website, create tickets or service requests with software vendor, develop scripts to fix data issues, make program changes, test fixes and apply the changes to production.

These tasks and activities will be completed with the help and under the guidance of the supervisor. Participation in team and / or project meetings, to schedule work and discuss status, will be required.

The position also requires staying abreast with changes in technology, programming languages, and software development tools.

Duties

VDart: Digital Consulting & Staffing Solutions

VDart is a leading digital consulting and staffing company founded in 2007 and headquartered in Alpharetta, Georgia. VDart is one of the top staffing companies in USA & also provides technology solutions, supporting businesses in their digital transformation journey. Digital consulting services, Digital consulting agency

Core Services:

• Digital consulting and transformation solutions
• Comprehensive staffing services (contract, temporary, permanent placements)
• Talent management and workforce solutions
• Technology implementation services

Key Industries: VDart primarily serves industries such as automotive and mobility, banking and finance, healthcare and life sciences, and energy and utilities. VDart - Products, Competitors, Financials, Employees, Headquarters Locations

Notable Partnership: VDart Digital has been a consistently strong supplier for Toyota since 2017 Digital Transformation Consulting & Strategy Services | VDart Digital , demonstrating their capability in large-scale digital transformation projects.

With over 17 years of experience, VDart combines staffing expertise with technology solutions to help organizations modernize their operations and build skilled workforce capabilities across multiple sectors.

Entity:

Job Family Group:

Job Description:

About bp/team

 Bp's Technology organization is the central organization for all software and platform development. We build all the technology that powers bp’s businesses, from upstream energy production to downstream energy delivery to our customers. We have a variety of teams depending on your areas of interest, including infrastructure and backend services through to customer-facing web and native applications. We encourage our teams to adapt quickly by using native AWS and Azure services, including serverless, and enable them to pick the best technology for a given problem. This is meant to empower our software and platform engineers while allowing them to learn and develop themselves.

Responsibilities
 

• Part of a cross-disciplinary team, working closely with other data engineers, software engineers, data scientists, data managers and business partners.
• Architects, designs, implements and maintains reliable and scalable data infrastructure to move, process and serve data.
• Writes, deploys and maintains software to build, integrate, manage, maintain, and quality-assure data at bp.
• Adheres to and advocates for software engineering standard methodologies (e.g. technical design, technical design review, unit testing, monitoring & alerting, checking in code, code review, documentation)
• Responsible for deploying secure and well-tested software that meets privacy and compliance requirements; develops, maintains and improves CI / CD pipeline.
• Responsible for service reliability and following site-reliability engineering best practices: on-call rotations for services they maintain, responsible for defining and maintaining SLAs. Design, build, deploy and maintain infrastructure as code.
• Containerizes server deployments. Actively contributes to improve developer velocity. Mentors others.

Qualifications

• BS degree or equivalent experience in computer science or related field.
• Deep and hands-on experience designing, planning, building, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments.
• Development experience in one or more object-oriented programming languages (e.g. Python, Scala, Java, C#)
• Sophisticated database and SQL knowledge
• Experience designing and implementing large-scale distributed data systems Deep knowledge and hands-on experience in technologies across all data lifecycle stages.
• Strong stakeholder management and ability to lead initiatives through technical influence
• Continuous learning and improvement mindset Desired
• No prior experience in the energy industry required

Travel Requirement

Relocation Assistance:

Remote Type:

Skills:

Legal Disclaimer:

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, socioeconomic status, neurodiversity/neurocognitive functioning, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please  contact us .

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.

Entity:

Job Family Group:

Job Description:

You will work with

For driving application simplification across the organization, focusing on reducing operational complexity and technical debt. They work closely with the wider digital delivery and digital core teams to identify and pursue simplification opportunities. The team collaborates with business partners to align simplification efforts with broader transformation goals and drive measurable improvements in operational efficiency.

Let me tell you about the role

A Data Engineer designs and builds scalable data management systems that support application simplification efforts. They develop and maintain databases and large-scale processing systems to enable efficient data collection, analysis, and integration. Key responsibilities include ensuring data accuracy for modeling and analytics, optimizing data pipelines for scalability, and collaborating with data scientists to drive data-driven decision-making. They play a crucial role in supporting the team’s goals by enabling seamless access to reliable data for simplification initiatives.

What you will deliver

As part of a cross-disciplinary team, you will collaborate with data engineers, software engineers, data scientists, data managers, and business partners to architect, design, implement, and maintain reliable and scalable data infrastructure for moving, processing, and serving data. You will write, deploy, and maintain software to build, integrate, manage, and assure the quality of data. Adhering to software engineering best practices, you will ensure technical design, unit testing, monitoring, code reviews, and documentation are followed. You will also ensure the deployment of secure, well-tested software that meets privacy and compliance standards, and improve the CI/CD pipeline. Additionally, you will be responsible for service reliability, on-call rotations, SLAs, and maintaining infrastructure as code. By containerizing server deployments and mentoring others, you'll actively contribute to improving developer velocity.

What you will need to be successful (experience and qualifications)

Essential

• Deep, hands-on expertise in designing, building, and maintaining scalable data infrastructure and products in complex environments
• Development experience in object-oriented programming languages (e.g., Python, Scala, Java, C#)
• Advanced knowledge of databases and SQL
• Experience designing and implementing large-scale distributed data systems
• Strong understanding of technologies across all stages of the data lifecycle
• Ability to manage stakeholders effectively and lead initiatives through technical influence
• Continuous learning approach
• BS degree in computer science or a related field (or equivalent experience)

About bp

Our purpose is to deliver energy to the world, today and tomorrow. For over 100 years, bp has focused on discovering, developing, and producing oil and gas in the nations where we operate. We are one of the few companies globally that can provide governments and customers with an integrated energy offering. Delivering our strategy sustainably is fundamental to achieving our ambition to be a net zero company by 2050 or sooner!

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform crucial job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

Travel Requirement

Relocation Assistance:

Remote Type:

Skills:

Legal Disclaimer:

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, socioeconomic status, neurodiversity/neurocognitive functioning, veteran status or disability status. Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process, please  contact us .

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.