1745 Data Engineer jobs in Bengaluru

Data Engineer- Lead Data Engineer

Bengaluru, Karnataka Paytm

Posted today

Job Viewed

Tap Again To Close

Job Description

Role Overview



We are seeking an experienced Lead Data Engineer to join our Data Engineering team at Paytm, India's leading digital payments and financial services platform. This is a critical role responsible for designing, building, and maintaining large-scale, real-time data streams that process billions of transactions and user interactions daily. Data accuracy and stream reliability are essential to our operations, as data quality issues can result in financial losses and impact customer trust.

As a Lead Data Engineer at Paytm, you will be responsible for building robust data systems that support India's largest digital payments ecosystem. You'll architect and implement reliable, real-time data streaming solutions where precision and data correctness are fundamental requirements . Your work will directly support millions of users across merchant payments, peer-to-peer transfers, bill payments, and financial services, where data accuracy is crucial for maintaining customer confidence and operational excellence.


This role requires expertise in designing fault-tolerant, scalable data architectures that maintain high uptime standards while processing peak transaction loads during festivals and high-traffic events. We place the highest priority on data quality and system reliability, as our customers depend on accurate, timely information for their financial decisions. You'll collaborate with cross-functional teams including data scientists, product managers, and risk engineers to deliver data solutions that enable real-time fraud detection, personalized recommendations, credit scoring, and regulatory compliance reporting.


Key technical challenges include maintaining data consistency across distributed systems with demanding performance requirements, implementing comprehensive data quality frameworks with real-time validation, optimizing query performance on large datasets, and ensuring complete data lineage and governance across multiple business domains. At Paytm, reliable data streams are fundamental to our operations and our commitment to protecting customers' financial security and maintaining India's digital payments infrastructure.


Key Responsibilities


Data Stream Architecture & Development Design and implement reliable, scalable data streams handling high-volume transaction data with strong data integrity controlsBuild real-time processing systems using modern data engineering frameworks (Java/Python stack) with excellent performance characteristicsDevelop robust data ingestion systems from multiple sources with built-in redundancy and monitoring capabilitiesImplement comprehensive data quality frameworks, ensuring the 4 C's: Completeness, Consistency, Conformity, and Correctness - ensuring data reliability that supports sound business decisionsDesign automated data validation, profiling, and quality monitoring systems with proactive alerting capabilities Infrastructure & Platform Management Manage and optimize distributed data processing platforms with high availability requirements to ensure consistent service deliveryDesign data lake and data warehouse architectures with appropriate partitioning and indexing strategies for optimal query performanceImplement CI/CD processes for data engineering workflows with comprehensive testing and reliable deployment proceduresEnsure high availability and disaster recovery for critical data systems to maintain business continuity


Performance & Optimization Monitor and optimize streaming performance with focus on latency reduction and operational efficiencyImplement efficient data storage strategies including compression, partitioning, and lifecycle management with cost considerationsTroubleshoot and resolve complex data streaming issues in production environments with effective response protocolsConduct proactive capacity planning and performance tuning to support business growth and data volume increases


Collaboration & Leadership Work closely with data scientists, analysts, and product teams to understand important data requirements and service level expectationsMentor junior data engineers with emphasis on data quality best practices and customer-focused approachParticipate in architectural reviews and help establish data engineering standards that prioritize reliability and accuracyDocument technical designs, processes, and operational procedures with focus on maintainability and knowledge sharing


Required Qualifications


Experience & Education Bachelor's or Master's degree in Computer Science, Engineering, or related technical field

7+ years (Senior) of hands-on data engineering experience

Proven experience with large-scale data processing systems (preferably in fintech/payments domain)

Experience building and maintaining production data streams processing TB/PB scale data with strong performance and reliability standards


Technical Skills & RequirementsProgramming Languages:

Expert-level proficiency in both Python and Java; experience with Scala preferred


Big Data Technologies: Apache Spark (PySpark, Spark SQL, Spark with Java), Apache Kafka, Apache Airflow

Cloud Platforms: AWS (EMR, Glue, Redshift, S3, Lambda) or equivalent Azure/GCP services

Databases: Strong SQL skills, experience with both relational (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Cassandra, Redis)

Data Quality Management: Deep understanding of the 4 C's framework - Completeness, Consistency, Conformity, and Correctness

Data Governance: Experience with data lineage tracking, metadata management, and data cataloging

Data Formats & Protocols: Parquet, Avro, JSON, REST APIs, GraphQL Containerization & DevOps: Docker, Kubernetes, Git, GitLab/GitHub with CI/CD pipeline experience

Monitoring & Observability: Experience with Prometheus, Grafana, or similar monitoring tools

Data Modeling: Dimensional modeling, data vault, or similar methodologies

Streaming Technologies: Apache Flink, Kinesis, or Pulsar experience is a plus

Infrastructure as Code: Terraform, CloudFormation (preferred)

Java-specific: Spring Boot, Maven/Gradle, JUnit for building robust data services


Preferred Qualifications


Domain Expertise

Previous experience in fintech, payments, or banking industry with solid understanding of regulatory compliance and financial data requirementsUnderstanding of financial data standards, PCI DSS compliance, and data privacy regulations where compliance is essential for business operationsExperience with real-time fraud detection or risk management systems where data accuracy is crucial for customer protection


Advanced Technical Skills (Preferred)


Experience building automated data quality frameworks covering all 4 C's dimensionsKnowledge of machine learning stream orchestration (MLflow, Kubeflow)Familiarity with data mesh or federated data architecture patternsExperience with change data capture (CDC) tools and techniques


Leadership & Soft Skills Strong problem-solving abilities with experience debugging complex distributed systems in production environmentsExcellent communication skills with ability to explain technical concepts to diverse stakeholders while highlighting business valueExperience mentoring team members and leading technical initiatives with focus on building a quality-oriented cultureProven track record of delivering projects successfully in dynamic, fast-paced financial technology environments


What We Offer


Opportunity to work with cutting-edge technology at scaleCompetitive salary and equity compensation

Comprehensive health and wellness benefits

Professional development opportunities and conference attendanceFlexible working arrangements

Chance to impact millions of users across India's digital payments ecosystem


Application Process


Interested candidates should submit:

Updated resume highlighting relevant data engineering experience with emphasis on real-time systems and data quality

Portfolio or GitHub profile showcasing data engineering projects, particularly those involving high-throughput streaming systems

Cover letter explaining interest in fintech/payments domain and understanding of data criticality in financial services

References from previous technical managers or senior colleagues who can attest to your data quality standards






PI1255d80c7d

This advertiser has chosen not to accept applicants from your region.

Data Engineer- Lead Data Engineer

Bengaluru, Karnataka Paytm

Posted today

Job Viewed

Tap Again To Close

Job Description

Role Overview We are seeking an experienced Lead Data Engineer to join our Data Engineering team at Paytm, India's leading digital payments and financial services platform. This is a critical role responsible for designing, building, and maintaining large-scale, real-time data streams that process billions of transactions and user interactions daily. Data accuracy and stream reliability are essential to our operations, as data quality issues can result in financial losses and impact customer a Lead Data Engineer at Paytm, you will be responsible for building robust data systems that support India's largest digital payments ecosystem. You'll architect and implement reliable, real-time data streaming solutions where precision and data correctness are fundamental requirements . Your work will directly support millions of users across merchant payments, peer-to-peer transfers, bill payments, and financial services, where data accuracy is crucial for maintaining customer confidence and operational excellence.This role requires expertise in designing fault-tolerant, scalable data architectures that maintain high uptime standards while processing peak transaction loads during festivals and high-traffic events. We place the highest priority on data quality and system reliability, as our customers depend on accurate, timely information for their financial decisions. You'll collaborate with cross-functional teams including data scientists, product managers, and risk engineers to deliver data solutions that enable real-time fraud detection, personalized recommendations, credit scoring, and regulatory compliance reporting.Key technical challenges include maintaining data consistency across distributed systems with demanding performance requirements, implementing comprehensive data quality frameworks with real-time validation, optimizing query performance on large datasets, and ensuring complete data lineage and governance across multiple business domains. At Paytm, reliable data streams are fundamental to our operations and our commitment to protecting customers' financial security and maintaining India's digital payments ResponsibilitiesData Stream Architecture & Development Design and implement reliable, scalable data streams handling high-volume transaction data with strong data integrity controlsBuild real-time processing systems using modern data engineering frameworks (Java/Python stack) with excellent performance characteristicsDevelop robust data ingestion systems from multiple sources with built-in redundancy and monitoring capabilitiesImplement comprehensive data quality frameworks, ensuring the 4 C's: Completeness, Consistency, Conformity, and Correctness - ensuring data reliability that supports sound business decisionsDesign automated data validation, profiling, and quality monitoring systems with proactive alerting capabilitiesInfrastructure & Platform Management Manage and optimize distributed data processing platforms with high availability requirements to ensure consistent service deliveryDesign data lake and data warehouse architectures with appropriate partitioning and indexing strategies for optimal query performanceImplement CI/CD processes for data engineering workflows with comprehensive testing and reliable deployment proceduresEnsure high availability and disaster recovery for critical data systems to maintain business continuityPerformance & Optimization Monitor and optimize streaming performance with focus on latency reduction and operational efficiencyImplement efficient data storage strategies including compression, partitioning, and lifecycle management with cost considerationsTroubleshoot and resolve complex data streaming issues in production environments with effective response protocolsConduct proactive capacity planning and performance tuning to support business growth and data volume increasesCollaboration & Leadership Work closely with data scientists, analysts, and product teams to understand important data requirements and service level expectationsMentor junior data engineers with emphasis on data quality best practices and customer-focused approachParticipate in architectural reviews and help establish data engineering standards that prioritize reliability and accuracyDocument technical designs, processes, and operational procedures with focus on maintainability and knowledge sharingRequired QualificationsExperience & Education Bachelor's or Master's degree in Computer Science, Engineering, or related technical field7+ years (Senior) of hands-on data engineering experienceProven experience with large-scale data processing systems (preferably in fintech/payments domain)Experience building and maintaining production data streams processing TB/PB scale data with strong performance and reliability standardsTechnical Skills & RequirementsProgramming Languages: Expert-level proficiency in both Python and Java; experience with Scala preferredBig Data Technologies: Apache Spark (PySpark, Spark SQL, Spark with Java), Apache Kafka, Apache AirflowCloud Platforms: AWS (EMR, Glue, Redshift, S3, Lambda) or equivalent Azure/GCP servicesDatabases: Strong SQL skills, experience with both relational (PostgreSQL, MySQL) and NoSQL databases (MongoDB, Cassandra, Redis)Data Quality Management: Deep understanding of the 4 C's framework - Completeness, Consistency, Conformity, and CorrectnessData Governance: Experience with data lineage tracking, metadata management, and data catalogingData Formats & Protocols: Parquet, Avro, JSON, REST APIs, GraphQLContainerization & DevOps: Docker, Kubernetes, Git, GitLab/GitHub with CI/CD pipeline experienceMonitoring & Observability: Experience with Prometheus, Grafana, or similar monitoring toolsData Modeling: Dimensional modeling, data vault, or similar methodologiesStreaming Technologies: Apache Flink, Kinesis, or Pulsar experience is a plusInfrastructure as Code: Terraform, CloudFormation (preferred)Java-specific: Spring Boot, Maven/Gradle, JUnit for building robust data servicesPreferred QualificationsDomain Expertise Previous experience in fintech, payments, or banking industry with solid understanding of regulatory compliance and financial data requirementsUnderstanding of financial data standards, PCI DSS compliance, and data privacy regulations where compliance is essential for business operationsExperience with real-time fraud detection or risk management systems where data accuracy is crucial for customer protectionAdvanced Technical Skills (Preferred) Experience building automated data quality frameworks covering all 4 C's dimensionsKnowledge of machine learning stream orchestration (MLflow, Kubeflow)Familiarity with data mesh or federated data architecture patternsExperience with change data capture (CDC) tools and techniquesLeadership & Soft Skills Strong problem-solving abilities with experience debugging complex distributed systems in production environmentsExcellent communication skills with ability to explain technical concepts to diverse stakeholders while highlighting business valueExperience mentoring team members and leading technical initiatives with focus on building a quality-oriented cultureProven track record of delivering projects successfully in dynamic, fast-paced financial technology environmentsWhat We Offer Opportunity to work with cutting-edge technology at scaleCompetitive salary and equity compensationComprehensive health and wellness benefitsProfessional development opportunities and conference attendanceFlexible working arrangementsChance to impact millions of users across India's digital payments ecosystemApplication Process Interested candidates should submit:Updated resume highlighting relevant data engineering experience with emphasis on real-time systems and data qualityPortfolio or GitHub profile showcasing data engineering projects, particularly those involving high-throughput streaming systemsCover letter explaining interest in fintech/payments domain and understanding of data criticality in financial servicesReferences from previous technical managers or senior colleagues who can attest to your data quality standards
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka Kyndryl

Posted 2 days ago

Job Viewed

Tap Again To Close

Job Description

**Who We Are**
At Kyndryl, we design, build, manage and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward - always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers and our communities.
**The Role**
Are you ready to dive headfirst into the captivating world of data engineering at Kyndryl? As a Data Engineer, you'll be the visionary behind our data platforms, crafting them into powerful tools for decision-makers. Your role? Ensuring a treasure trove of pristine, harmonized data is at everyone's fingertips.
As a Data Engineer at Kyndryl, you'll be at the forefront of the data revolution, crafting and shaping data platforms that power our organization's success. This role is not just about code and databases; it's about transforming raw data into actionable insights that drive strategic decisions and innovation.
In this role, you'll be engineering the backbone of our data infrastructure, ensuring the availability of pristine, refined data sets. With a well-defined methodology, critical thinking, and a rich blend of domain expertise, consulting finesse, and software engineering prowess, you'll be the mastermind of data transformation.
Your journey begins by understanding project objectives and requirements from a business perspective, converting this knowledge into a data puzzle. You'll be delving into the depths of information to uncover quality issues and initial insights, setting the stage for data excellence. But it doesn't stop there. You'll be the architect of data pipelines, using your expertise to cleanse, normalize, and transform raw data into the final dataset-a true data alchemist.
Armed with a keen eye for detail, you'll scrutinize data solutions, ensuring they align with business and technical requirements. Your work isn't just a means to an end; it's the foundation upon which data-driven decisions are made - and your lifecycle management expertise will ensure our data remains fresh and impactful.
So, if you're a technical enthusiast with a passion for data, we invite you to join us in the exhilarating world of data engineering at Kyndryl. Let's transform data into a compelling story of innovation and growth.
Your Future at Kyndryl
Every position at Kyndryl offers a way forward to grow your career. We have opportunities that you won't find anywhere else, including hands-on experience, learning opportunities, and the chance to certify in all four major platforms. Whether you want to broaden your knowledge base or narrow your scope and specialize in a specific sector, you can find your opportunity here.
**Who You Are**
You're good at what you do and possess the required experience to prove it. However, equally as important - you have a growth mindset; keen to drive your own personal and professional development. You are customer-focused - someone who prioritizes customer success in their work. And finally, you're open and borderless - naturally inclusive in how you work with others.
Required Technical and Professional Expertise
+ 6-8 years of experience working as a Data Engineer or in Azure cloud modernization
+ Good experience in Power BI for data visualization and dashboard development
+ Strong experience in developing data pipelines and using tools such as AWS Glue, Azure Databricks, Synapse, or Google Dataproc
+ Proficient in working with both relational and NoSQL databases, including PostgreSQL, DB2, and MongoDB
+ Excellent problem-solving, analytical, and critical thinking skills
+ Ability to manage multiple projects simultaneously while maintaining a high level of attention to detail
+ Expertise in data mining, data storage, and Extract-Transform-Load (ETL) processes
Preferred Technical and Professional Experience
+ Experience in Data Modelling, to create conceptual model of how data is connected and how it will be used in business processes
+ Professional certification, e.g., Open Certified Technical Specialist with Data Engineering Specialization
+ Cloud platform certification, e.g., AWS Certified Data Analytics - Specialty, Elastic Certified Engineer, Google Cloud Professional Data Engineer, or Microsoft Certified: Azure Data Engineer Associate
+ Understanding of social coding and Integrated Development Environments, e.g., GitHub and Visual Studio
+ Degree in a scientific discipline, such as Computer Science, Software Engineering, or Information Technology
**Being You**
Diversity is a whole lot more than what we look like or where we come from, it's how we think and who we are. We welcome people of all cultures, backgrounds, and experiences. But we're not doing it single-handily: Our Kyndryl Inclusion Networks are only one of many ways we create a workplace where all Kyndryls can find and provide support and advice. This dedication to welcoming everyone into our company means that Kyndryl gives you - and everyone next to you - the ability to bring your whole self to work, individually and collectively, and support the activation of our equitable culture. That's the Kyndryl Way.
**What You Can Expect**
With state-of-the-art resources and Fortune 100 clients, every day is an opportunity to innovate, build new capabilities, new relationships, new processes, and new value. Kyndryl cares about your well-being and prides itself on offering benefits that give you choice, reflect the diversity of our employees and support you and your family through the moments that matter - wherever you are in your life journey. Our employee learning programs give you access to the best learning in the industry to receive certifications, including Microsoft, Google, Amazon, Skillsoft, and many more. Through our company-wide volunteering and giving platform, you can donate, start fundraisers, volunteer, and search over 2 million non-profit organizations. At Kyndryl, we invest heavily in you, we want you to succeed so that together, we will all succeed.
**Get Referred!**
If you know someone that works at Kyndryl, when asked 'How Did You Hear About Us' during the application process, select 'Employee Referral' and enter your contact's Kyndryl email address.
Kyndryl is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, age, veteran status, or other characteristics. Kyndryl is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka Kyndryl

Posted 6 days ago

Job Viewed

Tap Again To Close

Job Description

**Who We Are**
At Kyndryl, we design, build, manage and modernize the mission-critical technology systems that the world depends on every day. So why work at Kyndryl? We are always moving forward - always pushing ourselves to go further in our efforts to build a more equitable, inclusive world for our employees, our customers and our communities.
**The Role**
Are you ready to dive headfirst into the captivating world of data engineering at Kyndryl? As a Data Engineer, you'll be the visionary behind our data platforms, crafting them into powerful tools for decision-makers. Your role? Ensuring a treasure trove of pristine, harmonized data is at everyone's fingertips.
As a Data Engineer at Kyndryl, you'll be at the forefront of the data revolution, crafting and shaping data platforms that power our organization's success. This role is not just about code and databases; it's about transforming raw data into actionable insights that drive strategic decisions and innovation.
In this role, you'll be engineering the backbone of our data infrastructure, ensuring the availability of pristine, refined data sets. With a well-defined methodology, critical thinking, and a rich blend of domain expertise, consulting finesse, and software engineering prowess, you'll be the mastermind of data transformation.
Your journey begins by understanding project objectives and requirements from a business perspective, converting this knowledge into a data puzzle. You'll be delving into the depths of information to uncover quality issues and initial insights, setting the stage for data excellence. But it doesn't stop there. You'll be the architect of data pipelines, using your expertise to cleanse, normalize, and transform raw data into the final dataset-a true data alchemist.
Armed with a keen eye for detail, you'll scrutinize data solutions, ensuring they align with business and technical requirements. Your work isn't just a means to an end; it's the foundation upon which data-driven decisions are made - and your lifecycle management expertise will ensure our data remains fresh and impactful.
So, if you're a technical enthusiast with a passion for data, we invite you to join us in the exhilarating world of data engineering at Kyndryl. Let's transform data into a compelling story of innovation and growth.
Your Future at Kyndryl
Every position at Kyndryl offers a way forward to grow your career. We have opportunities that you won't find anywhere else, including hands-on experience, learning opportunities, and the chance to certify in all four major platforms. Whether you want to broaden your knowledge base or narrow your scope and specialize in a specific sector, you can find your opportunity here.
**Who You Are**
You're good at what you do and possess the required experience to prove it. However, equally as important - you have a growth mindset; keen to drive your own personal and professional development. You are customer-focused - someone who prioritizes customer success in their work. And finally, you're open and borderless - naturally inclusive in how you work with others.
Required Technical and Professional Expertise
+ 4-5 years of experience working as a Data Engineer or in Azure cloud modernization
+ Good experience in Power BI for data visualization and dashboard development
+ Strong experience in developing data pipelines and using tools such as AWS Glue, Azure Databricks, Synapse, or Google Dataproc
+ Proficient in working with both relational and NoSQL databases, including PostgreSQL, DB2, and MongoDB
+ Excellent problem-solving, analytical, and critical thinking skills
+ Ability to manage multiple projects simultaneously while maintaining a high level of attention to detail
+ Expertise in data mining, data storage, and Extract-Transform-Load (ETL) processes
Preferred Technical and Professional Experience
+ Experience in Data Modelling, to create conceptual model of how data is connected and how it will be used in business processes
+ Professional certification, e.g., Open Certified Technical Specialist with Data Engineering Specialization
+ Cloud platform certification, e.g., AWS Certified Data Analytics - Specialty, Elastic Certified Engineer, Google Cloud Professional Data Engineer, or Microsoft Certified: Azure Data Engineer Associate
+ Understanding of social coding and Integrated Development Environments, e.g., GitHub and Visual Studio
+ Degree in a scientific discipline, such as Computer Science, Software Engineering, or Information Technology
**Being You**
Diversity is a whole lot more than what we look like or where we come from, it's how we think and who we are. We welcome people of all cultures, backgrounds, and experiences. But we're not doing it single-handily: Our Kyndryl Inclusion Networks are only one of many ways we create a workplace where all Kyndryls can find and provide support and advice. This dedication to welcoming everyone into our company means that Kyndryl gives you - and everyone next to you - the ability to bring your whole self to work, individually and collectively, and support the activation of our equitable culture. That's the Kyndryl Way.
**What You Can Expect**
With state-of-the-art resources and Fortune 100 clients, every day is an opportunity to innovate, build new capabilities, new relationships, new processes, and new value. Kyndryl cares about your well-being and prides itself on offering benefits that give you choice, reflect the diversity of our employees and support you and your family through the moments that matter - wherever you are in your life journey. Our employee learning programs give you access to the best learning in the industry to receive certifications, including Microsoft, Google, Amazon, Skillsoft, and many more. Through our company-wide volunteering and giving platform, you can donate, start fundraisers, volunteer, and search over 2 million non-profit organizations. At Kyndryl, we invest heavily in you, we want you to succeed so that together, we will all succeed.
**Get Referred!**
If you know someone that works at Kyndryl, when asked 'How Did You Hear About Us' during the application process, select 'Employee Referral' and enter your contact's Kyndryl email address.
Kyndryl is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, age, veteran status, or other characteristics. Kyndryl is also committed to compliance with all fair employment practices regarding citizenship and immigration status.
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bengaluru, Karnataka Red Hat

Posted 9 days ago

Job Viewed

Tap Again To Close

Job Description

The Data Engineer exercises judgment when following general instructions and works with minimal instruction to support the integration and automation of data solutions. This role focuses on data massaging, reconciliation, and analysis, resolving routine to semi-routine issues. Responsibilities include creating optimized SQL queries, managing data pipelines, and collaborating with cross-functional teams to ensure data accuracy and availability.
**What will you do:**
+ Write optimized and scalable complex SQL queries
+ Automate data processing tasks using Python, focusing on cleaning and merging datasets.
+ Manage data pipelines, including scheduling, monitoring, and debugging workflows.
+ Collaborate with data engineers and IT teams to maintain data accessibility for stakeholders.
+ Assist in developing automated tests to ensure the accuracy and integrity of data.
+ Participate in version control and CI/CD processes for deploying and testing pipeline changes across environments.
+ Work cross-functionally with analysts, engineers, and operations.
+ Data stewardship including: data governance, data compliance, data transformation, data cleanliness, data validation, data audit/maintenance.
+ Writing complex, highly-optimized SQL queries across large datasets, involved in SQL Query tuning and provided tuning recommendations
+ Experienced in Data Analytics, hands-on experience of various Python libraries such as NumPy and Pandas
+ Python development experience to massage, clean data and automate data extract and loads
+ Expertise to convert raw data to processed data by merging, finding outliers, errors, trends, missing values and distributions in the data
+ Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow, resolve performance tuning related issues and queries
+ Foster collaboration among Data engineers, IT & other business groups to ensure data is accessible to FP&A team
+ Scheduled a regular hot backup process and involved in the backup activities
+ Strong analytical and problem-solving skills with ability to represent complex algorithms in software
+ Develop automated unit tests, end-to-end tests, and integration tests to assist in quality assurance (QA) procedures
**What will you bring:**
+ Bachelor's or Master's degree in Computer Science, IT, Engineering or equivalent
+ 5+ years of experience as a Data Engineer, BI Engineer, Systems Analyst in a company with large, complex data sources
+ Working knowledge of DBT, Snowflake, Fivetran, Git and SQL or Python programming skills for data querying, cleaning, and presentation
+ Build highly available, reliable and secured API solutions, experience working with REST API design and Implementation
+ Working knowledge of relational databases (PostgreSQL, MSSQL, etc.), experience with AWS services including S3, Redshift, EMR and RDS
+ Ability to manage multiple projects at the same time in a fast-paced team environment, across time zones, and with different cultures, while maintaining ability to work as part of a team
+ The candidate must have good troubleshooting skills and be able to think through issues and problems in a logical manner and planning knowledge would be an added advantage
+ Detail-oriented and enthusiastic who is also focused and diligent on delivering results
**About Red Hat**
Red Hat ( is the world's leading provider of enterprise open source ( software solutions, using a community-powered approach to deliver high-performing Linux, cloud, container, and Kubernetes technologies. Spread across 40+ countries, our associates work flexibly across work environments, from in-office, to office-flex, to fully remote, depending on the requirements of their role. Red Hatters are encouraged to bring their best ideas, no matter their title or tenure. We're a leader in open source because of our open and inclusive environment. We hire creative, passionate people ready to contribute their ideas, help solve complex problems, and make an impact.
**Inclusion at Red Hat**
Red Hat's culture is built on the open source principles of transparency, collaboration, and inclusion, where the best ideas can come from anywhere and anyone. When this is realized, it empowers people from different backgrounds, perspectives, and experiences to come together to share ideas, challenge the status quo, and drive innovation. Our aspiration is that everyone experiences this culture with equal opportunity and access, and that all voices are not only heard but also celebrated. We hope you will join our celebration, and we welcome and encourage applicants from all the beautiful dimensions that compose our global village.
**Equal Opportunity Policy (EEO)**
Red Hat is proud to be an equal opportunity workplace and an affirmative action employer. We review applications for employment without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, ancestry, citizenship, age, veteran status, genetic information, physical or mental disability, medical condition, marital status, or any other basis prohibited by law.
**Red Hat does not seek or accept unsolicited resumes or CVs from recruitment agencies. We are not responsible for, and will not pay, any fees, commissions, or any other payment related to unsolicited resumes or CVs except as required in a written contract between Red Hat and the recruitment agency or party requesting payment of a fee.**
**Red Hat supports individuals with disabilities and provides reasonable accommodations to job applicants. If you need assistance completing our online job application, email** ** ** **. General inquiries, such as those regarding the status of a job application, will not receive a reply.**
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka ThermoFisher Scientific

Posted 12 days ago

Job Viewed

Tap Again To Close

Job Description

**Work Schedule**
Standard (Mon-Fri)
**Environmental Conditions**
Office
**Job Description**
**Job Summary:**
We are seeking a skilled and detail-oriented **Data Engineer** to join our data team. The ideal candidate will be responsible for building and maintaining scalable data pipelines, extracting data from diverse sources including APIs, databases, and flat files, and ensuring high data quality and reliability. You will work closely with analysts, data scientists, and engineers to power data-driven decision-making across the organization.
**Key Responsibilities:**
+ Design, develop, and maintain scalable and robust data pipelines for both batch and real-time processing.
+ Extract, transform, and load (ETL) data from a wide variety of structured and unstructured data sources including:
+ RESTful and SOAP APIs
+ Databases (SQL, NoSQL)
+ Cloud storage (e.g., S3, Google Cloud Storage)
+ File formats (e.g., JSON, CSV, XML, Parquet)
+ Web scraping tools where appropriate
+ Build reusable data connectors and integration solutions to automate data ingestion.
+ Collaborate with internal stakeholders to understand data requirements and ensure accessibility and usability.
+ Monitor and optimize pipeline performance and troubleshoot data flow issues.
+ Ensure data governance, security, and quality standards are applied across all pipelines.
+ Experience with data manipulation and analysis libraries such as Pandas, Polars, or Dask for handling large datasets efficiently.
+ Design and create data flow and architecture diagrams to visually represent data pipelines, system integrations, and data models, ensuring clarity and alignment among technical and non-technical stakeholders.
**Requirements:**
**Technical Skills:**
+ Proficiency in SQL and at least one programming language (Python, Java, Scala).
+ Experience with data pipeline and workflow tools (e.g., Apache Airflow, AWS Data Pipeline).
+ Knowledge of relational and non-relational databases. (e.g., Oracle, SqlServer, MongoDB).
+ Strong data modeling and data warehousing skills.
**Education & Experience:**
+ Bachelor's degree in Computer Science, Engineering, Information Systems, or related field (Master's a plus).
+ 5+ years of experience in a data engineering or similar role.
**Soft Skills:**
+ Strong analytical and problem-solving abilities.
+ Excellent communication and collaboration skills.
+ Detail-oriented and proactive mindset.
Thermo Fisher Scientific is an EEO/Affirmative Action Employer and does not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability or any other legally protected status.
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka NTT America, Inc.

Posted 13 days ago

Job Viewed

Tap Again To Close

Job Description

**Req ID:**
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.
We are currently seeking a Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).
**Key Skills & Competencies**
+ Advanced SQL development (joins, CTEs, window functions, optimization)
+ Experience with ETL/ELT processes and tools
+ Data modeling (dimensional and normalized
+ Familiarity with version control (e.g., Git) and CI/CD practices
+ Understanding of DWH architectures and data integration patterns
+ Ability to work with large datasets and performance-tune queries
+ Platform-agnostic mindset with readiness to adapt to Azure or AWS
**About NTT DATA**
NTT DATA is a $30 billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established and start-up companies. Our services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation and management of applications, infrastructure and connectivity. We are one of the leading providers of digital and AI infrastructure in the world. NTT DATA is a part of NTT Group, which invests over $3.6 billion each year in R&D to help organizations and society move confidently and sustainably into the digital future. Visit us at us.nttdata.com ( possible, we hire locally to NTT DATA offices or client sites. This ensures we can provide timely and effective support tailored to each client's needs. While many positions offer remote or hybrid work options, these arrangements are subject to change based on client requirements. For employees near an NTT DATA office or client site, in-office attendance may be required for meetings or events, depending on business needs. At NTT DATA, we are committed to staying flexible and meeting the evolving needs of both our clients and employees. NTT DATA recruiters will never ask for payment or banking information and will only use @nttdata.com and @talent.nttdataservices.com email addresses. If you are requested to provide payment or disclose banking information, please submit a contact us form, .
**_NTT DATA endeavors to make_** **_ **_accessible to any and all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please contact us at_** **_ **_._** **_This contact information is for accommodation requests only and cannot be used to inquire about the status of applications. NTT DATA is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status. For our EEO Policy Statement, please click here ( . If you'd like more information on your EEO rights under the law, please click here ( . For Pay Transparency information, please click here ( ._**
This advertiser has chosen not to accept applicants from your region.
Be The First To Know

About the latest Data engineer Jobs in Bengaluru !

Data Engineer

Bangalore, Karnataka NTT DATA North America

Posted 13 days ago

Job Viewed

Tap Again To Close

Job Description

**Req ID:**
NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.
We are currently seeking a Data Engineer to join our team in Bangalore, Karnātaka (IN-KA), India (IN).
**Key Skills & Competencies**
+ Advanced SQL development (joins, CTEs, window functions, optimization)
+ Experience with ETL/ELT processes and tools
+ Data modeling (dimensional and normalized
+ Familiarity with version control (e.g., Git) and CI/CD practices
+ Understanding of DWH architectures and data integration patterns
+ Ability to work with large datasets and performance-tune queries
+ Platform-agnostic mindset with readiness to adapt to Azure or AWS
**About NTT DATA**
NTT DATA is a $30 billion trusted global innovator of business and technology services. We serve 75% of the Fortune Global 100 and are committed to helping clients innovate, optimize and transform for long term success. As a Global Top Employer, we have diverse experts in more than 50 countries and a robust partner ecosystem of established and start-up companies. Our services include business and technology consulting, data and artificial intelligence, industry solutions, as well as the development, implementation and management of applications, infrastructure and connectivity. We are one of the leading providers of digital and AI infrastructure in the world. NTT DATA is a part of NTT Group, which invests over $3.6 billion each year in R&D to help organizations and society move confidently and sustainably into the digital future. Visit us at us.nttdata.com ( possible, we hire locally to NTT DATA offices or client sites. This ensures we can provide timely and effective support tailored to each client's needs. While many positions offer remote or hybrid work options, these arrangements are subject to change based on client requirements. For employees near an NTT DATA office or client site, in-office attendance may be required for meetings or events, depending on business needs. At NTT DATA, we are committed to staying flexible and meeting the evolving needs of both our clients and employees. NTT DATA recruiters will never ask for payment or banking information and will only use @nttdata.com and @talent.nttdataservices.com email addresses. If you are requested to provide payment or disclose banking information, please submit a contact us form, .
**_NTT DATA endeavors to make_** **_ **_accessible to any and all users. If you would like to contact us regarding the accessibility of our website or need assistance completing the application process, please contact us at_** **_ **_._** **_This contact information is for accommodation requests only and cannot be used to inquire about the status of applications. NTT DATA is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or protected veteran status. For our EEO Policy Statement, please click here ( . If you'd like more information on your EEO rights under the law, please click here ( . For Pay Transparency information, please click here ( ._**
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka Target

Posted 17 days ago

Job Viewed

Tap Again To Close

Job Description

**About Us: **
As a Fortune 50 company with more than 400,000 team members worldwide, Target is an iconic brand and one of America's leading retailers. At Target, we have a timeless purpose and a proven strategy and that hasn't happened by accident. Some of the best minds from diverse backgrounds come together at Target to redefine retail in an inclusive learning environment that values people and delivers world-class outcomes. That winning formula is especially apparent in Bengaluru, where Target in India operates as a fully integrated part of Target's global team and has more than 4,000 team members supporting the company's global strategy and operations. Joining Target means promoting a culture of mutual care and respect and striving to make the most meaningful and positive impact. Becoming a Target team member means joining a community that values diverse backgrounds. We believe your unique perspective is important, and you'll build relationships by being authentic and respectful. At Target, inclusion is part of the core value. We aim to create equitable experiences for all, regardless of their dimensions of difference. As an equal opportunity employer, Target provides diverse opportunities for everyone to grow and win
**Role Overview:**
- Assess client needs and convert business requirements into business intelligence (BI) solutions roadmap relating to complex issues involving long-term or multi-work streams.
- Analyze technical issues and questions identifying data needs and delivery mechanisms
- Implement data structures using best practices in data modeling, ETL/ELT processes, Spark, Scala, SQL, database, and OLAP technologies
- Manage overall development cycle, driving best practices and ensuring development of high quality code for common assets and framework components
- Develop test-driven solutions and provide technical guidance and heavily contribute to a team of high caliber Data Engineers by developing test-driven solutions and BI Applications that can be deployed quickly and in an automated fashion.
- Manage and execute against agile plans and set deadlines based on client, business, and technical requirements
- Drive resolution of technology roadblocks including code, infrastructure, build, deployment, and operations
- Ensure all code adheres to development & security standards
**About you:**
- 4 year degree or equivalent experience
- 1+ years of software development experience preferably in a data engineering/Hadoop development (Hive, Spark etc.)
- Hands on Experience in Object Oriented or functional programming such as Scala / Java / Python
- Knowledge or experience with a variety of database technologies (Postgres, Cassandra, SQL Server)
- Knowledge with design of data integration using API and streaming technologies (Kafka) as well as ETL and other data Integration patterns
- Experience with cloud platforms like Google Cloud, AWS, or Azure. Hands on Experience on BigQuery will be an added advantage
- Good understanding of distributed storage(HDFS, Google Cloud Storage, Amazon S3) and processing(Spark, Google Dataproc, Amazon EMR or Databricks)
- Experience with CI/CD toolchain (Drone, Jenkins, Vela, Kubernetes) a plus
- Familiarity with data warehousing concepts and technologies.
- Maintains technical knowledge within areas of expertise
- Constant learner and team player who enjoys solving tech challenges with global team.
- Hands on experience in building complex data pipelines and flow optimizations
- Be able to understand the data, draw insights and make recommendations and be able to identify any data quality issues upfront
- Experience with test-driven development and software test automation
- Follow best coding practices & engineering guidelines as prescribed
- Strong written and verbal communication skills with the ability to present complex technical information in a clear and concise manner to variety of audiences
This advertiser has chosen not to accept applicants from your region.

Data Engineer

Bangalore, Karnataka Huron Consulting Group

Posted 17 days ago

Job Viewed

Tap Again To Close

Job Description

Huron is redefining what a global consulting organization can be. Advancing new ideas every day to build even stronger clients, individuals and communities. We're helping our clients find new ways to drive growth, enhance business performance and sustain leadership in the markets they serve. And, we're developing strategies and implementing solutions that enable the transformative change they need to own their future.
As a member of the Huron corporate team, you'll help to evolve our business model to stay ahead of market forces, industry trends and client needs. Our accounting, finance, human resources, IT, legal, marketing and facilities management professionals work collaboratively to support Huron's collective strategies and enable real transformation to produce sustainable business results.
Join our team and create your future.
In the Data Engineer role, you will be working with the data engineers, MS Power BI Developers, data Architect, team leads and practice leadership to contribute to the operational effectiveness of the Software Engineering function, team, and strategy. This role is based in Bengaluru, India, and will work with Software Development and Corporate IT, and business/practice resources based in the US and India.
Under limited direction, this individual is responsible for designing and developing solutions that improve how Huron does business. This individual participates in the full software development life cycle including requirements development, analysis, design, implementation, and support. In this role, you will also create technical specifications based on conceptual design and business requirements and will consult with project and business teams to prototype, refine, test, and debug solutions. You will use current programming languages and technologies to build solutions and integrations between systems, develop documentation and procedures for installation, support, and maintenance. All solutions will be built based on Huron's enterprise platforms and technologies.
**Requirements and Preferences**
+ Bachelor's degree in Engineering **(** or related field) with 2+ years of relevant industry experience.
+ Hands-on experience in building data lake, data warehouse, and big data/analytics solutions.
+ Proficiency in ETL development, data modeling (dimensional and normalized), and batch data processing (Hadoop/Spark).
+ Strong programming skills in Python, SQL, and Bash, with the ability to create robust and reusable code.
+ Experience working with AWS (S3, Lambda, Glue, RDS, Redshift, Athena) and Azure (Data Factory, containers) ecosystems.
+ Demonstrated experience with Power BI, including building data models and dashboards, especially using data from Databricks.
**Preferred Certifications:**
+ Microsoft Certified: Power BI Data Analyst Associate (PL-300)
+ AWS Certified Developer - Associate (DVA-C01)
+ AWS Certified Data Analytics - Specialty (DAS-C01)
+ Azure Data Engineer Associate (DP-203)
**Position Level**
Analyst
**Country**
India
At Huron, we're redefining what a consulting organization can be. We go beyond advice to deliver results that last. We inherit our client's challenges as if they were our own. We help them transform for the future. We advocate. We make a difference. And we intelligently, passionately, relentlessly do great work.together.
Are you the kind of person who stands ready to jump in, roll up your sleeves and transform ideas into action? Then come discover Huron.
Whether you have years of experience or come right out of college, we invite you to explore our many opportunities. Find out how you can use your talents and develop your skills to make an impact immediately. Learn about how our culture and values provide you with the kind of environment that invites new ideas and innovation. Come see how we collaborate with each other in a culture of learning, coaching, diversity and inclusion. And hear about our unwavering commitment to make a difference in partnership with our clients, shareholders, communities and colleagues.
Huron Consulting Group offers a competitive compensation and benefits package including medical, dental, and vision coverage to employees and dependents; a 401(k) plan with a generous employer match; an employee stock purchase plan; a generous Paid Time Off policy; and paid parental leave and adoption assistance. Our Wellness Program supports employee total well-being by providing free annual health screenings and coaching, bank at work, and on-site workshops, as well as ongoing programs recognizing major events in the lives of our employees throughout the year. All benefits and programs are subject to applicable eligibility requirements.
Huron is fully committed to providing equal employment opportunity to job applicants and employees in recruitment, hiring, employment, compensation, benefits, promotions, transfers, training, and all other terms and conditions of employment. Huron will not discriminate on the basis of age, race, color, gender, marital status, sexual orientation, gender identity, pregnancy, national origin, religion, veteran status, physical or mental disability, genetic information, creed, citizenship or any other status protected by laws or regulations in the locations where we do business. We endeavor to maintain a drug-free workplace.
This advertiser has chosen not to accept applicants from your region.
 

Nearby Locations

Other Jobs Near Me

Industry

  1. request_quote Accounting
  2. work Administrative
  3. eco Agriculture Forestry
  4. smart_toy AI & Emerging Technologies
  5. school Apprenticeships & Trainee
  6. apartment Architecture
  7. palette Arts & Entertainment
  8. directions_car Automotive
  9. flight_takeoff Aviation
  10. account_balance Banking & Finance
  11. local_florist Beauty & Wellness
  12. restaurant Catering
  13. volunteer_activism Charity & Voluntary
  14. science Chemical Engineering
  15. child_friendly Childcare
  16. foundation Civil Engineering
  17. clean_hands Cleaning & Sanitation
  18. diversity_3 Community & Social Care
  19. construction Construction
  20. brush Creative & Digital
  21. currency_bitcoin Crypto & Blockchain
  22. support_agent Customer Service & Helpdesk
  23. medical_services Dental
  24. medical_services Driving & Transport
  25. medical_services E Commerce & Social Media
  26. school Education & Teaching
  27. electrical_services Electrical Engineering
  28. bolt Energy
  29. local_mall Fmcg
  30. gavel Government & Non Profit
  31. emoji_events Graduate
  32. health_and_safety Healthcare
  33. beach_access Hospitality & Tourism
  34. groups Human Resources
  35. precision_manufacturing Industrial Engineering
  36. security Information Security
  37. handyman Installation & Maintenance
  38. policy Insurance
  39. code IT & Software
  40. gavel Legal
  41. sports_soccer Leisure & Sports
  42. inventory_2 Logistics & Warehousing
  43. supervisor_account Management
  44. supervisor_account Management Consultancy
  45. supervisor_account Manufacturing & Production
  46. campaign Marketing
  47. build Mechanical Engineering
  48. perm_media Media & PR
  49. local_hospital Medical
  50. local_hospital Military & Public Safety
  51. local_hospital Mining
  52. medical_services Nursing
  53. local_gas_station Oil & Gas
  54. biotech Pharmaceutical
  55. checklist_rtl Project Management
  56. shopping_bag Purchasing
  57. home_work Real Estate
  58. person_search Recruitment Consultancy
  59. store Retail
  60. point_of_sale Sales
  61. science Scientific Research & Development
  62. wifi Telecoms
  63. psychology Therapy
  64. pets Veterinary
View All Data Engineer Jobs View All Jobs in Bengaluru