22,598 Big Data Engineer jobs in India

Big Data Engineer (PySpark)

Location: Pune/Nagpur (WFO)

Experience: 8 - 12 Years

Employment Type: Full-time

Job Overview

We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high performance and reliability.

Key Responsibilities

• Design, develop, and maintain data pipelines using PySpark and related Big Data technologies.
• Work with HDFS, Hive, Sqoop, and other tools in the Hadoop ecosystem.
• Write efficient HiveQL and SQL queries to handle large-scale datasets.
• Perform performance tuning and optimization of distributed data systems.
• Collaborate with cross-functional teams in an Agile environment to deliver high-quality solutions.
• Manage and schedule workflows using Apache Airflow or Oozie.
• Troubleshoot and resolve issues in data pipelines to ensure reliability and accuracy.

Required Skills

• Proven experience in Big Data Engineering with a focus on PySpark.
• Strong knowledge of HDFS, Hive, Sqoop, and related tools.
• Proficiency in SQL/HiveQL for large datasets.
• Expertise in performance tuning and optimization of distributed systems.
• Familiarity with Agile methodology and collaborative team practices.
• Experience with workflow orchestration tools (Airflow/Oozie).
• Strong problem-solving, analytical, and communication skills.

Good to Have

• Knowledge of data modeling and data warehousing concepts.
• Exposure to DevOps practices and CI/CD pipelines for data engineering.
• Experience with other Big Data frameworks such as Spark Streaming or Kafka.

Big Data Engineer (PySpark)

Location: Pune/Nagpur (WFO)

Experience: 8 - 12 Years

Employment Type: Full-time

Job Overview

We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high performance and reliability.

Key Responsibilities

• Design, develop, and maintain data pipelines using PySpark and related Big Data technologies.
• Work with HDFS, Hive, Sqoop , and other tools in the Hadoop ecosystem.
• Write efficient HiveQL and SQL queries to handle large-scale datasets.
• Perform performance tuning and optimization of distributed data systems.
• Collaborate with cross-functional teams in an Agile environment to deliver high-quality solutions.
• Manage and schedule workflows using Apache Airflow or Oozie .
• Troubleshoot and resolve issues in data pipelines to ensure reliability and accuracy.

Required Skills

• Proven experience in Big Data Engineering with a focus on PySpark.
• Strong knowledge of HDFS, Hive, Sqoop , and related tools.
• Proficiency in SQL/HiveQL for large datasets.
• Expertise in performance tuning and optimization of distributed systems.
• Familiarity with Agile methodology and collaborative team practices.
• Experience with workflow orchestration tools ( Airflow/Oozie ).
• Strong problem-solving, analytical, and communication skills.

Good to Have

• Knowledge of data modeling and data warehousing concepts.
• Exposure to DevOps practices and CI/CD pipelines for data engineering.
• Experience with other Big Data frameworks such as Spark Streaming or Kafka .

We are looking for passionate   B.Tech freshers   with strong programming skills in   Java   who are eager to start their career in   Big Data technologies . The role offers exciting opportunities to work on real-time big data projects, data pipelines, and cloud-based data solutions.

• Assist in designing, developing, and maintaining   big data solutions .
  

• Write efficient code in   Java   and integrate with big data frameworks.
  

• Support in building   data ingestion, transformation, and processing pipelines .
  

• Work with   distributed systems   and learn technologies like   Hadoop, Spark, Kafka, Hive, HBase .
  

• Collaborate with senior engineers on data-related problem-solving and performance optimization.
  

• Participate in   debugging, testing, and documentation   of big data workflows.
  

• Strong knowledge of   Core Java & OOPs concepts .
  

• Good understanding of   SQL and database concepts .
  

• Familiarity with   data structures & algorithms .
  

• Basic knowledge of   Big Data frameworks   (Hadoop/Spark/Kafka) is an added advantage.
  

• Problem-solving skills and eagerness to learn new technologies.
  

• Education:   B.Tech (CSE/IT or related fields).
  

• Batch:   (specific, e.g., 2024/2025 pass outs).
  

• Experience:   Fresher (0–1 year)
  

• Training and mentoring in   cutting-edge Big Data tools & technologies .
  

• Exposure to   live projects   from day one.
  

• A fast-paced, learning-oriented work culture.
  

Basic Qualifications:

Bachelors degree or higher in Computer Science, or equivalent degree and 3-10 years related working experience.

In-depth experience with a big data cloud platform, preferably Azure.

Strong grasp of programming languages (Python, PySpark, or equivalent) and a willingness to learn new ones.

Experience writing database-heavy services or APIs.

Experience building and optimizing data pipelines, architectures, and data sets.

Working knowledge of queueing, stream processing, and highly scalable data stores

Experience working with and supporting cross-functional teams.

Strong understanding of structuring code for testability.

Preferred Qualifications:

Professional experience implementing and maintaining MLOps pipelines in MLflow or AzureML.

Professional experience implementing data ingestion pipelines using Data Factory.

Professional experience with Databricks and coding with notebooks.

Professional experience processing and manipulating data using SQL and Python code.

Professional experience with user training, customer support, and coordination with cross-functional teams.

We are looking for passionate   B.Tech freshers   with strong programming skills in   Java   who are eager to start their career in   Big Data technologies . The role offers exciting opportunities to work on real-time big data projects, data pipelines, and cloud-based data solutions.

• Assist in designing, developing, and maintaining   big data solutions .
  

• Write efficient code in   Java   and integrate with big data frameworks.
  

• Support in building   data ingestion, transformation, and processing pipelines .
  

• Work with   distributed systems   and learn technologies like   Hadoop, Spark, Kafka, Hive, HBase .
  

• Collaborate with senior engineers on data-related problem-solving and performance optimization.
  

• Participate in   debugging, testing, and documentation   of big data workflows.
  

• Strong knowledge of   Core Java & OOPs concepts .
  

• Good understanding of   SQL and database concepts .
  

• Familiarity with   data structures & algorithms .
  

• Basic knowledge of   Big Data frameworks   (Hadoop/Spark/Kafka) is an added advantage.
  

• Problem-solving skills and eagerness to learn new technologies.
  

• Education:   B.Tech (CSE/IT or related fields).
  

• Batch:   (specific, e.g., 2024/2025 pass outs).
  

• Experience:   Fresher (0–1 year)
  

• Training and mentoring in   cutting-edge Big Data tools & technologies .
  

• Exposure to   live projects   from day one.
  

• A fast-paced, learning-oriented work culture.
  

• Design, develop, and implement robust Big Data solutions using technologies such as Hadoop, Spark, and NoSQL databases.
• Build and maintain scalable data pipelines for effective data ingestion, transformation, and analysis.
• Collaborate with data scientists, analysts, and cross-functional teams to understand business requirements and translate them into technical solutions.
• Ensure data quality and integrity through effective validation, monitoring, and troubleshooting techniques.
• Optimize data processing workflows for maximum performance and efficiency.
• Stay up-to-date with evolving Big Data technologies and methodologies to enhance existing systems.
• Implement best practices for data governance, security, and compliance.
• Document technical designs, processes, and procedures to support knowledge sharing across teams.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• 4+ years of experience as a Big Data Engineer or in a similar role.
• Strong proficiency in Big Data technologies (Hadoop, Spark, Hive, Pig) and frameworks.
• Extensive experience with programming languages such as Python, Scala, or Java.
• Knowledge of data modeling and data warehousing concepts.
• Familiarity with NoSQL databases like Cassandra or MongoDB.
• Proficient in SQL for data querying and analysis.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to work independently and effectively in a fast-paced environment.

Benefits

Competitive salary and benefits package.

Opportunity to work on cutting-edge technologies and solve complex challenges.

Dynamic and collaborative work environment with opportunities for growth and career advancement.

Regular training and professional development opportunities.

Description

Benefit from our flexible work options and career growth opportunities

As an Azure Senior Data Engineer specialized in Microsoft Azure tools, you will support the implementation of projects focused on collecting, aggregating, storing, reconciling, and making data accessible from disparate sources to enable analysis and decision making.

This role will also play a critical part in the data supply chain, by ensuring stakeholders can access and manipulate data for routine and ad hoc analysis. Additionally, you will support the full lifecycle of data from ingesting through analytics to action.

What will you do?

- Translate business requirements to technical solutions
- Support the planning and implementation of technical solutions
- Strong knowledge of data warehouse concepts and T-SQL relational/non-relational databases for data access and Advanced Analytics
- Knowledge of Power BI to enhance better data visualizations to the clients.

Some of the best things about working at Avanade:
- Opportunity to work for Microsoft’s Global Alliance Partner of the Year (14 years in a row), with exceptional development and training (minimum 8 hours per year for training and paid certifications)
- Real-time access to technical and skilled resources globally
- Dedicated career advisor to encourage your growth
- Engaged and helpful coworkers genuinely interested in you.