33,429 Big Data jobs in India

About the Company

Our client is a trusted global innovator of IT and business services, present in 50+ countries. They specialize in digital & IT modernization, consulting, managed services, and industry-specific solutions. With a commitment to long-term success, they empower clients and society to move confidently into the digital future.

Title: Senior Data Engineer

Location: Hyderabad,Chennai

Experience: 6 - 12 years

Employment Type: Permanent

Notice Period: Immediate Joiners

Key Responsibilities:

• Big Data Processing:
• Design, develop, and maintain scalable ETL/ELT pipelines using PySpark, Scala, or Java.
• Implement data transformation workflows to ensure clean and reliable datasets.
• Cloud Integration:
• Work extensively with AWS services such as EMR, S3, Glue, Lambda, Redshift, DynamoDB, and RDS to build and manage data solutions.
• Data Modeling & Optimization:
• Develop and optimize data models in Hadoop and Spark environments.
• Ensure performance tuning and cost optimization of cloud-based data platforms.
• Data Ingestion & Integration:
• Ingest data from various sources including APIs, RDBMS, and NoSQL databases.
• Implement data pipelines for batch and real-time processing.
• Collaboration & Documentation:
• Collaborate with cross-functional teams to understand data requirements and deliver solutions.
• Document data models, transformation logic, and technical specifications.
• Automation & Monitoring:
• Automate data workflows and ensure data pipelines are reliable, scalable, and error-free.
• Implement data monitoring, logging, and alerting to ensure pipeline reliability.

Required Skills & Qualifications:

• Technical Expertise:
• 6–12 years of experience in Big Data engineering with strong hands-on experience in AWS, PySpark, Scala, or Java.
• Proficiency in SQL and experience with Hadoop ecosystem components like Hive, HDFS, and Kafka.
• Cloud Services:
• Extensive experience with AWS services such as EMR, S3, Glue, Lambda, Redshift, DynamoDB, and RDS.
• Data Engineering Tools:
• Familiarity with data orchestration tools like Apache Airflow or Oozie.
• Experience with containerization technologies like Docker and Kubernetes is a plus.
• Soft Skills:
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to work independently and in a team-oriented environment.

Preferred Skills (Good to Have):

• Certifications: AWS Certified Solutions Architect, AWS Certified Big Data – Specialty, or Cloudera Certified Associate (CCA).
• Experience with CI/CD pipelines using tools like Jenkins or GitLab.
• Knowledge of data warehousing concepts and experience with platforms like Redshift or Snowflake.
• Familiarity with streaming technologies such as Apache Kafka or Kinesis.

Job Title :- Data Engineer - Pyspark

Experience: 5 to 8 Years

Location: Pune/Hyderabad

Job Description

Required Skills:

5+ years of experience in Big data and pyspark

Must-Have

Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL

Good-to-Have

Good Spark, Pyspark,Big Data experience

Spark UI/Optimization/debugging techniques

Good python scripting skills

Intermediate SQL exposure – Subquery, Joins, CTE’s

Database technologies

AWS EMR, S3, IAM, Lambda, SNS, SQS

Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL

Experience working on Data Engineering projects

Good Understanding of SQL

Good understanding of Unix and HDFS commands

Communication

Experience working on Data Analytics and Pyspark

Thanks & Regards ,

Manasi Prava Natha

Who we are

Alef Education began with a bold idea: that every learner deserves a personalised and meaningful education experience. What started in 2016 as a small pilot programme in Abu Dhabi has evolved into one of the world’s most dynamic EdTech companies—reshaping how millions of students engage with learning across the globe.

Today, Alef is proudly headquartered in the UAE, working hand-in-hand with ministries of education, schools, and teachers to bring smart, data-powered platforms into classrooms in over 14,000 schools.

Supporting over 1.1 million students and 50,000 teachers across the UAE, Indonesia & Morocco our AI-driven platforms generate 16+ million data points every day, helping drive smarter learning decisions. Whether it’s improving national exam results, boosting classroom engagement, or supporting educators with world-class tools, Alef is committed to impact at scale.

In 2024, Alef made history as the first EdTech company to list on the Abu Dhabi Securities Exchange (ADX), cementing our role as a regional innovator with global reach.

About The Role

As an ALEF Big Data Engineer you will have a strong understanding of big data technologies with an exceptional ability to code. You will provide technical leadership, working closely with the wider team to ensure high quality code is delivered in line with the project goals and delivery cycles. You will work closely with other teams to deliver rapid prototypes as well as production code for which you will ensure high accessibility standards are upheld. We expect familiarity with modern frameworks and languages, as well as working practices such as Clean Code, TDD, BDD, continuous integration, continuous delivery, and DevOps.

Key Responsibilities

Defining and developing services and solutions

• Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.
• Define, design, and run robust data pipelines/batch jobs in a production environment.
• Architecting highly scalable, highly concurrent, and low latency systems

Maintain, support, and enhance current systems.

• Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.
• Contribute feedback to improve the quality, readability, and testability of the code base within your team.
• Mentor and train other developers in a non-line management capacity.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.

Collaborating with Internal and external stakeholders

• Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.
• Work with third-party and other internal providers to support a variety of integrations.
• As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.
• Proactively advise on best practices.

To Be The Right Fit, You'll Need

• Degree in Computer Science, Software Engineering or related preferred
• Minimum of 5 years experience in a Big Data
• Follow Clean Code/Solid principles
• Adhere and use TDD/BDD.
• Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.
• Highly Proficient in either Functional Java or Scala, Python
• Knowledge of Azure Big Data/Analytics services – ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks
• Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs
• Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs
• In-depth knowledge of Hadoop technology ecosystem – HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet
• Experience debugging a complex multi-server service.
• In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)
• Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.
• Knowledge of relational and non-relational database systems
• Understanding continuous integration and delivery.
• Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).
• IDE Intellij or Eclipse.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.
• An ability to communicate technical concepts to a non-technical audience.
• Working knowledge of unix-like operating systems such as Linux and/or Mac OS X.
• Knowledge of the git version control system.
• Ability to quickly research and learn new programming tools and techniques.

Position Overview:

We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent.

You will work on live client projects, provide technical support, contribute to solution delivery, and serve as a go-to technical mentor for less experienced team members.

Key Responsibilities:

• Perform hands-on Big Data development work, including coding, testing, troubleshooting, and deploying solutions.
• Support ongoing client projects, addressing technical challenges and ensuring smooth delivery.
• Collaborate with junior engineers to guide them on coding standards, best practices, debugging, and project execution.
• Review code and provide feedback to junior engineers to maintain high quality and scalable solutions.
• Assist in designing and implementing solutions using Hadoop, Spark, Hive, HDFS, and Kafka.
• Lead by example in object-oriented development, particularly using Scala and Java.
• Translate complex requirements into clear, actionable technical tasks for the team.
• Contribute to the development of ETL processes for integrating data from various sources.
• Document technical approaches, best practices, and workflows for knowledge sharing within the team.

Required Skills and Qualifications:

• 8+ years of professional experience in Big Data development and engineering.
• Strong hands-on expertise with Hadoop, Hive, HDFS, Apache Spark, and Kafka.
• Solid object-oriented development experience with Scala and Java.
• Strong SQL skills with experience working with large data sets.
• Practical experience designing, installing, configuring, and supporting Big Data clusters.
• Deep understanding of ETL processes and data integration strategies.
• Proven experience mentoring or supporting junior engineers in a team setting.
• Strong problem-solving, troubleshooting, and analytical skills.
• Excellent communication and interpersonal skills.

Preferred Qualifications:

• Professional certifications in Big Data technologies (Cloudera, Databricks, AWS Big Data Specialty, etc.).
• Experience with cloud Big Data platforms (AWS EMR, Azure HDInsight, or GCP Dataproc).
• Exposure to Agile or DevOps practices in Big Data project environments.

What We Offer:

Opportunity to work on challenging, high-impact Big Data projects.

Leadership role in shaping and mentoring the next generation of engineers.

Supportive and collaborative team culture.

Flexible working environment

Competitive compensation and professional growth opportunities.

Coforge Ltd is Hiring for Big Data Engineer – AWS, Spark & Scala.

Must Have Skills:- AWS, Spark & Scala.

Experience Required: 3 to 6 Years

Job Locations: Pune, Hyderabad, Greater Noida Only.

Send your CV to:

For queries, contact via WhatsApp:

Key Responsibilities:-

• Design, develop, and optimize Big Data architectures leveraging AWS services for large-scale, complex data processing.

• Build and maintain data pipelines using Spark (Scala) for both structured and unstructured datasets.

• Architect and operationalize data engineering and analytics platforms (AWS preferred; Hortonworks, Cloudera, or MapR experience a plus).

• Implement and manage AWS services including EMR, Glue, Kinesis, DynamoDB, Athena, CloudFormation, API Gateway, and S3.

• Work on real-time streaming solutions using Kafka and AWS Kinesis.

• Support ML model operationalization on AWS (deployment, scheduling, and monitoring).

• Analyze source system data and data flows to ensure high-quality, reliable data delivery for business needs.

• Write highly efficient SQL queries and support data warehouse initiatives using Apache NiFi, Airflow, and Kylo.

• Collaborate with cross-functional teams to provide technical leadership, mentor team members, and strengthen the data engineering capability.

• Troubleshoot and resolve complex technical issues, ensuring scalability, performance, and security of data solutions.

Mandatory Skills & Qualifications:-

• Solid hands-on experience in Big Data Technologies (AWS, Scala, Hadoop, and Spark Mandatory)

• Proven expertise in Spark with Scala

• Hands-on experience with: AWS services (EMR, Glue, Lambda, S3, CloudFormation, API Gateway, Athena, Lake Formation)

Who we are

Alef Education began with a bold idea: that every learner deserves a personalised and meaningful education experience. What started in 2016 as a small pilot programme in Abu Dhabi has evolved into one of the world's most dynamic EdTech companies—reshaping how millions of students engage with learning across the globe.

Today, Alef is proudly headquartered in the UAE, working hand-in-hand with ministries of education, schools, and teachers to bring smart, data-powered platforms into classrooms in over 14,000 schools.

Supporting over 1.1 million students and 50,000 teachers across the UAE, Indonesia & Morocco our AI-driven platforms generate 16+ million data points every day, helping drive smarter learning decisions. Whether it's improving national exam results, boosting classroom engagement, or supporting educators with world-class tools, Alef is committed to impact at scale.

In 2024, Alef made history as the first EdTech company to list on the Abu Dhabi Securities Exchange (ADX), cementing our role as a regional innovator with global reach.

About The Role

As an ALEF Big Data Engineer you will have a strong understanding of big data technologies with an exceptional ability to code. You will provide technical leadership, working closely with the wider team to ensure high quality code is delivered in line with the project goals and delivery cycles. You will work closely with other teams to deliver rapid prototypes as well as production code for which you will ensure high accessibility standards are upheld. We expect familiarity with modern frameworks and languages, as well as working practices such as Clean Code, TDD, BDD, continuous integration, continuous delivery, and DevOps.

Key Responsibilities

Defining and developing services and solutions

• Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.
• Define, design, and run robust data pipelines/batch jobs in a production environment.
• Architecting highly scalable, highly concurrent, and low latency systems

Maintain, support, and enhance current systems.

• Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.
• Contribute feedback to improve the quality, readability, and testability of the code base within your team.
• Mentor and train other developers in a non-line management capacity.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.

Collaborating with Internal and external stakeholders

• Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.
• Work with third-party and other internal providers to support a variety of integrations.
• As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.
• Proactively advise on best practices.

To Be The Right Fit, You'll Need

• Degree in Computer Science, Software Engineering or related preferred
• Minimum of 5 years experience in a Big Data
• Follow Clean Code/Solid principles
• Adhere and use TDD/BDD.
• Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.
• Highly Proficient in either Functional Java or Scala, Python
• Knowledge of Azure Big Data/Analytics services – ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks
• Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs
• Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs
• In-depth knowledge of Hadoop technology ecosystem – HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet
• Experience debugging a complex multi-server service.
• In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)
• Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.
• Knowledge of relational and non-relational database systems
• Understanding continuous integration and delivery.
• Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).
• IDE Intellij or Eclipse.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.
• An ability to communicate technical concepts to a non-technical audience.
• Working knowledge of unix-like operating systems such as Linux and/or Mac OS X.
• Knowledge of the git version control system.
• Ability to quickly research and learn new programming tools and techniques.

Who we are

Alef Education began with a bold idea: that every learner deserves a personalised and meaningful education experience. What started in 2016 as a small pilot programme in Abu Dhabi has evolved into one of the world's most dynamic EdTech companies—reshaping how millions of students engage with learning across the globe.

Today, Alef is proudly headquartered in the UAE, working hand-in-hand with ministries of education, schools, and teachers to bring smart, data-powered platforms into classrooms in over 14,000 schools.

Supporting over 1.1 million students and 50,000 teachers across the UAE, Indonesia & Morocco our AI-driven platforms generate 16+ million data points every day, helping drive smarter learning decisions. Whether it's improving national exam results, boosting classroom engagement, or supporting educators with world-class tools, Alef is committed to impact at scale.

In 2024, Alef made history as the first EdTech company to list on the Abu Dhabi Securities Exchange (ADX), cementing our role as a regional innovator with global reach.

About The Role

As an ALEF Big Data Engineer you will have a strong understanding of big data technologies with an exceptional ability to code. You will provide technical leadership, working closely with the wider team to ensure high quality code is delivered in line with the project goals and delivery cycles. You will work closely with other teams to deliver rapid prototypes as well as production code for which you will ensure high accessibility standards are upheld. We expect familiarity with modern frameworks and languages, as well as working practices such as Clean Code, TDD, BDD, continuous integration, continuous delivery, and DevOps.

Key Responsibilities

Defining and developing services and solutions

• Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.
• Define, design, and run robust data pipelines/batch jobs in a production environment.
• Architecting highly scalable, highly concurrent, and low latency systems

Maintain, support, and enhance current systems.

• Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.
• Contribute feedback to improve the quality, readability, and testability of the code base within your team.
• Mentor and train other developers in a non-line management capacity.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.

Collaborating with Internal and external stakeholders

• Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.
• Work with third-party and other internal providers to support a variety of integrations.
• As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.
• Proactively advise on best practices.

To Be The Right Fit, You'll Need

• Degree in Computer Science, Software Engineering or related preferred
• Minimum of 5 years experience in a Big Data
• Follow Clean Code/Solid principles
• Adhere and use TDD/BDD.
• Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.
• Highly Proficient in either Functional Java or Scala, Python
• Knowledge of Azure Big Data/Analytics services – ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks
• Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs
• Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs
• In-depth knowledge of Hadoop technology ecosystem – HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet
• Experience debugging a complex multi-server service.
• In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)
• Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.
• Knowledge of relational and non-relational database systems
• Understanding continuous integration and delivery.
• Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).
• IDE Intellij or Eclipse.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.
• An ability to communicate technical concepts to a non-technical audience.
• Working knowledge of unix-like operating systems such as Linux and/or Mac OS X.
• Knowledge of the git version control system.
• Ability to quickly research and learn new programming tools and techniques.

Who we are

Alef Education began with a bold idea: that every learner deserves a personalised and meaningful education experience. What started in 2016 as a small pilot programme in Abu Dhabi has evolved into one of the world's most dynamic EdTech companies—reshaping how millions of students engage with learning across the globe.

Today, Alef is proudly headquartered in the UAE, working hand-in-hand with ministries of education, schools, and teachers to bring smart, data-powered platforms into classrooms in over 14,000 schools.

Supporting over 1.1 million students and 50,000 teachers across the UAE, Indonesia & Morocco our AI-driven platforms generate 16+ million data points every day, helping drive smarter learning decisions. Whether it's improving national exam results, boosting classroom engagement, or supporting educators with world-class tools, Alef is committed to impact at scale.

In 2024, Alef made history as the first EdTech company to list on the Abu Dhabi Securities Exchange (ADX), cementing our role as a regional innovator with global reach.

About The Role

As an ALEF Big Data Engineer you will have a strong understanding of big data technologies with an exceptional ability to code. You will provide technical leadership, working closely with the wider team to ensure high quality code is delivered in line with the project goals and delivery cycles. You will work closely with other teams to deliver rapid prototypes as well as production code for which you will ensure high accessibility standards are upheld. We expect familiarity with modern frameworks and languages, as well as working practices such as Clean Code, TDD, BDD, continuous integration, continuous delivery, and DevOps.

Key Responsibilities

Defining and developing services and solutions

• Define, design, and develop services and solutions around large data ingestion, storage, and management such as withRDBMS, No SQL DBs, Log Files, Events.
• Define, design, and run robust data pipelines/batch jobs in a production environment.
• Architecting highly scalable, highly concurrent, and low latency systems

Maintain, support, and enhance current systems.

• Contribute to paying down technical debt and use development approaches that minimize the growth of new technical debt.
• Contribute feedback to improve the quality, readability, and testability of the code base within your team.
• Mentor and train other developers in a non-line management capacity.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.

Collaborating with Internal and external stakeholders

• Participating in sprint planning to work with developers and project teams to ensure projects are deployable and monitorable from the outside.
• Work with third-party and other internal providers to support a variety of integrations.
• As part of the team, you may be expected to participate in some of the 2nd line in-house support and Out-of-Hours support rotas.
• Proactively advise on best practices.

To Be The Right Fit, You'll Need

• Degree in Computer Science, Software Engineering or related preferred
• Minimum of 5 years experience in a Big Data
• Follow Clean Code/Solid principles
• Adhere and use TDD/BDD.
• Outstanding ability to develop efficient, readable, highly optimized/maintainable and clear code.
• Highly Proficient in either Functional Java or Scala, Python
• Knowledge of Azure Big Data/Analytics services – ADLS (Azure Data Lake Storage), HDInsight, Azure Data Factory, Azure Synapse Analytics, Azure Fabric, Azure Event Hubs, Azure Stream Analytics, Azure Databricks
• Experience of Storing Data in systems such as Hadoop HDFS, ADLS, Event Hubs
• Experience of designing, setting up and running big data tech stacks such as Hadoop, Azure Databricks, Spark and distributed datastores such as Cassandra, DocumentDBs, MongoDB, Event Hubs
• In-depth knowledge of Hadoop technology ecosystem – HDFS, Spark, Hive, HBase, Event Hubs, Flume, Sqoop, Oozie, SPARK, Avro, Parquet
• Experience debugging a complex multi-server service.
• In depth knowledge and experience in IaaS/PaaS solutions (eg AWS Infrastructure hosting and managed services)
• Familiarity with network protocols - TCP/IP, HTTP, SSL, etc.
• Knowledge of relational and non-relational database systems
• Understanding continuous integration and delivery.
• Mocking (any of the following Mockito, ScalaTest Spock, Jasmine, Mocha).
• IDE Intellij or Eclipse.
• Build tools (One of SBT, Gradle, Maven).
• Ensure all software built is robust and scalable.
• An ability to communicate technical concepts to a non-technical audience.
• Working knowledge of unix-like operating systems such as Linux and/or Mac OS X.
• Knowledge of the git version control system.
• Ability to quickly research and learn new programming tools and techniques.