2749 Data Engineer jobs in Bengaluru

Profile: Big Data Engineer (System Design)
Experience:
5+ years

Location:
Bangalore

Work Mode:
Hybrid

About The Role
We're looking for an experienced
Big Data Engineer
with system design expertise to architect and build scalable data pipelines and optimize big data solutions.

Key Responsibilities

• Design, develop, and maintain data pipelines and ETL processes using Python, Hive, and Spark
• Architect scalable big data solutions with strong system design principles
• Build and optimize workflows using Apache Airflow
• Implement data modeling, integration, and warehousing solutions
• Collaborate with cross-functional teams to deliver data solutions

Must-Have Skills

• 5+ years as a Data Engineer with Python, Hive, and Spark
• Strong hands-on experience with Java
• Advanced SQL and Hadoop experience
• Expertise in Apache Airflow
• Strong understanding of data modeling, integration, and warehousing
• Experience with relational databases (PostgreSQL, MySQL)
• System design knowledge
• Excellent problem-solving and communication skills

Good to Have

• Docker and containerization experience
• Knowledge of Apache Beam, Apache Flink, or similar frameworks
• Cloud platform experience.

Skills:- Apache Hive, Apache Spark, Python, SQL, Hadoop and Systems design

JOB RESPONSIBILITY

· Experience with Big Data technologies will be a plus (Hadoop, Spark, Kafka, HBase, etc)

· Write SQL queries to validate the dashboard output

· Working experience with database environment - understanding relational database structure and hands-on SQL knowledge to extract/manipulate data for variance testing.

· Performing code reviews and pair programming

· Supporting and enhancing current applications

· Design, develop, test, and implement the application investigate and resolve complex issues while supporting existing applications.

QUALIFICATION

· B.Tech /B.E /MCA

EXPERIENCE

· 6+ year's experience in AWS Services: RDS, AWS Lambda, AWS Glue, Apache Spark, Kafka, Spark streaming, spark, Scala , Hive and AWS etc.

· 6+ year's experience SQL and NoSQL databases like MySQL, Postgres, Elasticsearch

· 6+ year's experience with Spark programming paradigms (batch and stream-processing)

· 6+ year's experience in Java, Scala. Familiarity with a scripting language like Python as well as Unix/Linux shells

· 6+ year's experience with Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.

SKILLS AND COMPETENCIES

· Profound understanding of Big Data core concepts and technologies – Apache Spark, Kafka, Spark streaming, spark, Scala , Hive and AWS etc.

· Solid experience and understanding of various core AWS services such as IAM, Cloud Formation, EC2, S3, EMR, Glue, Lambda, Athena, and Redshift. Data Engineer (Bigdata, Kafka, Spark streaming, spark, Scala and AWS).

· Experience in system analysis, design, development, and implementation of data ingestion pipeline in AWS.

· Programming experience with Python/Scala, Shell scripting.

· Experience with DevOps and Continuous Integration/Delivery (CI/CD) concepts and tools such as Bitbucket and Bamboo.

· Good understanding of business and operational processes.

· Capable of Problem / issue resolution, capable of thinking out of the box.

Job Type: Full-time

Benefits:

• Provident Fund
• Work from home

Work Location: In person

Job Title :- Data Engineer - Pyspark

Experience: 5 to 8 Years

Location: Pune/Hyderabad

Job Description

Required Skills:

5+ years of experience in Big data and pyspark

Must-Have

Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL

Good-to-Have

Good Spark, Pyspark,Big Data experience

Spark UI/Optimization/debugging techniques

Good python scripting skills

Intermediate SQL exposure – Subquery, Joins, CTE’s

Database technologies

AWS EMR, S3, IAM, Lambda, SNS, SQS

Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL

Experience working on Data Engineering projects

Good Understanding of SQL

Good understanding of Unix and HDFS commands

Communication

Experience working on Data Analytics and Pyspark

Thanks & Regards ,

Manasi Prava Natha

Hiring: Big Data Engineer (PySpark)

Client: (Details will be shared upon shortlisting)

Location: Bangalore (Onsite)

Experience: 6-8 years (Relevant)

Joining: Immediate to 15 days

Key Skills:

Strong hands-on experience in PySpark

Expertise in Big Data technologies

Solid understanding of SQL, Data Pipelines, and ETL

Interested candidates can share their resume with the following details to

Current CTC

Expected CTC

Location

We are seeking skilled Data Engineers with proven expertise in Apache Spark, Python, and SQL whom are Databricks Certified

Work from home

Job Title: Big Data Engineer

Location: Bengaluru

Experience Required: 5 to 8 years

Job Description:

We are looking for a skilled Big Data Engineer with strong hands-on experience in PySpark and SQL. This is a hybrid role, requiring the candidate to work both remotely and from the office as needed. The ideal candidate should be comfortable working on large-scale data pipelines and big data processing frameworks.

Key Responsibilities:

• Develop, optimize, and maintain scalable data pipelines using PySpark.
• Write efficient and complex SQL queries for data transformation and extraction.
• Work closely with data architects, analysts, and other stakeholders to deliver high-quality data solutions.
• Handle structured and unstructured data from various sources.
• Perform performance tuning and troubleshooting of big data jobs.

Required Skills & Experience:

• Hands-on experience with PySpark for big data processing.
• Strong expertise in SQL for data manipulation and analysis.
• Good understanding of distributed data systems and big data architecture (e.g., Hadoop, Spark, Hive, etc.).
• Ability to work independently in a hybrid setup.
• Comfortable taking ownership as a Big Data Engineer.

Preferred Qualifications:

• Experience with cloud platforms (AWS, Azure, or GCP) is a plus.
• Familiarity with version control systems (e.g., Git).
• Good communication and collaboration skills.

Job Type: Full-time

Pay: ₹120,000.00 per month

Experience:

• Work: 5 years (Required)

Work Location: In person

Key Skills:

Strong hands-on experience in PySpark

Expertise in Big Data technologies

Solid understanding of SQL, Data Pipelines, and ETL

Interested candidates can share their resume with the following details to

Current CTC

Expected CTC

Location

We're Hiring: Big Data Engineer – Full-Time

Looking for a skilled
Big Data Engineer
to join our team in
Bangalore

Role
: Big Data Engineer

Grade
: G4 |
TSR

Location
: Bangalore (Onsite/Hybrid – specify if needed)

Budget
: ₹13,39,200

Experience
: 4–6 years

Skills
:

• Azure Databricks (ADB)
• PySpark
• SQL
• Spark (Scala/Python)
• Terraform
• Airflow & Kubernetes
• Data Warehousing & ETL
• CI/CD (Git, Jenkins)

Job Overview
:

We're looking for professionals experienced in building Big Data applications using Azure Databricks and PySpark. Ideal candidates will have strong SQL skills, a good grasp of infrastructure tools (like Terraform), and experience with orchestration tools like Airflow.

Education
: Bachelor's in Computer Science or equivalent.

Technix India Solutions Private Limited, founded in 2004, delivering a wide spectrum of IT Consulting and Outsourcing services ranging from Product Development, Migration, Porting, Testing, Custom Software Development, and Distribution of Software for Desktop, Web, and Mobiles. We work on latest technologies such as React, JavaScript, Spring boot, Big Data, GO, PHP, AI and NLP, AWS Cloud and much more to deliver the best to our clients. We work for various Government Organizations such as Forest Department, Pollution Control Board, Cooperative Societies as well as Public Sector setups like Banks & Insurance. We have our presence in medium and small organizations across domains from Manufacturing Setup to Retail chains. We leverage Software, Web, and Mobile technology to provide our customers cost-effective solutions to maximize their profits. As a Big data Developer, You shall be a part of Development Team and working in SRE principles for a host of Client Requirement. You shall have access to latest tools and technology to assist you in delivering the desired results. You should be flexible to work in dynamic environment with ready to work in shifts. The hunger to learn and deliver can take you to new highs of Profession. Join us to build your future along

• Develop and maintain scalable and high-performance big data solutions
• Design, develop and implement ETL processes
• Perform data analysis and data profiling to ensure data quality and integrity
• Collaborate with cross-functional teams including data scientists, business analysts, and product
• Managers to identify and solve complex business problems
• Develop and maintain documentation, including technical specifications and user guides
• Conduct code reviews and mentor junior team members

• 4+ years of relevant experience in big data engineering
• Strong proficiency in SQL, Hive, Spark and basic Unix shell scripting
• Solid understanding of ETL processes
• Experience working with large data sets
• Familiarity with data modeling and data warehousing concepts
• Excellent problem-solving skills and attention to detail
• Strong communication skills and ability to work in a team environment

• Experience with Pig, Java, Go, NiFi and PySpark
• Worked on AMEX projects or experience in FSS domain
• Knowledge of distributed computing, Hadoop ecosystem, and NoSQL databases

• If you are passionate about big data and are looking for a challenging and rewarding opportunity, we encourage you to apply for this position.