Senior Backend Engineer (Spark, Scala, Python, AWS/GCP)
About the Role:
We are seeking a skilled and passionate Back-End Engineer with a strong background in data engineering to join our growing team. In this role, you will be responsible for designing, developing, and maintaining scalable and reliable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and other engineers to deliver impactful data-driven solutions. This position is available in both the United States and India.
Responsibilities:
* Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
* Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL) processes.
* Work with DataFrames to manipulate and analyze large datasets.
* Implement data storage and processing solutions using cloud technologies (preferably AWS or GCP).
* Build and maintain real-time data streaming pipelines using MSK/Kafka.
* Utilize S3 for data storage and retrieval.
* Work with data lake technologies like Iceberg.
* Ensure data quality, integrity, and security.
* Collaborate with data scientists and other engineers to understand data requirements and deliver solutions.
* Participate in code reviews and contribute to improving our development processes.
* Troubleshoot and resolve issues in data pipelines and back-end systems.
* Bachelor’s degree in computer science or a related field (or equivalent experience).
* 5+ years of experience in back-end development with a focus on data engineering.
Required Skills: * Strong proficiency in Spark, Python, Scala, and Java.
* Expertise in SQL and working with relational databases.
* Experience with DataFrames for data manipulation and analysis.
* Experience with cloud technologies (preferably AWS or GCP).
* Experience with message streaming platforms like MSK/Kafka.
* Experience with S3 or similar object storage.
* Experience with data lake technologies like Iceberg.
* Solid understanding of data warehousing concepts.
* Excellent communication and collaboration skills.
Bonus Points:
* Experience with data modelling and schema design.
* Experience with data governance and data quality management.
* Experience with DevOps practices and CI/CD pipelines.
* Relevant certifications (e.g., AWS Certified Data Engineer, Google Cloud Certified Professional Data Engineer).
Recommended Jobs
Fullstack Software Engineer, Demands Product Engineering
EvenUp is on a mission to support injury law firms across America in providing a consistent and high standard of representation, ensuring that every injury victim who seeks legal assistance can expec…
Principal Data Scientist - Engine Systems
WHY DATA SCIENCE & ANALYTICS? The Data Science & Analytics organization’s mission is to increase our speed, frequency and acumen of making decisions at scale by instilling a data-influenced approac…
Software Engineer - Data Platform
Design, develop, and maintain large-scale software platforms that serve millions of users globally. Contribute significantly to the architecture and evolution of our core data platform. Deliver high-p…
Principal Software Engineer, Ads Experience
With Roblox’s daily active users growing at a record pace (now over 150+ million), we are seeking a Principal Engineer to join the our Ads Experience team to help build the next generation of immersi…
Medical Surgical Med Surg Clinical Nurse RN Coordinator
Medical Surgical Clinical RN Nurse Coordinator — ignite nursing leadership in a high-growth Southern California hospital. The Medical Surgical Clinical RN Nurse Coordinator will drive Med-Surg excel…
Seasonal Part Time Sales Help
For California-based roles: Annual pay ranges may fall between $33,280 - $68,640 . You may also be offered a bonus, and other benefits. EEO Statement: Hat World, Inc., Lids Holdings, Inc., dba …
Senior Software Engineer
, permanent Position: Senior Software Engineer Direct Recruit Agency is seeking a highly skilled and experienced Senior Software Engineer to join our team on a full-time, permanent basis. As a …
Software Engineer I - Direct Issuance
StubHub is on a mission to redefine the live event experience on a global scale. Whether someone is looking to attend their first event or their hundredth, we’re here to delight them all the way fr…
Warehouse Loader (Full Time)
Reporting to the Operations Manager or Warehouse Manager, the full-time Warehouse Loader position is responsible for assisting in warehouse activities including loading, repacking, inventory, and mai…
Corporate development director - strategy
Company Description Guardant Health is a leading precision oncology company focused on guarding wellness and giving every person more time free from cancer. Founded in 2012, Guardant is transformi…