Data Engineer

job summary:
We are seeking a highly motivated Data Engineer with experience in Databricks on Microsoft Azure to join a fast-paced team responsible for modernizing enterprise data platforms supporting healthcare research and analytics."

The primary responsibility of this role is to migrate existing Oracle PL/SQL-based ETL processes to Databricks data pipelines. The successful candidate will analyze existing ETL workflows, redesign them using modern cloud technologies, and develop scalable, high-performance data pipelines while ensuring data quality and reliability. The individual will also support existing production ETL processes during the migration.

location: Pleasanton, California
job type: Contract
salary: $50 - 60 per hour
work hours: 9am to 5pm
education: Bachelors

responsibilities:
- Work with business users and stakeholders to gather requirements and translate them into technical solutions.

- Analyze existing Oracle SQL and PL/SQL ETL processes and migrate them to Databricks using Spark SQL and/or PySpark.

- Design, develop, test, deploy, and maintain scalable data pipelines on Microsoft Azure Databricks.

- Create and maintain technical design, data mapping, and system documentation.

- Validate migrated data with business users and subject matter experts to ensure business-level accuracy

- Implement incremental data processing strategies using Databricks and Delta Lake.

- Analyze and convert complex Oracle PL/SQL transformation logic (packages, procedures, functions) into scalable Databricks pipelines.

- Perform data validation and reconciliation between source (Oracle) and target (Databricks) to ensure data accuracy and completeness during migration.

- Provide ongoing support and maintenance for ETL pipelines post-migration in Databricks, including monitoring, troubleshooting, and enhancements.

- Optimize pipeline performance, troubleshoot production issues, perform root cause analysis, and implement solutions.

- Participate in source control, CI/CD, release management, and production support activities.

- Take ownership of assigned projects, manage priorities, and deliver high-quality solutions on schedule.

qualifications:
· 4-6 years of experience in ETL, database development, or data engineering.

· 4+ years of advanced SQL development and query optimization.

· 3+ years of Oracle PL/SQL development.

· 3+ years of experience developing data pipelines using Databricks on Microsoft Azure.

· Experience with Spark SQL, PySpark, Delta Lake, and Azure Data Lake Storage (ADLS).

· Experience migrating legacy ETL processes from Oracle or other relational databases to cloud-based data platforms.

· Experience with Git, source control, and CI/CD deployment processes.

· Strong analytical, troubleshooting, and problem-solving skills.

· Excellent communication skills and the ability to work independently and collaboratively.

Preferred:

· Experience in the healthcare, biomedical research, or clinical data domain.

· Experience converting Oracle PL/SQL packages, procedures, and functions into Databricks pipelines.

· Experience with Azure Data Factory, Databricks Workflows, or similar orchestration tools. Knowledge of dimensional data modeling and enterprise data warehousing.

· Experience with Python and modern software engineering practices.

· Experience with Power BI, Tableau, or similar reporting tools.

· Familiarity with machine learning or AI-enabled data engineering is a plus.

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact [email protected].


Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.

Qualified applicants in San Francisco with criminal histories will be considered for employment in accordance with the San Francisco Fair Chance Ordinance.


Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.


We will consider for employment all qualified Applicants, including those with criminal histories, in a manner consistent with the requirements of applicable state and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance.