Lead Data Engineer - DataBricks

Job Title: Lead Data Engineer – Databricks
<\/h2>

<\/p>

<\/div>

Job Summary
<\/h2>

We are seeking a Lead Data Engineer with deep expertise in Databricks to architect, build, and lead scalable data engineering solutions on cloud -based lakehouse platforms. The role combines hands -on technical leadership with solution design, mentoring, and close collaboration with architects, BI, and AI teams.
<\/p>

<\/div>

Key Responsibilities
<\/h2>

Technical Leadership & Architecture
<\/h3>

• Lead the design and implementation of Databricks Lakehouse architectures
  <\/p><\/li>

• Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
  <\/p><\/li>

• Drive architectural decisions for batch and streaming data pipelines
  <\/p><\/li>

• Establish coding standards, best practices, and reusable frameworks
  <\/p><\/li><\/ul>

  Data Engineering & Databricks
  <\/h3>

  • Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
    <\/p><\/li>

  • Optimize Spark jobs for performance, reliability, and cost
    <\/p><\/li>

  • Implement Delta Lake features (ACID, time travel, schema enforcement)
    <\/p><\/li>

  • Develop and manage Databricks workflows, jobs, and clusters
    <\/p><\/li><\/ul>

    Cloud & Platform Integration
    <\/h3>

    • Architect Databricks solutions on Azure (preferred) or AWS
      <\/p><\/li>

    • Integrate Databricks with cloud storage and data services
      <\/p>

      • Azure: ADLS, ADF, Synapse
        <\/p><\/li>

      • AWS: S3, Glue, Redshift
        <\/p><\/li><\/ul><\/li>

      • Enable BI and analytics consumption (Power BI, Tableau)
        <\/p><\/li><\/ul>

        Governance, Security & DevOps
        <\/h3>

        • Implement data governance using Unity Catalog
          <\/p><\/li>

        • Define RBAC, data access controls, and security best practices
          <\/p><\/li>

        • Enable CI/CD for Databricks using GitHub / Azure DevOps
          <\/p><\/li>

        • Use Infrastructure -as -Code (Terraform) for environment management
          <\/p><\/li><\/ul>

          Leadership & Collaboration
          <\/h3>

          • Lead, mentor, and grow data engineering teams
            <\/p><\/li>

          • Conduct design and code reviews
            <\/p><\/li>

          • Collaborate with Data Architects, Product Owners, and stakeholders
            <\/p><\/li>

          • Support production releases, monitoring, and incident resolution
            <\/p><\/li><\/ul>

            
            <\/div>

            Required Skills
            <\/h2>

            Databricks & Big Data
            <\/h3>

            • Expert -level Databricks experience (Azure or AWS)
              <\/p><\/li>

            • Strong Spark / PySpark / Spark SQL expertise
              <\/p><\/li>

            • Delta Lake and Lakehouse architecture
              <\/p><\/li>

            • Streaming (Structured Streaming) experience
              <\/p><\/li><\/ul>

              Cloud & Data Platforms
              <\/h3>

              • Strong experience with Azure or AWS cloud platforms
                <\/p><\/li>

              • Data orchestration tools (ADF, Airflow, or similar)
                <\/p><\/li>

              • Strong SQL and data modeling skills
                <\/p><\/li><\/ul>

                DevOps & Automation
                <\/h3>

                • Git -based version control
                  <\/p><\/li>

                • CI/CD pipelines for data engineering workloads
                  <\/p><\/li>

                • Terraform or similar IaC tools
                  <\/p><\/li><\/ul>

                  
                  <\/div>

                  Preferred Qualifications
                  <\/h2>

                  • Experience with MLflow and MLOps workflows
                    <\/p><\/li>

                  • Exposure to Microsoft Fabric or Snowflake
                    <\/p><\/li>

                  • Databricks certifications (Professional Data Engineer / Architect)
                    <\/p><\/li>

                  • Experience working in Agile environments
                    <\/p><\/li><\/ul>

                    
                    <\/div>

                    Education
                    <\/h2>

                    • Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
                      <\/p><\/li><\/ul>

                      
                      <\/div>

                      Quick Fit Indicators
                      <\/h3>

                      
                      <\/p>

                      ✔ Leads Databricks lakehouse implementations
                      <\/div>

                      ✔ Strong Spark optimization and governance expertise
                      <\/div>

                      ✔ Mentors and scales engineering teams
                      <\/div>

                      ✔ Owns delivery, quality, and platform reliability
                      <\/div>

                      
                      <\/p>

                      
                      <\/div><\/span>