Senior Data Platform / Data Product Engineering Lead - Irvine, CA | TCS Job

Senior Data Platform / Data Product Engineering Lead

Must Have Technical/Functional Skills

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

________________________________________

Required Skills & Experience

•10+ years of experience in Data Engineering / Data Platform roles

•Strong hands-on expertise in: 

oDatabricks (Delta Lake, workflows, DAG)

oApache Airflow / Astronomer

oPython, SQL, DBT ,AWS

•Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)

•Deep understanding of: 

oData governance (catalogs, lineage, contracts, metadata)

oData quality and masking techniques

oEnterprise data platforms and marketplace ecosystems

•Experience with API-based integrations (e.g., entitlement systems like AccessCentral)

•Monitoring/observability tools (e.g., Datadog)

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

________________________________________

Core Responsibilities

1. Platform Strategy & Self-Service Enablement

•Define and implement a self-service data platform strategy to reduce onboarding friction.

•Lead automated provisioning of: 

oDatabricks workspaces (via DevHub)

oAirflow/Astronomer environments

oAccess and entitlements (AccessCentral APIs)

•Establish isolated, stable development environments for federated teams.

•Drive platform observability by integrating metrics into tools lik e Datadog.

________________________________________

2. Data Discovery, Access & Governance

•Architect and implement enterprise-wide data discovery and marketplace enablement.

•Drive adoption of: 

oData contracts

oMetadata standards

oDomain-aligned catalogs (Unity Catalog)

•Enable secure access to curated, masked datasets in dev and production environments.

•Implement tagging, access patterns, and entitlement automation.

•Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned).

________________________________________

3. Data Engineering, Curation & Orchestration

•Lead design of scalable data ingestion, curation, and transformation frameworks.

•Build and standardize modular, reusable frameworks: 

oLaunchLake templates

oAirflow DAG libraries

oDBT-based transformation models

•Ensure: 

oData quality and consistency

oEmbedded governance and compliance policies

•Enable concurrent development using standardized patterns and environments.

________________________________________

4. CI/CD, Automation & Deployment

•Define and enforce standard CI/CD pipelines across data products: 

oHarness (or equivalent)

oDatabricks Asset Bundles (DAB)

•Automate: 

oDAG deployments (Airflow/Astronomer)

oDBT pipeline releases

•Reduce manual interventions and ensure consistent, repeatable deployments.

•Improve release reliability with feedback loops, notifications, and monitoring.

________________________________________

5. Data Product Publishing & Marketplace Enablement

•Drive publishing of data products to: 

oUnity Catalog

oEnterprise Data Marketplace

•Define and enforce: 

oDocumentation standards

oData ownership models

oVersioning and contract management

•Enable cross-domain data sharing with embedded governance and access controls.

________________________________________

6. Operations, Observability & Reliability

•Establish a scalable operating model for data product support.

•Implement: 

oMonitoring dashboards (Datadog)

oData quality frameworks

oUsage and performance metrics tracking

•Improve visibility into: 

oPipeline health

oData lineage

oAccess and consumption patterns

•Lead incident management, root cause analysis, and escalation processes.

________________________________________

7. Transformation, Roadmap & Innovation

•Drive execution of platform priorities such as: 

oData contract activation strategy

oDomain catalog integration

oData masking in development environments

oData quality frameworks

oDBT adoption and POCs

•Lead maturity uplift from: 

oManual, fragmented workflows → standardized, automated paved paths

•Champion continuous improvement and innovation in developer experience.

________________________________________

Salary Range- $100,000-$120,000 a year