Lead Databricks Engineer Fulltime

Role – Lead Databricks Engineer

Location: O’Fallon, MO (Onsite from Day 1 – No Remote)

Yrs. of experience: 10+ Yrs.

Mode of employment: Full-Time

Job Description:

Candidates are required to have these mandatory skills:

• Strong hands on in Pyspark and Apache Spark.

• Experience in Native Spark Migration to Databricks.

• Experience in Hadoop Migration to Databricks.

• Experience in Building Data Governance Solutions like Unity Catalog, StarBust …etc

• Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.

• Build CICD for Databricks in Azure Devops.

• Process near Real time Data thru Auto Loader, DLT Pipelines.

• Implement Security Layer in Delta Lake.

• Implement Massive Parallel Processing Layers in Spark SQL and PySpark.

• Implement Cost effective Infrastructure in Databricks.

• Experience In extracting logic and from on prem layers like SSIS, Stored procedures, Informatica, Vertica, Apache Hudi, Filesystems. etc into Pyspark.

• Ability to Build Solutions as Cloud Agnostic.

• Proven experience in designing and implementing complex data solutions aligned with business objectives.

• Expertise in data modelling, integration, security, and governance

• Hands-on experience with guiding the virtual data model definition, defining Data Virtualization architecture and deployment with focus on Azure, Databricks, PySpark technologies.

• Prior experience with establishing best practices for business optimizations.

• Experience with relational and non-relational data stores (Hadoop, SQL, Mongo DB), ETL or ELT tools (SSIS, Informatica, Matillion, DBT), DevOps, Data Lake and Data Fabric concepts

• In-depth experience with data governance, data integration and related technologies.

• Proficiency in a variety of database technologies, both relational and non-relational.

• Knowledge of cloud-based data solutions (e.g., AWS, Azure).

• Excellent collaboration and communication skills

Tagged as: Engineer