Databricks Architect

Onebridge is a Consulting firm with an HQ in Indianapolis, and clients dispersed throughout North America and beyond. We have an exciting opportunity for a highly skilled Databricks Architect to join an innovative and dynamic group of professionals at a company rated among the top “Best Places to Work” in Indianapolis since 2015.

Databricks Architect | About You

As a Databricks Architect, you will be a key player in shaping and designing scalable data solutions using Databricks. Collaborating with cross-functional teams, you will lead efforts to understand business requirements, optimize data workflows, and ensure the efficient processing of large datasets. Your role involves architecting and implementing complex data solutions on Databricks, ensuring architectural integrity, scalability, and adherence to best practices.

Databricks Architect | Day-to-Day

• Architect and design advanced ETL processes using Databricks to extract, transform, and load data from various sources.
• Lead collaboration with data scientists, analysts, and other stakeholders to understand and implement advanced data requirements.
• Provide expertise in optimizing and troubleshooting existing Databricks jobs for performance and reliability.
• Define and implement best practices for data quality, data governance, and data security within Databricks architecture.
• Work closely with cloud platforms to architect and manage robust Databricks clusters.
• Lead and participate in code reviews, documentation, and knowledge-sharing sessions.

Databricks Architect | Skills & Experience

• 8+ years of proven experience in Databricks architecture, data engineering, and migration from on-prem Hadoop to Databricks/AWS.
• Expert proficiency in Apache Spark, Databricks platform, and advanced programming languages such as Python or Scala.
• In-depth understanding of data modeling, complex ETL processes, data integration, and CI/CD pipeline in AWS technologies.
• Strong familiarity with cloud platforms and services, particularly AWS, Azure, or GCP.
• Extensive hands-on experience in Databricks fundamentals/architecture, including setting up and optimizing Databricks clusters and modules (Data Engineering, ML, and SQL warehouse).
• Knowledge and expertise in advanced concepts such as medallion architecture, DLT, unity catalog within Databricks, and core AWS services/architecture best practices.

A Best Place to Work in Indiana since 2015