Databricks Database Engineer

Overview/ Job Responsibilities

We are seeking a motivated and experienced Data Engineer to join our team. As a Data Engineer, you will be at the forefront developing ETL pipelines from various sources, data models for business users to create dashboards and ad-hoc reporting, along with efficient pipelines end-to-end for machine learning initiatives. In this role you will work closely with cross-functional teams to drive innovation and enhance data analytics capabilities utilizing Databricks. This position offers a unique opportunity to work with cutting-edge technologies and shape the future of our federal customer's data infrastructure.

 

If you are a motivated and experienced Data Engineer with expertise in Databricks and passion for building scalable and secure data solutions, we would like to hear from you.

 

Responsibilities:

 

• Design and implement data ingestion pipelines to efficiently ELT/ETL data from various sources into Databricks.
• Develop, implement, and validate logical and physical models while expanding on enterprise Lakehouse.
• Experience with AWS technologies, relational and non-relational databases (e.g., MySQL, NoSQL, Postgress; ideally MONGO experience).
• Work with scalable search technology (e.g. ElasticSearch, Solr).
• Support lead engineer in the architecture and engineering of complex data models, workflows, and customer support.
• Collaborate with cross-functional teams to gather requirements, understand data integration needs, and define data lake architecture and governance policies.
• Establish, maintain, and configure Databricks workspaces, clusters, and storage components, optimizing the solution for efficient data processing, query performance, and data governance.
• Develop and maintain data lake security frameworks, including access controls, encryption solutions, and data masking techniques to protect sensitive data.
• Develop code or scripts to ingest data from web-based APIs
• Collaborate with data engineers, data scientists, data analyst, and business partners to optimize data pipelines, develop data transformations, and ensure data quality and integrity.
• Monitor and tune Databricks workflows to ensure performance, reliability, and cost optimization, utilizing automated scaling and resource management techniques.
• Implement best practices for data governance, data cataloging, metadata management, and data lineage within Databricks, adhering to regulatory and compliance requirements.
• Collaborate with infrastructure teams to ensure Databricks infrastructure meets scalability and availability requirements, leveraging Databricks cluster management and AWS/Azure services.
• Develop and maintain documentation and guidelines related to the Databricks solution, including architecture diagrams, standards, and processes.
• Stay up to date with the latest advancements in Databricks, big data technologies, and cloud platforms, continuously evaluating and implementing new features and capabilities.
• Provide technical guidance and mentorship to junior data engineers and our customers to promote best practices and fostering a culture of continuous learning and growth.
• Collaborate with stakeholders to understand their data analytics and reporting needs and develop scalable data models and data transformation processes to support these requirements.
• Support data lake-related incident resolutions, troubleshooting data quality issues, performance bottlenecks, and other data-related challenges.
• Collaborate with data governance and compliance teams to ensure data privacy, security, and compliance guidelines are adhered to within the data lake solution.
• Participate in the evaluation and selection of new tools, technologies, and services to enhance the data lake infrastructure.
• Handle large data sets and scale their handling and storage.
• Author developer-friendly documentation (e.g., API documentation, deployment operations).
• Hands on experience writing code in Python/Pyspark.

Minimum Qualifications

 

• Bachelor's degree in computer science, information technology, or a related field. Equivalent experience will also be considered.
• Proven experience in creating data pipelines Databricks environment from scratch.
• Proven experience in optimizing existing data pipelines and data modeling.
• In-depth knowledge of Databricks, including but not limited to workspaces, clusters, storage, notebook development, and automation capabilities.
• Strong expertise in designing and implementing data ingestion pipelines, data transformations, and data quality processes using Databricks.
• Experience with big data technologies such as Apache Spark, Apache Hive, Delta Lake, and Hadoop