Data Engineering Analytics Intern

Description of Job Function:

Support the migration of existing Jupyter Notebook-based data workflows, including those involving Excel, databases, and Salesforce, into Azure Databricks. Focus on developing scalable, automated pipelines that deliver data to Excel files and Power BI dashboards. Apply AI expertise on a small-scale project or initiative.

 

Primary Responsibilities

Databricks Transition & Automation

• Convert Python and Pandas-based workflows into PySpark where appropriate.
• Migrate connection logic from local drivers to Databricks-supported methods (e.g., JDBC, ODBC, Databricks connectors for Salesforce, Azure SQL, etc.).
• Investigate and implement Delta Lake storage for versioned, searchable datasets.
• Schedule Databricks Jobs to run transformations without manual execution.
• Set up triggers (e.g., file arrival in Azure Data Lake Storage).
• Automate data exports to Excel and direct connections to Power BI datasets.

 

Best Practices & Documentation

• Research and propose best methods for ETL in Databricks for each data source.
• Write reusable Databricks notebooks or workflows with parameterization.
• Develop clear and accessible documentation to ensure team members can maintain workflows without requiring deep expertise in Databricks.

 

Skills Needed or Open to Development

• Python (Pandas → PySpark conversion experience is a plus)
• SQL (experience in a distributed environment like Spark SQL a plus)
• Databricks Basics (clusters, notebooks, jobs, repos, Delta Lake)
• Azure Services (Azure Data Lake, Azure SQL, Key Vault)
• API Integration (for Salesforce and other REST and ODBC data sources)
• Data pipeline scheduling & orchestration

 

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or veteran status.