Lead Data Engineer

Purpose of Position

The position plays a key role in providing technical leadership and direction for the development and maintenance of enterprise analytics deliverables, including but not limited operational data stores, data integrations and reports. The ideal candidate will be guide a team in a mixed technology environment to deliver data products providing decision support for business and customers. As part of a highly collaborative team, the role will interact with technical and business resources within and outside of IT organization. The ideal candidate is a committed, creative, self-motivated and passionate technologist who is interested in practicing current skills and learning new ones. 

Tasks and Responsibilities

This role is a highly technical and leadership position with responsibility to deliver, maintain and oversee data engineering components to meet the business objectives. The following duties are essential to the successful and satisfactory performance of this job. Other duties may be assigned. 
Technical Leadership 
 
•    Technical Leadership – 
o    Provide technical direction and oversight to a team of data engineers and developers. 
o    Set standards for data engineering best practices, code quality, and documentation. 
o    Mentor and coach junior and mid-level engineers, fostering skill development and knowledge sharing. 
o    Lead code and design reviews to ensure adherence to established standards 

•    Design –  
o    Partner with Business Stakeholders, Business Analysts, Data Engineers, Developers to design enterprise data warehouse components. 
o    Provide estimations, schedules and regular and timely updates to project managers & senior management as needed. 
o    Improve operations by conducting system analysis, recommending changes in policies and procedures 
•    Develop/Validate – 
o    Oversee and validate proposed design for accuracy and completeness of business use cases. 
o    Lead the development of data integration and transformation solutions to meet the input needs of the models. 
o    Guide the development and support batch jobs. 
o    Perform unit & regression testing. 
o    Perform code/peer reviews to ensure adherence to established design & development standards. 
o    Collaborate with development and quality assurance teams for testing and product quality improvements as needed. 
o    Produce deployment scripts, checklists, playbook & operations runbook in accordance with SDLC & change management requirements 

•    Deploy/Support –  
o    Oversee the deployment of database & data engineering components to higher environments, using CI/CD automation where applicable. 
o    Take measure to ensure adherence to committed service level agreements. 
o    Monitor the scheduled to jobs & performance of the platform for smooth operation. 
o    Independently and with support from other developers, troubleshoot and fix issues that arise with data and/or processes. 
o    Serve as techno-functional expert and assist developers and users in understanding data flow and application data 

Minimum Education

• Bachelor's degree from an accredited institution or equivalent industry experience

Minimum Experience

• Bachelor’s degree in related field (prefer CS major) 

• 10+ years of software development experience 

• 5+ years of enterprise ETL, analytics & reporting development experience in SSIS, SSAS, SSRS 

• 2+ Years experience in Databricks and expert level proficiency in DLT and Unity Catalog. preferably certified. 

• 2+ Years experience with Python Development 

• 5+ years of experience in RDBMS design and development. Must demonstrate a clear mastery of the logical and physical database design (for both transactional and data warehouse) and data normalization concepts 

• 5+ Expert SQL abilities and use of tools and platforms and troubleshooting skills. Must have extensive hands-on dev experience: T-SQL, UDF, UDAF, View, Trigger, Stored Proc,  

• 3+ years of experience in Azure using ADLS & Azure Data Factory and Automation 

• Experience in warehouse modeling using Data Vault 2.0, preferably certified.  

Knowledge, Skills, Abilities

• Demonstrated experience building and maintaining data products. 
  Experience working in visual studio development environment and with using DevOps platforms for code management and deployment using CI/CD techniques 
  Familiarity with SDLC and agile methodologies and tools such as TFS, Git or SCRUM  
  Ability to take a project from scoping requirements through actual launch of the project 
  Experience in communicating with users, other technical teams and management to collect requirements, identify tasks, provide estimates and meet production deadlines 
  Experience with professional software engineering best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing and operations 
  Understand and work in an Agile development environment 
  Experience in designing & building BI Reporting solutions, preferably using Power BI. 
  Experience with Apache Spark and PySpark 
  Experience in building data engineering platforms in Python for data science or data analytic groups  
  Experience with NoSQL 
  Experience with consuming data from streaming platforms like Kafka. 
  System and networking fundamentals 
  Knowledge/experience in Education or Aviation industry 
  Experience with AI: Exposure to AI concepts, tools, and frameworks, preferably using Databricks. 
  Strong leadership and mentoring skills 
  Proactive self-starter with a strong work-ethics, accountability and responsibility for the accuracy and completeness of assigned work and results 
  Excellent Communication and Customer Service Skills. 
  Must be able to work individually and as part of a team to achieve common goals. 
  Ability to effectively manage time and prioritize work load with competing/changing priorities. 
  Mentor junior and mid-level engineers 
  Comply with all laws, regulations and company policies 

Physical Demands and Work Environment

The physical demands and work environment described here are representative of those that must be met and/or encountered by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. While performing the duties of this job, the employee is regularly required to use hands to finger, handle, or feel; reach with hands and arms; and communicate. The employee may be required to stand, walk, and sit. Specific vision abilities required by this job include the ability to view monitors, technical documents, and reference material. The employee must occasionally lift or move up to 25 pounds. The noise level in the work environment is usually low to moderate.