Principal Data Engineer

Principal / Lead Data Engineer ( Microsoft Azure / Databricks / Data Factory )
Location: Sydney (Hybrid Work)
Position Type: Full-Time, Permanent
Salary: Up to $220,000 base DOE + Superannuation

An exciting opportunity has arisen for a hands-on, proactive Principal / Lead Data Engineer to join a leading Insurance company based in Sydney.
In this role, you will be responsible for leading a team of 2-4 data engineers,
working as part of a global team on high-impact data engineering initiatives.

This is an excellent opportunity to own and shape the data engineering function, drive innovation, and play a key role in building real-time data solutions using Microsoft Azure and Generative AI technologies.

  
Key Responsibilities for the Principal / Lead Data Engineer include :

• Lead a team of 2-4 data engineers, guiding and mentoring them to deliver high-quality data engineering solutions in a fast-paced environment.
• Take ownership of building, optimizing, and scaling real-time data pipelines using Microsoft Azure, including Azure Databricks, Azure Data Factory, and Power BI.
• Work with real-time data use cases: scoring data in real-time, ensuring that information is transferred and processed quickly, with an emphasis on low-latency data flow.
• Play a key role in the development and maintenance of war boards in contact centres worldwide, enabling real-time dashboards and reporting with near-zero delays.
• Focus on solving complex data challenges related to processing thousands of calls coming through call centers, including real-time transcriptions and passing the data for further processing.
• Partner with the Data Science team and be involved in a journey of real-time training and AI model development, including working on training your own ChatGPT model for specific business use cases.
• Explore and implement Generative AI applications within the data engineering function, particularly around real-time transcriptions and training models that will drive future capabilities.
• Collaborate with the broader technology team to continuously improve data quality, security, and pipeline performance.
• Be proactive in problem-solving, owning data engineering challenges and delivering impactful solutions under tight deadlines.

  
Key Skills & Experience required for the Principal / Lead Data Engineer include :

• Hands-on experience with Microsoft Azure, ideally including Azure Databricks, Azure Data Factory, and Power BI.
• Strong proficiency in Python and SQL for building scalable and efficient data pipelines.
• Experience in real-time data processing, with a focus on minimizing delays (e.g., 1-second latency in scoring data, real-time reporting, etc.).
• Familiarity with call centre data and technologies for real-time transcriptions and natural language processing is a nice to have.
• Ability to collaborate with Data Science teams to solve complex challenges, including exploring Generative AI applications.
• Experience deploying ML models in practice is desirable, particularly around the integration of models into real-time data workflows.
• Experience leading or mentoring others, ensuring high performance and fostering a collaborative culture.
• A passion for tackling complex data challenges and being part of a transformative journey towards real-time, AI-driven solutions.
• Experience in the insurance industry or similar sectors is a plus but not essential.

  
Why should the Principal / Lead Data Engineer join? :

• A great opportunity to own and shape the future of data engineering within a global organization, driving key innovations in real-time data solutions.
• Work on exciting real-time data use cases, including scoring, transcription, and AI model training for large-scale operations like contact centres.
• Play a key role in the development and deployment of Generative AI technologies, with a focus on practical, impactful use cases.
• Enjoy the flexibility of a hybrid work environment, balancing work and life commitments.
• Competitive salary of up to $220k base plus superannuation.

  
This is a unique opportunity for a proactive, hands-on Data Engineering Lead to take ownership of critical data initiatives, solve complex challenges, and work alongside a talented team on innovative real-time data and AI applications. If you're ready to shape the future of data engineering and play a pivotal role in driving transformative solutions, apply now!
  
Keywords: Microsoft Azure / Power BI / Azure Data Factory / Databricks / SQL / Python / Insurance / Principal Data Engineer / Lead Data Engineer