Principal Machine Learning & Data Engineering Lead
Role Overview
The Principal Machine Learning & Data Engineering Lead is a senior technical leadership role responsible for defining and executing the AI, machine learning, and data engineering strategy within the Intelligent Data division. The role combines deep expertise in machine learning, cloud-native architectures, real-time streaming platforms, and large-scale data engineering to design, build, and scale advanced AI-driven data ecosystems.
Operating at the intersection of data engineering, machine learning, and platform architecture, this role is responsible for delivering robust, secure, and scalable data platforms that enable advanced analytics, automation, artificial intelligence, and business decision-making. The successful candidate will provide technical leadership, drive innovation, mentor engineering teams, and champion emerging technologies across machine learning, generative AI, streaming data, cloud platforms, and distributed systems.
Key Responsibilities
Strategic Leadership & Architecture
-
Serve as the technical authority for the design, architecture, and implementation of scalable AI, machine learning, and data engineering platforms.
-
Define and execute the organisation’s AI, machine learning, and intelligent data strategy.
-
Lead the resolution of complex technical and architectural challenges across data, analytics, and AI ecosystems.
-
Translate business and technical requirements into scalable, high-performing, and future-ready solutions.
-
Evaluate and champion emerging technologies, frameworks, and tools to maintain technical leadership and innovation.
Data Engineering & Platform Development
-
Design, build, and maintain scalable data platforms, analytics frameworks, and modern data architectures.
-
Develop and support secure, repeatable, and optimised batch and real-time data processing solutions.
-
Architect and maintain reliable data pipelines integrating on-premise systems with AWS cloud environments.
-
Develop and manage ETL and ELT processes using Talend or similar data integration technologies.
-
Design and optimise distributed data platforms using technologies such as Spark, Hadoop, EMR, and cloud-native services.
-
Contribute to database design, performance optimisation, governance, operational management, business continuity, and disaster recovery practices.
-
Support production data environments and provide leadership in operational support and problem resolution.
Machine Learning & AI Solutions
-
Architect and build end-to-end machine learning ecosystems and production-grade AI platforms.
-
Develop intelligent applications leveraging machine learning, advanced analytics, and predictive modelling.
-
Drive innovation initiatives involving Large Language Models (LLMs), Generative AI, AI assistants, and emerging AI capabilities.
-
Implement machine learning workflows and platforms using tools such as SageMaker, Jupyter Notebooks, and cloud-native AI services.
-
Manage machine learning lifecycle activities, including model deployment, monitoring, governance, and scaling.
-
Work with structured, semi-structured, and unstructured datasets to build advanced AI solutions.
Real-Time Data & Streaming
-
Design and implement streaming data platforms and event-driven architectures.
-
Lead the adoption and optimisation of technologies such as Kafka, Flink, Beam, or similar real-time processing frameworks.
-
Develop intelligent real-time analytics, telemetry, fraud detection, and automation solutions.
-
Ensure scalable data movement and processing across distributed cloud environments.
Engineering Excellence & Team Leadership
-
Mentor, coach, and develop engineering, machine learning, and data teams.
-
Establish and maintain high standards for software engineering, code quality, testing, automation, and maintainability.
-
Drive DevOps, DataOps, MLOps, CI/CD, and infrastructure automation practices.
-
Foster a culture of innovation, continuous learning, collaboration, and technical excellence.
-
Conduct architecture reviews, code reviews, and technical design sessions.
Stakeholder & Industry Engagement
-
Collaborate closely with executives, business stakeholders, clients, and engineering teams to align technology initiatives with business objectives.
-
Act as a trusted advisor on AI, machine learning, cloud, and data engineering capability development.
-
Represent the organisation through presentations, technical forums, conferences, publications, and industry engagement.
-
Communicate complex technical concepts effectively to both technical and non-technical audiences.
Essential Skills & Experience
-
8+ years of experience in Data Engineering, Machine Learning Engineering, Software Engineering, or related technical roles.
-
Proven experience designing and delivering end-to-end AI, machine learning, and data platform solutions.
-
Strong expertise in Python for data engineering, machine learning, and platform development.
-
Experience with multiple programming languages such as Java, Go, C#, or JavaScript.
-
Deep experience with AWS cloud technologies, including S3, Lambda, EMR, EC2, RDS, DynamoDB, VPC, and related services.
-
Strong understanding of cloud-native architecture principles across AWS, Azure, and/or GCP.
-
Hands-on experience with machine learning platforms and tooling, including SageMaker, Jupyter Notebooks, and MLOps frameworks.
-
Expertise in real-time streaming technologies such as Kafka, Flink, Beam, or equivalent event-driven platforms.
-
Experience with Spark, PySpark, Hadoop, EMR, and distributed data processing frameworks.
-
Strong experience developing and supporting ETL/ELT pipelines and data transformation processes.
-
Experience designing and operating highly scalable distributed systems and microservices architectures.
-
Strong knowledge of Kubernetes, Docker, containerisation, and orchestration technologies.
-
Advanced understanding of SQL, NoSQL, graph databases, data modelling, and large-scale data architecture.
-
Experience implementing CI/CD, DataOps, DevOps, and automation practices.
-
Strong analytical, troubleshooting, problem-solving, and solution-design capabilities.
-
Proven experience leading technical teams, architecture initiatives, and strategic technology programmes.
Desirable Skills & Experience
-
Experience with Large Language Models (LLMs), Generative AI, AI agents, and intelligent assistants.
-
Data Science
-
Knowledge of advanced AI techniques, including semi-supervised learning and modern machine learning methodologies.
-
Experience implementing enterprise-scale real-time analytics, telemetry, fraud detection, or streaming intelligence solutions.
-
Exposure to business intelligence, analytics, and visualisation platforms.
-
Experience contributing to thought leadership through conference speaking, publications, blogs, or industry presentations.
-
Experience working in advanced data engineering, AI, or intelligent data consulting environments.
-
Experience implementing cloud-native MLOps and AI governance frameworks.
Qualifications
-
Bachelor's degree in Computer Science, Software Engineering, Data Science, Information Technology, or a related field.
-
Postgraduate qualification (Honours, Master's, or equivalent) in a relevant discipline would be advantageous.
-
AWS, Azure, GCP, Machine Learning, Data Engineering, Data Science, or other relevant technical certifications are highly desirable.
Behavioural Competencies
-
Strategic thinker with strong business and technical acumen.
-
Exceptional leadership, mentoring, and people development capabilities.
-
Strong communication, presentation, and stakeholder engagement skills.
-
Proven ability to influence technical direction and drive organisational change.
-
Innovative and forward-thinking mindset with a passion for emerging technologies.
-
Strong analytical and problem-solving abilities.
-
High attention to quality, scalability, security, and operational excellence.
-
Ability to work effectively across multiple teams and stakeholder groups.
-
Adaptable, resilient, and committed to continuous learning and improvement.
What We Offer
-
Competitive salary and comprehensive benefits package.
-
Opportunity to lead and shape enterprise AI, machine learning, and data engineering capabilities.
-
Exposure to cutting-edge cloud, AI, streaming data, and advanced analytics technologies.
-
Collaborative, innovative, and high-performing engineering culture.
-
Professional growth, leadership development, and career advancement opportunities.
-
The opportunity to deliver impactful, real-world AI and data-driven solutions at scale.
