Senior Principal Machine Learning Engineer
About this opportunity
At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work, but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.
Role Overview
We are looking for a Senior Principal Machine Learning Engineer to join the AI Engineering team, with a primary focus on Hands-On Engineering (MLE) and MLOps & Platform Reliability and GenAI & Agentic Systems. This posting is at level R4 on our engineering ladder, see the level framing below for the expected scope of ownership and impact.
Level framing: Recognized expert who mentors others; makes key technical decisions impacting multiple teams; leads resolution of highly complex challenges; cross-functional influence; may engage with external partners as an authority.
Core Responsibilities
Design and build production backend systems for agentic AI applications, API layers, backend-for-frontend (BFF) patterns, session/state management, and streaming for concurrent multi-user workloads.
Lead resolution of highly complex, cross-system technical challenges.
Set the technical bar for hands-on engineering practice across the team.
Own GitOps and engineering-governance standards across the team (version control, CI/CD, review, promotion).
Make key MLOps/platform decisions that affect multiple teams (e.g. deployment topology, observability strategy).
Drive triage and resolution of complex, cross-system production incidents.
Own architecture for agentic AI systems, including routing, policy enforcement, and RAG/knowledge-registry design.
Lead evaluation and adoption of new agent frameworks, LLMOps practices, and GenAI tooling across the team.
Implement authentication/authorization flows for agentic systems (OAuth/OIDC, on-behalf-of token exchange, secure credential storage).
Mentor other engineers on architecture, design patterns, and production engineering practices, and act as a role model for engineering rigor across the wider team.
Key Tools & Technologies
Cloud & Data Infra: AWS (EC2/ECS, S3, Lambda, IAM, CloudWatch or equivalent); Databricks & Unity Catalog
Foundational: Python; Git/GitHub; software engineering best practices; testing; SQL
MLOps & Deployment: Docker, Kubernetes, Prefect; CI/CD (GitHub Actions); production monitoring; model versioning & lineage; GitOps governance
GenAI & Agentic Architecture: Claude or comparable LLMs; LangGraph or comparable agent frameworks; RAG architectures; vector databases (e.g. Pinecone); prompt engineering & evaluation
Required Qualifications
11–15 years of hands-on experience, with demonstrated growth into architecture-level ownership spanning multiple teams or systems.
Strong proficiency in Python and a track record of writing clean, testable, production-quality code.
Demonstrated experience owning CI/CD, containerisation, and orchestration for production ML/AI systems.
Proven experience developing or deploying LLM-based applications, including prompt engineering, RAG, or agentic workflows.
Strong working knowledge of containerisation (Docker), orchestration (Kubernetes), and CI/CD pipelines.
Working knowledge of AWS cloud services and Databricks/Unity Catalog or equivalent enterprise data platforms.
Excellent verbal and written communication skills.
Experience working in Agile/Scrum environments.
Education
Bachelor's or Master's degree in Computer Science, Computer Applications, or a related technical field.
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation ) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.
#WeAreLilly
Job details
How this role compares
Computed from every other active Information Technology role in our database, not just this employer's listings.
We currently track 577 comparable Principal Information Technology roles across 55 biopharma companies.
Salary context
107 of 577 peers report a salary range (USD, annualized)
Peers share this role's job function and a matching or adjacent seniority level -- not necessarily the same therapeutic area or country.
Where these roles are based
Top locations among the 577 comparable roles
+ 26 more countries
Seniority mix
577 of 577 peers have a known seniority level
Therapeutic area mix
5 of 577 peers have a known therapeutic area; the rest are genuinely unlabeled, not hidden
Similar opportunities
The closest matches from our peer group, ranked by how similar they are, not how well you'd qualify for them -- treat this as market context, not a guaranteed shortlist; a weak match is labeled as one below.
Notify me about similar jobs
Get an email when we spot other openings like this one – same job function, comparable seniority, roles you'd actually want to see.
Or follow a whole field: get every new Information Technology job in India, and choose locations and companies.
How we calculate "similar"
No black box, no LLM guesswork: a deterministic score built from four normalized attributes. Here's this role's own peer group at different match levels, so you can see the mechanism, not just the result.
Every comparison starts from the same 100-point budget: 25 for working in the same function, 40 for the same therapeutic area, 20 for the same or adjacent seniority, 15 for the same country. A dimension we can't confirm on both sides contributes nothing, never a guess, never a free pass.
0 points, never a partial guess. A role we know almost nothing about beyond its function bottoms out at 25%; it never inflates to 100% just because there's little to compare against. Seniority uses a defined ladder (Associate → Manager → Associate Director → Senior → Principal → Director → Senior Director → Executive/VP) so "Director" and "Senior Director" count as adjacent, but "Director" and "Executive/VP" do not.