Posted 1h ago
Loading this job…
LLM Evaluation Harness Intern
Posted 2h ago0 views
MailerMen ships prompt changes to production most weeks, and right now nobody can prove that a change made things better.
The problem you will own
Over this six month internship you will build the evaluation harness that sits between our prompt library and the outbound email assistant. That means assembling a golden set of roughly 400 real support and outreach threads, writing graders in Python for tone, factual grounding and format compliance, and wiring it all into a nightly job that scores every candidate prompt against the current champion.
How the team works
You will sit with the applied AI pod in our Bengaluru office three days a week. Expect review sessions where you defend a scoring rubric and somebody senior asks whether your judge model is measuring what you think it is measuring. You will learn quickly why one flaky grader can waste an entire sprint.
What you walk away with
By the end you will have a harness other engineers depend on, a written report on which prompt patterns actually survived regression testing, and a much sharper instinct for spotting an output that is confidently wrong.
Responsibilities
- Curate and label a golden dataset of around 400 real email threads for prompt regression testing
- Write Python graders that score model outputs on tone, factual grounding and JSON schema compliance
- Run A/B comparisons between candidate prompts and the current production champion
- Investigate disagreements between the LLM judge and human raters, and document the root cause
- Maintain the nightly evaluation job and triage failures before the team's morning standup
- Publish a weekly summary of which prompt variants moved which metric
Requirements
- Final year student or recent graduate in any discipline, with strong written English
- Comfortable writing Python scripts that call an API and parse JSON responses
- Have used ChatGPT, Claude or a similar model beyond casual chat and can explain a prompt you improved
- Detail oriented enough to notice when two outputs differ in a way that matters
- Available to work from our Bengaluru office three days a week
Skills
Benefits
- Monthly stipend paid on the first working day
- One to one mentorship from the engineer who owns our production prompt stack
- Internship certificate on completion
- Strong performers are considered for a full time applied AI role