Linguist

Remote

1 Year Contract

Years of Experience: 4-6 year of experience

Must-Have Skills

1 Data Annotation

2 SQL Python

3 Linguistic Background

Nice-to-have Skills

1 Computational Linguistics (Master's preferred)

2 Statistics & Data Science

Reponsibilites

● Dataset Building & Curation: Receive requests from engineers and research scientists, then design and execute synthetic data collections or sample from existing data to produce clean, labeled datasets for LLM training and evaluation.

● Annotation Queue Management: Stand up and maintain grading queues — define rubrics, write annotation guidelines, onboard raters, and iterate on queue design to keep pace with rapidly evolving LLM features.

● Quality Auditing & Error Attribution: Review annotated data daily, flag inconsistencies, run inter-rater reliability checks, and trace quality issues back to specific LLM subcomponents (e.g., factuality, coherence, brevity).

● Data Analysis & Reporting: Use AI to write SQL queries and Python scripts to pull metrics, compute agreement scores, and generate dashboards; summarize findings in weekly reports shared with cross-functional partners.

● Experiment Design: Plan and run controlled experiments to test rater calibration, guideline clarity, or model performance changes — then present results and recommendations in team meetings.

Guideline Authoring: Draft, test, and refine annotation guidelines for new LLM capabilities on tight timelines, ensuring raters can consistently apply them at scale.

● Cross-Functional Collaboration: Attend syncs with engineering, research, and PM stakeholders to align on data priorities, surface quality insights, and inform roadmap decisions for smart glasses LLM features.

Skills:

• Basic Python, basic SQL with strong desire to grow required for this position. Intermediate level preferred.

• Basic knowledge of linguistics (some of Phonetics, Syntax, Semantics, Dialectology) required.

• Ability to analyze numerical rating results and draw conclusions based on data.

• Basic stats / math for data analysis.

• Written and oral communication skills necessary for communicating on project milestones and for reporting findings .

Education/Experience:

• Bachelor’s degree in Linguistics, Computational Linguistics, Speech Science, or related field OR graduate degree in linguistics OR industry similar experience required.

• Master’s degree or beyond preferred OR equivalent industry experience.

• Additional learning via Coursera, Udemy, or other online resources is great too

PRI Global jobs

Never miss a role

Linguist