Director, Research - AI Evals

About the role

This role focuses on establishing and overseeing the evaluation of AI-powered experiences within Figma's products. The Director will be responsible for defining quality metrics, designing measurement strategies, and translating evaluation results into actionable insights for product decisions. This includes developing frameworks, rubrics, and datasets, and deciding when to use human judgment versus automated or model-based evaluation methods.

A key aspect of this position involves collaborating with engineering teams to set up consistent and repeatable evaluation pipelines. The goal is to make evaluation a standard part of how AI features are developed and released, ensuring quality control throughout the process. The Director will also be responsible for creating clear reports and dashboards that help stakeholders make informed go/no-go and prioritization decisions.

Beyond direct evaluation tasks, the Director will work to standardize the definition of quality across different teams, promoting a unified approach to evaluation rather than fragmented efforts. This involves advocating for evaluation as a strategic component of the product development process and gaining buy-in from various stakeholders to ensure evaluation standards are adopted and maintained.

The annual base salary for this position ranges from $258,000 to $348,000 USD.

Skills that matter here

• AI evaluation: The role requires direct, hands-on experience in evaluating AI/LLM-powered products, including designing and running human evaluation programs and constructing benchmarks.
• Management: Candidates need at least two years of management experience to lead a small team and oversee evaluation operations.
• Qualitative and quantitative methods: The position demands strength in both research methodologies, along with comfort in analyzing data and understanding model behavior.
• Stakeholder management: Success in this role involves gaining buy-in from executive and cross-disciplinary stakeholders, articulating user stories, and inspiring action based on evaluation findings.

Who this role suits

• A person with a strong background in product, research, or applied research, specifically within AI-powered product environments.
• Someone who can identify and prioritize ambiguous quality questions, then design appropriate evaluations to build confidence in product decisions.
• An individual adept at both qualitative and quantitative analysis, comfortable interpreting data and understanding complex model behaviors.
• A leader capable of influencing and gaining consensus from diverse teams and executive leadership on evaluation standards and strategic direction.

From the employer

What you'll do at Figma:

• Own AI evaluation methods and operations for Figma's AI-powered experiences — define quality dimensions, design how we measure them, and turn results into decision-ready signal.
• Build and maintain evaluation frameworks, rubrics, golden datasets, and quality bars, combining human evaluation with automated/model-based approaches (e.g., LLM-as-judge) where appropriate.
• Partner with engineering to stand up repeatable, reproducible evaluation pipelines and regression testing, so evaluation is a routine part of how AI features are built and shipped.
• Produce clear readouts and dashboards that let stakeholders confidently make go/no-go and prioritization decisions.
• Socialize a shared definition of quality so evaluation standards are adopted across teams rather than re-invented — and advocate for evaluation as a strategic partner in the product process.
• Manage a small team to execute our AI evals in partnership with contractors, internal staff, and/or LLMs.

We'd love to hear from you if you have:

• 10+ years of experience in product, research, applied research, or a closely related field, including 2+ years of management experience.
• Direct, hands-on experience owning the evaluation of AI/LLM-powered products.
• Expertise designing and running AI evaluation — human evaluation programs, rubric and benchmark/golden-dataset construction, inter-rater reliability — and sound judgment about when and how to apply automated/model-based approaches (e.g., LLM-as-judge), including their limitations.
• Strength across both qualitative and quantitative methods, comfort with data and metrics, and the ability to reason about model behavior.
• Demonstrated success in identifying the riskiest assumptions behind an ambiguous quality question, prioritizing them, and designing right-sized evaluation to build confidence.
• A proven track record of gaining buy-in from executive and cross-disciplinary stakeholders — transcending methodology to articulate a larger user story and the "so what" to inspire action.

Figma offers:

• Equity to employees.
• Competitive package of additional benefits, including health, dental & vision, retirement with company contribution, parental leave & reproductive or family planning support, mental health & wellness benefits, generous PTO, company recharge days, a learning & development stipend, a work from home stipend, and cell phone reimbursement.
• Sales incentive pay for most sales roles and an annual bonus plan for eligible non-sales roles.
• Annual Base Salary Range: $258,000 — $348,000 USD.