E-commerce Evaluation Specialist
Mercor
| Company | Mercor |
| Category | Marketing |
| Location | IN |
| Remote | — |
| Employment | Not stated |
| Level | Mid |
| Salary | Not stated by the employer |
| Posted | 5 Aug 2026 |
| Last verified | 12 Aug 2026 |
| Source | The employer's own careers page (company_site) |
Description
**About The Job**
**Mercor**
connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include
**Benchmark**
,
**General Catalyst**
,
**Peter Thiel**
,
**Adam D'Angelo**
,
**Larry Summers**
, and
**Jack Dorsey**
.
**Position:**
Shopify Storefront Agent Evaluator
**Type:**
**Contract**
**Compensation:**
**$40–$50/hour**
**Location:**
**Remote**
**Commitment:**
**up to 5 hours/week**
**Role Responsibilities**
- Evaluate the Storefront Agent by simulating real shopper interactions, including product discovery, comparisons, and inquiries about sizing, stock, shipping, and returns.
- Grade responses based on Relevance, Accuracy, and Safety to ensure they meet shopper expectations and adhere to store policies.
- Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.
- Work independently to deliver precise and repeatable judgments on agent performance.
**Qualifications**
**Must-Have**
- Consumer e-commerce fluency — you shop online regularly and understand effective product recommendations.
- Careful, consistent judgment — apply the rubric uniformly across responses.
- Clear, concise written English — articulate reasons for response failures in a sentence or two.
**Preferred**
- Hands-on Shopify merchant experience — familiarity with shopper behavior on Shopify stores.
- Experience using AI shopping assistants or chat agents — understand common failure points.
- Prior work evaluating, red-teaming, or annotating AI model outputs.
- Familiarity with retail operations: catalog and variant structure, inventory, shipping, and returns policy.
**Application Process (Takes 20–30 mins to complete)**
- Upload resume
- AI interview based on your resume
- Submit form
**Resources & Support**
- For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
- For any help or support, reach out to: [email protected]
*PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.*
,