Should you outsource data annotation? In-house, managed or crowd

Four models, including the hybrid

The choice is not binary. Four arrangements cover nearly every project, and they differ mainly in who writes the guideline, who checks the work, and who answers the annotators’ questions when an item does not fit the rules.

The crowd option has narrowed. Amazon Mechanical Turk, the platform behind much of the research below, closed on 30 September 2026. Research-panel marketplaces such as Prolific remain, with published pay floors, and Toloka, which began as a crowdsourcing platform, now leads with expert data for AI labs. Our guide to labelling companies and the list of Scale AI alternatives name the providers; this piece is about whether, and how, to hand the work over.

A fifth arrangement, a platform licence with your own staff labelling inside it, counts as in-house for this decision: the tool changes, but the ownership of quality does not.

Dataset Finder, which publishes this blog, lets teams order labelling from Innovatiana’s managed annotators, so we have a stake in the answer; the studies cited here are independent.

What the evidence says about quality

The most direct comparison is by Rädsch and colleagues, published in Nature Machine Intelligence in 2023. A biomedical segmentation task was given to 156 annotators from four professional annotation companies and to 708 MTurk crowdworkers, under three versions of the instructions: 14,040 annotated images in all. Given minimal text instructions, the professionals reached a median Dice score of 0.93 against 0.67 for MTurk, and their odds of a severe error, a score of zero, were 0.09 times those of a crowdworker.

The second finding is as useful as the first. Adding example images to the instructions reduced severe errors for every provider, while lengthening the text alone did not reliably help. A managed workforce with poor instructions gives back part of its advantage; a crowd with good instructions narrows the gap without closing it.

The older, more optimistic result is Snow et al. (2008). On five short language tasks, MTurk labels agreed well with expert gold labels, and for affect recognition an average of four non-expert labels per item matched one expert. The whole study bought 21,690 labels for $25.82. Two caveats: the tasks were quick judgements rather than skilled drawing, and the price reflected pay that Hara et al. later measured at a median of about $2 an hour, with only 4% of workers earning more than $7.25.

Read together, crowds can match experts on simple judgements if you pay for redundancy. On spatial or specialist work, trained annotators are measurably better, and nothing in the literature suggests that extra crowd labels close that gap cheaply.

Where domain knowledge cannot be outsourced

Sambasivan and colleagues interviewed 53 practitioners building high-stakes AI in India, East and West African countries and the US, and 92% reported at least one data cascade: a problem introduced upstream, in data work, that surfaced later as model failure. Domain experts were mostly brought in for collection or troubleshooting rather than throughout, and field partners often absorbed data work as extra duty, without adequate pay.

None of this rules out outsourcing; it decides which part stays. In the common hybrid, in-house experts write the guideline, label a gold set, review a sample and adjudicate disagreements, while a managed team does the volume against that gold. The guideline is where the expertise is transferred. The annotators Rädsch’s team surveyed, 298 professionals in all, named labelling instructions as the primary cause of problems in their daily work, and 76% of the MICCAI challenge tasks the team analysed reported no labelling instructions at all. Our annotation guidelines template is a starting point.

Three signs that the work, or at least its judgement, must stay close to you:

• The definitions are still moving. If the class list or the edge-case rules change every few weeks, each change has to reach every annotator the same day, and someone has to relabel what came before.
• Rare cases carry the risk. Clinical findings, fraud patterns, safety-critical defects: the hard minority of items needs a person who can be held responsible for the call.
• The label is an opinion. Legal relevance, medical severity, or a preference between two specialist answers is expert judgement, not perception, and it should be made by the experts whose judgement the model is meant to reproduce.

Working conditions and the feedback loop

How annotators are employed affects what reaches you. Miceli and Posada studied three platforms in Venezuela and an outsourcing firm in Argentina through 55 interviews and 210 instruction documents, and found that precarious conditions and economic dependence made workers obedient to instructions. An obedient annotator does not tell you the guideline is wrong. They apply it, and the error is consistent enough to look like signal.

Wang, Prabhat and Sambasivan reached a similar conclusion from 25 annotators in Indian firms: their work “is dictated by the interests, priorities and values of others”. In Rädsch’s data, professional annotators labelled as their main source of income and spent more hours a week on it than the MTurk crowdworkers, who in turn had longer histories of labelling work.

The practical consequence is a short loop between annotators and the model team, whichever model you choose:

• a written channel for questions, answered within a working day, with each answer folded back into the guideline;
• a weekly review of disagreements and rejected items, with someone from the model team present;
• named, trained annotators on long-running work, so that what they learn about your data stays with the project;
• pay you would defend in public: Prolific, for instance, sets a minimum of £6 / $8 an hour and recommends at least £9 / $12.

The costs and risks teams forget

In-house annotation looks cheaper because an employee’s hourly cost is visible and everything around it is not. A vendor’s rate already includes management, tooling and slack, so compare it with a fully loaded internal cost, not with a salary. The rate mechanics are in our data labelling pricing guide.

Data protection settles some cases outright. Under the GDPR, an annotation vendor handling personal data is normally a processor, which needs a contract meeting Article 28; annotators outside the EEA bring in the transfer rules of Chapter V; and health, biometric and other special-category data fall under Article 9. An open crowd shows each item to unknown people on their own devices, which is hard to reconcile with any of that. A managed vendor can offer named staff, controlled devices and a known location: ask for the sub-processor list and where the annotators actually sit. Our GDPR and training data piece covers the questions to settle first.

The line items most often missing from an in-house estimate:

• Recruitment and training, including the weeks before a new annotator reaches full speed.
• Management: someone schedules batches, answers questions, runs QA and keeps the guideline current.
• Tooling: hosted seats from $23 per user a month (CVAT Online, billed yearly) or $99 a month for up to 12 users (Label Studio Starter Cloud), or a server you maintain.
• Idle time and churn: demand arrives in bursts, and a team sized for the peak waits between them or leaves.
• QA hours: gold sets, review and adjudication are real time, whoever does them.
• Compliance work: access controls, audits and contracts apply to in-house and outsourced teams alike.

A decision matrix

Score each factor for your project. If expertise or sensitivity lands in the in-house column, keep at least the guideline, the gold set and adjudication in-house, whatever happens to the volume.

Then run a paid pilot of a week with two providers, scored against the same gold items. It will tell you more than any comparison page, including this one.

A pilot that settles the question takes four measurements: accuracy against your gold items, agreement between two annotators on a double-labelled sample, seconds per label, and the number and quality of the questions annotators ask. The last is the one teams forget. A team that asks nothing in its first week has either met a perfect guideline or is not reading it closely; the questions show where your definitions are ambiguous, and they are cheaper to answer in a pilot than to discover in a trained model. Put the in-house option through the same test with the same gold items, so that you compare measured numbers rather than a quote with a salary.

Questions people ask