AI-Assisted Abstract Screening: How It Works and When to Trust It

Why Abstract Screening Is the Bottleneck

A well-built systematic review search is deliberately over-inclusive. Missing an eligible study is a worse failure than retrieving irrelevant ones, so search strategies across databases such as PubMed are tuned for sensitivity rather than precision. The predictable consequence is that a search returning several thousand records in order to find a few dozen eligible studies is entirely normal, not a sign that anything went wrong.

Convention, reflected in the Cochrane Handbook, is that at least two reviewers screen every title and abstract independently, with disagreements resolved by discussion or a third reviewer. The redundancy exists because individual reviewers make errors: attention fades over long screening sessions, interpretations of criteria drift, and a study excluded by mistake vanishes silently from the review. Dual independent screening is an error-catching mechanism, not a formality.

Multiply thousands of records by two readers, each making a considered judgment per abstract, and screening becomes one of the most time-consuming stages between running the search and reading full texts. This is the stage that AI-assisted screening targets, and it is worth understanding precisely what these tools do before deciding how far to trust them.

How LLM-Based Screening Actually Works

Earlier screening automation was mostly machine-learning classification with active learning: the tool ranked records by predicted relevance, a human screened from the top, and the model retrained as labels accumulated. Large language model (LLM) screening works differently. The model reads each title and abstract directly against your written eligibility criteria, with no project-specific training labels required.

Eligibility criteria become a structured prompt

Your PICO elements (population, intervention, comparator, outcomes) plus design and language restrictions are translated into explicit instructions: each criterion becomes a condition the model checks against the abstract text. The precision of that translation matters more than anything else in the pipeline. If your criteria say "adults" without specifying how to treat mixed-age samples, the model will resolve that ambiguity inconsistently, exactly as two human reviewers would. Many teams find that writing criteria precisely enough for a model surfaces ambiguities that would otherwise emerge as mid-screen disagreements.

Three-way decisions with a stated rationale

For each record, the output is a decision, include, exclude, or uncertain, plus a rationale keyed to specific criteria, for example "exclude: no comparator arm reported." The rationale serves two purposes: it makes every decision auditable, and it lets you catch misreadings quickly, because a wrong rationale is easier to spot than a wrong bare label. Screening tools such as AutoEvidence structure their output this way for exactly that reason. The uncertain category is not a weakness of the approach; it is the honest category. A system that never says uncertain should worry you more than one that says it often.

The Dual-Pass Pattern and Cohen's Kappa

The human convention of two independent reviewers maps naturally onto AI screening: run two passes, each blind to the other's decisions — ideally with different models or differently framed prompts, because two runs of the same model share its systematic blind spots, so their agreement is weaker evidence than agreement between two independent humans. Records both passes include move forward, records both passes exclude are set aside, and everything else, discordant decisions plus anything flagged uncertain, goes to a human. This mirrors the logic of dual human review: the point was never that any single reviewer is reliable, but that independent errors rarely coincide.

Agreement between passes is conventionally measured with Cohen's kappa, which corrects raw agreement for the agreement expected by chance. The correction matters in screening because the classes are wildly imbalanced: when the large majority of records are excludes, two screeners who exclude nearly everything will show high raw agreement even if they agree on almost nothing else. Widely used rule-of-thumb labels for kappa look like this:

Treat these bands honestly for what they are: conventions, not statistically derived thresholds. Kappa is also sensitive to prevalence, and when eligible records are rare, the usual situation in abstract screening, kappa can look mediocre despite very high raw agreement. Report both numbers, and read kappa as a diagnostic signal rather than a pass/fail test. In practice, low agreement between two AI passes usually means the eligibility criteria are ambiguous. The productive response is to sharpen the criteria and re-run, not to average over the disagreement.

Where AI Screening Fails

The failure modes of LLM screening are specific and mostly predictable. Knowing them in advance lets you design the workflow so that failures are caught rather than absorbed.

• ·Ambiguous abstracts. Abstracts routinely omit the very details your criteria depend on: study design, comparator, age range, outcome timing. No reader, human or model, can extract information that is not there. These records belong in the uncertain pile, where full-text retrieval settles the question. A model forced into a binary include/exclude decision on an ambiguous abstract is guessing.
• ·Missing abstracts. Older database records, conference proceedings, and some registry entries carry a title and nothing else. A registration on ClinicalTrials.gov is structured data, not an abstract, and title-only screening is guesswork regardless of who performs it. Route these records to full-text retrieval or human judgment rather than letting a model manufacture confidence from a title.
• ·Non-English nuance. If your review includes non-English studies, and it generally should, since language restrictions can introduce bias, be cautious about letting AI make silent exclusion decisions on non-English abstracts. Method-specific terminology and reporting conventions vary across languages, and a misread design descriptor is enough to exclude an eligible trial.
• ·Criteria drift. Human teams quietly refine their interpretation of criteria as edge cases accumulate, which means record 200 and record 2,000 are not judged by quite the same standard. A fixed prompt does not drift, which is a genuine consistency advantage, but it cuts both ways: when you deliberately revise criteria mid-screen, you must re-run every previously screened record under the new criteria. With AI this re-run is cheap; skipping it silently reintroduces the inconsistency you were trying to avoid.
• ·Plausible but wrong rationales. A model can produce a fluent, confident rationale that misreads the abstract, mistaking a protocol paper for a results paper, or a secondary analysis for a new trial. Fluency is not accuracy. This is precisely why rationales must be spot-checked rather than admired.

Why Human Adjudication Remains Mandatory

Screening errors are asymmetric. A false include costs a few minutes at full-text review, where it will be caught by the normal workflow. A false exclude disappears without a trace: no downstream step re-examines excluded abstracts, so the error propagates straight into the synthesis and ultimately into certainty-of-evidence judgments under frameworks such as GRADE. Any sane configuration of AI screening therefore calibrates toward inclusion, accepting more full-text work in exchange for fewer silent losses.

The practical rules follow from that asymmetry. Every uncertain record and every discordant record between passes gets a human decision, without exception. Beyond that, audit a random sample of the concordant excludes, the records both passes agreed to remove, before trusting the remainder; agreement between two automated passes is evidence of consistency, not proof of correctness. And remember that humans make the final eligibility call on every included study at full-text stage regardless. The right mental model is an additional screener whose work is systematically checked, not an oracle whose output is accepted.

Reporting AI Involvement in Your Methods Section

Transparency is what separates defensible AI-assisted screening from an unauditable shortcut. The PRISMA 2020 statement (Page MJ et al., BMJ 2021;372:n71) already anticipates automation: the selection-process item asks for details of any automation tools used, and the flow diagram includes a slot for records excluded by automation. Registering your review prospectively in PROSPERO with the planned AI workflow described, before screening begins, protects you from the suspicion that the method was chosen to fit the results.

At minimum, your methods section and supplementary material should allow a reader to reconstruct what the AI did and what humans did:

• ·The model and version used, and the dates screening was performed
• ·The exact prompt, including the full eligibility criteria text, as supplementary material
• ·The decision rules: how many passes, what happened to uncertain records, and what triggered human review
• ·Agreement statistics between passes, both raw agreement and Cohen's kappa
• ·The human oversight performed: who adjudicated disagreements, and what sample of exclusions was audited
• ·Counts for the PRISMA flow diagram, including any records excluded by automation

Platforms built for systematic review work, AutoEvidence among them, log per-record decisions and rationales as an exportable audit trail, which turns this reporting into a mechanical export rather than an after-the-fact reconstruction. Reporting expectations for AI-assisted methods are still evolving; the EQUATOR Network is the place to check for relevant reporting guidelines as they mature.

A Practical Checklist Before You Trust the Output

1. 1.Write eligibility criteria explicit enough that a new co-reviewer could apply them from the text alone, then pilot the prompt on a set of records you have already screened by hand.
2. 2.Run two independent AI passes, blind to each other, and compute raw agreement and Cohen's kappa.
3. 3.Send every uncertain and every discordant record to human adjudication, without exception.
4. 4.Audit a random sample of concordant excludes before accepting the rest.
5. 5.Freeze the criteria once screening starts, or re-run all previously screened records whenever they change.
6. 6.Log every decision and rationale, report per PRISMA 2020, and describe the planned workflow in your PROSPERO registration.

AI-assisted screening earns trust the same way a new human reviewer does: through calibration against known decisions, measured agreement, and spot audits of its work. Trust is a property of the workflow you build around the model, not a property of the model itself. Built that way, AI screening compresses the most tedious stage of a systematic review while keeping every consequential judgment where it belongs, with the review team.