Search import wizard: reference formats, library folders and optional PDFs

Import a systematic search, with or without its PDFs, from the reference formats researchers actually export, and say exactly what was imported, omitted, matched, skipped and verified.

Rollout: delivered behind the default-off Release flag searchImportWizard (feature search-import-wizard, lifecycle Developing, permission ProjectImportSearchPolicy). With the flag off, users get the legacy "Import a search" dialog (no wizard, no new formats). The PDF step also needs bulkPdfUpload and the project's bulk PDF upload permission. The parsers and diagnostics behind it ship unflagged (D2), so the new formats are only offered inside the wizard. User guide: Import a search with the wizard. Parser detail (pipeline passes, limits, error model, per-format mapping): Reference-library import.

Delivered by the search import programme (2026-10-27 to 2026-10-30):

• Plan and slices: #4575 (plan), #4576 (S1 diagnostics), #4577 (S2 attachment references), #4579 (S3 EndNote XML), #4591 (S4 RIS L1), #4578 (S5 PubMed NBIB), #4614 (S6 Zotero RDF), #4595 (S7 web shared core), #4608 (S8 wizard), #4628 (S9 end-to-end), #4597 (S10 streaming record pipeline), on the m3e-angular foundation #4598 and served-PDF verification #4571.
• Follow-ups: #4633 and #4661 (EndNote nesting bound and namespaced-record refusal), #4644 (uncached bulk PDF upload context), #4655 (Zotero RDF shared reader), #4656 (browser upload robustness), #4657 (CSV bounds, value-free errors and the signed upload size), #4658 (upload outcomes and per-search upload state), #4662 and #4663 (#3977 data-export download and project navigation race).

Journeys

1. File only. Import a reference library file. Citations import whether or not PDFs are supplied.
2. File, then optional PDFs. When the file contains attachment references, the wizard offers an optional PDF step, previews the match, explains missing files and allows citation-only completion.
3. Library folder. Accept a dropped or selected exported library folder holding a reference file and PDF subfolders. Relative paths are preserved, candidate reference files are found (the user is asked when there is more than one), and a file and folder picker is offered where folder drag-and-drop is unavailable.
4. Standalone bulk PDF upload for an existing search stays. It shares matching, validation, progress and availability logic with journeys 2 and 3.

Formats: RIS (common format and tested exporter variants, Zotero included), EndNote XML from EndNote, Zotero and other compatible exporters, Zotero RDF (Zotero's export vocabulary only), PubMed NBIB (citations only) and CSV/TSV. Capability (whether a PDF step is offered) is detected from the file's content, never from the format name.

Decisions

All approved by the owner on 2026-10-27 unless stated.

ID	Decision
P-1	PDFs are matched after the import (studies exist only then); the browser preview before import is an estimate at most
P-2	After matching, the wizard stops on the preview; uploading never starts without confirmation
P-3	The standalone upload's overwrite policy is hidden in the wizard (a new search has no PDFs)
P-4	Closing the dialog keeps the chosen PDFs and the run in a root-level store; only a reload or closed tab loses them; the search row and a snackbar lead back to the match
P-5	"Retry PDFs" reuses the chosen folder and starts a fresh upload of the matched files only; after a reload it falls back to choosing the folder again. Amended 2026-11-03 (study-management slice 1, SM-1): Retry is offered only before any server outcome or when the server's canRetry allows it, and never re-sends an Infected upload's files (that offers "Choose another folder"). A server-reported failure shows the standalone result panel's headline, reason, access summary and partial-delivery line (bulk-pdf-upload/core/failure-presentation.ts); "No PDFs reached final storage" is the wizard's own copy only for a failure before any server outcome
P-6	Default-off Release flag searchImportWizard; flag off keeps the legacy dialog; the PDF step also needs bulkPdfUpload and the project's permission
P-7	Sentence-case button labels; test ids follow the existing page object
P-8	The wizard is built with m3e-angular components (#4598, which superseded the #4485 experiment)
P-9	"Create new stage…" opens the existing Create stage dialog (saved immediately, inactive) and the new stage is auto-selected
P-10	A file without attachment references offers only Skip at the PDF step and explains the manual route; it never promises file-name links that don't exist. RIS is not such a format: L1 attachments are kept for matching. PubMed is citation-only
P-11	Attached and verified are separate: the run is not finished until the served-PDF availability check (#4571) passes
D1	A record with several attachments: the first PDF in record order becomes PdfRelativePath; every other attachment is reported per study in the import diagnostics and the match preview, never dropped silently. The same file listed twice in one record (normaliser-equal paths) is one attachment. Several PDFs per study is a follow-up model change
D2	Parser and diagnostics changes ship unflagged (additive, parity-tested). New formats are offered only inside the wizard, behind searchImportWizard
D3	The "PDFs matched" snackbar shows anywhere in the app
D5	EndNote non-journal publication name: the book title (secondary-title) for sections and proceedings, not the record's own title
D6	Owner-supplied private library exports are diagnostic evidence only and are never committed; tests use synthetic fixtures
D7	The wizard uses the m3e-angular sidebar stepper (MatExpStepper/MatExpStep) inside mat-exp-dialog-layout (ADR-027)
D8	Parsers stream wherever possible, follow the record-reader/processor pipeline and share one error model; see the streaming record pipeline
D9	RIS's limit is raised to 256 MiB / 100,000 records, as EndNote XML and Zotero RDF; PubMed keeps 64 MiB / 50,000. The upload-signing endpoint signs the declared size as the upload's Content-Length and refuses one over the format's limit (#4657); the size stays optional until every web client sends it
EndNote	EndNote XML runs on the record pipeline; the legacy parser is deleted, and any producer is accepted when the structure is EndNote XML
#4658	(Owner, 2026-10-28) The legacy dialog's upload outcome fix ships unflagged in #4658: "success" there means the file reached SyRF, and the dialog says processing continues until the job completes

Attachment references and paths

Every parser turns attachment references into one AttachmentReference (C#, mirrored in TypeScript, both tested against one shared vector file):

Raw reference	Classification	Stored path
internal-pdf://<rest>	relative to the library's PDF root	<rest>
files/<n>/x.pdf, PDF/x.pdf, x.pdf	relative	as written, \ folded to /
file://…, C:\…, \\server\…, leading /	local absolute: refused	none; counted diagnostic
http://, https://, other schemes	remote: refused, never fetched	none; counted diagnostic
a .. segment that escapes the root	traversal: refused	none; counted diagnostic
relative, extension not .pdf	non-PDF attachment	reported, not attached

• Paths are never URL-decoded: no exporter seen writes them encoded, and decoding would corrupt a file name with a literal %. Percent signs, spaces and Unicode are kept as written.
• Matching uses PdfPathNormalizer (NFC, punctuation fold, Unicode full case fold, separator fold), so the browser preview, the agent and Project Management agree.
• Two selected files whose normalised paths collide are refused; they are never collapsed.
• One file referenced by several studies is attached to each of them, and the preview says so.

Library root

Reference paths are relative to the exported library's root, which differs by exporter (EndNote and Zotero's EndNote XML: the PDF folder; Zotero RIS and RDF: the export folder that contains files/). The browser resolves the root by matching, not by folder name:

• For each selected PDF path and each reference path, a selected path that ends with /<reference path> votes for its prefix as the root. Only references with at least one folder segment vote; a bare x.pdf never votes, because a common file name would vote for many prefixes. A reference that matches under several candidate prefixes casts no vote.
• The prefix with the most votes wins; a tie is shown to the user with the candidates.
• When no reference votes (every reference is a bare file name), each folder that directly holds selected PDFs is a candidate, and a bare name matches only a PDF directly in it. The candidate with the most matches wins; the folder the user selected wins a tie that includes it; any other tie, or no match, is shown to the user.
• Files outside the root are out of selection and never uploaded.

The #4589 root-hint follow-up retains this matching contract: no stored Base or IsInternalPdf field is needed. A format hint cannot disambiguate arbitrary user-selected parent folders or duplicate paths; the selected-file matches and existing tie handling remain authoritative. The library-folder intake specs cover EndNote PDF and RIS files/ layouts; root-resolution specs cover competing prefixes.

Import diagnostics

Parsers emit structured warnings (code, field, cause, record number). The import job stores an aggregate: one group per field and cause with the number of affected records, up to five example record numbers, and the total warning count, kept separate from any capped sample. The searches page shows the groups, expandable to the examples, and never presents a warning count as a failed-study count. It distinguishes citations imported, optional metadata omitted, attachment references missing or refused, unmatched files skipped, and PDFs failing availability. The original upload is kept and labelled "your original file, kept as uploaded"; fields omitted from studies are still in it but are not searchable as study fields. Transfers say "uploaded", never "copied".

Rollback: the parser and diagnostics changes are unflagged (D2), so their rollback is reverting the PR. Imported studies keep their data and no stored data changes shape, so no migration is needed either way.

Upload outcomes

The reference-file upload (project-detail.effects.ts addSearch$, shared by the wizard and the legacy dialog) runs one upload per search (mergeMap); a retry replaces only that search's earlier attempt. It dispatches only that search's progress, its upload error, and searchReferenceFileSent when S3 accepts the file; the import job itself always comes from the server. Upload state is kept per search (ui.search.bySearchId) and reset per search, so one dialog or run never clears another's. Resetting an active upload defers removal until it is sent or fails; settled entries are removed immediately. See upload-state ownership and background runs.

Stopping the wait removes that search's browser upload entry immediately, even if the UI previously marked it for cleanup on settlement. Closing a legacy upload in the background retains its entry until the upload settles. Late outcomes cannot recreate an entry removed by either path.

• Wizard: a failed upload ends the run in error; when S3 accepts the file the run leaves sending without waiting for a job push. After one minute of sending, Stop waiting is available, including when reopened from another project. It aborts this tab's hashing/signing/PUT subscription, clears that search's upload entry and releases the wizard (including its chosen PDFs), so another import can start. It does not undo an import the server has already accepted: the message asks the user to check their searches before trying again. Late responses cannot restore the abandoned run. The action disappears once S3 answers; it is not a cancellation action for server-side processing. Background parse/save counts in another project refresh when that project is reopened; no extra project subscription or polling is added. Import-only rows remain visible even before the project has a completed search. If signing created a job but the file never arrives, the server can keep that job in Uploading and its row says Transferring...; stopping this tab does not claim the server cancelled it. The existing delete action remains available for an import-only row, subject to its normal guards.
• Legacy dialog: a file that never reached SyRF is shown with Retry; "Transferring…", then the job's "File received. Processing…"; only a Complete job says the search was created.

Requirement-to-test map

Requirement	Tests
File-only import; optional PDF step when references exist; missing files explained	per-format fixtures without attachments; capability detection, wizard step and skip specs; E2E file-only journey
Library folder, relative paths, several candidates, picker fallback, root resolution	intake specs (one, several, none); root-vote specs (votes, multi-prefix references, ties, bare-names-only fallback); picker spec
Standalone upload shares the core	standalone dialog specs on the shared core
RIS variants and L1 attachments	RisParserParityTestBase, RisParityCorpus, RisRecordReaderTests
EndNote XML from any producer; tagged text refused	processor and orchestrator tests, Zotero-shaped fixture; server and browser refusal tests
Zotero RDF vocabulary and attachments	ZoteroRdfRecordProcessorParseImplementationTests (synthetic ZoteroRdfFixtures); RecordPipelineBoundedMemoryTests
PubMed citation-only	NBIB fixture set; no-attachment capability test
Paths: separators, Unicode, spaces, %, case, traversal, absolute, remote	shared attachment-reference vectors (C# and TypeScript)
Only matched valid PDFs uploaded; uploaded, attached and verified distinct; safe retry	match preview specs; wizard result and upload-outcome specs; E2E
Stop waiting after a minute of sending; only that transfer stops; late replies cannot revive it	real-effect upload-outcome specs (hanging PUT/signature, retry, server acceptance); concurrent/hash effect specs; rendered wizard action
Field-level diagnostics, aggregated, total vs sample	aggregation tests; DTO mapping; searches page spec
Very large libraries, bounded memory, one error model	RecordPipelineBoundedMemoryTests, RecordPipelineImportTests, ImportErrorCollectorTests
Isolated end-to-end journeys (library folder, file-only, later standalone upload, PDF retry, flag off)	search-import-wizard-library.spec.ts, search-import-wizard-file-only.spec.ts and search-import-wizard-flag.spec.ts (e2e/tests/, Bulk PDF lane where PDFs are involved)

Known gaps and follow-ups

• Several PDFs per study (D1): the first is attached, the rest reported.
• Local absolute and remote attachment references are refused, not fetched or read.
• EndNote tagged text, EndNote library binaries (.enl, .enlx), PubMed XML and arbitrary RDF are refused with guidance.
• EndNote XML exporters other than EndNote and Zotero import, but only those two are tested with real exports.
• Follow-ups are tracked on #4589, including making the signed upload size required after the client compatibility floor and a real S3 check that the signature enforces it are proved.