Daily production increases the chance that a copied draft or incorrect file lands under a new route. A duplicate-body scan catches that mechanical failure before publication.
Normalize conservatively
Remove frontmatter, formatting tokens, and known shared calls to action, but retain headings and substantive prose. Over-normalizing can make distinct articles look identical or hide a copied section.
Compare exact and near matches
Start with a cryptographic hash for exact duplicates. Then use paragraph overlap or similarity review to flag near matches. Treat the score as a prompt for editorial inspection, not an automatic deletion rule.
Review intentional reuse
Definitions, legal language, and process templates may repeat appropriately. Record the approved shared component and focus review on whether the core answer, examples, and decision guidance are unique.
Stop on unexplained matches
If two canonical routes carry the same substantive body, pause the batch. Identify the correct source, preserve existing pages, and repair the candidate before rebuilding the route inventory.
