What is website crawling?
Crawling begins from a site or path and discovers additional pages within the configured scope. Those pages can then be processed into cleaner representations for downstream use.
Crawl vs scrape
Choose scrape for one known page. Choose crawl when discovery across a documentation site, knowledge base, content hub, or other multi-page source is part of the job.
Control crawl scope
Keep crawls focused. Limit paths, page counts, and unwanted sections whenever the API options and target site structure allow it. Smaller, purposeful crawls are easier to validate and maintain.
Crawling for RAG
Documentation, support centers, and knowledge hubs are common sources for retrieval systems. Clean content should still be deduplicated, chunked, metadata-tagged, and refreshed according to the product's needs.