Focused crawling refers to a crawler that focuses on particularly relevant and current websites. Accordingly, the focused crawler is in contrast to the universal search engine, whose goal is to crawl as many websites represented on the Internet as possible as well as to index them. The crawler limits itself to a specific niche on the web and indexes and crawls this niche in particular detail. Non-topic websites are ignored and accordingly not crawled.
In the case of focused crawling, websites are indexed and crawled preferentially if they are considered to be particularly important on the WWW. If these websites are kept particularly up-to-date, they are indexed by the Google crawler significantly more often than those that are rarely or not at all updated.
Goals for search engine optimization
In search engine optimization, focused crawling naturally aims to achieve a better ranking. Webmasters are therefore faced with the challenge of regularly publishing unique and up-to-date content in order to receive special attention from the focused crawler. If pages are updated less frequently, this can result in a reduced frequency of visits by the crawler and thus also a poorer ranking.
Thematically relevant sub-pages:
