Definition of web crawler
: a computer program that automatically and systematically searches web pages for certain keywords
- Each search engine has its own proprietary computation (called an "algorithm") that ranks websites for each keyword or combination of keywords. These algorithms use Web crawlers … that collect data from your website to determine where it ranks with respect to search terms.
- —Julie Brinton
