Class Web.Crawler.Crawler

Method create

Web.Crawler.Crawler Web.Crawler.Crawler(Queue _queue, function(:void) _page_cb, function(:void) _error_cb, function(:void) _done_cb, function(:void) _prepare_cb, string|array(string)|Standards.URI|array(Standards.URI) start_uri, mixed ... _args)

Parameter _page_cb

function called when a page is retreived. Arguments are: Standards.URI uri, mixed data, mapping headers, mixed ... args. should return an array containing additional links found within data that will be analyzed for insertion into the crawler queue (assuming they are allowed by the allow/deny rulesets.

Parameter _error_cb

function called when an error is received from a server. Arguments are: Standards.URI real_uri, int status_code, mapping headers, mixed ... args. Returns void.

Parameter _done_cb

function called when crawl is complete. Accepts mixed ... args and returns void.

Parameter _prepare_cb

argument called before a uri is retrieved. may be used to alter the request. Argument is Standards.URI uri. Returns array with element 0 of Standards.URI uri, element 1 is a header mapping for the outgoing request.

Parameter start_uri

location to start the crawl from.

Parameter _args

optional arguments sent as the last argument to the callback functions.