net.nutch.parse.html
Class RobotsMetaProcessor

java.lang.Object
  net.nutch.parse.html.RobotsMetaProcessor

public class RobotsMetaProcessor

extends Object

Class for parsing META Directives from DOM trees. This class currently handles Robots META directives (all, none, nofollow, noindex), finding BASE HREF tags, and HTTP-EQUIV no-cache instructions.

Nested Class Summary

static class

RobotsMetaProcessor.RobotsMetaIndicator Utility class with indicators for the robots directives "noindex" and "nofollow", and HTTP-EQUIV/no-cache

Constructor Summary

RobotsMetaProcessor()

Method Summary

static void

getRobotsMetaDirectives(RobotsMetaProcessor.RobotsMetaIndicator robotsMeta, Node node, URL currURL) Sets the indicators in robotsMeta to appropriate values, based on any META tags found under the given node.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RobotsMetaProcessor

public RobotsMetaProcessor()

Method Detail

getRobotsMetaDirectives

public static final void getRobotsMetaDirectives(RobotsMetaProcessor.RobotsMetaIndicator robotsMeta,
                                                 Node node,
                                                 URL currURL)

Sets the indicators in robotsMeta to appropriate values, based on any META tags found under the given node.