0 #-ˆl–äY>””†Ù”â�nãa¸&TÅ£Ø ƒu‘;׿ÕŸ-a ´iÐ$ã@s\dZddlZddlZddlZdgZe dd¡ZGdd„dƒZGdd„dƒZ Gd d „d ƒZ dS) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt éNÚRobotFileParserÚ RequestRatezrequests secondsc@sreZdZdZddd„Zdd„Zdd„Zd d „Zd d „Zd d„Z dd„Z dd„Z dd„Z dd„Z dd„Zdd„ZdS)rzs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. ÚcCs2g|_g|_d|_d|_d|_| |¡d|_dS)NFr)ÚentriesÚsitemapsÚ default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked©ÚselfÚurl©rú*/usr/lib64/python3.9/urllib/robotparser.pyÚ__init__s zRobotFileParser.__init__cCs|jS)z·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r ©r rrrÚmtime%szRobotFileParser.mtimecCsddl}| ¡|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)Útimer )r rrrrÚmodified.szRobotFileParser.modifiedcCs&||_tj |¡dd…\|_|_dS)z,Sets the URL referring to a robots.txt file.ééN)rÚurllibÚparseÚurlparseÚhostÚpathr rrrr 6szRobotFileParser.set_urlc Csˆztj |j¡}WnTtjjyf}z8|jdvr8d|_n|jdkrR|jdkrRd|_WYd}~n&d}~00|  ¡}|  |  d¡  ¡¡dS)z4Reads the robots.txt URL and feeds it to the parser.)i‘i“Ti�iôNzutf-8) rZrequestÚurlopenrÚerrorÚ HTTPErrorÚcoderr ÚreadrÚdecodeÚ splitlines)r ÚfÚerrÚrawrrrr!;s zRobotFileParser.readcCs,d|jvr|jdur(||_n |j |¡dS©NÚ*)Ú useragentsrrÚappend)r ÚentryrrrÚ _add_entryHs  zRobotFileParser._add_entrycCsPd}tƒ}| ¡|D�]}|sP|dkr4tƒ}d}n|dkrP| |¡tƒ}d}| d¡}|dkrn|d|…}| ¡}|s|q| dd¡}t|ƒdkr|d ¡ ¡|d<tj   |d ¡¡|d<|ddkrú|dkrä| |¡tƒ}|j   |d¡d}q|ddk�r.|dk�r6|j   t|dd ƒ¡d}q|dd k�rb|dk�r6|j   t|dd ƒ¡d}q|dd k�r |dk�r6|d ¡ ¡�ršt|dƒ|_d}q|dd k�r|dk�r6|d d¡}t|ƒdk�r|d ¡ ¡�r|d ¡ ¡�rtt|dƒt|dƒƒ|_d}q|ddkr|j  |d¡q|dk�rL| |¡dS)z”Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. rréú#Nú:z user-agentZdisallowFZallowTz crawl-delayz request-rateú/Zsitemap)ÚEntryrr,ÚfindÚstripÚsplitÚlenÚlowerrrÚunquoter)r*Ú rulelinesÚRuleLineÚisdigitÚintÚdelayrÚreq_rater)r ÚlinesÚstater+ÚlineÚiZnumbersrrrrQsj              ÿ  zRobotFileParser.parsecCs |jr dS|jrdS|jsdStj tj |¡¡}tj dd|j|j |j |j f¡}tj  |¡}|sfd}|j D]}| |¡rl| |¡Sql|jrœ|j |¡SdS)z=using the parsed robots.txt decide if useragent can fetch urlFTrr0)rr r rrrr7Ú urlunparserÚparamsÚqueryÚfragmentÚquoterÚ applies_toÚ