online #ˆÃÒ „ ,¶ÏÑÅÏ4)#§ ñ§OÕ ùýpãó˜—dZddlZddlZddlZdgZejdd¦«ZGd„d¦«ZGd„d¦«Z Gd „d ¦«Z dS) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt éNÚRobotFileParserÚ RequestRatezrequests secondscó\—eZdZdZdd„Zd„Zd„Zd„Zd„Zd„Z d „Z d „Z d „Z d „Z d „Zd„ZdS)rzs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. Úcó„—g|_g|_d|_d|_d|_| |¦«d|_dS)NFr)ÚentriesÚsitemapsÚ default_entryÚ disallow_allÚ allow_allÚset_urlÚ last_checked©ÚselfÚurls ú?/opt/alt/python-internal/lib64/python3.11/urllib/robotparser.pyÚ__init__zRobotFileParser.__init__sG€ØˆŒ ؈Œ Ø!ˆÔØ!ˆÔ؈ŒØ � Š �SÑÔÐØˆÔÐÐócó—|jS)z·Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r©rs rÚmtimezRobotFileParser.mtime%s €ðÔ Ð rcó@—ddl}| ¦«|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)Útimer)rrs rÚmodifiedzRobotFileParser.modified.s#€ð ˆ ˆ ˆ Ø ŸIšI™KœKˆÔÐÐrcó|—||_tj |¦«dd…\|_|_dS)z,Sets the URL referring to a robots.txt file.ééN)rÚurllibÚparseÚurlparseÚhostÚpathrs rr zRobotFileParser.set_url6s4€àˆŒÝ%œ|×4Ò4°SÑ9Ô9¸!¸A¸#Ô>шŒ �4”9�9�9rcó°— tj |j¦«}| ¦«}| | d¦« ¦«¦«dS#tjj $rK}|j dvrd|_ n)|j dkr|j dkrd|_ Yd}~dSYd}~dSYd}~dSYd}~dSd}~wwxYw)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i‘i“Ti�iôN) rÚrequestÚurlopenrÚreadrÚdecodeÚ splitlinesÚerrorÚ HTTPErrorÚcoder r )rÚfÚrawÚerrs rr&zRobotFileParser.read;só€ð 9Ý”×&Ò& t¤xÑ0Ô0ˆAð—&’&‘(”(ˆCØ �JŠJ�s—z’z 'Ñ*Ô*×5Ò5Ñ7Ô7Ñ 8Ô 8Ð 8Ð 8Ð 8øõŒ|Ô%ð &ð &ð &ØŒx˜:Ð%Ð%Ø$(�Ô!Ð!Ø”˜S’� S¤X°¢^ ^Ø!%�”������ð!�����ð"Ð!Ð!Ð!Ð!Ð!Ø%3 ^ ^ ^ ^ ^øøøøð &øøøs‚$A6Á6C .CÃCcóp—d|jvr|j€ ||_dSdS|j |¦«dS©NÚ*)Ú useragentsr rÚappend)rÚentrys rÚ _add_entryzRobotFileParser._add_entryHsM€Ø �%Ô"Ð "Ð "àÔ!Ð)à%*�Ô"Ð"Ð"ð*Ð)ð ŒL× Ò  Ñ &Ô &Ð &Ð &Ð &rcó<—d}t¦«}| ¦«|D�]V}|sB|dkrt¦«}d}n+|dkr%| |¦«t¦«}d}| d¦«}|dkr |d|…}| ¦«}|sŒƒ| dd¦«}t |¦«dk�r©|d ¦«