fjffk简单的咖啡店科开发2方法2
环滁皆山也。其西南诸峰，林壑尤美，望之蔚然而深秀者，琅琊也。山行六七里，渐闻水声潺潺，而泻出于两峰之间者，酿泉也
???????????????????????????????
??????????????/?????????????????
............/...................
.........................../...
???????????????????????????????
>>>>>>>>>>>>>/>>>>>>>>>>>>>>>>>
ÿØÿà JFIF ÿÛ „ ( % !1!%)+//. 383,7(-.+
-% %-////---/-.+/--+------/------/--0+--/-/-----.-----ÿÀ ¥2" ÿÄ ÿÄ J ! 1AQ"aq2‘#BR‚¡ÁÑ3br’¢±Âð$CSƒ²á4c“%DsÓñÿÄ ÿÄ * !1AQa‘"2q3±ð#b¡ÿÚ ? ¼QxJQa Íuò¸Zö Úü8,Ð Úú
"SS n<rçù–´âE—^ªBÖ9À\†¸ÔÁTÃÛ5
ëd´³Í#Ý;Þ38œî ¶ H£M:wÎ3…³…âpÔF&‚FK¸9„â4àGEõªfÿ ‘ñ(ß wpŽF|È¥ù®häðÍѶ¹‘[ÒinÙW¶ùñ Y˜Q{›K"išÒ[Ú8žë\F¹@-? v"ÔU”,ìöžkÿ {I‡£šÍ?e
ríV
?>
.........................................
.............................................................................
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>/>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
dsdfREADME.md 0000644 00000006451 15266562307 0006045 0 ustar 00 # Parse `robots.txt`, `robots` meta and headers
[](https://packagist.org/packages/spatie/robots-txt)
[](https://github.com/spatie/robots-txt/actions/workflows/run-tests.yml)
[](https://scrutinizer-ci.com/g/spatie/robots-txt)
[](https://packagist.org/packages/spatie/robots-txt)
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers.
## Support us
[
](https://spatie.be/github-ad-click/robots-txt)
We invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).
We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).
## Installation
You can install the package via composer:
```bash
composer require spatie/robots-txt
```
## Usage
``` php
$robots = Spatie\Robots\Robots::create();
$robots->mayIndex('https://www.spatie.be/nl/admin');
$robots->mayFollowOn('https://www.spatie.be/nl/admin');
$robotsTxt = new RobotsTxt('
User-agent: *
Disallow: /admin
Crawl-delay: 1.5
');
$robotsTxt->allows('/admin', 'google'); // false
$robotsTxt->whyDisallows('/admin', 'google')[0]->userAgent; // '*'
$robotsTxt->crawlDelay('/admin', '*'); // '1.5'
```
You can also specify a user agent:
``` php
$robots = Spatie\Robots\Robots::create('UserAgent007');
```
By default, `Robots` will look for a `robots.txt` file on `https://host.com/robots.txt`.
Another location can be specified like so:
``` php
$robots = Spatie\Robots\Robots::create()
->withTxt('https://www.spatie.be/robots-custom.txt');
$robots = Spatie\Robots\Robots::create()
->withTxt(__DIR__ . '/public/robots.txt');
```
### Testing
``` bash
composer test
```
### Changelog
Please see [CHANGELOG](CHANGELOG.md) for more information what has changed recently.
## Contributing
Please see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.
## Security Vulnerabilities
Please review [our security policy](../../security/policy) on how to report security vulnerabilities.
## Postcardware
You're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.
Our address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium.
We publish all received postcards [on our company website](https://spatie.be/en/opensource/postcards).
## Credits
- [Brent Roose](https://github.com/brendt)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.
src/RobotsMeta.php 0000644 00000003477 15266562307 0010152 0 ustar 00 robotsMetaTagProperties = $this->findRobotsMetaTagProperties($html);
}
public function mayIndex(): bool
{
return ! $this->noindex();
}
public function mayFollow(): bool
{
return ! $this->nofollow();
}
public function noindex(): bool
{
return $this->robotsMetaTagProperties['noindex'] ?? false;
}
public function nofollow(): bool
{
return $this->robotsMetaTagProperties['nofollow'] ?? false;
}
#[ArrayShape(['noindex' => 'bool', 'nofollow' => 'bool'])]
protected function findRobotsMetaTagProperties(string $html): array
{
$metaTagLine = $this->findRobotsMetaTagLine($html);
return [
'noindex' => $metaTagLine
? strpos(strtolower($metaTagLine), 'noindex') !== false
: false,
'nofollow' => $metaTagLine
? strpos(strtolower($metaTagLine), 'nofollow') !== false
: false,
];
}
protected function findRobotsMetaTagLine(string $html): ?string
{
if (preg_match('/\/mis', $html, $matches)) {
return $matches[0];
}
return null;
}
}
src/Disallow.php 0000644 00000000347 15266562307 0007642 0 ustar 00 includeGlobalGroup = false;
return $this;
}
public function includeGlobalGroup(): self
{
$this->includeGlobalGroup = true;
return $this;
}
public function withPartialMatches(): self
{
$this->matchExactly = false;
return $this;
}
public function exactMatchesOnly(): self
{
$this->matchExactly = true;
return $this;
}
public static function readFrom(string $source): self
{
$content = @file_get_contents($source);
return new self($content !== false ? $content : '');
}
public function __construct(public string $content)
{
$this->disallowsPerUserAgent = $this->getDisallowsPerUserAgent($content);
$this->allowsPerUserAgent = $this->getAllowsPerUserAgent($content);
$this->crawlDelaysPerUserAgent = $this->getCrawlDelaysPerUserAgent($content);
}
public static function create(string $source): self
{
if (
strpos($source, 'http') !== false
&& strpos($source, 'robots.txt') !== false
) {
return self::readFrom($source);
}
return new self($source);
}
public function allows(string $url, ?string $userAgent = '*'): bool
{
$requestUri = '';
$parts = parse_url($url);
if ($parts !== false) {
if (isset($parts['path'])) {
$requestUri .= $parts['path'];
}
if (isset($parts['query'])) {
$requestUri .= '?'.$parts['query'];
} elseif ($this->hasEmptyQueryString($url)) {
$requestUri .= '?';
}
}
$disallowsPerUserAgent = $this->includeGlobalGroup
? $this->disallowsPerUserAgent
: array_filter($this->disallowsPerUserAgent, fn ($key) => $key !== '*', ARRAY_FILTER_USE_KEY);
$allowsPerUserAgent = $this->includeGlobalGroup
? $this->allowsPerUserAgent
: array_filter($this->allowsPerUserAgent, fn ($key) => $key !== '*', ARRAY_FILTER_USE_KEY);
$normalizedUserAgent = strtolower(trim($userAgent ?? ''));
$disallows = $this->matchExactly
? $this->getDisallowsExactly($normalizedUserAgent, $disallowsPerUserAgent)
: $this->getDisallowsContaining($normalizedUserAgent, $disallowsPerUserAgent);
$allows = $this->matchExactly
? $this->getDisallowsExactly($normalizedUserAgent, $allowsPerUserAgent)
: $this->getDisallowsContaining($normalizedUserAgent, $allowsPerUserAgent);
if ($this->pathIsExplicitAllowed($url, $allows)) {
return true;
}
$isDenied = $this->pathIsDenied($requestUri, $disallows);
if ($isDenied) {
$isAllowedWeight = $this->pathMatchWeight($requestUri, $allows);
if ($isAllowedWeight === 0) {
return false;
}
$isDisallowedWeight = $this->pathMatchWeight($requestUri, $disallows);
return $isAllowedWeight > $isDisallowedWeight;
}
return ! $isDenied;
}
/**
* @return Disallow[]|null Null if path is allowed, or a list of reasons if path is disallowed
*/
public function whyDisallows(string $path, string $userAgent): ?array
{
if ($this->allows($path, $userAgent)) {
return null;
}
$reasons = $this->getDisallows($path, $userAgent);
if ($userAgent !== '*') {
$newDisallows = $this->getDisallows($path, '*');
$reasons = array_merge($reasons, $newDisallows);
}
return $reasons;
}
/**
* @return Disallow[]
*/
protected function getDisallows(string $path, string $userAgent): array
{
$reasons = [];
foreach (array_keys($this->disallowsPerUserAgent[$userAgent] ?? []) as $disallowedPath) {
if (str_starts_with($path, $disallowedPath)) {
$reasons[] = new Disallow($userAgent, $disallowedPath);
}
}
return $reasons;
}
public function crawlDelay(string $userAgent = '*'): ?string
{
$crawlDelaysPerUserAgent = $this->includeGlobalGroup
? $this->crawlDelaysPerUserAgent
: array_filter($this->crawlDelaysPerUserAgent, fn ($key) => $key !== '*', ARRAY_FILTER_USE_KEY);
$normalizedUserAgent = strtolower(trim($userAgent ?? ''));
if (isset($crawlDelaysPerUserAgent[$normalizedUserAgent])) {
return $crawlDelaysPerUserAgent[$normalizedUserAgent];
} elseif (isset($crawlDelaysPerUserAgent['*'])) {
return $crawlDelaysPerUserAgent['*'];
}
return null;
}
protected function pathMatchWeight(string $requestUri, array $itemsPerUseragent): int
{
$weight = 0;
foreach ($itemsPerUseragent as $item => $strlen) {
$startsWith = str_starts_with($requestUri, $item);
if ($startsWith && $strlen > $weight) {
$weight = (int) $strlen;
}
}
return $weight;
}
protected function getDisallowsExactly(string $userAgent, array $disallowsPerUserAgent): array
{
return $disallowsPerUserAgent[$userAgent] ?? $disallowsPerUserAgent['*'] ?? [];
}
protected function getDisallowsContaining(string $userAgent, array $disallowsPerUserAgent): array
{
$disallows = [];
foreach ($disallowsPerUserAgent as $userAgentKey => $disallowsPerUserAgentKey) {
$contains = strpos($userAgent, $userAgentKey) !== false;
if ($contains || $userAgentKey === '*') {
$disallows = [...$disallows, ...$disallowsPerUserAgentKey];
}
}
return $disallows;
}
protected function pathIsExplicitAllowed(string $requestUri, array $allows): bool
{
return in_array($requestUri, array_keys($allows));
}
protected function pathIsDenied(string $requestUri, array $disallows): bool
{
foreach (array_keys($disallows) as $disallow) {
if ($disallow === '') {
continue;
}
$stopAtEndOfString = false;
if ($disallow[-1] === '$') {
// if the pattern ends with a dollar sign, the string must end there
$disallow = substr($disallow, 0, -1);
$stopAtEndOfString = true;
}
// convert to regexp
$disallowRegexp = preg_quote($disallow, '/');
// the pattern must start at the beginning of the string...
$disallowRegexp = '^'.$disallowRegexp;
// ...and optionally stop at the end of the string
if ($stopAtEndOfString) {
$disallowRegexp .= '$';
}
// replace (preg_quote'd) stars with an eager match
$disallowRegexp = str_replace('\\*', '.*', $disallowRegexp);
// enclose in delimiters
$disallowRegexp = '/'.$disallowRegexp.'/';
if (preg_match($disallowRegexp, $requestUri) === 1) {
return true;
}
}
return false;
}
/**
* Checks for an empty query string.
*
* This works around the fact that parse_url() will not set the 'query' key when the query string is empty.
* See: https://bugs.php.net/bug.php?id=78385
*/
protected function hasEmptyQueryString(string $url): bool
{
if ($url === '') {
return false;
}
if ($url[-1] === '?') { // ends with ?
return true;
}
if (strpos($url, '?#') !== false) { // empty query string, followed by a fragment
return true;
}
return false;
}
protected function getDisallowsPerUserAgent(string $content): array
{
$lines = explode(PHP_EOL, $content);
$lines = array_filter($lines);
$disallowsPerUserAgent = [];
$currentUserAgents = [];
$treatAllowDisallowLine = false;
foreach ($lines as $line) {
if ($this->isComment($line)) {
continue;
}
if ($this->isEmptyLine($line)) {
continue;
}
if ($this->isUserAgentLine($line)) {
if ($treatAllowDisallowLine) {
$treatAllowDisallowLine = false;
$currentUserAgents = [];
}
$disallowsPerUserAgent[$this->parseUserAgent($line)] = [];
$currentUserAgents[] = &$disallowsPerUserAgent[$this->parseUserAgent($line)];
continue;
}
if ($this->isDisallowLine($line)) {
$treatAllowDisallowLine = true;
}
if ($this->isAllowLine($line)) {
$treatAllowDisallowLine = true;
continue;
}
$disallowUrl = $this->parseDisallow($line);
foreach ($currentUserAgents as &$currentUserAgent) {
$currentUserAgent[$disallowUrl] = strlen($disallowUrl);
}
}
return $disallowsPerUserAgent;
}
protected function getAllowsPerUserAgent(string $content): array
{
$lines = explode(PHP_EOL, $content);
$lines = array_filter($lines);
$allowsPerUserAgent = [];
$currentUserAgents = [];
$treatAllowDisallowLine = false;
foreach ($lines as $line) {
if ($this->isComment($line)) {
continue;
}
if ($this->isEmptyLine($line)) {
continue;
}
if ($this->isUserAgentLine($line)) {
if ($treatAllowDisallowLine) {
$treatAllowDisallowLine = false;
$currentUserAgents = [];
}
$allowsPerUserAgent[$this->parseUserAgent($line)] = [];
$currentUserAgents[] = &$allowsPerUserAgent[$this->parseUserAgent($line)];
continue;
}
if ($this->isDisallowLine($line)) {
$treatAllowDisallowLine = true;
continue;
}
if ($this->isAllowLine($line)) {
$treatAllowDisallowLine = true;
}
$allowUrl = $this->parseAllow($line);
foreach ($currentUserAgents as &$currentUserAgent) {
$currentUserAgent[$allowUrl] = strlen($allowUrl);
}
}
$allowsPerUserAgent = array_filter($allowsPerUserAgent, function ($item) {
return count($item) > 0;
});
return $allowsPerUserAgent;
}
protected function getCrawlDelaysPerUserAgent(string $content): array
{
$lines = explode(PHP_EOL, $content);
$lines = array_filter($lines);
$crawlDelaysPerUserAgent = [];
$currentUserAgents = [];
$isUserAgentListGoing = false;
foreach ($lines as $line) {
if ($this->isComment($line)) {
continue;
}
if ($this->isEmptyLine($line)) {
continue;
}
if ($this->isUserAgentLine($line)) {
if (! $isUserAgentListGoing) {
$isUserAgentListGoing = true;
$currentUserAgents = [];
}
$userAgent = $this->parseUserAgent($line);
$currentUserAgents[] = $userAgent;
continue;
}
$isUserAgentListGoing = false;
if ($this->isCrawlDelayLine($line)) {
$crawlDelay = $this->parseCrawlDelay($line);
foreach ($currentUserAgents as $currentUserAgent) {
$crawlDelaysPerUserAgent[$currentUserAgent] = $crawlDelay;
}
}
}
return $crawlDelaysPerUserAgent;
}
protected function isComment(string $line): bool
{
return strpos(trim($line), '#') === 0;
}
protected function isEmptyLine(string $line): bool
{
return trim($line) === '';
}
protected function isUserAgentLine(string $line): bool
{
return strpos(trim(strtolower($line)), 'user-agent') === 0;
}
protected function parseUserAgent(string $line): string
{
return trim(str_replace('user-agent', '', strtolower(trim($line))), ': ');
}
protected function parseDisallow(string $line): string
{
return trim(substr_replace(trim($line), '', 0, 8), ': ');
}
protected function parseAllow(string $line): string
{
return trim(substr_replace(trim($line), '', 0, 6), ': ');
}
protected function parseCrawlDelay(string $line): string
{
return trim(substr_replace(trim($line), '', 0, 11), ': ');
}
protected function isDisallowLine(string $line): string
{
return trim(substr(str_replace(' ', '', strtolower(trim($line))), 0, 8), ': ') === 'disallow';
}
protected function isAllowLine(string $line): string
{
return trim(substr(str_replace(' ', '', strtolower(trim($line))), 0, 6), ': ') === 'allow';
}
protected function isCrawlDelayLine(string $line): string
{
return trim(substr(str_replace(' ', '', strtolower(trim($line))), 0, 11), ': ') === 'crawl-delay';
}
}
src/Robots.php 0000644 00000004265 15266562307 0007337 0 ustar 00 robotsTxt = $source;
} elseif (is_string($source)) {
$this->robotsTxt = RobotsTxt::readFrom($source);
} else {
$this->robotsTxt = null;
}
}
public function withTxt(RobotsTxt|string $source): self
{
$this->robotsTxt = $source instanceof RobotsTxt
? $source
: RobotsTxt::readFrom($source);
return $this;
}
public static function create(?string $userAgent = null, ?string $source = null): self
{
return new self($userAgent, $source);
}
public function mayIndex(string $url, ?string $userAgent = null): bool
{
$userAgent = $userAgent ?? $this->userAgent;
$robotsTxt = $this->robotsTxt ?? RobotsTxt::create($this->createRobotsUrl($url));
if (! $robotsTxt->allows($url, $userAgent)) {
return false;
}
$content = $robotsTxt->content;
if ($content === false) {
throw new InvalidArgumentException("Could not read url `{$url}`");
}
return RobotsMeta::create($content)->mayIndex()
&& RobotsHeaders::create($http_response_header ?? [])->mayIndex();
}
public function mayFollowOn(string $url): bool
{
$content = @file_get_contents($url);
if ($content === false) {
throw new InvalidArgumentException("Could not read url `{$url}`");
}
return
RobotsMeta::create($content)->mayFollow()
&& RobotsHeaders::create($http_response_header ?? [])->mayFollow();
}
protected function createRobotsUrl(string $url): string
{
$robotsUrl = parse_url($url, PHP_URL_SCHEME).'://'.parse_url($url, PHP_URL_HOST);
if ($port = parse_url($url, PHP_URL_PORT)) {
$robotsUrl .= ":{$port}";
}
return "{$robotsUrl}/robots.txt";
}
}
src/RobotsHeaders.php 0000644 00000005752 15266562307 0010635 0 ustar 00 robotHeadersProperties = $this->parseHeaders($headers);
}
public function mayIndex(string $userAgent = '*'): bool
{
return $this->none($userAgent) ? false : ! $this->noindex($userAgent);
}
public function mayFollow(string $userAgent = '*'): bool
{
return $this->none($userAgent) ? false : ! $this->nofollow($userAgent);
}
public function noindex(string $userAgent = '*'): bool
{
return
$this->robotHeadersProperties[$userAgent]['noindex']
?? $this->robotHeadersProperties['*']['noindex']
?? false;
}
public function nofollow(string $userAgent = '*'): bool
{
return
$this->robotHeadersProperties[$userAgent]['nofollow']
?? $this->robotHeadersProperties['*']['nofollow']
?? false;
}
public function none(string $userAgent = '*'): bool
{
return
$this->robotHeadersProperties[$userAgent]['none']
?? $this->robotHeadersProperties['*']['none']
?? false;
}
protected function parseHeaders(array $headers): array
{
$robotHeaders = $this->filterRobotHeaders($headers);
return array_reduce($robotHeaders, function (array $parsedHeaders, $header) {
$header = $this->normalizeHeaders($header);
$headerParts = explode(':', $header);
$userAgent = count($headerParts) === 3
? trim($headerParts[1])
: '*';
$options = end($headerParts);
$parsedHeaders[$userAgent] = [
'noindex' => strpos(strtolower($options), 'noindex') !== false,
'nofollow' => strpos(strtolower($options), 'nofollow') !== false,
'none' => strpos(strtolower($options), 'none') !== false,
];
return $parsedHeaders;
}, []);
}
protected function filterRobotHeaders(array $headers): array
{
return array_filter($headers, function ($header) use ($headers) {
$headerContent = $this->normalizeHeaders($headers[$header] ?? []);
return strpos(strtolower($header), 'x-robots-tag') === 0
|| strpos(strtolower($headerContent), 'x-robots-tag') === 0;
}, ARRAY_FILTER_USE_KEY);
}
protected function normalizeHeaders($headers): string
{
return implode(',', (array) $headers);
}
}
src/WhyDisallows.php 0000644 00000000257 15266562307 0010515 0 ustar 00
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
.php-cs-fixer.dist.php 0000644 00000002345 15266562307 0010622 0 ustar 00 in([
__DIR__ . '/src',
__DIR__ . '/tests',
])
->name('*.php')
->ignoreDotFiles(true)
->ignoreVCS(true);
return (new PhpCsFixer\Config())
->setRules([
'@PSR12' => true,
'array_syntax' => ['syntax' => 'short'],
'ordered_imports' => ['sort_algorithm' => 'alpha'],
'no_unused_imports' => true,
'not_operator_with_successor_space' => true,
'trailing_comma_in_multiline' => true,
'phpdoc_scalar' => true,
'unary_operator_spaces' => true,
'binary_operator_spaces' => true,
'blank_line_before_statement' => [
'statements' => ['break', 'continue', 'declare', 'return', 'throw', 'try'],
],
'phpdoc_single_line_var_spacing' => true,
'phpdoc_var_without_name' => true,
'class_attributes_separation' => [
'elements' => [
'method' => 'one',
],
],
'method_argument_space' => [
'on_multiline' => 'ensure_fully_multiline',
'keep_multiple_spaces_after_comma' => true,
],
'single_trait_insert_per_statement' => true,
])
->setFinder($finder);