Other packages > Find by keyword >

robotstxt  

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
View on CRAN: Click here


Download and install robotstxt package within the R console
Install from CRAN:
install.packages("robotstxt")

Install from Github:
library("remotes")
install_github("cran/robotstxt")

Install by package version:
library("remotes")
install_version("robotstxt", "0.7.15")



Attach the package and use:
library("robotstxt")
Maintained by
Pedro Baltazar
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2016-02-08
Latest Update: 2020-09-03
Description:
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
How to cite:
Pedro Baltazar (2016). robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker. R package version 0.7.15, https://cran.r-project.org/web/packages/robotstxt. Accessed 06 Nov. 2024.
Previous versions and publish date:
0.1.2 (2016-02-08 13:37), 0.3.2 (2016-12-05 18:28), 0.4.0 (2017-07-16 22:24), 0.4.1 (2017-09-01 22:39), 0.5.2 (2017-11-12 18:45), 0.6.0 (2018-02-11 15:19), 0.6.2 (2018-07-18 23:30), 0.7.4 (2020-05-31 23:30), 0.7.7 (2020-06-27 09:00), 0.7.8 (2020-07-25 23:30), 0.7.13 (2020-09-03 21:30)
Other packages that cited robotstxt R package
View robotstxt citation profile
Other R packages that robotstxt depends, imports, suggests or enhances
Complete documentation for robotstxt
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

equatiomatic  
Transform Models into 'LaTeX' Equations
The goal of equatiomatic is to reduce the painassociated with writing LaTeX formulas from fitted mod ...
Download / Learn more Package Citations See dependency  
mailR  
A Utility to Send Emails from R
Interface to Apache Commons Email to send emails from R. ...
Download / Learn more Package Citations See dependency  
seriation  
Infrastructure for Ordering Objects Using Seriation
Infrastructure for ordering objects with an implementation of several seriation/sequencing/ordinati ...
Download / Learn more Package Citations See dependency  
memoise  
'Memoisation' of Functions
Cache the results of a function so that when you call it again with the same arguments it returns t ...
Download / Learn more Package Citations See dependency  
ddp  
Desirable Dietary Pattern
The desirable Dietary Pattern (DDP)/ PPH score measures the variety of food consumption. The (weigh ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  

23,092

R Packages

198,677

Dependencies

62,675

Author Associations

23,089

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA