Other packages > Find by keyword >

spiderbar  

Parse and Test Robots Exclusion Protocol Files and Rules
View on CRAN: Click here


Download and install spiderbar package within the R console
Install from CRAN:
install.packages("spiderbar")

Install from Github:
library("remotes")
install_github("cran/spiderbar")

Install by package version:
library("remotes")
install_version("spiderbar", "0.2.5")



Attach the package and use:
library("spiderbar")
Maintained by
Bob Rudis
[Scholar Profile | Author Map]
First Published: 2017-09-25
Latest Update: 2023-02-11
Description:
The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
How to cite:
Bob Rudis (2017). spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules. R package version 0.2.5, https://cran.r-project.org/web/packages/spiderbar. Accessed 21 Apr. 2025.
Previous versions and publish date:
0.2.0 (2017-09-25 10:24), 0.2.1 (2017-11-17 13:40), 0.2.2 (2019-08-19 18:50), 0.2.3 (2020-05-30 06:50), 0.2.4 (2021-05-16 07:00)
Other packages that cited spiderbar R package
View spiderbar citation profile
Other R packages that spiderbar depends, imports, suggests or enhances
Complete documentation for spiderbar
Functions, R codes and Examples using the spiderbar R package
Some associated functions: can_fetch . crawl_delays . print.robxp . robxp . sitemaps . spiderbar . 
Some associated R codes: RcppExports.R . spiderbar-package.R .  Full spiderbar package functions and examples
Downloads during the last 30 days
03/2203/2303/2403/2503/2603/2703/2803/2903/3003/3104/0104/0204/0304/0404/0504/0604/0704/0804/0904/1004/1104/1204/1304/1404/1504/1604/1704/1804/1904/20Downloads for spiderbar2030405060708090100110120TrendBars

Today's Hot Picks in Authors and Packages

bigsparser  
Sparse Matrix Format with Data on Disk
Provide a sparse matrix format with data stored on disk, to be used in both R and C++. This is inte ...
Download / Learn more Package Citations See dependency  
bssn  
Birnbaum-Saunders Model
It provides the density, distribution function, quantile function, random number generator, reliabil ...
Download / Learn more Package Citations See dependency  
discretecdAlgorithm  
Coordinate-Descent Algorithm for Learning Sparse Discrete Bayesian Networks
Structure learning of Bayesian network using coordinate-descent algorithm. This algorithm is design ...
Download / Learn more Package Citations See dependency  
r2resize  
In-Text Resize for Images, Tables and Fancy Resize Containers in 'shiny', 'rmarkdown' and 'quarto' Documents
Automatic resizing toolbar for containers, images and tables. Various resizable or expandable contai ...
Download / Learn more Package Citations See dependency  
HellCor  
The Hellinger Correlation
Empirical value of the Hellinger correlation, a measure of dependence between two continuous random ...
Download / Learn more Package Citations See dependency  

24,098

R Packages

207,311

Dependencies

65,069

Author Associations

24,099

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA