Other packages > Find by keyword >

spiderbar  

Parse and Test Robots Exclusion Protocol Files and Rules
View on CRAN: Click here


Download and install spiderbar package within the R console
Install from CRAN:
install.packages("spiderbar")

Install from Github:
library("remotes")
install_github("cran/spiderbar")

Install by package version:
library("remotes")
install_version("spiderbar", "0.2.5")



Attach the package and use:
library("spiderbar")
Maintained by
Bob Rudis
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2017-09-25
Latest Update: 2023-02-11
Description:
The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
How to cite:
Bob Rudis (2017). spiderbar: Parse and Test Robots Exclusion Protocol Files and Rules. R package version 0.2.5, https://cran.r-project.org/web/packages/spiderbar
Previous versions and publish date:
0.2.0 (2017-09-25 10:24), 0.2.1 (2017-11-17 13:40), 0.2.2 (2019-08-19 18:50), 0.2.3 (2020-05-30 06:50), 0.2.4 (2021-05-16 07:00)
Other packages that cited spiderbar R package
View spiderbar citation profile
Other R packages that spiderbar depends, imports, suggests or enhances
Functions, R codes and Examples using the spiderbar R package
Some associated functions: can_fetch . crawl_delays . print.robxp . robxp . sitemaps . spiderbar . 
Some associated R codes: RcppExports.R . spiderbar-package.R .  Full spiderbar package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

steepness  
Testing Steepness of Dominance Hierarchies
The steepness package computes steepness as a property of dominance hierarchies. Steepness is define ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
mistral  
Methods in Structural Reliability
Various reliability analysis methods for rare event inference (computing failure probability and qua ...
Download / Learn more Package Citations See dependency  
ftaproxim  
Fault Tree Analysis Based on Proxel Simulation
Calculation and plotting of instantaneous unavailabilities of basic events along with the top event ...
Download / Learn more Package Citations See dependency  
rdbnomics  
Download DBnomics Data
R access to hundreds of millions data series from DBnomics API (). ...
Download / Learn more Package Citations See dependency  
critpath  
Setting the Critical Path in Project Management
Solving the problem of project management using CPM (Critical Path Method), PERT (Program Evaluation ...
Download / Learn more Package Citations See dependency  

22,114

R Packages

188,080

Dependencies

55,244

Author Associations

22,115

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net. Contact Us / Suggestions / Concerns