Other packages > Find by keyword >

tokenizers  

Fast, Consistent Tokenization of Natural Language Text
View on CRAN: Click here


Download and install tokenizers package within the R console
Install from CRAN:
install.packages("tokenizers")

Install from Github:
library("remotes")
install_github("cran/tokenizers")

Install by package version:
library("remotes")
install_version("tokenizers", "0.3.0")



Attach the package and use:
library("tokenizers")
Maintained by
Lincoln Mullen
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2016-04-02
Latest Update: 2022-12-22
Description:
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words.The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages forfast yet correct tokenization in 'UTF-8'.
How to cite:
Lincoln Mullen (2016). tokenizers: Fast, Consistent Tokenization of Natural Language Text. R package version 0.3.0, https://cran.r-project.org/web/packages/tokenizers. Accessed 15 Jul. 2026.
Previous versions and publish date:
(2026-07-09 07:14), 0.1.0 (2016-04-02 23:34), 0.1.1 (2016-04-04 08:37), 0.1.2 (2016-04-14 18:19), 0.1.3 (2016-08-18 23:27), 0.1.4 (2016-08-29 22:59), 0.2.0 (2018-03-21 15:43), 0.2.1 (2018-03-29 22:07), 0.2.3 (2022-09-23 22:00)
Other packages that cited tokenizers R package
View tokenizers citation profile
Other R packages that tokenizers depends, imports, suggests or enhances
Complete documentation for tokenizers
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

r2resize  
In-Text Resize for Images, Tables and Fancy Resize Containers in 'shiny', 'rmarkdown' and 'quarto' Documents
Automatic resizing toolbar for containers, images and tables. Various resizable or expandable contai ...
Download / Learn more Package Citations See dependency  
footBayes  
Fitting Bayesian and MLE Football Models
This is the first package allowing for the estimation, visualization and prediction of the most wel ...
Download / Learn more Package Citations See dependency  
PermAlgo  
Permutational Algorithm to Simulate Survival Data
This version of the permutational algorithm generates a dataset in which event and censoring times ...
Download / Learn more Package Citations See dependency  
gscaLCA  
Generalized Structure Component Analysis- Latent Class Analysis & Latent Class Regression
Execute Latent Class Analysis (LCA) and Latent Class Regression (LCR) by using Generalized Structu ...
Download / Learn more Package Citations See dependency  
gamlss.add  
Extra Additive Terms for Generalized Additive Models for Location Scale and Shape
Interface for extra smooth functions including tensor products, neural networks and decision trees. ...
Download / Learn more Package Citations See dependency  
SECFISH  
Disaggregate Variable Costs
These functions were developed within SECFISH project (Strengthening regional cooperation in the are ...
Download / Learn more Package Citations See dependency  

27,806

R Packages

239,283

Dependencies

73,837

Author Associations

27,807

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA