Other packages > Find by keyword >

tokenizers  

Fast, Consistent Tokenization of Natural Language Text
View on CRAN: Click here


Download and install tokenizers package within the R console
Install from CRAN:
install.packages("tokenizers")

Install from Github:
library("remotes")
install_github("cran/tokenizers")

Install by package version:
library("remotes")
install_version("tokenizers", "0.3.0")



Attach the package and use:
library("tokenizers")
Maintained by
Lincoln Mullen
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2016-04-02
Latest Update: 2022-12-22
Description:
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words.The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages forfast yet correct tokenization in 'UTF-8'.
How to cite:
Lincoln Mullen (2016). tokenizers: Fast, Consistent Tokenization of Natural Language Text. R package version 0.3.0, https://cran.r-project.org/web/packages/tokenizers. Accessed 27 Oct. 2026.
Previous versions and publish date:
(2026-07-09 07:14), 0.1.0 (2016-04-02 23:34), 0.1.1 (2016-04-04 08:37), 0.1.2 (2016-04-14 18:19), 0.1.3 (2016-08-18 23:27), 0.1.4 (2016-08-29 22:59), 0.2.0 (2018-03-21 15:43), 0.2.1 (2018-03-29 22:07), 0.2.3 (2022-09-23 22:00)
Other packages that cited tokenizers R package
View tokenizers citation profile
Other R packages that tokenizers depends, imports, suggests or enhances
Complete documentation for tokenizers
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

usemodels  
Boilerplate Code for 'Tidymodels' Analyses
Code snippets to fit models using the tidymodels framework can be easily created for a given data se ...
Download / Learn more Package Citations See dependency  
SGB  
Simplicial Generalized Beta Regression
Main properties and regression procedures using a generalization of the Dirichlet distribution calle ...
Download / Learn more Package Citations See dependency  
exams2learnr  
Interface for 'exams' Exercises in 'learnr' Tutorials
Automatic generation of quizzes or individual questions for 'learnr' tutorials based on 'R/exams' ex ...
Download / Learn more Package Citations See dependency  
spanish  
Translate Quantities from Strings to Integer and Back. Misc Functions on Spanish Data
Character vector to numerical translation in Euros from Spanish spelled monetary quantities. Reverse ...
Download / Learn more Package Citations See dependency  
r2resize  
In-Text Resize for Images, Tables and Fancy Resize Containers in 'shiny', 'rmarkdown' and 'quarto' Documents
Automatic resizing toolbar for containers, images and tables. Various resizable or expandable contai ...
Download / Learn more Package Citations See dependency  

29,218

R Packages

247,686

Dependencies

77,281

Author Associations

29,219

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA