Other packages > Find by keyword >

tokenizers  

Fast, Consistent Tokenization of Natural Language Text
View on CRAN: Click here


Download and install tokenizers package within the R console
Install from CRAN:
install.packages("tokenizers")

Install from Github:
library("remotes")
install_github("cran/tokenizers")

Install by package version:
library("remotes")
install_version("tokenizers", "0.3.0")



Attach the package and use:
library("tokenizers")
Maintained by
Lincoln Mullen
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2016-04-02
Latest Update: 2022-12-22
Description:
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words.The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages forfast yet correct tokenization in 'UTF-8'.
How to cite:
Lincoln Mullen (2016). tokenizers: Fast, Consistent Tokenization of Natural Language Text. R package version 0.3.0, https://cran.r-project.org/web/packages/tokenizers. Accessed 07 Oct. 2026.
Previous versions and publish date:
(2026-07-09 07:14), 0.1.0 (2016-04-02 23:34), 0.1.1 (2016-04-04 08:37), 0.1.2 (2016-04-14 18:19), 0.1.3 (2016-08-18 23:27), 0.1.4 (2016-08-29 22:59), 0.2.0 (2018-03-21 15:43), 0.2.1 (2018-03-29 22:07), 0.2.3 (2022-09-23 22:00)
Other packages that cited tokenizers R package
View tokenizers citation profile
Other R packages that tokenizers depends, imports, suggests or enhances
Complete documentation for tokenizers
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

ggTimeSeries  
Time Series Visualisations Using the Grammar of Graphics
Provides additional display mediums for time series visualisations. ...
Download / Learn more Package Citations See dependency  
skewlmm  
Scale Mixture of Skew-Normal Linear Mixed Models
It fits scale mixture of skew-normal linear mixed models using an expectation–maximization (EM) ty ...
Download / Learn more Package Citations See dependency  
splm  
Econometric Models for Spatial Panel Data
ML and GM estimation and diagnostic testing of econometric models for spatial panel data. ...
Download / Learn more Package Citations See dependency  
plaqr  
Partially Linear Additive Quantile Regression
Estimation, prediction, thresholding, transformation, and plotting for partially linear additive qua ...
Download / Learn more Package Citations See dependency  
nextGenShinyApps  
Craft Exceptional 'R Shiny' Applications and Dashboards with Novel Responsive Tools
Nove responsive tools for designing and developing 'Shiny' dashboards and applications. The scripts ...
Download / Learn more Package Citations See dependency  
PairedData  
Paired Data Analysis
Many datasets and a set of graphics (based on ggplot2), statistics, effect sizes and hypothesis test ...
Download / Learn more Package Citations See dependency  

28,905

R Packages

247,686

Dependencies

76,495

Author Associations

28,906

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA