Other packages > Find by keyword >

tok  

Fast Text Tokenization
View on CRAN: Click here


Download and install tok package within the R console
Install from CRAN:
install.packages("tok")

Install from Github:
library("remotes")
install_github("cran/tok")

Install by package version:
library("remotes")
install_version("tok", "0.2.1")



Attach the package and use:
library("tok")
Maintained by
Daniel Falbel
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2023-07-06
Latest Update: 2025-08-27
Description:
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
How to cite:
Daniel Falbel (2023). tok: Fast Text Tokenization. R package version 0.2.1, https://cran.r-project.org/web/packages/tok. Accessed 06 Mar. 2026.
Previous versions and publish date:
0.1.0 (2023-07-06 15:00), 0.1.1 (2023-08-18 01:30), 0.1.2 (2024-06-27 13:30), 0.1.3 (2024-07-06 15:40), 0.1.4 (2024-09-04 16:10), 0.2.0 (2025-08-27 11:10)
Other packages that cited tok R package
View tok citation profile
Other R packages that tok depends, imports, suggests or enhances
Complete documentation for tok
Functions, R codes and Examples using the tok R package
Some associated functions: encoding . tokenizer . 
Some associated R codes: encoding.R . extendr-wrappers.R . tokenizer.R .  Full tok package functions and examples
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

DatabionicSwarm  
Swarm Intelligence for Self-Organized Clustering
Algorithms implementing populations of agents that interact with one another and sense their environ ...
Download / Learn more Package Citations See dependency  
EMVS  
The Expectation-Maximization Approach to Bayesian Variable Selection
An efficient expectation-maximization algorithm for fitting Bayesian spike-and-slab regularization p ...
Download / Learn more Package Citations See dependency  
lbfgs  
Limited-memory BFGS Optimization
A wrapper built around the libLBFGS optimization library by Naoaki Okazaki. The lbfgs package implem ...
Download / Learn more Package Citations See dependency  
openxlsx  
Read, Write and Edit xlsx Files
Simplifies the creation of Excel .xlsx files by providing a high level interface to writing, stylin ...
Download / Learn more Package Citations See dependency  
r2resize  
In-Text Resize for Images, Tables and Fancy Resize Containers in 'shiny', 'rmarkdown' and 'quarto' Documents
Automatic resizing toolbar for containers, images and tables. Various resizable or expandable contai ...
Download / Learn more Package Citations See dependency  
solitude  
An Implementation of Isolation Forest
Isolation forest is anomaly detection method introduced by the paper Isolation based Anomaly Detecti ...
Download / Learn more Package Citations See dependency  

26,264

R Packages

223,360

Dependencies

70,244

Author Associations

26,265

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA