Other packages > Find by keyword >

tok  

Fast Text Tokenization
View on CRAN: Click here


Download and install tok package within the R console
Install from CRAN:
install.packages("tok")

Install from Github:
library("remotes")
install_github("cran/tok")

Install by package version:
library("remotes")
install_version("tok", "0.2.1")



Attach the package and use:
library("tok")
Maintained by
Daniel Falbel
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2023-07-06
Latest Update: 2025-08-27
Description:
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
How to cite:
Daniel Falbel (2023). tok: Fast Text Tokenization. R package version 0.2.1, https://cran.r-project.org/web/packages/tok. Accessed 04 Aug. 2026.
Previous versions and publish date:
(2026-07-09 07:14), 0.1.0 (2023-07-06 15:00), 0.1.1 (2023-08-18 01:30), 0.1.2 (2024-06-27 13:30), 0.1.3 (2024-07-06 15:40), 0.1.4 (2024-09-04 16:10), 0.2.0 (2025-08-27 11:10), 0.2.1 (2025-09-30 15:10), 0.2.2 (2026-04-22 16:40)
Other packages that cited tok R package
View tok citation profile
Other R packages that tok depends, imports, suggests or enhances
Complete documentation for tok
Functions, R codes and Examples using the tok R package
Some associated functions: encoding . tokenizer . 
Some associated R codes: encoding.R . extendr-wrappers.R . tokenizer.R .  Full tok package functions and examples
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

ggconf  
Simpler Appearance Modification of "ggplot2"
A flexible interface for ggplot2theme potentially saving 50 of your typing. ...
Download / Learn more Package Citations See dependency  
rCMA  
R-to-Java Interface for 'CMA-ES'
Tool for providing access to the Java version 'CMAEvolutionStrategy' of Nikolaus Hansen. 'CMA-ES' i ...
Download / Learn more Package Citations See dependency  
epigraphdb  
Interface Package for the 'EpiGraphDB' Platform
The interface package to access data from the 'EpiGraphDB' platform. It p ...
Download / Learn more Package Citations See dependency  
Maintainer: Yi Liu (view profile)
nextGenShinyApps  
Craft Exceptional 'R Shiny' Applications and Dashboards with Novel Responsive Tools
Nove responsive tools for designing and developing 'Shiny' dashboards and applications. The scripts ...
Download / Learn more Package Citations See dependency  
sValues  
Measures of Sturdiness of Regression Coefficients
Implements the s-values proposed by Ed. Leamer. It provides a context-minimal approach for sensitivi ...
Download / Learn more Package Citations See dependency  
GLMMadaptive  
Generalized Linear Mixed Models using Adaptive Gaussian Quadrature
Fits generalized linear mixed models for a single grouping factor under maximum likelihood approxim ...
Download / Learn more Package Citations See dependency  

28,083

R Packages

239,283

Dependencies

74,457

Author Associations

28,084

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA