Other packages > Find by keyword >

tok  

Fast Text Tokenization
View on CRAN: Click here


Download and install tok package within the R console
Install from CRAN:
install.packages("tok")

Install from Github:
library("remotes")
install_github("cran/tok")

Install by package version:
library("remotes")
install_version("tok", "0.1.4")



Attach the package and use:
library("tok")
Maintained by
Daniel Falbel
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2023-07-06
Latest Update: 2023-08-17
Description:
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
How to cite:
Daniel Falbel (2023). tok: Fast Text Tokenization. R package version 0.1.4, https://cran.r-project.org/web/packages/tok. Accessed 22 Dec. 2024.
Previous versions and publish date:
0.1.0 (2023-07-06 15:00), 0.1.1 (2023-08-18 01:30), 0.1.2 (2024-06-27 13:30), 0.1.3 (2024-07-06 15:40), 0.1.4 (2024-09-04 16:10)
Other packages that cited tok R package
View tok citation profile
Other R packages that tok depends, imports, suggests or enhances
Complete documentation for tok
Functions, R codes and Examples using the tok R package
Some associated functions: encoding . tokenizer . 
Some associated R codes: encoding.R . extendr-wrappers.R . tokenizer.R .  Full tok package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

dmlalg  
Double Machine Learning Algorithms
Implementation of double machine learning (DML) algorithms in R, based on Emmenegger and Buehlmann ...
Download / Learn more Package Citations See dependency  
composits  
Compositional, Multivariate and Univariate Time Series Outlier Ensemble
A compositional multivariate and univariate time series outlier ensemble.It uses the four R packages ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
wordspace  
Distributional Semantic Models in R
An interactive laboratory for research on distributional semantic models ('DSM', see < ...
Download / Learn more Package Citations See dependency  
tropAlgebra  
Tropical Algebraic Functions
It includes functions like tropical addition, tropical multiplication for vectors and matrices. In t ...
Download / Learn more Package Citations See dependency  
elect  
Estimation of Life Expectancies Using Multi-State Models
Functions to compute state-specific and marginal life expectancies. The computation is based on a fi ...
Download / Learn more Package Citations See dependency  

23,394

R Packages

201,798

Dependencies

63,416

Author Associations

23,395

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA