Other packages > Find by keyword >

tok  

Fast Text Tokenization
View on CRAN: Click here


Download and install tok package within the R console
Install from CRAN:
install.packages("tok")

Install from Github:
library("remotes")
install_github("cran/tok")

Install by package version:
library("remotes")
install_version("tok", "0.1.4")



Attach the package and use:
library("tok")
Maintained by
Daniel Falbel
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2023-07-06
Latest Update: 2023-08-17
Description:
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
How to cite:
Daniel Falbel (2023). tok: Fast Text Tokenization. R package version 0.1.4, https://cran.r-project.org/web/packages/tok. Accessed 08 Apr. 2025.
Previous versions and publish date:
0.1.0 (2023-07-06 15:00), 0.1.1 (2023-08-18 01:30), 0.1.2 (2024-06-27 13:30), 0.1.3 (2024-07-06 15:40), 0.1.4 (2024-09-04 16:10)
Other packages that cited tok R package
View tok citation profile
Other R packages that tok depends, imports, suggests or enhances
Complete documentation for tok
Functions, R codes and Examples using the tok R package
Some associated functions: encoding . tokenizer . 
Some associated R codes: encoding.R . extendr-wrappers.R . tokenizer.R .  Full tok package functions and examples
Downloads during the last 30 days
03/1003/1103/1503/1903/2103/2203/2403/2804/0204/0604/07Downloads for tok12345678910TrendBars

Today's Hot Picks in Authors and Packages

icr  
Compute Krippendorff's Alpha
Provides functions to compute and plot Krippendorff's inter-coder reliability coefficient alpha and ...
Download / Learn more Package Citations See dependency  
seasonalclumped  
Toolbox for Clumped Isotope Seasonality Reconstructions
Compiles a set of functions and dummy data that simplify reconstructions of seasonal temperature va ...
Download / Learn more Package Citations See dependency  
ssc  
Semi-Supervised Classification Methods
Provides a collection of self-labeled techniques for semi-supervised classification. In semi-superv ...
Download / Learn more Package Citations See dependency  
sybil  
Efficient Constrained Based Modelling
This Systems Biology Package Gelius-Dietrich et. al. 2012 doi10.11861752-0509-7-125 implements algor ...
Download / Learn more Package Citations See dependency  
dave  
Functions for "Data Analysis in Vegetation Ecology"
A collection of functions accompanying the book "Data Analysis in Vegetation Ecology". 3rd ed. CABI, ...
Download / Learn more Package Citations See dependency  
nextGenShinyApps  
Craft Exceptional 'R Shiny' Applications and Dashboards with Novel Responsive Tools
Nove responsive tools for designing and developing 'Shiny' dashboards and applications. The scripts ...
Download / Learn more Package Citations See dependency  

24,012

R Packages

207,311

Dependencies

64,867

Author Associations

24,013

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA