Other packages > Find by keyword >

tok  

Fast Text Tokenization
View on CRAN: Click here


Download and install tok package within the R console
Install from CRAN:
install.packages("tok")

Install from Github:
library("remotes")
install_github("cran/tok")

Install by package version:
library("remotes")
install_version("tok", "0.1.1")



Attach the package and use:
library("tok")
Maintained by
Daniel Falbel
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2023-07-06
Latest Update: 2023-08-17
Description:
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
How to cite:
Daniel Falbel (2023). tok: Fast Text Tokenization. R package version 0.1.1, https://cran.r-project.org/web/packages/tok
Previous versions and publish date:
0.1.0 (2023-07-06 15:00)
Other packages that cited tok R package
View tok citation profile
Other R packages that tok depends, imports, suggests or enhances
Functions, R codes and Examples using the tok R package
Some associated functions: encoding . tokenizer . 
Some associated R codes: encoding.R . extendr-wrappers.R . tokenizer.R .  Full tok package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

rdbnomics  
Download DBnomics Data
R access to hundreds of millions data series from DBnomics API (). ...
Download / Learn more Package Citations See dependency  
critpath  
Setting the Critical Path in Project Management
Solving the problem of project management using CPM (Critical Path Method), PERT (Program Evaluation ...
Download / Learn more Package Citations See dependency  
mistral  
Methods in Structural Reliability
Various reliability analysis methods for rare event inference (computing failure probability and qua ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
steepness  
Testing Steepness of Dominance Hierarchies
The steepness package computes steepness as a property of dominance hierarchies. Steepness is define ...
Download / Learn more Package Citations See dependency  
ftaproxim  
Fault Tree Analysis Based on Proxel Simulation
Calculation and plotting of instantaneous unavailabilities of basic events along with the top event ...
Download / Learn more Package Citations See dependency  

22,114

R Packages

188,080

Dependencies

55,244

Author Associations

22,115

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net. Contact Us / Suggestions / Concerns