Other packages > Find by keyword >

tokenizers.bpe  

Byte Pair Encoding Text Tokenization
View on CRAN: Click here


Download and install tokenizers.bpe package within the R console
Install from CRAN:
install.packages("tokenizers.bpe")

Install from Github:
library("remotes")
install_github("cran/tokenizers.bpe")

Install by package version:
library("remotes")
install_version("tokenizers.bpe", "0.1.3")



Attach the package and use:
library("tokenizers.bpe")
Maintained by
Jan Wijffels
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2019-08-02
Latest Update: 2023-09-15
Description:
Unsupervised text tokenizer focused on computational efficiency. Wraps the 'YouTokenToMe' library <https://github.com/VKCOM/YouTokenToMe> which is an implementation of fast Byte Pair Encoding (BPE) <https://aclanthology.org/P16-1162/>.
How to cite:
Jan Wijffels (2019). tokenizers.bpe: Byte Pair Encoding Text Tokenization. R package version 0.1.3, https://cran.r-project.org/web/packages/tokenizers.bpe
Previous versions and publish date:
0.1.0 (2019-08-02 11:40), 0.1.1 (2023-01-06 11:20), 0.1.2 (2023-09-14 16:10)
Other packages that cited tokenizers.bpe R package
View tokenizers.bpe citation profile
Other R packages that tokenizers.bpe depends, imports, suggests or enhances
Functions, R codes and Examples using the tokenizers.bpe R package
Some associated functions: belgium_parliament . bpe . bpe_decode . bpe_encode . bpe_load_model . 
Some associated R codes: RcppExports.R . data.R . pkg.R . youtokentome.R .  Full tokenizers.bpe package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

critpath  
Setting the Critical Path in Project Management
Solving the problem of project management using CPM (Critical Path Method), PERT (Program Evaluation ...
Download / Learn more Package Citations See dependency  
mistral  
Methods in Structural Reliability
Various reliability analysis methods for rare event inference (computing failure probability and qua ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
rdbnomics  
Download DBnomics Data
R access to hundreds of millions data series from DBnomics API (). ...
Download / Learn more Package Citations See dependency  
steepness  
Testing Steepness of Dominance Hierarchies
The steepness package computes steepness as a property of dominance hierarchies. Steepness is define ...
Download / Learn more Package Citations See dependency  
ftaproxim  
Fault Tree Analysis Based on Proxel Simulation
Calculation and plotting of instantaneous unavailabilities of basic events along with the top event ...
Download / Learn more Package Citations See dependency  

22,114

R Packages

188,080

Dependencies

55,244

Author Associations

22,115

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net. Contact Us / Suggestions / Concerns