Other packages > Find by keyword >

sentencepiece  

Text Tokenization using Byte Pair Encoding and Unigram Modelling
View on CRAN: Click here


Download and install sentencepiece package within the R console
Install from CRAN:
install.packages("sentencepiece")

Install from Github:
library("remotes")
install_github("cran/sentencepiece")

Install by package version:
library("remotes")
install_version("sentencepiece", "0.2.4")



Attach the package and use:
library("sentencepiece")
Maintained by
Jan Wijffels
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2020-06-04
Latest Update: 2022-11-13
Description:
Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) . Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) .
How to cite:
Jan Wijffels (2020). sentencepiece: Text Tokenization using Byte Pair Encoding and Unigram Modelling. R package version 0.2.4, https://cran.r-project.org/web/packages/sentencepiece. Accessed 18 Sep. 2026.
Previous versions and publish date:
(2026-07-09 07:03), 0.1.1 (2020-06-04 12:10), 0.1.2 (2020-06-08 23:40), 0.2.1 (2021-12-21 17:00), 0.2.2 (2022-11-09 09:00), 0.2.3 (2022-11-13 10:30), 0.2.4 (2025-11-27 21:20), 0.2 (2021-12-15 00:00)
Other packages that cited sentencepiece R package
View sentencepiece citation profile
Other R packages that sentencepiece depends, imports, suggests or enhances
Complete documentation for sentencepiece
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

downlit  
Syntax Highlighting and Automatic Linking
Syntax highlighting of R code, specifically designed for the needs of 'RMarkdown' packages like 'pk ...
Download / Learn more Package Citations See dependency  
eyelinker  
Import ASC Files from EyeLink Eye Trackers
Imports plain-text ASC data files from EyeLink eye trackers into (relatively) tidy data frames for ...
Download / Learn more Package Citations See dependency  
data360r  
Wrapper for 'TCdata360' and 'Govdata360' API
Makes it easy to engage with the Application Program Interface (API) of the 'TCdata360' and 'Govdat ...
Download / Learn more Package Citations See dependency  
r2resize  
In-Text Resize for Images, Tables and Fancy Resize Containers in 'shiny', 'rmarkdown' and 'quarto' Documents
Automatic resizing toolbar for containers, images and tables. Various resizable or expandable contai ...
Download / Learn more Package Citations See dependency  
hmeasure  
The H-Measure and Other Scalar Classification Performance Metrics
Classification performance metrics that are derived from the ROC curve of a classifier. The package ...
Download / Learn more Package Citations See dependency  
injectoR  
R Dependency Injection
R dependency injection framework. Dependency injection allows a program design to follow the depend ...
Download / Learn more Package Citations See dependency  

28,565

R Packages

239,283

Dependencies

75,677

Author Associations

28,566

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA