Other packages > Find by keyword >

sentencepiece  

Text Tokenization using Byte Pair Encoding and Unigram Modelling
View on CRAN: Click here


Download and install sentencepiece package within the R console
Install from CRAN:
install.packages("sentencepiece")

Install from Github:
library("remotes")
install_github("cran/sentencepiece")

Install by package version:
library("remotes")
install_version("sentencepiece", "0.2.4")



Attach the package and use:
library("sentencepiece")
Maintained by
Jan Wijffels
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2020-06-04
Latest Update: 2022-11-13
Description:
Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) . Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) .
How to cite:
Jan Wijffels (2020). sentencepiece: Text Tokenization using Byte Pair Encoding and Unigram Modelling. R package version 0.2.4, https://cran.r-project.org/web/packages/sentencepiece. Accessed 16 Jul. 2026.
Previous versions and publish date:
(2026-07-09 07:03), 0.1.1 (2020-06-04 12:10), 0.1.2 (2020-06-08 23:40), 0.2.1 (2021-12-21 17:00), 0.2.2 (2022-11-09 09:00), 0.2.3 (2022-11-13 10:30), 0.2.4 (2025-11-27 21:20), 0.2 (2021-12-15 00:00)
Other packages that cited sentencepiece R package
View sentencepiece citation profile
Other R packages that sentencepiece depends, imports, suggests or enhances
Complete documentation for sentencepiece
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

Rnmr1D  
Perform the Complete Processing of a Set of Proton Nuclear Magnetic Resonance Spectra
Perform the complete processing of a set of proton nuclear magnetic resonance spectra from the free ...
Download / Learn more Package Citations See dependency  
wordnet  
WordNet Interface
An interface to WordNet using the Jawbone Java API to WordNet. WordNet (< ...
Download / Learn more Package Citations See dependency  
tibble  
Simple Data Frames
Provides a 'tbl_df' class (the 'tibble') with stricter checking and better formatting than the tradi ...
Download / Learn more Package Citations See dependency  
tarchetypes  
Archetypes for Targets
Function-oriented Make-like declarative pipelines for Statistics and data science are supported in t ...
Download / Learn more Package Citations See dependency  
bbdetection  
Identification of Bull and Bear States of the Market
Implements two algorithms of detecting Bull and Bear markets in stock prices: the algorithm of Pagan ...
Download / Learn more Package Citations See dependency  
schoolmath  
Functions and Datasets for Math Used in School
Contains functions and datasets for math taught in school. A main focus is set to prime-calculation. ...
Download / Learn more Package Citations See dependency  

27,806

R Packages

239,283

Dependencies

73,837

Author Associations

27,807

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA