Other packages > Find by keyword >

rtiktoken  

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View on CRAN: Click here


Download and install rtiktoken package within the R console
Install from CRAN:
install.packages("rtiktoken")

Install from Github:
library("remotes")
install_github("cran/rtiktoken")

Install by package version:
library("remotes")
install_version("rtiktoken", "0.0.7")



Attach the package and use:
library("rtiktoken")
Maintained by
David Zimmermann-Kollenda
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2024-11-06
Latest Update: 2025-04-14
Description:
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
How to cite:
David Zimmermann-Kollenda (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.0.7, https://cran.r-project.org/web/packages/rtiktoken. Accessed 04 Jul. 2026.
Previous versions and publish date:
0.0.6 (2024-11-06 16:40)
Other packages that cited rtiktoken R package
View rtiktoken citation profile
Other R packages that rtiktoken depends, imports, suggests or enhances
Complete documentation for rtiktoken
Functions, R codes and Examples using the rtiktoken R package
Full rtiktoken package functions and examples
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

missCompare  
Intuitive Missing Data Imputation Framework
Offers a convenient pipeline to test and compare various missing data imputation algorithms on simu ...
Download / Learn more Package Citations See dependency  
PELVIS  
Probabilistic Sex Estimate using Logistic Regression, Based on VISual Traits of the Human Os Coxae
An R-Shiny application implementing a method of sexing the human os coxae based on logistic regressi ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
musica  
Multiscale Climate Model Assessment
Provides functions allowing for (1) easy aggregation of multivariate time series into custom time sc ...
Download / Learn more Package Citations See dependency  
multiwayvcov  
Multi-Way Standard Error Clustering
Exports two functions implementing multi-way clustering using the method suggested by Cameron, Gelb ...
Download / Learn more Package Citations See dependency  
SurvCorr  
Correlation of Bivariate Survival Times
Estimates correlation coefficients with associated confidence limits for bivariate, partially censo ...
Download / Learn more Package Citations See dependency  

27,653

R Packages

236,180

Dependencies

73,674

Author Associations

27,536

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA