Other packages > Find by keyword >

rtiktoken  

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View on CRAN: Click here


Download and install rtiktoken package within the R console
Install from CRAN:
install.packages("rtiktoken")

Install from Github:
library("remotes")
install_github("cran/rtiktoken")

Install by package version:
library("remotes")
install_version("rtiktoken", "0.11.0.3")



Attach the package and use:
library("rtiktoken")
Maintained by
David Zimmermann-Kollenda
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2024-11-06
Latest Update: 2025-04-14
Description:
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
How to cite:
David Zimmermann-Kollenda (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.11.0.3, https://cran.r-project.org/web/packages/rtiktoken. Accessed 05 Oct. 2026.
Previous versions and publish date:
(2026-08-31 20:40), 0.0.6 (2024-11-06 16:40), 0.0.7 (2025-04-15 00:50), 0.11.0.2 (2026-08-21 19:40)
Other packages that cited rtiktoken R package
View rtiktoken citation profile
Other R packages that rtiktoken depends, imports, suggests or enhances
Complete documentation for rtiktoken
Functions, R codes and Examples using the rtiktoken R package
Full rtiktoken package functions and examples
Downloads during the last 30 days

Today's Hot Picks in Authors and Packages

genie  
Fast, Robust, and Outlier Resistant Hierarchical Clustering
Includes the reference implementation of Genie - a hierarchical clustering algorithm that links two ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
correlation  
Methods for Correlation Analysis
Lightweight package for computing different kinds of correlations, such as partial correlations, Ba ...
Download / Learn more Package Citations See dependency  
cdlTools  
Tools to Download and Work with USDA Cropscape Data
Downloads USDA National Agricultural Statistics Service (NASS) cropscape data for a specified state ...
Download / Learn more Package Citations See dependency  

28,905

R Packages

247,686

Dependencies

76,495

Author Associations

28,906

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA