Other packages > Find by keyword >

rtiktoken  

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View on CRAN: Click here


Download and install rtiktoken package within the R console
Install from CRAN:
install.packages("rtiktoken")

Install from Github:
library("remotes")
install_github("cran/rtiktoken")

Install by package version:
library("remotes")
install_version("rtiktoken", "0.0.6")



Attach the package and use:
library("rtiktoken")
Maintained by
David Zimmermann-Kollenda
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2024-11-06
Latest Update: 2024-11-06
Description:
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
How to cite:
David Zimmermann-Kollenda (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.0.6, https://cran.r-project.org/web/packages/rtiktoken. Accessed 03 Jan. 2025.
Previous versions and publish date:
No previous versions
Other packages that cited rtiktoken R package
View rtiktoken citation profile
Other R packages that rtiktoken depends, imports, suggests or enhances
Complete documentation for rtiktoken
Functions, R codes and Examples using the rtiktoken R package
Full rtiktoken package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
kosel  
Variable Selection by Revisited Knockoffs Procedures
Performs variable selection for many types of L1-regularised regressions using the revisited knockof ...
Download / Learn more Package Citations See dependency  
autonewsmd  
Auto-Generate Changelog using Conventional Commits
Automatically generate a changelog file (NEWS.md / CHANGELOG.md) from the git history using convent ...
Download / Learn more Package Citations See dependency  
ICglm  
Information Criteria for Generalized Linear Regression
Calculate various information criteria in literature for "lm" and "glm" objects. ...
Download / Learn more Package Citations See dependency  
DiscreteLaplace  
Discrete Laplace Distributions
Probability mass function, distribution function, quantile function, random generation and estimatio ...
Download / Learn more Package Citations See dependency  
NBBDesigns  
Neighbour Balanced Block Designs (NBBDesigns)
Neighbour-balanced designs ensure that no treatment is disadvantaged unfairly by its surroundings. T ...
Download / Learn more Package Citations See dependency  

23,440

R Packages

202,297

Dependencies

63,567

Author Associations

23,434

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA