Other packages > Find by keyword >

rtiktoken  

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View on CRAN: Click here


Download and install rtiktoken package within the R console
Install from CRAN:
install.packages("rtiktoken")

Install from Github:
library("remotes")
install_github("cran/rtiktoken")

Install by package version:
library("remotes")
install_version("rtiktoken", "0.0.6")



Attach the package and use:
library("rtiktoken")
Maintained by
David Zimmermann-Kollenda
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2024-11-06
Latest Update: 2024-11-06
Description:
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
How to cite:
David Zimmermann-Kollenda (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.0.6, https://cran.r-project.org/web/packages/rtiktoken. Accessed 28 Jan. 2025.
Previous versions and publish date:
No previous versions
Other packages that cited rtiktoken R package
View rtiktoken citation profile
Other R packages that rtiktoken depends, imports, suggests or enhances
Complete documentation for rtiktoken
Functions, R codes and Examples using the rtiktoken R package
Full rtiktoken package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
fabMix  
Overfitting Bayesian Mixtures of Factor Analyzers with Parsimonious Covariance and Unknown Number of Components
Model-based clustering of multivariate continuous data using Bayesian mixtures of factor analyzers ( ...
Download / Learn more Package Citations See dependency  
fontcm  
Computer Modern font for use with extrafont package
Computer Modern font for use with extrafont package ...
Download / Learn more Package Citations See dependency  
MetaIntegration  
Ensemble Meta-Prediction Framework
An ensemble meta-prediction framework to integrate multiple regression models into a current study. ...
Download / Learn more Package Citations See dependency  
waiter  
Loading Screen for 'Shiny'
Full screen and partial loading screens for 'Shiny' with spinners, progress bars, and notifications. ...
Download / Learn more Package Citations See dependency  
eiopaR  
Access to RFR (Risk-Free Rate) Curves Produced by the EIOPA
Provides EIOPA (European Insurance And Occupational Pensions Authority) risk-free rates. Please note ...
Download / Learn more Package Citations See dependency  

23,580

R Packages

204,057

Dependencies

63,980

Author Associations

23,581

Publication Badges

© Copyright since 2022. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA