TokenEvaluator.Net is a simple and useful library designed to measure and calculate the token count of given text inputs, as per the specifics of the language model specified by the user. This tool is crucial for efficient resource management when dealing with AI language models, such as OpenAI’s GPT-3.5-turbo and others.
By providing a comprehensive and detailed evaluation of the token count, this library assists developers in understanding the cost, performance, and optimization aspects of their AI language model interactions.
Whether you’re running an AI chatbot, a content generator, or any application that leverages AI language models, understanding your token usage is fundamental. TokenEvaluator.Net fills this gap, offering a clear, accurate, and easy-to-use solution.
Unlock the power of accurate token count understanding with TokenEvaluator.Net - the essential tool for AI developers.
These are the currently supported tokenizers:
These are the currently supported vision models:
Based on the OpenAI API documentation for Vision enabled models (as of 04/12/2023), to calculate the token count of an image, you need to consider the size of the image and the detail option on each image_url block.
TokenEvaluator.Net can be used via dependency injection, or an instance can be created using a tightly-coupled factory class.
If you want to be able to inject an instance of this client into multiple methods, then you can make use of the libraries dependency injection extension to add all of the required interfaces and implementations to your service collection.
using TokenEvaluator.Net.Dependency;
// Init a service collection, use the extension method to add the library services.
IServiceCollection services = new ServiceCollection();
services.AddTokenEvaluator.NetServices();
services.AddSingleton<ITokenEvaluatorClient, TokenEvaluatorClient>();
var serviceProvider = services.BuildServiceProvider();
Then simply inject the service into your class constructors like so:
internal const string GeneratedText = "The quick, brown fox—enamored by the moonlit night—jumped over 10 lazily sleeping dogs near 123 Elm St. at approximately 7:30 PM. Isn't text tokenization interesting?";
public ClassConstructor(ITokenEvaluatorClient tokenClient)
{
// Set token encoding type
tokenClient.SetDefaultTokenEncoding(EncodingType.Cl100kBase);
var tokenCount = tokenClient.EncodedTokenCount(GeneratedText);
// or choose a supported model
tokenClient.SetDefaultTokenEncodingForModel(ModelType.TextDavinci003);
var tokenCount = tokenClient.EncodedTokenCount(GeneratedText);
}
Using this as a concrete, tightly-coupled implementation is fairly straightforward. Simply use the below code and all internal interface and service references will be initialised and tightly-coupled. This is difficult to write tests for within your application, but ultimately is the easiest way to implement the client.
using TokenEvaluator.Net;
var client = TokenEvaluatorClientFactory.Create();
client.SetDefaultTokenEncoding(EncodingType.Cl100kBase);
var tokenCount = client.EncodedTokenCount(Constants.GeneratedText);
EncodedTokenCount allows the use of unsafe encoding, these methods use the unsafe keyword and are not recommended for use in production environments. They are however useful for benchmarking and testing purposes; refer to the Microsoft documentation on unsafe code for more information. Microsoft Docs: Unsafe code, pointer types, and function pointers
the ‘unsafe’ parameter defaults to false, but can be set to true if required.
internal const string GeneratedText = "The quick, brown fox—enamored by the moonlit night—jumped over 10 lazily sleeping dogs near 123 Elm St. at approximately 7:30 PM. Isn't text tokenization interesting?";
public ClassConstructor(ITokenEvaluatorClient tokenClient)
{
// Set token encoding type
tokenClient.SetDefaultTokenEncoding(EncodingType.Cl100kBase);
var tokenCount = tokenClient.EncodedTokenCount(GeneratedText, unsafe: true, useParallelProcessing: false);
// or choose a supported model
tokenClient.SetDefaultTokenEncodingForModel(ModelType.TextDavinci003);
var tokenCount = tokenClient.EncodedTokenCount(GeneratedText, unsafe: true, useParallelProcessing: false);
}
EncodedTokenCount, Encode, and Decode Methods allow developers to make use of parallel processing (This utilises the parallel threading library), this is useful for large text inputs and can significantly reduce the time taken to encode the text. This is not recommended for use in production environments, but is useful for benchmarking and testing purposes.
The ‘useParallelProcessing’ parameter defaults to true, but can be set to false if required.
internal const string GeneratedText = "The quick, brown fox—enamored by the moonlit night—jumped over 10 lazily sleeping dogs near 123 Elm St. at approximately 7:30 PM. Isn't text tokenization interesting?";
public ClassConstructor(ITokenEvaluatorClient tokenClient)
{
// Set token encoding type
tokenClient.SetDefaultTokenEncoding(EncodingType.Cl100kBase);
var tokenCount = tokenClient.EncodedTokenCount(GeneratedText, unsafe: false,useParallelProcessing: true);
var tokens = tokenClient.Encode(GeneratedText, useParallelProcessing: true);
var decodedText = tokenClient.Decode(tokens, useParallelProcessing: true);
}
For the purposes of openness and transparency included below are a number of benchmark tests. The library used to run these is included within the Benchmark folder.
These need some review and assessment to determine realistically which is the most efficient, but the results are included below for reference.
BenchmarkDotNet v0.13.6, Windows 11 (10.0.22621.1848/22H2/2022Update/SunValley2)
11th Gen Intel Core i7-11700K 3.60GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK 7.0.304
[Host] : .NET 7.0.7 (7.0.723.27404), X64 RyuJIT AVX2 [AttachedDebugger]
DefaultJob : .NET 7.0.7 (7.0.723.27404), X64 RyuJIT AVX2
Method | Mean | Min | Q1 | Median | Max | Op/s | Gen0 | Gen1 | Allocated |
---|---|---|---|---|---|---|---|---|---|
TiktokenSharp CountSpeed | 1,574.6 μs | 1,526.2 μs | 1,541.8 μs | 1,563.4 μs | 1,639.0 μs | 635.1 | 248.0469 | 177.7344 | 2031.03 KB |
SharpToken CountSpeed | 2,377.3 μs | 2,290.6 μs | 2,331.4 μs | 2,360.9 μs | 2,498.6 μs | 420.6 | 324.2188 | 226.5625 | 2658.29 KB |
TokenEvaluatorNet Managed NonParallel CountSpeed | 1,621.9 μs | 1,543.0 μs | 1,605.4 μs | 1,616.0 μs | 1,693.9 μs | 616.6 | 238.2813 | 171.8750 | 1954.78 KB |
TokenEvaluatorNet Unsafe NonParallel CountSpeed | 597.7 μs | 576.0 μs | 592.1 μs | 595.3 μs | 617.0 μs | 1,673.0 | 47.8516 | - | 391.77 KB |
TokenEvaluatorNet Managed Parallel CountSpeed | 1,640.2 μs | 1,592.7 μs | 1,617.5 μs | 1,638.2 μs | 1,686.8 μs | 609.7 | 238.2813 | 171.8750 | 1954.78 KB |
TokenEvaluatorNet Unsafe Parallel CountSpeed | 593.6 μs | 578.7 μs | 590.8 μs | 592.9 μs | 602.8 μs | 1,684.6 | 47.8516 | - | 391.77 KB |
TikToken CountSpeed | 604.6 μs | 592.2 μs | 597.3 μs | 605.3 μs | 620.2 μs | 1,653.9 | 47.8516 | - | 391.74 KB |
Method | Mean | Min | Q1 | Median | Max | Op/s | Gen0 | Gen1 | Allocated |
---|---|---|---|---|---|---|---|---|---|
TiktokenSharp EncodeDecode | 1,841.2 μs | 1,789.1 μs | 1,809.6 μs | 1,836.9 μs | 1,921.1 μs | 543.1 | 289.0625 | 164.0625 | 2383.89 KB |
SharpToken EncodeDecode | 2,717.7 μs | 2,531.2 μs | 2,634.8 μs | 2,705.1 μs | 3,076.4 μs | 368.0 | 339.8438 | 226.5625 | 2802.97 KB |
TokenEvaluatorNet EncodeDecode | 2,559.7 μs | 2,356.4 μs | 2,501.5 μs | 2,547.7 μs | 2,811.4 μs | 390.7 | 375.0000 | 371.0938 | 3005.72 KB |
TikToken Unsafe EncodeDecode | 934.8 μs | 895.7 μs | 919.7 μs | 929.4 μs | 980.7 μs | 1,069.8 | 75.1953 | 7.8125 | 618.92 KB |