Class CharBasedTokenCounter

java.lang.Object

ai.redouble.nucleo.harness.models.CharBasedTokenCounter

All Implemented Interfaces:

TokenCounter

public class CharBasedTokenCounter
extends Object
implements TokenCounter

Character-count / 4 fallback counter for providers with no known tokenizer. Image and file heuristics are configurable via constructor arguments.

Since:

0.1 (2026-04-16)

Author:

Andrey Santrosyan

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_IMAGE_BASE_TOKENS
static final int	DEFAULT_PDF_TOKENS_PER_KB

Constructor Summary

Constructors

Constructor	Description
CharBasedTokenCounter(int imageBaseTokens, int pdfTokensPerKB)

Method Summary

Modifier and Type	Method	Description
int	countFileTokens(ContentBlocks.FileBlock file)
int	countImageTokens(ContentBlocks.ImageBlock image)
int	countTokens(Message message)
int	countTokens(String text)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_IMAGE_BASE_TOKENS

public static final int DEFAULT_IMAGE_BASE_TOKENS

See Also:

• Constant Field Values

DEFAULT_PDF_TOKENS_PER_KB

public static final int DEFAULT_PDF_TOKENS_PER_KB

See Also:

• Constant Field Values

Constructor Details

CharBasedTokenCounter

public CharBasedTokenCounter(int imageBaseTokens,
 int pdfTokensPerKB)

Method Details

countTokens

public int countTokens(String text)

Specified by:

countTokens in interface TokenCounter

countTokens

public int countTokens(Message message)

Specified by:

countTokens in interface TokenCounter

countImageTokens

public int countImageTokens(ContentBlocks.ImageBlock image)

Specified by:

countImageTokens in interface TokenCounter

countFileTokens

public int countFileTokens(ContentBlocks.FileBlock file)

Specified by:

countFileTokens in interface TokenCounter