Dittin.AI
  • 👁️Introduction
    • What is Dittin AI?
    • Why should you choose Dittin AI?
    • Contact
    • Legal
    • Community Guidelines
  • 🚀Getting Started
    • Signup & Login
    • Password Recovery
    • Edit your profile
  • 📕Some Concepts
    • DittinLLM
    • Credits
  • 🤖Chatbot
    • Create Chatbot
    • Collection
    • Chatbot Interaction
      • Selecting a Chatbot
      • Start a New Chat/ Continue Chat
      • Restart Chat
      • Deleting a Chat
      • Share a public chat
      • Let Chatbot Continue Generating
      • Regenerate
      • Backtracing
      • Edit
      • Rename
  • ⚙️Chat Setting
    • Chat Setting
    • Message Instruction
    • Max AI Response Length(Premium)
    • User Profile
    • Memory Chip
  • AI IMAGE GENERATION
    • Generate AI Image
    • Gallery
    • Beginner's Guide to Prompting for Image Generation
  • 🏆Creator ecosystem
    • What is a creator?
    • Creator leaderboard
    • View creators
    • Follow creators
    • Block creators
    • Comment section
    • Notifications for creators
  • 📖Knowledge Base
    • Introduction
    • What is LLM?
    • What is LLM prompt?
    • What is LLM Token
    • LLM Context
Powered by GitBook
On this page
  1. Knowledge Base

What is LLM Token

PreviousWhat is LLM prompt?NextLLM Context

Last updated 10 months ago

LLM is subject to a maximum limit of tokens it can accommodate, often expressed as numbers like 4k or 8k, which means it can handle a maximum of 4k or 8k tokens.

How are tokens counted?

In LLM, "tokens" refer to the smallest units into which text is broken down. For English, a token usually corresponds to a word, for example, "hello" is one token, and "world" is another token. However, for languages with more characters like Chinese, one character may correspond to one token.

Additionally, there are some special cases to consider. Punctuation marks are typically treated as separate tokens, such as periods, commas, etc. Special character combinations like URLs or email addresses are considered as a single token.

It's important to note that for longer text sequences, the number of tokens can be quite substantial. When counting tokens, we can simply consider it as the length of the text. Therefore, when we mention how many tokens are equivalent to a character, in English, it's usually a one-to-one relationship, but it may vary for languages with more characters.

📖