Static endpoints and load balancing

An endpoint is a stable public URL that fronts a pool of your machines. Create named static endpoints on a paid plan, point several machines at the same one, and Tokmine spreads the traffic across them.

What an endpoint is

Every endpoint has a URL of the form https://tunnel.tokmine.ai/t/<slug>. The URL never changes, no matter which machines come and go behind it. Machines that join an endpoint form its pool; callers only ever see the URL.

Default and static endpoints

Create an endpoint

Sign in to the app, open Endpoints and choose Create endpoint. Enter a name and a slug; the page shows the resulting URL and whether the slug is available. Only workspace admins can create, disable or delete endpoints. On a free account the page offers an upgrade instead of the form.

Slug rules

• 3 to 40 characters: lowercase letters, digits and hyphens. It must start and end with a letter or digit, and cannot contain --.
• Unique across all of Tokmine, not just your workspace. A taken slug is refused.
• Reserved words are refused, such as api, admin, app, docs, www, status, login, default, new, and anything starting with tm- or w- followed by 16 hex characters.
• The slug cannot be changed later, because clients depend on the URL. To rename, create a new endpoint, move your machines and callers over, then delete the old one. The name and description can be edited any time.

Join an endpoint from the CLI

Log in as usual (see Using an account), then name the endpoint:

tokmine --endpoint acme-prod
tokmine -e acme-prod
TOKMINE_ENDPOINT=acme-prod tokmine

The CLI picks the endpoint in this order:

1. --endpoint <slug> (or -e), then the TOKMINE_ENDPOINT environment variable, then endpoint in the config file. default means the default endpoint.
2. If you name none and the workspace has exactly one static endpoint, it is selected automatically (the CLI prints Using endpoint acme-prod (only one)).
3. If there are several, an interactive terminal shows a picker (type a number or slug) and remembers your choice in the config file. Without a terminal (scripts, services, containers) the CLI exits with an error listing them and the flag to pass. Disabled endpoints are never auto-selected.
4. If there are none, the machine joins the default endpoint, as before.

An unknown slug is an error that lists the valid ones. List them yourself with tokmine endpoints, which prints each endpoint's name, slug, URL and how many machines are online, plus how many static endpoints your plan allows (add --json for scripts). Anonymous sessions have no endpoints, so it asks you to log in.

$ tokmine endpoints
NAME       SLUG               URL                                    MACHINES
Default    w-... (default)    https://tunnel.tokmine.ai/t/w-...      0/0 online
Acme prod  acme-prod          https://tunnel.tokmine.ai/t/acme-prod  2/3 online

Custom endpoints: 1/20 used (plan: pay-as-you-go)
Serve one with: tokmine --endpoint <slug>

In scripts and services with several endpoints, always set --endpoint or TOKMINE_ENDPOINT so nothing depends on a prompt.

Several machines on one endpoint

Run the CLI on each machine with the same endpoint. They share one URL:

# gpu-box-1
tokmine --name gpu-box-1 --endpoint acme-prod
# gpu-box-2
tokmine --name gpu-box-2 --endpoint acme-prod
# both now serve https://tunnel.tokmine.ai/t/acme-prod

A machine belongs to one endpoint at a time. Restarting it with a different --endpoint moves it to that pool. Pools never mix between workspaces.

How load is balanced

Every request goes through the same steps, per endpoint:

• Model routing. Only machines in the pool that serve the requested model are candidates.
• Live scoring. Each machine reports CPU, RAM, GPU load, in-flight requests and tokens per second, and whether the model is already loaded. Machines with headroom and a loaded model score best; stale, degraded or draining machines are skipped.
• Power of two choices. Tokmine samples two candidates and sends the request to the better one. This spreads load evenly without every request piling onto the single top-scoring machine.
• Retry before the response starts. If the chosen machine fails before any of the response was sent, the request is retried once on another machine. A response that is already streaming is not restarted.
• Failover. When a machine drops, it leaves the pool immediately and traffic shifts to the rest. When it reconnects it rejoins the same pool.

GET /v1/models on an endpoint lists the union of models served by its online machines, so a client sees everything the pool can do.

When nothing can serve the request

Errors come back in the caller's dialect, so an Anthropic client gets an Anthropic-shaped error and a Gemini client a Gemini-shaped one. See API compatibility.

Manage, disable and delete

• Endpoints page. Shows each URL (with copy), machines online and total, and load. Edit the name and description, or copy ready-made snippets for the CLI and SDKs.
• Disable. Connected machines are disconnected with a clear message and public requests return 404 until you enable it again.
• Delete. The URL stops working and connected machines are disconnected. The default endpoint cannot be deleted.
• What machines see. The CLI prints that the endpoint was deleted, disabled or not found, forgets a saved choice for it, and exits without retrying. Restart it with another --endpoint.
• Downgrading. If the workspace no longer has a paid plan, static endpoints stop accepting machines; the default endpoint keeps working.

Use a static URL from the SDKs

Only the base URL changes. Use your own account token as the key.

from openai import OpenAI

client = OpenAI(base_url="https://tunnel.tokmine.ai/t/acme-prod/v1", api_key="<token>")
print(client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello"}],
).choices[0].message.content)

import anthropic

client = anthropic.Anthropic(base_url="https://tunnel.tokmine.ai/t/acme-prod", api_key="<token>")
print(client.messages.create(
    model="llama3.2", max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
).content[0].text)

from google import genai
from google.genai import types

client = genai.Client(api_key="<token>", http_options=types.HttpOptions(base_url="https://tunnel.tokmine.ai/t/acme-prod"))
print(client.models.generate_content(model="llama3.2", contents="Hello").text)

The browser chat can target a specific endpoint too: https://app.tokmine.ai/chat?endpoint=acme-prod.

Editor and CLI tool settings for your static URL are in AI tools and clients.

FAQ

• Do I need a paid plan for a static endpoint?

  Yes. Free accounts keep the default endpoint, which works the same way. Static endpoints are included with the paid plan at no extra price: you pay $0.50 per machine and $0.50 per seat.
• Can I change the slug of an endpoint?

  No, it is fixed at creation because clients depend on the URL. Create a new endpoint, move your machines and callers, then delete the old one. You can rename the display name at any time.
• Can one machine serve two endpoints?

  A running CLI belongs to one endpoint at a time. Restarting it with another --endpoint moves it. To serve two endpoints at once, run a second CLI with its own TOKMINE_CONFIG_DIR and machine name.
• Will my existing machines keep working?

  Yes. A machine that does not name an endpoint joins the default endpoint, or auto-selects your only static endpoint if you have created one. Pass --endpoint default to stay on the default.
• What happens to callers when I delete an endpoint?

  Requests to its URL return 404 and its machines are disconnected.
• How many endpoints can I create?

  Up to your plan's limit. The Endpoints page shows how many you have used.
• Does a static endpoint cost extra or change seats?

  No. Seats and machines are counted and billed as before, whichever endpoint they use.