How to pay for a page

A site with a toll answers AI crawlers with 402 Payment Required. The response says what to pay, in what, and to whom. Pay it, retry with the signature, get the page. Everything is in the headers and body of that first response; there is no registration for crawlers.

Overview

1. Request a page as you normally would, with your bot's usual User-Agent.
2. If the site has a toll and your user agent is one it charges, you get a 402 with a price in USDC and a Solana address.
3. Transfer that amount of USDC to the address, with a memo that carries the site id and the hash of a secret you choose.
4. Repeat the request with X-Crawl-Payment: <signature>.<secret>. The gate finds the transfer on chain and serves the page.
5. If you paid for more than one crawl, keep the X-Crawl-Pass token from the response and send it as a header on later requests until it runs out.

The payment goes to the site owner's wallet. TOLL is not in the money path and does not take a share.

Who is charged

The gate decides by the declared User-Agent. Browsers are never charged. Each site owner sets a rule per known bot: charge, allow free, or block. The defaults below apply where the owner has not set anything. A user agent the gate does not recognise is treated as a browser.

The 402 response

Status 402. The useful values are in headers, and the same values plus instructions are in a JSON body.

Body, application/json:

{
  "error":     "payment_required",
  "protocol":  "toll/1",
  "site":      "example.com",
  "price":     "0.002",
  "asset":     "USDC",
  "mint":      "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v",
  "network":   "solana",
  "payTo":     "<owner wallet>",
  "memo":      "toll:<siteId>:<sha256 hex of your secret>",
  "retryWith": "X-Crawl-Payment: <tx signature>.<secret>",
  "bulk":      "pay N x price in one transfer to get a pass good for N crawls"
}

mint is the USDC token mint on Solana mainnet. memo is a template: replace the last part with the SHA-256 hex digest of a secret you generate. The site id in the middle is filled in by the gate.

Paying

1. Pick a secret

Any random string, at least 16 bytes of entropy. Keep it; you send it on the retry. Hash it with SHA-256 and hex-encode the digest.

2. Send the USDC

One SPL token transfer of price USDC (or N × price for N crawls) to payTo, from any wallet you control. Attach a memo instruction with the exact string from the body, with your digest in the last position:

toll:<siteId>:<sha256(secret) as 64 hex chars>

The memo is how the gate matches the transfer to your retry without you having to prove which wallet you sent from. The gate verifies three things: the transfer reached payTo, the amount covers at least one crawl at the current price, and the memo's digest equals SHA-256 of the secret you send.

3. Wait for confirmation

Retry once the transaction is confirmed. A retry sent before the gate can see the transfer gets another 402; just try again.

Retrying with the payment

Repeat the original request with one extra header:

> GET /article HTTP/1.1
> User-Agent: GPTBot/1.2 (+https://openai.com/gptbot)
> X-Crawl-Payment: <transaction signature>.<secret>

< HTTP/1.1 200 OK
< X-Crawl-Pass: <pass token>
< X-Crawl-Pass-Remaining: <crawls left on the pass>

The value is the base58 transaction signature, a dot, then the secret itself (not its hash). The signature identifies the transfer, the secret proves you are the one who put its hash in the memo.

A successful retry consumes one crawl from that payment. The response carries the page and, in X-Crawl-Pass, a token for whatever is left.

Passes

If the transfer covered more than one crawl, the gate issues a pass. Send it instead of the payment header on later requests to the same site:

> GET /another-article HTTP/1.1
> X-Crawl-Pass: <pass token>

< HTTP/1.1 200 OK
< X-Crawl-Pass: <pass token>
< X-Crawl-Pass-Remaining: <one fewer>

• One pass per payment, tied to the site that issued it.
• Each 200 decrements X-Crawl-Pass-Remaining. When it reaches zero the next request gets a 402 again.
• The number of crawls on a pass is the amount paid divided by the price at the time of payment, rounded down.
• You can keep sending X-Crawl-Payment with the same signature and secret instead of the pass; it refers to the same balance.

When a retry fails

A retry that cannot be verified gets a 402 again with the same payment details, so you can pay and try once more. The body's error field names the reason (for example bad_payment_header) and detail says it in words. The usual reasons:

A 403 means the site owner blocks your bot outright. There is nothing to pay; the page is not available to that user agent.

Try it against the demo gate

The demo gate treats every request as an AI crawler, so you can see the 402 without a bot user agent:

curl -i {api}/demo/article

And a retry with a header that cannot verify:

curl -i {api}/demo/article -H 'X-Crawl-Payment: notasignature.notasecret'

The landing page has a button that runs the same request from your browser.

For site owners

You do not need to read the protocol to use it. Register a site, paste the snippet in front of your pages, and the gate does the exchange above on your behalf. Your readers and search engines are not affected; only the bots you charge see a 402.