How to try open-weight models through OpenRouter

Downloading a model takes time and disk space. Before doing that, you can try a hosted copy against a few tasks you care about.

OpenRouter gives you one API for models served by different providers. Some of those models have downloadable weights. Others are closed models available only through an API.

We’ll ask two open-weight models to classify the same support message, then inspect their answers and reported usage. The program checks a concrete result: a billing category and a summary of at most 12 words.

This is a hosted test. Your prompt goes to OpenRouter and the inference provider handling the request. Use sample data while you learn.

Pick comparable models

Let’s compare qwen/qwen3-14b and qwen/qwen3-32b.

These IDs come from OpenRouter’s model catalog API. Both models belong to the same family, with different sizes. Keeping the task fixed lets us compare whether the larger one handles this particular request better.

Their availability and prices can change. Our script will check the live catalog before making a paid request.

An OpenRouter ID identifies a hosted offering. To download the weights later, follow the model’s Hugging Face reference and choose a format supported by your local runtime. A GGUF conversion and an MLX conversion are different downloads.

My open weights guide explains what the model creator releases and what the license allows. OpenRouter access does not grant a separate license to redistribute the weights.

Create an API key

Create an OpenRouter account, add credit if required, and create a key on the API keys page.

Set a small spending limit on the key for this experiment. An output-token limit reduces the size of one response, but it does not cap the cost of every request you might make with that key.

You need a recent Node.js release. On macOS, Homebrew can install it:

brew install node

On Omarchy, use its package helper:

omarchy pkg add nodejs

Create the project:

mkdir model-comparison
cd model-comparison

Create .env with your key:

OPENROUTER_API_KEY=your-openrouter-key

Replace the value with the key you created. Put .env in .gitignore before saving this project in Git:

.env

We will use Node’s built-in fetch() and environment-file support, so this project needs no npm packages.

Fetch the catalog before spending money

Create compare.mjs with this complete program:

const ids = ['qwen/qwen3-14b', 'qwen/qwen3-32b']
const key = process.env.OPENROUTER_API_KEY

const prompt = `Return only JSON with keys category and summary.
Category must be billing, bug, or question.
Summary must have at most 12 words.
Message: I paid twice for my subscription. Can you refund one payment?`

async function readJson(url, options = {}) {
  const response = await fetch(url, {
    ...options,
    signal: AbortSignal.timeout(120_000)
  })
  const result = await response.json()

  if (!response.ok || result.error) {
    throw new Error(`API request failed: ${response.status} ${result.error?.message || ''}`)
  }

  return result
}

const catalog = await readJson('https://openrouter.ai/api/v1/models')
const selected = ids.map(id => {
  const model = catalog.data.find(item => item.id === id)
  if (!model) throw new Error(`Model is unavailable: ${id}`)
  return model
})

for (const model of selected) {
  console.log({
    id: model.id,
    weights: model.hugging_face_id,
    inputDollarsPerMillionTokens: Number((model.pricing.prompt * 1_000_000).toFixed(4)),
    outputDollarsPerMillionTokens: Number((model.pricing.completion * 1_000_000).toFixed(4))
  })
}

if (!process.argv.includes('--run')) {
  console.log('Review prices, then add --run to make paid requests.')
  process.exit(0)
}

if (!key) throw new Error('Set OPENROUTER_API_KEY in .env')

for (const model of selected) {
  const started = performance.now()
  const result = await readJson('https://openrouter.ai/api/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${key}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      model: model.id,
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 1024,
      reasoning: { enabled: false },
      stream: false
    })
  })

  const choice = result.choices?.[0]
  const answer = choice?.message?.content
  console.log(`\n${model.id}`)
  console.log(answer || '(No visible answer)')
  console.log({
    milliseconds: Math.round(performance.now() - started),
    finishReason: choice?.finish_reason,
    usage: result.usage
  })

  try {
    const value = JSON.parse(answer)
    const valid = choice.finish_reason === 'stop' &&
      value !== null && typeof value === 'object' && !Array.isArray(value) &&
      Object.keys(value).sort().join(',') === 'category,summary' &&
      value.category === 'billing' &&
      typeof value.summary === 'string' && value.summary.trim().length > 0 &&
      value.summary.trim().split(/\s+/).length <= 12
    console.log({ passesThisExample: valid })
  } catch {
    console.log({ passesThisExample: false })
  }
}

The first run fetches the public catalog and prints current prices:

node --env-file=.env compare.mjs

{
  id: 'qwen/qwen3-14b',
  weights: 'Qwen/Qwen3-14B',
  inputDollarsPerMillionTokens: 0.1,
  outputDollarsPerMillionTokens: 0.22
}
{
  id: 'qwen/qwen3-32b',
  weights: 'Qwen/Qwen3-32B',
  inputDollarsPerMillionTokens: 0.08,
  outputDollarsPerMillionTokens: 0.28
}
Review prices, then add --run to make paid requests.

The larger model is cheaper on input and more expensive on output.

The catalog expresses basic prompt and completion prices per token. Multiplying them by a million makes them easier to read. These are advertised rates, so use the returned usage and your account activity to inspect the actual charge.

Send the requests

After reviewing prices, run:

node --env-file=.env compare.mjs --run

This sends two requests, one after the other. Each asks the model to classify the same fictional support message.

The request uses OpenRouter’s chat completions format. The reasoning setting disables optional reasoning for this comparison. Check supported settings when replacing these model IDs with other families.

The script requires a completed response, parses the answer as JSON, and checks the exact keys, billing category, and nonempty summary of at most 12 words.

Here’s what I got. I removed some of the fields inside usage to keep it short:

qwen/qwen3-14b
{
  "category": "billing",
  "summary": "User paid twice for subscription and requests a refund."
}
{
  milliseconds: 687,
  finishReason: 'stop',
  usage: {
    prompt_tokens: 56,
    completion_tokens: 25,
    total_tokens: 81,
    cost: 0.0000111
  }
}
{ passesThisExample: true }

qwen/qwen3-32b
```json
{
  "category": "billing",
  "summary": "Request refund for duplicate subscription payment"
}
```
{
  milliseconds: 725,
  finishReason: 'stop',
  usage: {
    prompt_tokens: 56,
    completion_tokens: 26,
    total_tokens: 82,
    cost: 0.00001176
  }
}
{ passesThisExample: false }

Both models picked billing and wrote a good summary. The larger one failed anyway, because it wrapped its answer in a Markdown code fence and JSON.parse() throws on that.

The prompt said “Return only JSON”, and the smaller model followed it. If the next part of your application expects plain JSON, that matters more than which summary reads better.

The check only looks at the shape, the category and the length of the summary. Read the summaries yourself too.

Understand the usage

OpenRouter documents the fields in its usage accounting guide. They include input and output token counts, and cost information when available.

In my run each request cost about a thousandth of a cent: $0.0000111 for the smaller model and $0.00001176 for the larger one.

Keep the full usage object while experimenting. Tokens spent on reasoning can affect usage even when the visible answer is short. A response ending with a token-limit finish reason needs a separate look before you call it a model failure.

The elapsed time includes the network request and provider processing. Here it was 687ms against 725ms, too close to call from one request each. OpenRouter can also route a model to different providers, so performance can vary between runs.

The program stops if a request fails. If you rerun it after the first response succeeded, that first model receives another paid request. For larger experiments, save successful results and resume only the missing work.

Make the comparison useful

I would replace the example with a small collection of fictional messages resembling my app’s workload. Include an ambiguous request and a message that contains instructions trying to change the output format.

Write the expected category before reading each answer. For a message that fits more than one category, decide how your application should handle the ambiguity before counting either model as correct.

Use the same prompts for each model and keep failures alongside successful answers. Extend the comparison with summaries judged against a human-written expectation.

A hosted result helps decide whether a model deserves more testing. Running a quantized copy on your laptop introduces another variable. Once you download it, repeat the same requests locally before choosing it for an application. How to run an LLM locally covers that next step.

Want me to talk about your product? You can sponsor this site.