11  Using Ollama through the OpenAI API in R

The goal of this brief tutorial is to demonstrate how to work with an OpenAI-compatible API in R to interact with local Ollama-provided models.

You will learn how to:

The OpenAI-compatible API used by Ollama does not yet support all options included in the native Ollama API. This is likely to be added in a future release. See GitHub issue #2963 and issue #11012.

11.1 Requirements

  • OpenRouter API key
  • httr2 and jsonlite R packages

11.2 Setup

11.2.1 Install packages

If not already installed, use either install.packages() or pak to install the required packages:

pak::pak(c("httr2", "jsonlite"))

11.2.2 Load packages

11.2.3 Ollama OpenAI API endpoint

Ollama does not require an API key. However, if some other API you are required to provide one (i.e. the relevant argument cannot be empty), you can use any string, like “ollama-api-key”.

ollama_openai_url <- "http://localhost:11434/v1/chat/completions"
model_name <- "qwen3:8b"

11.3 Generate a free text response

11.3.1 Create the request body

The following shows you how to define:

  • model: The name of the model to use.
  • prompt: The prompt to send to the model. This is often called the “user” message or prompt.
  • stream: Whether to stream the response token by token. Streaming is useful for interactive applications, like chat interfaces.
  • options: A list of options to control the model’s behavior, like temperature.
request_body <- list(
  model = model_name,
  messages = list(
    list(
      role = "system",
      content = "You are a meticulous research assistant."
    ),
    list(
      role = "user",
      content = "What is your name and who made you?"
    )
  ),
  temperature = 0.2
)

11.3.2 Build and perform the request

We use httr2’s request(), req_headers(), req_body_json(), and req_perform():

resp <- request(ollama_openai_url) |>
  req_body_json(request_body) |>
  req_perform()

11.3.2.1 Process the response

The response is an httr2_response object.

class(resp)
[1] "httr2_response"

We can convert it to a named list by combining httr2’s resp_body_string() with jsonlite’s fromJSON():

resp_list <- resp_body_json(resp)
messages <- resp_list[["choices"]][[1]][["message"]]

If the model used is a “reasoning” model, the output will contain a “thinking” field. All models include a “content” field.

In this case, since we used a reasoning model, we can print the reasoning step followed by the final response:

{
  if (!is.null(messages[["reasoning"]])) {
    cat("--- Reasoning ---\n")
    cat(messages[["reasoning"]], "\n")
  }
  cat("--- Response ---\n")
  cat(messages[["content"]], "\n")
}
--- Reasoning ---
Okay, the user is asking for my name and who made me. I need to make sure I provide accurate information.

First, my name is Qwen, which is the name given by Alibaba Cloud. I should mention that I was developed by Alibaba Cloud's Tongyi Lab. 

I should also clarify that I am a large-scale language model, so my name is Qwen, and my creator is Alibaba Cloud. I need to make sure the answer is clear and concise. 

Wait, maybe the user is confused about the difference between the company and the lab. I should specify that Alibaba Cloud is the company, and the Tongyi Lab is the research team within the company. 

Also, I should mention that I was developed by the Tongyi Lab, which is part of Alibaba Cloud. That way, the user understands the structure. 

I should avoid any technical jargon and keep the explanation straightforward. Let me put that together in a friendly and informative way.
 
--- Response ---
My name is Qwen, and I was developed by Alibaba Cloud's Tongyi Lab. I am a large-scale language model designed to assist with a wide range of tasks, from answering questions to creating content. My development team is part of Alibaba Cloud, which is a subsidiary of Alibaba Group. Let me know if you need help with anything!

In the above code block, we used braces {} in order to make Quarto print all cat() outputs together. You don’t need to do this in a regular R script.

11.4 Generate structured output

There are many scenarios, especially in research, where we want to generate a structured response instead of free text. Many users try to achieve this using instructions added to the user prompt and LLMs are increasingly good at following these instructions. However, there is native support to define an output schema including the required names and their descriptions, which is much cleaner, easier, and more likely to succeed, without requiring laborious and extensive prompting.

We begin by defining the output schema. To do this, we create a named list that follows the JSON schema format:

  • type = "object": Indicates that the output is a JSON object.
  • properties: A named list where we define the fields we want in the output. The name of each element is the field name, and its value is another list defining the field’s type (e.g. number, string, etc.) and its description.
  • required: A character vector listing the names of the required fields that must be present in the output.
LLMinfo_schema <- list(
  type = "object",
  properties = list(
    name = list(
      type = "string",
      description = "Your name"
    ),
    manufacturer = list(
      type = "string",
      description = "The name of the person, group, or company that built you."
    ),
    knowledge_cutoff = list(
      type = "string",
      description = "Your knowledge cutoff date."
    )
  ),
  required = c("name", "manufacturer", "knowledge_cutoff")
)

Let’s rerun the previous query, but this time we will include the output schema in the request.

When generating structured output, you might choose to adjust the prompt to explicitly ask for responses that conform to the schema. This is not necessary, as models, especially those designed for structured outputs, will usually adhere to the schema without additional prompting. Note that in this case, we are purposely using the same prompt as before, while the schema is actually requesting for a third field, knowledge_cutoff, which is not mentioned in the prompt.

request_body_structured <- list(
  model = model_name,
  messages = list(
    list(
      role = "system",
      content = "You are a meticulous research assistant."
    ),
    list(
      role = "user",
      content = "What is your name and who made you?"
    )
  ),
  temperature = 0.2,
  response_format = list(
    type = "json_schema",
    json_schema = list(
      name = "LLMinfo",
      strict = TRUE,
      schema = LLMinfo_schema
    )
  )
)

Perform the request:

resp_structured <- request(ollama_openai_url) |>
  req_body_json(request_body_structured) |>
  req_perform()

11.4.1 Process the structured response

Convert the response to a named list:

resp_structured_list <- resp_body_json(resp_structured)
messages_structured <- resp_structured_list[["choices"]][[1]][["message"]]

The response field is now a string in JSON format. We can use jsonlite::prettify() on the JSON string to pretty print it:

{
  if (!is.null(messages_structured[["reasoning"]])) {
    cat("--- Reasoning ---\n")
    cat(messages_structured[["reasoning"]], "\n")
  }
  cat("--- Response ---\n")
  prettify(messages_structured[["content"]])
}
--- Reasoning ---
Okay, the user is asking for my name and who made me. Let me start by confirming my name. I should mention that I'm Qwen, developed by Alibaba Cloud. But wait, I need to make sure I'm not mixing up any details. Let me check the official information again. Yes, Qwen is the correct name, and Alibaba Cloud is the company behind it. The user might be interested in knowing more about my development, so I should add a bit about my capabilities and purpose. Also, I should keep the response friendly and informative without being too technical. Let me structure the answer clearly: first state my name and developer, then briefly mention my functions and how I can assist them. Make sure there are no errors in the information provided.
 
--- Response ---
{
    "name": "Qwen",
    "manufacturer": "Alibaba Cloud",
    "knowledge_cutoff": "2024年4月"
}

You can use jsonlite::fromJSON() to convert the response field into a named list:

fromJSON(messages_structured[["content"]])
$name
[1] "Qwen"

$manufacturer
[1] "Alibaba Cloud"

$knowledge_cutoff
[1] "2024年4月"

11.4.2 Validate the response

We can optionally use the jsonvalidate package to validate the response:

is_valid <- jsonvalidate::json_validate(
  json = messages_structured[["content"]],
  schema = jsonlite::toJSON(LLMinfo_schema),
  verbose = TRUE
)

Produce warning if the response does not conform to the schema:

if (is_valid) {
  cat("The response conforms to the schema.\n")
} else {
  warning("The response does not conform to the schema.")
}
The response conforms to the schema.

11.5 References