App: Local large language model (llm2)

The llm2 app is one of the apps that provide text processing functionality using Large language models in Nextcloud and act as a text processing backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`, the mail app and :ref:`other apps making use of the core Text Processing API<tp-consumer-apps>`. The llm2 app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.

This app uses llama.cpp under the hood and is thus compatible with any model in gguf format.

However, we primarily test with the models listed below. Output quality will differ depending on which model you use and downstream tasks like summarization or Context Chat may not work on other models. We thus recommend the following models:

Multimodal chat

With a multimodal-capable model, llm2 supports multimodal chat in the :ref:`Nextcloud Assistant app<ai-app-assistant>`: users can attach images (and with Gemma 4, also audio) to chat messages so the model can reason about their content. Text files can also be attached as context.

The following shipped models support multimodality:

Vision-capable models also register the Analyze images and OCR task types in addition to the usual text-processing tasks.

Multimodal models need a matching multimodal projector (mmproj) gguf file next to the model file. The projector files for the shipped Qwen 3.5 and Gemma 4 models are downloaded automatically with the models. If you supply your own multimodal model, place its mmproj file in /nc_app_llm2_data as well and set mmproj_path in the model's JSON configuration (see :ref:`Configuring alternate models <ai-app-llm2-configuring-alternate-models>` below).

Multilinguality

This app supports input and output in languages other than English if the underlying model supports the language.

Qwen 3.5 supports the following 201 languages: <https://qwen.ai/blog?id=qwen3.5>

Gemma 4 provides multilingual support, offering high-quality performance in 35 languages and basic support for 140 languages.

Llama 3.1 supports the following languages:

Note that other languages may work as well, but only the above languages are guaranteed to work with Llama 3.1.

Requirements

Installation

  1. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed

  2. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`

  3. Install the "Local large language model" ExApp via the "Apps" page in the Nextcloud web admin user interface

Supplying alternate models

This app allows supplying alternate LLM models as gguf files in the /nc_app_llm2_data directory of the docker container.

  1. Download a gguf model e.g. from huggingface

  2. Copy the gguf file to /nc_app_llm2_data inside the docker container

  3. Restart the llm2 ExApp

  4. Select the new model in the Nextcloud AI admin settings

Configuring alternate models

Since every model requires slightly different inference parameters, you can pass along a configuration file for the alternate model files you supply.

The configuration file for a model file must have the same name as the model file but must end in .json instead of .gguf.

The strings {system_prompt} and {user_prompt} are variables that will be filled in by the app, so they must be part of your prompt template.

Here is an example config file for Llama 2:

{
  "prompt": "<|im_start|> system\n{system_prompt}\n<|im_end|>\n<|im_start|> user\n{user_prompt}\n<|im_end|>\n<|im_start|> assistant\n",
  "loader_config": {
     "n_ctx": 4096,
     "max_tokens": 2048,
     "stop": ["<|im_end|>"]
  }
}

Here is an example configuration for Llama 3:

{
  "prompt": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>\n{user_prompt}<|eot_id|>\n<|start_header_id|>assistant<|end_header_id|>\n",
  "loader_config": {
      "n_ctx": 8000,
      "max_tokens": 4000,
      "stop": ["<|eot_id|>"],
      "temperature": 0.3
  }
}

Scaling

It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 1000 user requests per hour. However, this number is based on theory and we do appreciate real-world feedback on this. If you would like to scale up your language model usage, we recommend using an :ref:`AI as a Service provider<ai-ai_as_a_service>` or hosting a service compatible with the OpenAI API yourself that can be scaled up and connecting nextcloud to it via the integration_openai app.

App store

You can also find the app in our app store, where you can write a review: https://apps.nextcloud.com/apps/llm2

Repository

You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: https://github.com/nextcloud/llm2

Nextcloud customers should file bugs directly with our Support system.

Known Limitations

Addendum: Running with a fully open model

If you would like to use a fully open model that scores a green score on our Ethical AI rating, we recommend the following model:

What makes OLMo a fully open model?

  • The code for training, fine-tuning and inference of the model is publicly available and fully open source

  • The training data with which the model is pretrained is publicly available

  • The model itself is publicly available and fully open source

  • The instruction tuning data is publicly available

  • The Reinforcement learning model is publicly available and fully open source

Limitations

  • OLMo currently only works well with English language input

  • In our tests it sometimes produced hallucinated or garbled output; make sure to thoroughly test the model for your use case

  • It cannot use tools, so cannot be used in conjunction with :ref:`Context Agent <ai-app-context_agent>`