I made a mild how-to, for replicating my setup. Please note that the Nvidia driver setup is a SEPARATE "huge deal" that is out of scope for this how-to. You'll know the system is working if you see action in response to a prompt while running nvtop: I run the open-webui stack found here: https://github.com/open-webui/open-webui In order to have ollama (the LLM backend) available for my other fun projects and to ensure that the GPU is utilized by ollama, make sure you run the api and gpu compose files: docker compose -f docker-compose.yaml -f docker-compose.api.yaml -f docker-compose.gpu.yaml up -d --build I've catted my own versions of these files below: (base) house@chonkers:~/open-webui$ cat docker-compose.gpu.yaml services: ollama: # GPU support deploy: resources: reservations: devices: - driver: ${OLLAMA_GPU_DRIVER-nvidia} count: all capabilities: - gpu (base) house@chonkers:~/open-webui$ cat docker-compose.api.yaml version: '3.8' services: ollama: # Expose Ollama API outside the container stack ports: - ${OLLAMA_WEBAPI_PORT-11434}:11434 (base) house@chonkers:~/open-webui$ cat docker-compose.yaml version: '3.8' services: ollama: volumes: - ollama:/root/.ollama container_name: ollama pull_policy: always tty: true restart: unless-stopped image: ollama/ollama:${OLLAMA_DOCKER_TAG-latest} open-webui: build: context: . args: OLLAMA_BASE_URL: '/ollama' dockerfile: Dockerfile image: ghcr.io/open-webui/open-webui:${WEBUI_DOCKER_TAG-main} container_name: open-webui volumes: - open-webui:/app/backend/data depends_on: - ollama ports: - 3131:8080 environment: - 'OLLAMA_BASE_URL=http://ollama:11434' - 'WEBUI_SECRET_KEY=' extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped volumes: ollama: {} open-webui: {} (base) house@chonkers:~/open-webui$ On Thu, Jun 13, 2024 at 11:48 AM John Stoffel <john@stoffel.org> wrote:

Thanks for sending out this info Jansen, now to find time to actually play with this stuff!

...

(can this email address be added to the mailing list?)

...

the self-hosted webui i use is here: https://github.com/open-webui/open-webui

...

ollama! https://github.com/ollama/ollama

...

here's the docker for ollama: https://hub.docker.com/r/ollama/ollama

...

On Tue, May 14, 2024 at 5:32 PM Kevin Harrington < mad.hephaestus@gmail.com> wrote:

...

---------- Forwarded message --------- From: Tim Keller via WLUG <wlug@lists.wlug.org> Date: Tue, May 14, 2024, 4:49 PM Subject: [WLUG] LLM's and graphical interfaces. To: Worcester Linux Users' Group General Discussion <

wlug@lists.wlug.org>

...

Cc: Tim Keller <turbofx@gmail.com>

...

A couple meetings ago Jansen showed up a really cool graphical

interface for LLM.

...

Does anybody remember what it was?

...

Thanks, Tim.

...

-- I am leery of the allegiances of any politician who refers to their

constituents as

...

"consumers". _______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings:

https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/

...

Web Forum/Archive:

https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/XVOQBA...

...

_______________________________________________ WLUG mailing list -- wlug@lists.wlug.org To unsubscribe send an email to wlug-leave@lists.wlug.org Create Account: https://wlug.mailman3.com/accounts/signup/ Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/4ZQIWW...