June 2024 - WLUG - wlug.mailman3.com

Re: LLM's and graphical interfaces.

by Jansen Smith

I made a mild how-to, for replicating my setup. Please note that the Nvidia driver setup is a SEPARATE "huge deal" that is out of scope for this how-to. You'll know the system is working if you see action in response to a prompt while running nvtop: I run the open-webui stack found here: https://github.com/open-webui/open-webui In order to have ollama (the LLM backend) available for my other fun projects and to ensure that the GPU is utilized by ollama, make sure you run the api and gpu compose files: docker compose -f docker-compose.yaml -f docker-compose.api.yaml -f docker-compose.gpu.yaml up -d --build I've catted my own versions of these files below: (base) house@chonkers:~/open-webui$ cat docker-compose.gpu.yaml services: ollama: # GPU support deploy: resources: reservations: devices: - driver: ${OLLAMA_GPU_DRIVER-nvidia} count: all capabilities: - gpu (base) house@chonkers:~/open-webui$ cat docker-compose.api.yaml version: '3.8' services: ollama: # Expose Ollama API outside the container stack ports: - ${OLLAMA_WEBAPI_PORT-11434}:11434 (base) house@chonkers:~/open-webui$ cat docker-compose.yaml version: '3.8' services: ollama: volumes: - ollama:/root/.ollama container_name: ollama pull_policy: always tty: true restart: unless-stopped image: ollama/ollama:${OLLAMA_DOCKER_TAG-latest} open-webui: build: context: . args: OLLAMA_BASE_URL: '/ollama' dockerfile: Dockerfile image: ghcr.io/open-webui/open-webui:${WEBUI_DOCKER_TAG-main} container_name: open-webui volumes: - open-webui:/app/backend/data depends_on: - ollama ports: - 3131:8080 environment: - 'OLLAMA_BASE_URL=http://ollama:11434' - 'WEBUI_SECRET_KEY=' extra_hosts: - host.docker.internal:host-gateway restart: unless-stopped volumes: ollama: {} open-webui: {} (base) house@chonkers:~/open-webui$ On Thu, Jun 13, 2024 at 11:48 AM John Stoffel <john(a)stoffel.org> wrote: > > Thanks for sending out this info Jansen, now to find time to actually > play with this stuff! > > > (can this email address be added to the mailing list?) > > > the self-hosted webui i use is here: > > https://github.com/open-webui/open-webui > > > ollama! > > https://github.com/ollama/ollama > > > here's the docker for ollama: > > https://hub.docker.com/r/ollama/ollama > > > On Tue, May 14, 2024 at 5:32 PM Kevin Harrington < > mad.hephaestus(a)gmail.com> wrote: > > > ---------- Forwarded message --------- > > From: Tim Keller via WLUG <wlug(a)lists.wlug.org> > > Date: Tue, May 14, 2024, 4:49 PM > > Subject: [WLUG] LLM's and graphical interfaces. > > To: Worcester Linux Users' Group General Discussion < > wlug(a)lists.wlug.org> > > Cc: Tim Keller <turbofx(a)gmail.com> > > > A couple meetings ago Jansen showed up a really cool graphical > interface for LLM. > > > Does anybody remember what it was? > > > Thanks, > > Tim. > > > -- > > I am leery of the allegiances of any politician who refers to their > constituents as > > "consumers". > > _______________________________________________ > > WLUG mailing list -- wlug(a)lists.wlug.org > > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > > Create Account: https://wlug.mailman3.com/accounts/signup/ > > Change Settings: > https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > > Web Forum/Archive: > > > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/XVOQB… > > _______________________________________________ > > WLUG mailing list -- wlug(a)lists.wlug.org > > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > > Create Account: https://wlug.mailman3.com/accounts/signup/ > > Change Settings: > https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > > Web Forum/Archive: > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/4ZQIW… >

June 23, 2024

This is for Brendan! Running VMs with GPU pass through

by John Stoffel

I haven't spent any time on this, but saw this article and thought of Brendan right off the bat. https://www.theregister.com/2024/06/19/proxmox_xcp_ng_gpu_passthrough/ This might be what he needs to run stuff at home on his GPU pass-through attempts. John

June 21, 2024

Re: Reminder: Meeting June 13th 2024 at 7PM! Topic: Your own personal LLM!

by John Stoffel

Hi all, I think we had a good meeting last night, it was certainly neat seeing how well ollama2 and ollama3 are at answering questions. It was also interesting to see how it could start going off the rails a little bit when asked interesting questions, because it couldn't (in my mind!) reason about things properly. Can you tell I'm an AI skeptic in some ways? It will make big changes at some point, but I don't think it will be as much as people think right away. It's still on the hype phase, and we haven't gotten to the reality crash yet. So hopefully we'll get some more details posted so we can all play around with ollama3 on our own systems. It's a bummer I can't use my AMD GPUs since that's all I have, no nVidia stuff at all in my house. And for next month, what do people want to talk about? I can try to resurrect my talk on KiCad, updated for v8 that was released ealier this year. Or I can talk about 'duc' which is the software package I help maintain. It's for indexing large large filesystems and displaying disk usage via CLI, GUI or web pages. It's really useful for when you have a 12Tb filesystem with 30 million files, because plain 'du' just takes way too long. https://github.com/zevv/duc is where you can find it. It's been quiet for a long time, but I've been recently getting ready to update and push out a new release. Hopefully this month if I'm lucky! John

June 14, 2024

Re: Reminder: Meeting June 13th 2024 at 7PM! Topic: Your own personal LLM!

by Keith Wright

Tim Keller via WLUG <wlug(a)lists.wlug.org> writes: > We've got a meeting tomorrow!! > Date/Time: June 13th 2024 @ 7pm. > Virtual Location: (back to the old haunt) https://meet.jit.si/WlugMA Tried a few times to get in between 6:55 -- 7:10. It asks for permission to share mike/video I say yes, it asks do I want to be moderator, I say no, it says you will get in when someone lets you in and displays the circling circle.... I'll try again. Then give up. -- Keith

June 13, 2024

Re: LLM's and graphical interfaces.

by John Stoffel

I'm on the jitsi.org meeting, but the only one right now. More are welcome to join Tim and I! Tim is running late...

June 13, 2024

Re: LLM's and graphical interfaces.

by John Stoffel

Thanks for sending out this info Jansen, now to find time to actually play with this stuff! > (can this email address be added to the mailing list?) > the self-hosted webui i use is here: > https://github.com/open-webui/open-webui > ollama! > https://github.com/ollama/ollama > here's the docker for ollama: > https://hub.docker.com/r/ollama/ollama > On Tue, May 14, 2024 at 5:32 PM Kevin Harrington <mad.hephaestus(a)gmail.com> wrote: > ---------- Forwarded message --------- > From: Tim Keller via WLUG <wlug(a)lists.wlug.org> > Date: Tue, May 14, 2024, 4:49 PM > Subject: [WLUG] LLM's and graphical interfaces. > To: Worcester Linux Users' Group General Discussion <wlug(a)lists.wlug.org> > Cc: Tim Keller <turbofx(a)gmail.com> > A couple meetings ago Jansen showed up a really cool graphical interface for LLM. > Does anybody remember what it was? > Thanks, > Tim. > -- > I am leery of the allegiances of any politician who refers to their constituents as > "consumers". > _______________________________________________ > WLUG mailing list -- wlug(a)lists.wlug.org > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > Create Account: https://wlug.mailman3.com/accounts/signup/ > Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > Web Forum/Archive: > https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/XVOQB… > _______________________________________________ > WLUG mailing list -- wlug(a)lists.wlug.org > To unsubscribe send an email to wlug-leave(a)lists.wlug.org > Create Account: https://wlug.mailman3.com/accounts/signup/ > Change Settings: https://wlug.mailman3.com/postorius/lists/wlug.lists.wlug.org/ > Web Forum/Archive: https://wlug.mailman3.com/hyperkitty/list/wlug@lists.wlug.org/message/4ZQIW…

June 13, 2024

Reminder: Meeting June 13th 2024 at 7PM! Topic: Your own personal LLM!

by Tim Keller

Hey Gang! We've got a meeting tomorrow!! Date/Time: June 13th 2024 @ 7pm. Location: Technocopia, 44 Portland St, 6th floor, Worcester MA. Virtual Location: (back to the old haunt) https://meet.jit.si/WlugMA <https://meet.jit.si/WlugMA> Considering how much AI has been in the news, I thought it would be cool to do a talk/demo about setting up your own personal Large Language Model! I'll either use my laptop of scrounge up a piece of good enough hardware to show it off.. hell I could even spin up a droplet and run it somewhere.. we'll figure that out. Jansen sent me some links to the cool stuff he uses and I'll definitely be leaning on his expertise and others. Here's what he wrote: the self-hosted webui i use is here: https://github.com/open-webui/open-webui ollama! https://github.com/ollama/ollama here's the docker for ollama: https://hub.docker.com/r/ollama/ollama I've experimented with the ollama models and they're cool. As usual we'll nerd out, wrestle with some LLM stuff and have a great time. Afterwards we'll head off to the Boyton or somewhere else for some dinner and keep the geekery in full swing. I'll bring some snacks and refreshments as well. Later, Tim. -- I am leery of the allegiances of any politician who refers to their constituents as "consumers".

June 12, 2024

Re: Hey Everybody! WLUG Meeting this week!

by Gregory Boyce

On Mon, Jun 10, 2024 at 6:01 PM John Stoffel <john(a)stoffel.org> wrote: > >>>>> "Gregory" == Gregory Boyce <gregory.boyce(a)gmail.com> writes: > > > On Mon, Jun 10, 2024, 2:42 PM John Stoffel via WLUG <wlug(a)lists.wlug.org> > wrote: > >>>>>> "Tim" == Tim Keller <turbofx(a)gmail.com> writes: > > >> Currently we're having issues getting the Turing Pi 2 to boot, so > >> doing a show off of Proxmox I could definitely do as well. > > > I'ev got proxmox running at owkr with some basic VMs on it for > > testing. I might be able to show it off doing some basic stuff like > > live migrations. > > > I recently switched my home server to proxmox, and I've been > > spending time using terraform and packer to build and deploy VMs. > > Are you running a cluster or just one system? I've got a three node > cluster at $WORK, but thinking of looking at their SCALE product. But > of course while VMware is going to shaft us pricewise, the manglement > might not want to leave the devil they know. Just a single system at the moment. Another issue is deploying OVA files. I find that ProxMOX is a pain > in terms of .iso and .ova files, you need to first add them into it's > own repository, then you can deploy from them. VMware just lets you > browse for the files and deploy. > Yeah, that makes sense. The interface isn't especially user friendly either. It takes me a while to remember where the interface for uploading ISO files is whenever I need to use it. That's one thing that Scale Computing does well. The interface for uploading images is pretty straightforward, and it will convert images from various formats for you automatically. Other than the current lack of support for compressed images. > Though I also admit that .ova deployments can be annoyingly tough, but > I'm running VCenter 7.0.x and a mix of ESXi 6.7 and 7.0 on my > systems. Legacy hardware which I can't move up. Another reason to > hate vendor designed stuff, it locks you in. > > > I'd be happy to share code if anyone is interested. > > Please do! > This is very much a work in progress as I work on learning the tools. https://gitlab.com/techstoa/proxmox -- Greg

June 11, 2024

Re: Hey Everybody! WLUG Meeting this week!

by John Stoffel

>>>>> "Gregory" == Gregory Boyce <gregory.boyce(a)gmail.com> writes: > On Mon, Jun 10, 2024, 2:42 PM John Stoffel via WLUG <wlug(a)lists.wlug.org> wrote: >>>>>> "Tim" == Tim Keller <turbofx(a)gmail.com> writes: >> Currently we're having issues getting the Turing Pi 2 to boot, so >> doing a show off of Proxmox I could definitely do as well. > I'ev got proxmox running at owkr with some basic VMs on it for > testing. I might be able to show it off doing some basic stuff like > live migrations. > I recently switched my home server to proxmox, and I've been > spending time using terraform and packer to build and deploy VMs. Are you running a cluster or just one system? I've got a three node cluster at $WORK, but thinking of looking at their SCALE product. But of course while VMware is going to shaft us pricewise, the manglement might not want to leave the devil they know. Another issue is deploying OVA files. I find that ProxMOX is a pain in terms of .iso and .ova files, you need to first add them into it's own repository, then you can deploy from them. VMware just lets you browse for the files and deploy. Though I also admit that .ova deployments can be annoyingly tough, but I'm running VCenter 7.0.x and a mix of ESXi 6.7 and 7.0 on my systems. Legacy hardware which I can't move up. Another reason to hate vendor designed stuff, it locks you in. > I'd be happy to share code if anyone is interested. Please do!

June 10, 2024

Re: Hey Everybody! WLUG Meeting this week!

by Gregory Boyce

On Mon, Jun 10, 2024, 2:42 PM John Stoffel via WLUG <wlug(a)lists.wlug.org> wrote: > >>>>> "Tim" == Tim Keller <turbofx(a)gmail.com> writes: > > > Currently we're having issues getting the Turing Pi 2 to boot, so > > doing a show off of Proxmox I could definitely do as well. > > I'ev got proxmox running at owkr with some basic VMs on it for > testing. I might be able to show it off doing some basic stuff like > live migrations. > I recently switched my home server to proxmox, and I've been spending time using terraform and packer to build and deploy VMs. I'd be happy to share code if anyone is interested. >

June 10, 2024