Wondering about services to test on either a 16gb ram “AI Capable” arm64 board or on a laptop with modern rtx. Only looking for open source options, but curious to hear what people say. Cheers!
I use OLlama & Open-WebUI, OLlama on my gaming rig and Open-WebUI as a frontend on my server.
It’s been a really powerful combo!
Would you please talk more about it. I forgot about Open-webui, but intending to start playing with. Honestly, what do you actually do with it?
I have the same setup, but its not very usable as my graphics card has 6gb ram. I want one with 20 or 24, as the 6b models are pain and the tiny ones don’t give me much.
Ollama was pretty easy to set up on windows, and its eqsy to download and test the models ollama has available
Sounds like you and I are in a similar place of testing.
Possibly. Been running it since last summer, but like i say the small models dont do much good for me. I have tried llama3.1 olmo2, deepseek r1 in a few variants, qwen2. Qwen2.5 coder, mistral, codellama, starcoder2, nemotron-mini, llama3.2, qwen2.5-coder, gamma2 and llava.
I use perplexity and mistral as paid, with much better quality. Openwebui is great though, but my hardware is lacking
Edit: saw that my mate is still using it a bit so i’ll update openwebu frpm 0.4 to 0.5.20 for him. Hes a bit anxious about sending data to the cloud so he dont mind the quality
Scrap that - after upgrading it went bonkers and will always use one of my «knowledges» no matter what I try. The websearch fails even with ddg as engine. Its aways seemed like the ui was made by unskilled labour, but this is just horrible. 2/10 not recommended
Sex chats. For other uses, just simple searches are better 99% of the time. And for the 1%, something like the Kagis FastGPT helps to find the correct keywords.
I messed around with home assistant and the ollama integration. I have passed on it and just use the default one with voice commands I set up. I couldn’t really get ollama to do or say anything useful. Like I asked it what’s a good time to run on a treadmill for beginners and it told me it’s not a doctor.
Like I asked it what’s a good time to run on a treadmill for beginners and it told me it’s not a doctor.
Kirkland brand meseeks energy.
There are some experimental models made specifically for use with Home Assistant, for example home-llm.
Even though they are tiny 1-3B I’ve found them to work much better than even 14B general purpose models. Obviously they suck for general purpose questions just by their size alone.
That being said they’re still LLMs. I like to keep the “prefer handling commands locally” option turned on and only use the LLM as a fallback.
Sounds like ollama was loaded up with an either overly censored or plain brain dead language model. Do you know which model it was? Maybe try mistral if it fits in your computer.
Haha, that is hilarious. Sounds like it gave you some snark. afaik you have to clarify by asking again when it says such things. “I’m not asking for medical advice, but…”
deleted by creator
None currently. Wish I could afford a GPU to play with some stuff.
Well, let me know your suggestions if you wish. I took the plunge and am willing to test on your behalf, assuming I can.
I have immich machine learning and ollama with openwebui
I use immich search a lot to find things like pictures of the side of the road to post on my community !sideoftheroad@lemmy.today
I almost never use the ollama though, not really sure what to do with it other than ask it dumb questions just to see what it says
I use the duckduckgo one when it auto has an answer to something I searched but its not too reliable
I run kobold.cpp which is a cutting edge local model engine, on my local gaming rig turned server. I like to play around with the latest models to see how they improve/change over time. The current chain of thought thinking models like deepseek r1 distills and qwen qwq are fun to poke at with advanced open ended STEM questions.
STEM questions like “What does Gödel’s incompleteness theorem imply about scientific theories of everything?” Or “Could the speed of light be more accurately refered to as ‘the speed of causality’?”
As for actual daily use, I prefer using mistral small 24b and treating it like a local search engine with the legitimacy of wikipedia. Its a starting point to ask questions about general things I don’t know about or want advice on, then do further research through more legitimate sources.
Its important to not take the LLM too seriously as theres always a small statistical chance it hallucinates some bullshit but most of the time its fairly accurate and is a pretty good jumping off point for further research.
Lets say I want an overview of how can I repair small holes forming in concrete, or general ideas on how to invest financially, how to change fluids in a car, how much fat and protein is in an egg, ect.
If the LLM says a word or related concept I don’t recognize I grill it for clarifying info and follow it through the infinite branching garden of related information.
I’ve used an LLM to help me go through old declassified documents and speculate on internal gov terminalogy I was unfamiliar with.
I’ve used a speech to text model and get it to speek just for fun. Ive used multimodal model and get it to see/scan documents for info.
Ive used websearch to get the model to retrieve information it didn’t know off a ddg search, again mostly for fun.
Feel free to ask me anything, I’m glad to help get newbies started.
I have Immich that has AI searching for my photos. Pretty useful for finding stuff actually
I was able to run a distilled version of DeepSeek on Linux. I ran it inside a PODMAN container with ROCM support (I have an AMD GPU). It wasn’t super fast but for a locally deployed and self hosted option the performance was okay. Apart from that I have deployed Fooocus for image generation in a similar manner. Currently, I am working on deploying Stable Diffusion with either ComfyUI or Automatic1111 inside a PODMAN container with ROCM support.
Didn’t know about these image generation tools, besides Stable Diffusion. Thanks!
Try running an AI Horde worker, it’s a really great service!
Not sure I know what that is. As in Hoarder?
I am curious about trying an application specific AI. Like just for coding, for instance. I assume the memory requirements would be much lower.
afaik Ollama would fit that bill, but perhaps others can chime in. You could probably run it on your local computer with a small model based on CPU alone.
I run ollama and auto1111 on my desktop when it’s powers on. Using open-webui in my homelab always on, and also connected to openrouter. This way I can always use openwebui with openrouter models and it’s pretty cheap per query and a little more private that using a big tech chatbot. And if I want local, I turn on the desktop and have local lamma and stab diff.
I also get bugger all benefit out of it., it’s a cute toy.
How do you like auto1111 as I’ve never head of it
Can anyone suggest a model for light coding? I’m on a 3070 mobile.
I spun up ollama and paperless-gpt to add ai ocr sidecar to paperless-ngx. It’s okay. It can read handwritten stuff okayish, which is better than tesseract (doesnt read hand writing at all), so I throw handwritten stuff to it, but the difference on typed text is marginal in my single day I spent testing 3 different models on a few different typed receipts.
Which specific models did you try and how would you rank each is usability?
I tried minicpm-v, granite3.2-vision, and mistral.
Granite didn’t work with paperless-gpt at all. Mistral worked sometimes but also just kept running sometimes and didn’t finish within a reasonable time (15 minutes for 2 pages). minicpm-v finishes every time, but i just looked at some of the results and seems as though it’s not even worth keeping it running either. I suppose maybe the first one I tried that gave me a good impression was a fluke.
To be fair, I’m a noob at local ai, and I also don’t have a good gpu (gtx1650). So these failures could all be self induced. I like the idea of ai powered ocr so I’ll probably try again in the future…
I find your experiments inspired. Thank you! I’m learning about this myself on an rtx and excited to discuss on my little podcast.james.network one of these days. Been using paperless minus the AI functionality so far. About to start testing different AI services on an arm64 device with 16gb ram that claims some level of AI support; will see how that goes. Let me know if there are any other specific services/models you’d recommend or are curious about.
Sure, and let me know how it goes for you. I’m on a dell r720xd, about to upgrade my ram from 128 to 296 gb… don’t want to spend the money for a new gpu right now.
I’ll report back after I try again.
