I mean that’s fine and its nice to have a future upgrade path for the mobo, but reality is that AM4 would also perform great for running all of the games that OP listed and likely future games as well. They did specify “upgradable build” so I am not really recommending AM4, just using it as a comparison that the one they have chosen is definitely relevant and suitable
Yes, that hardware is certainly relevant. AM4 platform is still relevant IMO
You can overwrite the model by using the same name instead of creating one with a new name if it bothers you. Either way there is no duplication of the llm model file
What I am talking about is when layers are split across GPUs. I guess this is loading the full model into each GPU to parallelize layers and do batching
Can you try setting the num_ctx and num_predict using a Modelfile with ollama? https://github.com/ollama/ollama/blob/main/docs/modelfile.md#parameter
Are you using a tiny model (1.5B-7B parameters)? ollama pulls 4bit quant by default. It looks like vllm does not used quantized models by default so this is likely the difference. Tiny models are impacted more by quantization
I have no problems with changing num_ctx or num_predict
Models are computed sequentially (the output of each layer is the input into the next layer in the sequence) so more GPUs do not offer any kind of performance benefit
Ummm… did you try /set parameter num_ctx # and /set parameter num_predict #? Are you using a model that actually supports the context length that you desire…?
you can tell whoever wrote this has never run that command
Uh… isn’t that a good thing?
My guess is an x86 32bit machine
4690k was solid! Mine is retired, though. Now I selfhost on ARM
No touchpads? Not even a contender lmao
The current state of gaming is that many games perform better on linux than on windows. What planet is this guy on?
https://en.m.wikipedia.org/wiki/Dunbar's_number
Shit, forget phones and AI, let’s go even more basic. Your brain still has essentially the same neocortex that people around 250,000 years ago had, and it is evolved to only be capable of processing/understanding a maximum of around 150 interconnected social relationships, the number of people that a hunter-gatherer 250,000 years ago could expect to know and interact with over their lifetime.
We haven’t had time to adjust to meeting and knowing more than about 150 people total in your lifetime. How many contacts are in your phone right now? I had like 500 facebook friends as a teenager and they were all people I knew outside of social media… Our current lives are extremely different than the life that our brain is equipped for.
The technological progress is what is not normal. Modern humans have been walking around, living their lives for 300,000 years. Agriculture is less than 12,000 years old, basically still brand new in comparison to the span of time that people just like us have existed for.
For nearly all of human history, generation after generation after generation for thousands of years lived very similar lifestyles with marginally improved, but familiar technology.
It is only in the last few hundred years, a tiny tiny sliver of the human timeline, that we have seen rapid technological progress that has completely changed the way people live their lives from one generation to the next. Lifestyle changes and paradigm shifts that used to take many many generations now are seemingly happening several times within an individual’s lifetime.
We have barely even had any time to adapt to agriculture, let alone capitalism or air travel or instant global telecommunication or AI, etc. So don’t be too hard on yourself about feeling fatigued. I feel it too. We are living in an alien world that we aren’t really “meant for”. You’re “supposed to be” a hunter-gatherer.
I just stick to AMD, especially on Linux. The official AMD driver is open source on Linux, included in mainline kernel, and performance is better than their Windows diver now