wowitsbase's comments

wowitsbase · 2026-09-09T22:11:50

Somewhat, but most of the optimizations are cuda based.

I would guess it would be a little faster, like 20-40%, but not 200% like on Nvidia.

wowitsbase · 2026-09-09T21:52:17

A small developer team and I had worked on this for a while for personal reasons, so I have decided to port it to cpp and release it to the general public. The stats are in the github. I promise if you have a gpu this will improve your speeds by at least 50%, even if you believe your configuration is optimized.

wowitsbase · 2026-09-07T21:36:00

Speculative decoding was on in all of my tests, the gain came from moving the round's inputs off host memory.

wowitsbase · 2026-09-07T21:34:04

agreed, if not for this project I've been making I would at least be using base llama.cpp

Personally I think it comes down to simplicity, but there's no reason for it's performance drops compared to llama.cpp while it's a wrapper of it.