I am using this: https://github.com/oobabooga/text-generation-webui … It is running great with my AMD 7900XT. It also ran great with my 5700xt. It sets up itself within a conda virtual environment so it takes all mess out of getting the packages to work correctly. It can use NVIDIA cards too.

Once you get it installed you can then get your models from huggingface.co

I’m on arch, btw. ;)

Edit: I just went and reinstalled it and saw it supports these gpus