diff --git a/website/blog/2024-05-01-vulkan-support/completion.png b/website/blog/2024-05-01-vulkan-support/completion.png new file mode 100644 index 000000000..55d1e313c --- /dev/null +++ b/website/blog/2024-05-01-vulkan-support/completion.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4aa971d90eca2da311b6f07d59130c2969790999e2ff1023d481df4aeece19d4 +size 6127 diff --git a/website/blog/2024-05-01-vulkan-support/index.md b/website/blog/2024-05-01-vulkan-support/index.md new file mode 100644 index 000000000..3c440ff1b --- /dev/null +++ b/website/blog/2024-05-01-vulkan-support/index.md @@ -0,0 +1,61 @@ +--- +title: 'Vulkan Support: LLMs for Everyone' +authors: [boxbeam] +tags: [deployment] +--- + +It has long been the case that machine learning models are run on the GPU to improve their performance. +The GPU is far more effective at the kinds of computations needed for AI than the CPU, and so GPU compute libraries +such as Cuda and ROCm are typically used. + +However, requiring the support of these libraries can restrict which graphics cards are compatible, leaving many +with older or less popular cards unable to run LLMs efficiently. + +Tabby is happy to announce that we now support Vulkan, a graphics library created primarily for games. Its original purpose +means that it is designed to work on a very broad range of cards, and leveraging it to host LLMs means that we can now +offer GPU acceleration to people whose cards are not supported by Cuda and ROCm. + +Vulkan works on basically any GPU, so if you have previously been forced to host local models on your CPU, now is the time +to see what Tabby with Vulkan can do for you! + +## Vulkan Installation + +To begin, first make sure that you have Vulkan installed. + +For Windows users, Vulkan may be natively supported. Otherwise, the Vulkan SDK can be downloaded at https://vulkan.lunarg.com/sdk/home#windows. + +For Linux users, Vulkan can be installed through your package manager: +- Arch Linux: vulkan-icd-loader (universal), and also install vulkan-radeon (for AMD) or vulkan-nouveau (for Nvidia) +- Debian Linux: libvulkan1 + +![Vulkan installed on Arch Linux](./vulkan-installed-on-arch.png) + +## Tabby Installation + +To start using Tabby with Vulkan, first download one of the pre-built Vulkan binaries for your platform: +- Linux: https://github.com/TabbyML/tabby/releases/download/v0.10.0/tabby_x86_64-manylinux2014-vulkan +- Windows: https://github.com/TabbyML/tabby/releases/download/v0.10.0/tabby_x86_64-windows-msvc-vulkan.exe + +## Running + +Once you've installed the appropriate binary, you can simply run it from the command line: + +For Windows, open a command prompt and navigate to the download folder, then run: + +``` +tabby_x84_64-windows-msvc-vulkan serve --model StarCoder-1B --device vulkan +``` + +For Linux: + +``` +./tabby_x64_64-manylinux2014-vulkan serve --model StarCoder-1B --device vulkan +``` + +When it starts, you should see a printout indicating that Vulkan has found your card and is working properly: + +![Vulkan running on Linux](./vulkan-running.png) + +Now enjoy your speedy completions! + +![Completion example](./completion.png) diff --git a/website/blog/2024-05-01-vulkan-support/vulkan-installed-on-arch.png b/website/blog/2024-05-01-vulkan-support/vulkan-installed-on-arch.png new file mode 100644 index 000000000..bc8025163 --- /dev/null +++ b/website/blog/2024-05-01-vulkan-support/vulkan-installed-on-arch.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b3eba27c4b08d17a753e71d9ed08dc59fb2460db5a54fc759a95b91da7618c30 +size 16694 diff --git a/website/blog/2024-05-01-vulkan-support/vulkan-running.png b/website/blog/2024-05-01-vulkan-support/vulkan-running.png new file mode 100644 index 000000000..de5d13672 --- /dev/null +++ b/website/blog/2024-05-01-vulkan-support/vulkan-running.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0ed0a074e8cb4ea35a8f3174257527c46fdbdbd6f64ac912ff71102e36e1a46c +size 46399