oneuptime/LLM
Simon Larsen a34a6657f0 feat: Add max_new_tokens parameter to job function
The code changes in `app.py` add a new parameter `max_new_tokens` to the `job` function. This parameter allows the maximum number of new tokens to be specified when calling the function. This change enhances the flexibility and control of the `job` function by allowing the maximum number of new tokens to be adjusted as needed.
2024-07-08 18:46:22 +00:00
..
.dockerignore chore: Update Dockerfile and docker-compose.llm.yml for LLM models 2024-06-28 17:59:43 +00:00
app.py feat: Add max_new_tokens parameter to job function 2024-07-08 18:46:22 +00:00
Dockerfile.tpl chore: Update Dockerfile and docker-compose.llm.yml for LLM models 2024-06-28 17:59:43 +00:00
Readme.md chore: Update Hugging Face clone URL in test-release.yaml 2024-06-28 13:09:54 +01:00
requirements.txt
tsconfig.json

LLM

Development Guide

Step 1: Downloading Model from Hugging Face

Please make sure you have git lfs installed before cloning the model.

git lfs install
cd ./LLM/Models
# Here we are downloading the Meta-Llama-3-8B-Instruct model
git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

You will be asked for username and password. Please use Hugging Face Username as Username and, Hugging Face API Token as Password.

Step 2: Install Docker.

Install Docker and Docker Compose

sudo apt-get update
sudo curl -sSL https://get.docker.com/ | sh  

Install Rootless Docker

sudo apt-get install -y uidmap
dockerd-rootless-setuptool.sh install

See if the installation works

docker --version
docker ps 

# You should see no containers running, but you should not see any errors. 

Step 3: Insall nvidia drivers on the machine to use GPU

Step 4: Run the test workload to see if GPU is connected to Docker.

docker run --rm -it --gpus=all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

You have configured the machine to use GPU with Docker.

Build

  • Download models from meta
  • Once the model is downloaded, place them in the Llama/Models folder. Please make sure you also place tokenizer.model and tokenizer_checklist.chk in the same folder.
  • Edit Dockerfile to include the model name in the MODEL_NAME variable.
  • Docker build
npm run build-ai

Run

npm run start-ai    

After you start, run nvidia-smi to see if the GPU is being used. You should see the python process running on the GPU.