> For the complete documentation index, see [llms.txt](https://docs.denvrdata.com/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.denvrdata.com/docs/overview/getting-started/launch-vllm-as-a-custom-application.md). # Launch vLLM as a custom application This tutorial walks you through deploying language model inference endpoints on Denvr Dataworks using the Custom Application feature. We'll set up a [vLLM](https://docs.vllm.ai/) server hosting `mistralai/Mistral-7B-v0.1` with OpenAI-compatible APIs, leveraging Denvr Dataworks's GPU infrastructure and container orchestration capabilities. For detailed information about the Mistral-7B model, including its architecture, capabilities, and licensing terms, visit the [official model page on Hugging Face](https://huggingface.co/mistralai/Mistral-7B-v0.1). > **💡 Using Different Models**: To use a different model, replace `mistralai/Mistral-7B-v0.1` in the command override, select the appropriate hardware, accept any license agreements on the model's Hugging Face page, and update API examples accordingly. *** ### Understanding Custom Applications in Denvr Dataworks Denvr Dataworks's **Custom Application** feature is a powerful deployment method that gives you complete control over container configuration. Unlike other applications with predefined settings, custom applications allow you to: * **Bring Your Own Container**: Use any Docker image from public or private registries * **Reverse Proxy with API Authentication**: Configure an optional reverse proxy layer with API key authentication for secure access control * **Readiness Monitoring**: Configure a readiness port that gets pinged to verify your application is ready * **Command Override**: Replace the default container entry point with your own commands * **Environment Customization**: Set custom environment variables for your specific needs * **User Scripts**: Upload custom scripts that are mounted at `/etc/script/user-scripts` and executable from your command override > **ℹ️ Deployment Requirement**: Custom applications are currently only available to launch through reserved resource pools. On-demand resource pools are not yet supported for custom application deployments. ### Prerequisites For this tutorial you need: * A Denvr Dataworks account with **Custom Application** deployment capability (found through the **Application Catalog**) * A Hugging Face account with a User Access Token (you'll create this in the next section) * Command line access with `curl` for testing the deployment (or any HTTP client) #### Hugging Face Model Access This tutorial requires a Hugging Face token to be configured as an environment variable in your custom application. **Creating Your Hugging Face Token:** 1. Log into your [Hugging Face account](https://huggingface.co/) 2. Navigate to **Settings** → **Access Tokens** 3. Create a new token with **Read** permissions 4. Copy the token value for use in Denvr Dataworks's environment variable configuration **Model License Requirements:** Before deployment, visit the [Mistral-7B model page](https://huggingface.co/mistralai/Mistral-7B-v0.1) and accept any required license agreements. vLLM will not complete initialization if your account hasn't accepted the model's usage terms. ### Create the Deployment Using Denvr Dataworks #### Step 1: Navigate to Application Catalog 1. **Login to Denvr Dataworks**: Access your organization's Denvr Dataworks console 2. **Navigate to Applications**: Go to **Applications** → **Catalog** from the main navigation 3. **Select Custom Application**: Look for the **Custom Application** card 4. **Initialize Configuration**: Click to start the custom application deployment #### Step 2: Basic Information Configuration * **Application Name**: Use a descriptive, systematic naming convention: * Example: `vllm-[model-name]-[environment]-[version]` * **Reserved Resource Pools Only**: Custom applications are currently only allowed through reserved resource pools #### Step 3: Custom Application Configuration In the **Application** section (custom application specific fields): * **Image Repository Type**: Select `Public` (since we're using the official Docker Hub image) * **Container Image URL**: `docker.io/vllm/vllm-openai:v0.10.1.1` #### Step 4: Hardware Configuration In the **Instance Type Configuration** section, select a GPU with sufficient VRAM for your model: **For Mistral-7B (14-16GB VRAM required):** * **Recommended**: A100 MIG `3g.20gb` (20GB) - Cost-effective for 7B models * **Alternative**: A100 Full (40GB) - Higher throughput > **💡 VRAM Estimates**: Requirements assume FP16 inference. Using quantization (4-bit/8-bit) can significantly reduce VRAM needs. #### Step 5: Environment Variables In the **Environment Variables** section, configure the following variables: **Required:** * `HF_TOKEN` - Your Hugging Face User Access Token obtained in the prerequisites **Recommended - Cache Configuration:** To avoid filling up the limited root filesystem storage, configure caching to use direct-attached storage: ```env HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxx XDG_CACHE_HOME=/mnt/direct-attached/.cache HF_HOME=/mnt/direct-attached/.cache/huggingface HUGGINGFACE_HUB_CACHE=/mnt/direct-attached/.cache/huggingface/hub CUDA_CACHE_PATH=/mnt/direct-attached/.cache/nv/ComputeCache PIP_CACHE_DIR=/mnt/direct-attached/.cache/pip VLLM_CACHE_ROOT=/mnt/direct-attached/.cache/vllm VLLM_ASSETS_CACHE=/mnt/direct-attached/.cache/vllm/assets ``` > **💡 Cache Benefits**: Configuring cache directories in direct-attached storage prevents filling up the limited root filesystem. Model downloads, compiled CUDA kernels, and other cached data can consume significant space and should be stored on direct-attached storage. #### Step 6: Advanced Settings In the **Advanced Settings** section, configure both the command override and port settings: **Command Override Configuration:** In the **Command Override** field, enter the following command. You can add additional vLLM engine arguments (like `--max-model-len 8192` or `--enable-prefix-caching`) after `--port 8000` and before the `&` symbol to customize performance and behavior. For a complete list of available vLLM engine arguments, see the [vLLM Engine Arguments documentation](https://docs.vllm.ai/en/latest/configuration/engine_args.html). ```bash ["/bin/bash", "-c", "vllm serve mistralai/Mistral-7B-v0.1 --download-dir /mnt/direct-attached/vllm-cache --host 127.0.0.1 --port 8000 & sleep infinity"] ``` > **ℹ️ About Command Override**: This field allows you to specify exactly how the container should start, overriding the default Docker image command. #### Step 7: Proxy Settings In the **Proxy Settings** section, configure the reverse proxy for your deployment: **Port Configuration:** * **Proxy Port**: Set to `8000` (required to enable the reverse proxy and route external traffic to your vLLM service) **API Authentication (Optional but Recommended):** * **API Keys**: Add your API keys in the **API Keys** field to enable secure access through the reverse proxy. > **ℹ️ Proxy Port**: This setting is required to enable the reverse proxy layer. When set, all external traffic routes through the reverse proxy to your vLLM service on this port. > **💡 API Key Best Practices**: Use descriptive prefixes like `sk-prod-`, `sk-staging-`, or `sk-dev-` to identify different keys. Keep these keys secure and never commit them to version control. > **ℹ️ Without API Keys**: If you don't configure any API keys but set the Proxy Port, requests will pass through the reverse proxy without authentication requirements. #### Step 9: Deploy the Application 1. Review all configuration settings 2. Click **Deploy Application** 3. Monitor the deployment status in the **Applications Overview** or **Application Details** 4. To view runtime logs, click on your deployment name and navigate to the **Runtime Logs** tab to monitor the model download and vLLM initialization progress The vLLM initialization process will: * Pull the vLLM Docker image * Start the container with your specified configuration * Download the Mistral-7B model weights from Hugging Face * Initialize the vLLM server with the OpenAI-compatible API > **ℹ️ Deployment Time**: Initial deployment may take 5-10 minutes as the system downloads the \~14GB model weights. The application will show as **ONLINE**, but the vLLM OpenAI-compatible endpoints will not be accessible until initialization is complete. You can monitor progress by viewing the runtime logs in the application details page. ### Alternative: Deploy Using API You can also use the Denvr Dataworks API directly to create Custom Applications. #### Using the Swagger API Interface 1. **Access Swagger UI**: Navigate to 2. **Login**: Log in with your user information (tenancy name, username/email, and password) 3. **Find the Endpoint**: Locate `POST /api/v1/servers/applications/CreateCustomApplication` 4. **Try It Out**: Click "Try it out" to enable the request body editor 5. **Configure Request**: Customize and paste the JSON configuration below into the request body 6. **Execute**: Click "Execute" to create your custom application #### JSON Configuration Template ```json { "name": "vllm-mistral-7b-v1", "cluster": "Msc1", "hardwarePackageName": "g-nvidia-1xa100-20gb-pcie-6vcpu-45gb", "imageUrl": "docker.io/vllm/vllm-openai:v0.10.1.1", "imageCmdOverride": [ "/bin/bash", "-c", "vllm serve mistralai/Mistral-7B-v0.1 --download-dir /mnt/direct-attached/vllm-cache --host 127.0.0.1 --port 8000 & sleep infinity" ], "environmentVariables": { "HF_TOKEN": "hf_xxxxxxxxxxxxxxxxxxxxx", "XDG_CACHE_HOME": "/mnt/direct-attached/.cache", "HF_HOME": "/mnt/direct-attached/.cache/huggingface", "HUGGINGFACE_HUB_CACHE": "/mnt/direct-attached/.cache/huggingface/hub", "CUDA_CACHE_PATH": "/mnt/direct-attached/.cache/nv/ComputeCache", "PIP_CACHE_DIR": "/mnt/direct-attached/.cache/pip", "VLLM_CACHE_ROOT": "/mnt/direct-attached/.cache/vllm", "VLLM_ASSETS_CACHE": "/mnt/direct-attached/.cache/vllm/assets" }, "imageRepository": { "hostname": "https://index.docker.io/v1/", }, "resourcePool": "your-reserved-resource-pool", "proxyPort": 8000, "proxyApiKeys": ["your-api-key1", "your-api-key2s"], "persistDirectAttachedStorage": false, "personalSharedStorage": true, "tenantSharedStorage": true, "securityContext": { "runAsRoot": true, "containerUid": null, "containerGid": null } } ``` **Key Parameters to Customize:** * `name`: Your application name * `cluster`: Your cluster name (e.g., "Msc1") * `hardwarePackageName`: Match available hardware in your cluster * `resourcePool`: Your reserved resource pool name * `HF_TOKEN`: Your Hugging Face token * `proxyApiKeys`: Array of API keys for reverse proxy authentication ### Accessing Your vLLM Deployment #### Finding Your Application's DNS Address 1. Navigate to **Applications** → **Overview** 2. Find your vLLM deployment in the list - the DNS address is displayed in the overview table 3. Alternatively, click on the deployment name to view **Application Details** where you'll also find the DNS address Once you have the DNS address, you can access your vLLM endpoint at that address (e.g., `https://abc-123-456-789-012.cloud.denvrdata.com`) #### Authentication If you configured API keys, your vLLM endpoint will require authentication through the reverse proxy. Include one of your configured API keys in all requests as a bearer token: ```bash -H "Authorization: Bearer your-api-key" ``` Replace `your-api-key` with one of the API keys you set (e.g., `API_KEY_1`, `API_KEY_2`, etc.). > **ℹ️ Without API Keys**: If you didn't configure any API keys but set the Proxy Port, requests will pass through the reverse proxy without authentication requirements. ### Testing Your Deployment #### 1. Health Check First, verify that your vLLM server is running by checking the health endpoint: ```bash curl https://your-dns-address/health \ -H "Authorization: Bearer your-api-key" ``` A successful response returns `HTTP/1.1 200 OK` with no body, indicating the server is healthy and ready to accept requests. #### 2. List Available Models Once the health check passes, verify your model is loaded and accessible: ```bash curl https://your-dns-address/v1/models \ -H "Authorization: Bearer your-api-key" ``` This should return a response showing the `mistralai/Mistral-7B-v0.1` model is available: ```json { "object": "list", "data": [ { "id": "mistralai/Mistral-7B-v0.1", "object": "model", "created": 1737380356, "owned_by": "vllm", "root": "mistralai/Mistral-7B-v0.1", "parent": null, "max_model_len": 32768, "permission": [ { "id": "modelperm-e3d0f87e19a548b2be64ca274a4550a6", "object": "model_permission", "created": 1737380356, "allow_create_engine": false, "allow_sampling": true, "allow_logprobs": true, "allow_search_indices": false, "allow_view": true, "allow_fine_tuning": false, "organization": "*", "group": null, "is_blocking": false } ] } ] } ``` ### Using Your vLLM Deployment #### Using curl (Command Line) The simplest way to test your deployment is with curl commands: **Text Completion** ```bash curl https://your-dns-address/v1/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-api-key" \ -d '{ "model": "mistralai/Mistral-7B-v0.1", "prompt": "Explain machine learning in simple terms:", "max_tokens": 150, "temperature": 0.7 }' ``` **Metrics** vLLM exposes Prometheus metrics for monitoring performance and resource utilization: ```bash curl https://your-dns-address/metrics \ -H "Authorization: Bearer your-api-key" ``` This returns metrics like request counts, GPU memory usage, and latency measurements that you can use for monitoring and optimization. > **ℹ️ Note**: If you didn't configure API keys, omit the `-H "Authorization: Bearer your-api-key"` header from all commands. #### Simple Python Client Here's a simple Python client to get you started with your vLLM deployment: ```python import requests def test_vllm_deployment(endpoint_url, api_key): """Simple function to test your vLLM deployment.""" # Test basic completion response = requests.post( f"{endpoint_url}/v1/completions", headers={"Authorization": f"Bearer {api_key}"}, json={ "model": "mistralai/Mistral-7B-v0.1", "prompt": "Explain machine learning in simple terms:", "max_tokens": 100, "temperature": 0.7 } ) if response.status_code == 200: result = response.json() print("Generated text:", result['choices'][0]['text']) return True else: print(f"Error: {response.status_code} - {response.text}") return False # Example usage if __name__ == "__main__": endpoint_url = "https://your-dns-address" # Replace with your actual DNS address api_key = "your-api-key" # Replace with one of your configured API keys test_vllm_deployment(endpoint_url, api_key) ``` > **ℹ️ Note**: If you didn't configure API keys, you can omit the `Authorization` header or pass `None` as the `api_key`. #### Using with OpenAI SDK Your vLLM deployment is compatible with the OpenAI Python SDK: ```python from openai import OpenAI # Initialize client with your vLLM endpoint client = OpenAI( base_url="https://your-dns-address/v1", api_key="your-api-key", # Use one of your configured API keys ) # Generate text completion completion = client.completions.create( model="mistralai/Mistral-7B-v0.1", prompt="Explain the benefits of containerized ML inference:", max_tokens=150, temperature=0.7 ) print(completion.choices[0].text) ``` > **ℹ️ Note**: If you didn't configure API keys, you can set `api_key="not-needed"` or any string value. > > **💡 Base vs Instruction Models**: `mistralai/Mistral-7B-v0.1` is a base model, so we use `client.completions.create()` with a `prompt`. For instruction-tuned models (like `mistralai/Mistral-7B-Instruct-v0.1`), you would use `client.chat.completions.create()` with `messages`. ### Quick Reference #### Essential curl Commands Once your deployment is running, use these commands to interact with your vLLM endpoint. Replace `your-dns-address` with your deployment's DNS address and `your-api-key` with one of your configured API keys: ```bash # Health check curl https://your-dns-address/health \ -H "Authorization: Bearer your-api-key" # List available models curl https://your-dns-address/v1/models \ -H "Authorization: Bearer your-api-key" # Text completion curl https://your-dns-address/v1/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-api-key" \ -d '{ "model": "mistralai/Mistral-7B-v0.1", "prompt": "Your prompt here", "max_tokens": 100, "temperature": 0.7 }' ``` #### Common Request Parameters Key parameters for the `/v1/completions` endpoint: * **`prompt`**: The input text to generate completion for (required) * **`max_tokens`**: Maximum tokens to generate (optional, defaults to model's max length) * **`temperature`**: Randomness control, 0.0-2.0 (default: 1.0) * **`stream`**: Enable streaming responses (default: false) See the [vLLM OpenAI-Compatible Server docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) for all parameters. #### Troubleshooting | Issue | Solution | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Connection refused | Check the runtime logs to see if vLLM is still initializing | | Unauthorized error | Verify your API key matches one of the configured API\_KEY\_\* seen in the proxy settings | | Model not found | Ensure model name exactly matches: `mistralai/Mistral-7B-v0.1` | | RuntimeError: Engine core initialization failed. | This is likely an out-of-memory error — choose a GPU with more VRAM, use a smaller model or reduce batch size, or enable quantization (e.g., 4-bit) |