FREE 32GB VRAM Server for Running AI Models Locally – Transcript

00:00What if I told you that you could run powerful large AI models completely free? Even if your computer doesn't have a dedicated GPU, no expensive graphics card, no powerful PC, and no paid cloud subscription required. In this video, I'll show you how to use Kaggle's free cloud GPUs to run AI models and access them directly from your own computer.

00:21Today, I want to show you step-by-step how to run large AI models using Kaggle's free GPU resources. We'll start by creating a Kaggle notebook and enabling GPU acceleration. Then, we'll install Ollama, download an AI model from Hugging Face, and use a Cloudflare tunnel to connect the model to our local computer. By the end of this tutorial, you'll be able to interact with a powerful AI model directly from your terminal without needing a high-end GPU

00:48in your own computer. I'll provide all the necessary commands in the video description, so you can easily follow along. Let's get started. Create a Kaggle notebook. First, open the Kaggle website. Don't worry, I'll leave the link in the description below, so you can access it easily. Before we start, we need to make sure GPU access is enabled for your Go to your Kaggle account settings and complete the required phone verification if you haven't already. This step is important

01:14because Kaggle may require account verification before allowing to use its free GPU resources. Once your account is verified, return to the Kaggle homepage. Now, click on create and select new notebook. This will create a new Kaggle notebook where we'll set everything up.

01:31Enable the free GPU. Now that our notebook is ready, let's enable GPU acceleration. On the right-hand side, open the notebook settings. Find the acceleration option and select GPU T4 2 if it is available. This gives us access to two Nvidia T4 GPUs, each with 16 GB of data. Keep in mind that this doesn't mean every model can use the combined 32 GB automatically. Memory usage depends on how the model loaded and distributed

01:58across the GPUs. Once you have selected the GPU, we are ready to install Ollama. Install Ollama. Now, click add code to create a new code cell. I'm going to paste the first command here. Don't worry about typing everything manually.

02:12I'll include all the commands in the video description. Run the cell and wait for it to finish. Next, create another code cell. This time, we'll install Ollama, which allows us to download and run AI models. Paste the installation command and click run. Wait a few moments while Kaggle installs Ollama.

02:30Once the installation is complete, you should see a confirmation message. Perfect. Ollama is now installed. Start the Ollama server. Next, we need to start the Ollama server. Create another code cell and paste the server startup command. Now, click run.

02:45Once the server starts successfully, Ollama will read to load and third-party AI models. Keep the server running in the background because we will need it for the next steps. Find an AI model on Hugging Face. Now comes the interesting part, choosing the AI model we want to run. For this, we are going to use Hugging Face. If you are already familiar with Hugging Face, you probably know that it hosts thousands of AI models. If you're new to it, don't worry. I'll leave the website link in

03:11the description below. Open Hugging Face and click on models. From the available filters, select text generation. You can also sort the results by the most downloaded models to explore popular options. If you're looking for a specific type of model, you can use the search bar to narrow down the results.

03:28For example, you can search for models with fewer content restriction, that is what you want to experiment with. Once you find a model you like, open its model page. Now, look for a compatible GGUF version that can be loaded through Ollama. For this tutorial, I'm going to choose a Q4KM quantized model.

03:46This is often a good balance between model quality, memory usage, and performance. However, the actual speed and memory requirements will depend on the model size. Copy the model reference or download command. Now, let's go back to Kaggle. Download and load the AM model. Back in our Kaggle notebook, create another code cell. Paste the command for the model we selected. Now, click run. Ollama will begin downloading the files. Depending on the model size

04:13and Kaggle's network speed, this might take a few minutes. Once the download is complete, Ollama will load the model so we can test it. You can see that our model is now responding successfully. That is great. Since our goal is to access this model from our own computer, we don't need to continue chatting with it inside the notebook. We can stop the interactive model session while keeping the Ollama server running. Now, we are ready to connect it to our local machine. Set up Cloudflare tunnel. To

04:39access our Kaggle-hosted AM model from our computer, we're going to use Cloudflare tunnel. This allows us to create a public HTTPS endpoint that forwards requests to the Ollama server running inside Kaggle. Create another code cell and paste the Cloudflare installation command. Run the cell and wait for the installation to finish.

04:59Create one more cell. Paste the command that starts the Cloudflare tunnel and connects it to the Ollama server. Now, run the command. After a few moments, Cloudflare should generate a temporary public URL. This is the URL we'll use it to connect our local computer to the AM model. Copy the URL and keep it somewhere safe.

05:18Important: A temporary Cloudflare tunnel URL may be publicly accessible. Don't share it openly or use it to expose sensitive data. For a secure setup, configure authentication and access controls. Also, remember that the tunnel will only work while the Kaggle session, Ollama server, and Cloudflare tunnel remain active. Connect the model to our local computer. Now that everything is running on Kaggle, let's switch to our local computer. For this demonstration, I'm using Mac. Open your terminal. If

05:47you're using a coding agent that supports custom OpenAI compatible endpoints, open its model configuration settings. In this tutorial, we are using local coding agents custom model endpoint option. If you haven't installed the coding agent yet, I've already created a separate tutorial showing how to install and configure it.

06:05You can check that video using the link in the description below. Now, let's continue. Find the option to add a custom endpoint. When prompted for the endpoint URL, paste the Cloudflare tunnel URL we copied earlier. At the end of the URL, add /v1. This points the client to Ollama's OpenAI compatible API. Press enter to continue. Next, application may ask for an API key. If you're using a compatible Ollama

06:31endpoint without API key authentication, the client may accept a placeholder value such as Ollama. Remember, this placeholder is not a security mechanism. A public endpoint should be protected separately. Now, continue to the model selection screen.

06:46Select the AI model. Here, the application should detect the models available through our custom endpoint. Look at the list and find the model we downloaded earlier. You should see the same model name including its quantization variant where applicable.

07:01For this demonstration, we are selecting our Q4 KM model. Choose the model and confirm your selection. If the application asks for any additional option settings, you can keep the default unless your chosen model requests something different. Now, our custom model is configured. Let's test it.

07:18Test the AI model. Open the coding agent from your terminal. You should now see the model we configured. Let's start with a simple prompt. I'm going to type, "Hi, what can you help me with?" Press enter, and there you go. Your AI model is responding directly inside the terminal. Now, let's ask another question. "Which AI model are you using?" You can see that the model is generating a response. Keep in mind that a model's self-reported identity isn't always reliable. The model name shown in

07:45the client configuration or written by the API is a better way to verify which model is actually running. The important thing is that the same model is running on Kaggle's cloud GPUs while we are accessing it from our own computer. Our local computer is simply sending requests and if displaying the responses. The heavy AI processing happening remotely on Kaggle. This means we can experiment with model that might otherwise be too demanding for our local hardware. And that's how you can use

08:11Kaggle's free GPU resources to run powerful AI models remotely. We've successfully installed Ollama, loaded an AI model from Hugging Face, created a Cloudflare tunnel, and connected the model to our local terminal. The best part is that we didn't need to buy an expensive graphics card or upgrade our computer. Just remember that Kaggle has GPU usage quotas and session limits. So, this setup is best suited for learning and experimentation and temporary projects rather than continuous production use. And that's it. You can

08:39now run your powerful AI models for free using Kaggle's cloud GPU without needing an expensive computer. If you found this video helpful, give it a like and share with someone who wants to run AI models for free. Don't forget to subscribe for more tutorials. Thank you for watching, and I'll see you in the next video.