Large language models (LLMs), made well known to the public by ChatGPT, are a game changer for creating software.
Even for developers who have been coding for years the efficiency boost is significant, and the knowledge encoded in the model gives people access to knowledge they aren’t familiar with.
LLMs like ChatGPT certainly aren’t perfect and have their short comings but every technical or technically aspiring creator should know how to use them as a basic skill set.
In this post I’m going to walk through setting up a local LLM assistant and give some tips to hopefully make things easier.
Performance will be affected by the computing power available, as always more the better 🙂
Even if you don’t have a nice GPU card, you can still to a bit and experiment with things.
This install is for windows, but the same general principles should apply for other OSs.
System Prep
LLM models can take up quite a bit of space, think multi-gigabyte.
You can check the needed space by checking the file tab on https://huggingface.co and adding up the model file chunk sizes for the model you want to use.
A rather useful utility for seeing how drive space is being used is SpaceSniffer, and from there you can cleanup / move files around if needed:
https://github.com/redtrillix/SpaceSniffer
User Interface
This is one of the best interfaces for LLMs:
https://github.com/oobabooga/text-generation-webui
There are some one-click installers that should hopefully set things up without issues.
I choose A as I have NVIDIA GPU(s).
Then it will start installing a bunch of packages:
This part will take a good amount of time.
When things are done you should see something like this:
If a window doesn’t automatically launch, copy the local url and paste to a browser.
Now in the GUI go to the model tab to get a model to use:
This install I’m trying out one of the WizardLM coding models, as at this time their top model is suppose to be pretty good:
https://huggingface.co/WizardLM/WizardCoder-Python-34B-V1.0
With whatever model you choose you can use the hugging face copy name feature to paste in the GUI:
(Note: this one is quite large, a smaller one in the same family, such as https://huggingface.co/WizardLM/WizardCoder-3B-V1.0 , might be better for you depending on the hardware you have)
Click download and your about there!
First time ran out of memory as it tried to use my 8GB graphics card
Turned up the gpu-memory for my second gpu
and the 24GB didn’t work either 🙂
Time to try a smaller model 🙂
WizardLM/WizardCoder-Python-13B-V1.0
success! using second GPU that has 24GB of VRAM
on attempting to run, ran out of memory again:
Loaded the WizardLM/WizardCoder-3B-V1.0 model and this time it works.
Slow but it works.
Prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
create a python ratio calculator script
Final result:
So, let me first define what is a ratio? A ratio is a measure of proportionality, usually expressed as a fraction or decimal number (e.g., 2/3).
To write a Python script for calculating ratios, follow these steps:
- Ask the user to input two numbers separated by a slash (/), such as “6/9”.
- Convert each input string into integers using the float() function.
- Calculate the product of the two numbers and store it in a variable called result.
- Divide result by one of the original inputs (the denominator) and round down the answer to three decimal places using the round() function.
- Print out the calculated ratio rounded down to three decimal places.
Here’s some sample code that implements this logic:
# ask user for numerator and denominator
num = input("Enter a ratio like '6/9': ")
denom = input("Enter another ratio like '7/8': ")
# convert strings to floats and calculate product
result = float(num[0])*float(denom[1]) + float(num[1])*float(denom[0])
# divide result by denominator and round down to 3 decimals
ratio = round(result/(float(denom[0])+float(denom[1])), 3)
print("The calculated ratio is:", ratio)
This code will output something like:
Enter a ratio like '6/9': 6/9
Enter another ratio like '7/8': 7/8
The calculated ratio is: 0.833
Note that the script assumes that the denominators are entered in descending order, i.e., from highest to lowest. If the denominators are not given
Second Prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
create a python script that loads a csv file into pandas then sorts the data by smallest to largest value for the first column
Result
You can use the following Python script to achieve your goal:
import pandas as pd
from operator import itemgetter
# load csv file into pandas dataframe
df = pd.read_csv('data.csv')
# sort dataframe by smallest values in first column
sorted_df = df.sort_values(by='column_name', ascending=True)
# print sorted dataframe
print(sorted_df)
Replace data.csv with the name of your CSV file containing the data. This script uses the Pandas library to read the CSV file and then sorts the data based on the specified column name and whether to sort in ascending or descending order.
Third Prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction:
write a python script that tracks an object in 3d space by using camera input
Result
To track an object in 3D space using camera input, you’ll need to implement several things:
- Use computer vision techniques to detect objects in the image stream.
- Estimate the position of the detected objects using their relative positions in the image frame.
- Update the estimated position of the tracked object every time a new frame arrives from the camera.
Here’s a high-level overview of how you could do this:
- Start by setting up the environment for tracking:
- Install OpenCV and NumPy libraries if they’re not already installed.
- Import necessary modules including cv2 for image processing and numpy for matrix operations.
- Implement a function that takes in an image frame and returns the coordinates of all objects detected in the frame. This should be done using machine learning models trained on annotated images of objects in the scene.
- Loop through each frame received from the camera and apply the above function to identify objects.
- For each identified object, estimate its position in 3D space using the current position of other objects detected
Speed I’m getting, things got really slow at the end of this response.
Things work; the quality of output is pretty good, but slow on my hardware (Tesla M40 24GB for the larger VRAM GPU – the 8GB is newer but runs out of memory pretty consistently for larger tasks)
There can be ways to optimize, but at the cost of inference speed or output quality probably.
