My #AI Notes – Sharing tips and tricks from my AI playground

Just starting this page to share what AI tools I have been playing with. Kind of a journal, but might write up separate articles if I end up using any of these in my regular workflow.

20240727
Use “–verbose” option with ollama to display the performance of the model.
e.g. ollama run llama3.1:70b-instruct-q4_0 –verbose

total duration: 1m29.230942s
load duration: 40.905541ms
prompt eval count: 26 token(s)
prompt eval duration: 2.024605s
prompt eval rate: 12.84 tokens/s
eval count: 1013 token(s)
eval duration: 1m27.162336s
eval rate: 11.62 tokens/s

Can’t run the 405B model on my Mac Studio M1 Ultra with 64GB RAM.
Will try this soon, to distribute the load to 2 (or more) Macs.
Eco Labs + MLX.
https://x.com/ac_crypto/status/1815969489990869369
https://x.com/awnihannun/status/1815972700097261953

Hard to say that I need the largest model. But the idea of being able run that at home seems cool. The 70B model is the one I use on my laptop. Waiting for codellama to be updated to use 3.1.

20240724
Llama 3.1 is out, available in 8B, 70B, and 405B versions. It’s big. I need to cleanup my Mac Studio to play with the 405B, currently just testing with the 70B version.

To cleanup Ollama a little, use “ollama list” to show the currently installed models and “ollama rm <model name>” to remove a model. New 405B will take up 231GB.

20240723
Getting Copilot setup again. Want to play around with coding with AI again. (Kind of annoying that I don’t get the 30-day free trial as participated in the year-long Technical Preview.)
Try Codeium too.
I am setting it up for Xcode and VSCode. You will need https://github.com/intitni/CopilotForXcode and https://github.com/intitni/CodeiumForXcode

Will need to install nvm and node; use “which node” to check the path of your node installation.

20240517
Been playing with Llama 3 on my Mac. Easy to install with Ollama https://ollama.com. Might try out Gemma and Mistral eventually too. Llama 3 isn’t great for Chinese output. There is another tweaked model. https://huggingface.co/shenzhi-wang/Llama3-70B-Chinese-Chat , will figure out how to use that soon.

20240516
Using Audio Hijack from Rogue Amoeba to transcribe some old voice recording. It’s not bad. Powered by Whisper https://github.com/openai/whisper
Also found a free app, Aiko, that use Whisper to do the same on the Mac. https://apps.apple.com/us/app/aiko/id1672085276