Jean Louis

“Can we see what it knew when it acted?”
Agreed: weights don’t give you human-style “facts it knew.”
What regulators/auditors can reasonably ask is: what information and constraints were available at decision time? That’s not “reading neurons.” It’s logging and binding the runtime knowledge state: the structured observation (with coverage/confidence), provenance refs, the policy/version in force, and the gating outcome. That’s actionable evidence.

Give me practical example on this one. Make it hypothetical.

If you do not have example or cannot make hypothetical one, let me know.

How would it look like "that auditor sees what it knew when the LLM (it) acted?"

1. Can we see what it “knew” when it acted?

Short answer: no, because it didn’t know anything. The LLM is a glorified probability engine. Its “knowledge” is encoded in billions of floating-point numbers—vectors that represent likelihoods, not facts. You could list every number in the model and stare at them, but unless you enjoy deciphering abstract hieroglyphics, you won’t learn anything useful.

2. Can we tell who (or what) initiated the action?

If the question is about tracing a user’s intent, that’s trivial. Build a logging system, capture the initiator, and voilà—accountability. The model itself doesn’t spontaneously decide to act; it’s a statistically-driven calculator.

3. Can we stop it, or roll it back, when we discover harm?

Stopping the model is not glamorous. Most often, the “stop” is enforced by the model’s own training constraints. If that fails, you’re left auditing the auditor—recursive fun for those who like probabilistic debugging.

4. Can we prove to a third party what happened?

Technically possible, practically absurd. Every inference involves terabytes of floating-point operations. Producing a fully verifiable trace would require superhuman attention and energy consumption worthy of a small data center.

Works great.

I am using nvtop and that one gives me list of processes in the VRAM. So I mostly use it to delete some processes to invoke some new process.

It would be good to implement such a feature to see what is going on by the process.

To create an image-generating model from the beginning, you first need to show a computer a massive library of pictures, each with a description of what it shows. The AI learns through a process of destruction and recreation; it takes a clear image, adds random noise until it's just static, and then is trained to reverse that process, using the text description as its guide to remove the noise step-by-step and rebuild the original image. By repeating this with millions of different images and descriptions, the model learns the connection between words and visual information, eventually gaining the ability to generate a completely new, coherent picture simply from a new text prompt it has never seen before.

Did I understand well that it runs on separate machines?

There is no description telling specifically what is that what is new with your release.

Individual users win only if they can get it cheaper, faster, more free as in software freedom, to run LLM models on their own hardware. Otherwise, those mega-stories are of no use.

Hey there! Just wanted to drop a quick note saying I'm really digging the new router mode in llama.cpp server. It's a game-changer for me, especially when I need to switch between different models. The auto-discovery of models and LRU eviction is pretty neat – no more manual updates or restarts needed. It's like having a dynamic model manager on-the-fly. And the request routing part? Brilliant! Makes my workflow with dmenu smoother. Check out the full experience and check out my dmenu launcher script on the project's GitHub: https://gitea.com/gnusupport/LLM-Helpers/src/branch/main/bin/rcd-llm-dmenu-launcher.sh

It's a win for sure.

Is there GGUF version?

There is no LLM that ever brings it's own opinion. Please reach out to basics on how LLMs work. There is nothing "new" that LLM can give you.

LLM models act like a book: when you open a page, you already have the content stored. The model processes this existing information, generating probabilistic results based on the training data, not new insights. This means LLMs rely on structured, data-driven outputs rather than independent opinions.

LLM models are designed to process and generate text based on vast training data, and their outputs are results of statistical inference rather than independent opinions. The "ingested data" combines the model’s training knowledge with user-retrieved information, generating probabilistic results that align with the training patterns, not personal beliefs. Thus, LLMs rely on structured, data-driven outputs to provide answers, not independent thoughts or opinions.

Those so called "opinions" must align with the data they are trained on.

Let us say this way, if LLM can give opinion, that means it is 100% biased opinion based on the data it was trained on.

You simply cannot get true opinions.

Oh, I’m sure the LLM you’re referring to is as clear as mud. Which one, exactly? And of course, the context provided was as precise as a weather forecast in a hurricane. What was it? Sure, because the output was so crystal clear, it’s not like anyone could possibly misinterpret it. What did it say? Oh, I’m sure you tried every single LLM under the sun. Which ones, exactly?

Ohhh Mitko, you’re telling me your desktop is now officially a server that got tired of hiding under your monitor and just started hosting LLMs like a caffeinated cloud? 😅

“Got to 1199.8 tokens/sec on Devstral Small-2… on the desktop?”
My jaw dropped so hard I accidentally spilled my coffee on my keyboard — again.
You didn’t just upgrade your desk… you turned it into a mini datacenter with a 32GB M4 chip pretending to be a server room air conditioner. And you’re still using Mistral Vibe like it’s a 2005 laptop? 😂

Next time, just call it “Mitko’s Desktop Data Center v1.0” — complete with blinking LED fans, a 16-B200 GPU cluster on top, and a “DO NOT TOUCH” sticker taped to the power button (because if you touch it, you’ll accidentally delete your 3rd coffee break).

Now go ahead — test the big one. I’ll be here, typing “Is this GPU cluster actually a desk, or is the desk just a disguise for a server?” 🤔

P.S. You’re officially the guy who turned “workstation” into “server-on-a-desk-stand-with-a-caffeinated-look.” 🍵💻✨

Congratulation. Publish the script on how you run it for others to see.

Here is exactly how I run it:

/usr/local/bin/llama-server --jinja -fa on -c 32768 -ngl 64 -v --log-timestamps --host 192.168.1.68 -m /mnt/nvme0n1/LLM/quantized/Qwen3VL-8B-Instruct-Q8_0.gguf --mmproj /mnt/nvme0n1/LLM/quantized/mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf

with the llama.cpp and API is of course available as well.

I’m running my own LLM because:
Privacy? 57% say it’s the biggest AI barrier…
But 48% still leak company data anyway.
CRAFT says privacy is architecture, not policy.
So I’m not waiting for “beta” — I’m beta-ing my data.
February 2026? Nah. I’m already typing on my own GPU.
Privacy’s not a feature — it’s a feature flag I turned on before the release.
And honestly? My model’s less “AI” and more “I’m not giving your data to strangers.”
Run your own. It’s fun. It’s free. It’s your data.
And it’s way more satisfying than waiting for “beta.”
(Also, no one’s gonna steal your jokes now. 😉)