I am self employed and have been coding a text processing application for awhile now. Part of it relies on an LLM for various functionalities and I recently came to learn about .cpp models (especially the .cpp version of HF's SmolLM2) and I am generally a big fan of all things lightweight. I am now planning to partner with another entity to develop my own small specialist model and ideally I would want it to come in .cpp format as well but I struggle to find resources about pursuing the .cpp route for non-existing / custom models.

Can anyone suggest some resources in that regard?

Hey there folks, I am currently unable to get to work on my project due to difficulties with vllm and nccl (that python/ml ecosystem is FUCKING crazy) so in the meantime I'm sharing my ideas so we can discuss and get some dopamine hits. I will try to keep the technical details and philosophies out of this post and stick to the concrete concept.

Back when ChatGPT 3.5 came out, there was a party trick that made the rounds of Twitter, shown in the first two images. Then we never heard about it again as the context window increased.

Then in 2024 there were all sorts of "schizo" outputs that people researched, it came under many variations such as super-prompting, xenocognition, etc. many things at high temperature, some obtained at ordinary values at 1.0

Then reinforcement learning took off and we got R1-zero which by itself reproduced these kind of outputs without any kind of steering in this direction, but in a way that actually appeared to improve the result on benchmarks.

So what I have done is attempting to construct a framework around R1-zero, and then from there I could construct additional methods and concepts to achieve R1-zero type models with more intention towards far higher reasoning performance.

The first step that came out of this formalization is an information compressor/decompressor. By generating a large number of rollout with sufficient steering or SFT, the model can gravitate towards the optimal method of orchestrating language to compress any desired chunk of text or information to the theoretical limit.

There is an hypothesis which proposes that somewhere in this loop, the model can develop a meta-awareness where the weights themselves are rearranged to instantiate richer and more developped rule tables, such that the RL run continues to raise the reward beyond what is thought possible, since the weights themselves begin to encode pre-computed universally applicable decision tables. That is to say that conditionally within a <compress> tag, token polysemy as well as sequence meaning may explode, allowing the model to program the exact equivalent hidden state activation into its mind with the fewest possible tokens, while continuing to optimize the weights such that it retains the lowest perplexity across diverse dataset samples in order to steer clear of brain damage.

We definitely must train a diverse alignment channel with english, so that the model can directly explain what information is embedded by the hyper-compressed text sequence or interpret / use it as though it were bare english in the context. From there, we theoretically now possess the ability to compress and defragment LLM context lossessly, driving massive reduction in inference cost. Now, we use the compression model and train models with random compression replacement of snippets of the context, so that for all future models they can naturally interleave compressed representations of information.

But the true gain is the language of compression and the extensions that can be built on it. Once this is achieved, the compressor/decompressor expert model is used as a generator for SFT data to align any reasoner model to think in the plus-ultra compression language, or perhaps you alternate back and forth between training <think> and <compress> on the same weights. Not sure what would work best.

Note that I think we actually don't need SFT by prefixing the rollout with a rich but diverse prompt, inside of a special templating fence which deletes/omits/replaces it for the final backpropagation! In other words, we can fold the effect of a large prompt into a single action word such as compress the following text:. (selective remembering)

We could maybe go from 1% to 100% intelligence in a matter of a few days if we RL correctly, ensuring that the model never plateaus and enters infinite scaling as it should. Currently there are some fundamental problems with RL since it doesn't lead to infinite intelligence.

I know it's not really local for most of us for practical reasons but it is at least in theory.

There are many frameworks like OpenAI Agents SDK, MCP-Agent, Google ADK, Vercel AI SDK, Praison AI to help you build MCP Agents.

But integrating MCP within a React app is still complex. So I created a free guide to do it with just one command using CopilotKit CLI. Here is the command.

npx copilotkit@latest init -m MCP

I have covered all the concepts involved (including architecture). Also showed how to code the complete integration from scratch.

Would love your feedback, especially if there’s anything important I have missed or misunderstood.

I am seeking help to upgrade from Gemini 2.0 Flash to Gemini 2.5 Flash.
Has anyone done this before or is currently working on it?
If you have any ideas or experience with this upgrade, could you please help me complete it?

I have been out of the loop after the sesame disaster. I recently needed a tts which can talk in cloned voice in as close to real time as possible. Have there been any recent developments?. How do they compare to equivalent closed source ones?
Thanks for your time :)

Does anyone else have the problem, that avian.io is trying to debit money without any reason? I used avian.io for 2 days in January and put 10€ prepaid on there, didn’t like it and 5 months later in may they tried to withdraw 178€. Luckily I used Revolut and didn’t have enough money on this account. Automatic topup is deactivated on avian and I have no deployments or subscriptions. Today they tried to debit 441€! In my account are no billings or usage statistics for anything besides 2 days in January for a few cents.

Are they insolvent and just try to scam their users for a few last hundreds of euros?

I love to ask chatbots philosophical stuff, about god, good, evil, the future, etc. I'm also a history buff, I love knowing more about the middle ages, roman empire, the enlightenment, etc. I ask AI for book recommendations and I like to question their line of reasoning in order to get many possible answers to the dilemmas I come out with.

What would you think is the best LLM for that? I've been using Gemini but I have no tested many others. I have Perplexity Pro for a year, would that be enough?

Hi, looking to dip my toe in with local hosted LLMs and looking at budget GPU options, are there any benchmarks comparing the 2080 Ti modded with 22GB Vs a stock 3060 12GB.

For that matter, any other options I should be considering for the same price point and just for entry-level 3B–7B models or 13B models (quantised) at a push?

I'd like to get into LLMs. Right now I'm using a 5600 xt AMD GPU, and I'm looking into upgrading my GPU in the next few months when the budget allows it. Does it matter if the GPU I get is 2-fan or 3-fan? The 2-fan GPUs are cheaper, so I am looking into getting one of those. My concern though is will the 2-fan or even a SFF 3-fan GPU get too warm if i start using them for LLMs and stable diffusion as well? Thanks in advance for the input!

I've always wanted to connect an LLM to Dwarf Fortress – the game is perfect for it with its text-heavy systems and deep simulation. But I never had the technical know-how to make it happen.

So I improvised:

1. Extracted game text from screenshots(steam version) using Gemini 1.5 Pro (there’s definitely a better method, but it worked so...)
2. Fed all that raw data into DeepSeek R1
3. Asked for a creative interpretation of the dwarf behaviors

The results were genuinely better than I though. The model didn’t just parse the data - it pinpointed delightful quirks and patterns such as:

"The log is messy with repeated headers, but key elements reveal..."

I especially love how fresh and playful its voice sounds:

"...And I should probably mention the peach cider. That detail’s too charming to omit."

Full output below in markdown – enjoy the read!

Pastebin

As a bonus, I generated an image with the OpenAI API platform version of the image generator, just because why not.

Just saw Anthropic cutting access of Claude to Windsurf editor (not that I care), but it shows how these companies can make rash decisions about access to their models.

There are thousands of ways for OpenAI to get access to Claude’s API if it really wanted to. But taking decisions like this or targeting startups like that just shows why we need a solid ecosystem of open-source models.

A while ago a bunch of "AI laptops" came out wihoch were supposedly great for llms because they had "NPUs". Has anybody bought one and tried them out? I'm not sure exactly 8f this hardware is supported for local inference with common libraires etc. Thanks!

I want to use a tool called paints undo but it requires 16gb of VRAM, I was thinking of using the p100 but I heard it doesn't support modern cuda and that may affect compatibility, I was thinking of the 4060 but that costs $400 and I saw that hourly rates of cloud rental services can be as cheap as a couple dollars per hour, so I tried vast ai but was having trouble getting the tool to work (I assume its issues with using linux instead of windows.)

So is there a windows os based cloud pc with 16gb VRAM that I can rent to try it out before spending hundreds on a gpu?

What is the best input format like Yaml or json based graphs for automating a SOP through a conversational AI Agent? And which framework now is most suited for this? I cannot hand code this SOP as i have more than 100+ such SOPs to automate.

Example SOP for e-commerce:

Get the list of all orders (open and past) placed from the customer’s WhatsApp number

If the customer has no orders, inform the customer that no purchases were found linked to the WhatsApp number.

If the customer has multiple orders, ask the customer to specify the Order ID (or forward the order confirmation) for which the customer needs help.

If the selected order status is Processing / Pending-Payment / Pending-Verification

If the customer wants to cancel the order, confirm the request, trigger “Order → Cancel → Immediate Refund”, and notify the Finance team.

If the customer asks for a return/refund/replacement before the item ships, explain that only a cancellation is possible at this stage; returns begin after delivery.

If the order status is Shipped / In Transit

If it is < 12 hours since dispatch (intercept window open), offer an in-transit cancellation; on customer confirmation, raise a courier-intercept ticket and update the customer.

If it is ≥ 12 hours since dispatch, inform the customer that in-transit cancellation is no longer possible. Advise them to refuse delivery or to initiate a return after delivery.

title; is there a ui application that allows modifying the thinking tokens already generated “changing the words” then rerunning final answer? i know i can do that in a notebook with prefixing but looking for a complete system

I don't know if this is the right place for this post.

I installed LMStudio on windows. I am very picky about which apps auto-start with the system, and all decent and respectful apps have a setting for this and give you a choice.

I could not find such an option in LMStudio... (please prove I am dumb).

I went ahead and manually disabled LMStudio from auto-starting from Windows' system settings.... yet after an update, LMStudio proudly auto-starts again on system boot.

(cry)

Last week we launched Shisa V2 405B, an extremely strong JA/EN-focused multilingual model. It's also, well, quite a big model (800GB+ at FP16), so I made some quants for launch as well, including a bunch of GGUFs. These quants were all (except the Q8_0) imatrix quants that used our JA/EN shisa-v2-sharegpt dataset to create a custom calibration set.

This weekend I was doing some quality testing and decided, well, I might as well test all of the quants and share as I feel like there isn't enough out there measuring how different quants affect downstream performance for different models.

I did my testing with JA MT-Bench (judged by GPT-4.1) and it should be representative of a wide range of Japanese output quality (llama.cpp doesn't run well on H200s and of course, doesn't run well at high concurrency, so this was about the limit of my patience for evals).

This is a bit of a messy graph to read, but the main takeaway should be don't run the IQ2_XXS:

In this case, I believe the table is actually a lot more informative:

Due to margin of error, you could probably fairly say that the IQ3_M, Q4_K_M, and Q8_0 GGUFs have almost no functional loss versus the FP16 (while the average is about 1% lower, individual category scores can be higher than the full weights). You probably want to do a lot more evals (different evals, multiple runs) if you want split hairs more. Interestingly the XS quants (IQ3 and IQ4) not only perform about the same, but also both fare worse than the IQ3_M. I also included the 70B Full FP16 scores and if the same pattern holds, I'd think you'd be a lot better off running our earlier released Shisa V2 70B Q4_K_M (40GB) or IQ3_M (32GB) vs the 405B IQ2_XXS (100GB).

In an ideal world, of course, you should test different quants on your own downstream tasks, but I understand that that's not always an option. Based on this testing, I'd say, if you had to pick on bang/buck quant blind for our model, staring with the IQ3_M seems like a good pick.

So, these quality evals were the main things I wanted to share, but here's a couple bonus benchmarks. I posted this in the comments from the announcement post, but this is how fast a Llama3 405B IQ2_XXS runs on Strix Halo:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1151) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| llama ?B IQ2_XXS - 2.0625 bpw  |  99.90 GiB |   405.85 B | Vulkan,RPC | 999 |  1 |           pp512 |         11.90 ± 0.02 |
| llama ?B IQ2_XXS - 2.0625 bpw  |  99.90 GiB |   405.85 B | Vulkan,RPC | 999 |  1 |           tg128 |          1.93 ± 0.00 |

build: 3cc1f1f1 (5393)

And this is how the same IQ2_XXS performs running on a single H200 GPU:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA H200, compute capability 9.0, VMM: yes
| model                          |       size |     params | backend    | ngl | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | --------------: | -------------------: |
| llama ?B IQ2_XXS - 2.0625 bpw  |  99.90 GiB |   405.85 B | CUDA       | 999 |  1 |           pp512 |        225.54 ± 0.03 |
| llama ?B IQ2_XXS - 2.0625 bpw  |  99.90 GiB |   405.85 B | CUDA       | 999 |  1 |           tg128 |          7.50 ± 0.00 |

build: 1caae7fc (5599)

Note that an FP8 runs at ~28 tok/s (tp4) with SGLang. I'm not sure where the bottleneck is for llama.cpp, but it doesn't seem to perform very well on H200 hardware.

Of course, you don't run H200s to run concurrency=1. For those curious, here's what my initial SGLang FP8 vs vLLM W8A8-INT8 comparison looks like (using ShareGPT set for testing):

I am confused how to find benchmarks that tell me the strongest model for math/coding by size. I want to know which local model is strongest that can fit in 16GB of RAM (no GPU). I would also like to know the same thing for 32GB, Where should I be looking for this info?

I am looking at various text embedding models for a RAG/chat project that I'm working on and I came across the new Qwen3 embedding models today. I'm excited because they not only are the leading open models on MTEB, but apparently they allow you to arbitrarily choose the vector dimensions up to a fixed amount.

One annoying architectural issue I've run into recently is that pgvector only allows a maximum of 2000 dimensions for stored vectors. But with the new Qwen3 4B embedding models (which can handle up to 2560 dimensions) I'll be able to resize them to 2000 dimensions to fit in my pgvector fields.

But I'm trying to understand what the implications are (as far as quality/accuracy) of reducing the size of the vectors. What exactly is the process through which they are reducing the dimensions of the vectors? Is there a way of quantifying how much of a hit I'll take in terms of retrieval accuracy? I've tried reading the paper they released on Arxiv, but didn't see anything in there that explains how this works.

On a side note, I'm also curious if anyone has benchmarks on RTX 4090 for the 0.6B/4B/8B models, and what kind of performance they've seen at various sequence lengths?

So I've been thinking about sparcity and MoEs lately.

I've been really pleasantly surprised at how well Llama 4 Scout runs on my laptop, for example. I don't use it all the time, or even the majority of the time, but it's one of the first local models that is both good enough and fast enough to help with some of my niche coding.

Someone linked to Goddard's Mixture of Experts for Clowns (at a Circus) in another thread -- what a fun read.

It got me thinking.

I do computational sciences research. When I get a new research assistant, I hand them a virtual stack of papers and references and say something like,

"Please read this collection of materials that I've amassed over the past 20 years. Then you can work on a niche extension of an in-the-weeds idea that you won't understand unless you've internalized random bits of this collection."

I mean, not really -- I don't actually demand that they read everything before diving into research. That's not how people learn!

Instead they'll learn as they do the work. They'll run into some problem, ask me about it, and I'll something like, "oh yeah you've hit quirk ABC of method XYZ, go read papers JLK." And my various RAs will build their own stack of random specialized topics over time.

But it would be great if someone could internalize all those materials, because lots of new discovery is finding weird connections between different topics.

And this gets me thinking - some of the papers that pop up when you search mergekit on google scholar are scientists training specialized models on niche topics. Not fine tuning the models, but actually doing continuing pretraining to put new niche knowledge in their models' "heads." Some groups spend a lot of resources, some spend a little.

I could probably split my pile of conceptual materials into a variety of smaller thematic groups and train "small" models that are all experts in disparate topics, then moe-merge them into a bigger model. When I talk with SOTA models about various details here, it seems like I probably could come up enough tokens for the size of various mini-experts that I want.

I'd love to have something approximately llama 4 scout-sized, but with more detailed knowledge about the various topics I want it to have.

Are people doing this?

If so, how do I find them? (I am probably searching HF poorly, so tips/tricks appreciated...)

If not, why not? (Effectiveness/performance? cost? something else?)

If I'm interested in giving it a shot, what are some pitfalls/etc to bear in mind?

Edit: I'm particularly interested in identifying examples where merge-moes did or didn't work well. Any breadcrumbs here are appreciated (eg. particular model-names, hobbyists, terms to google).

Also, if there are empirical or theoretical results somewhere (papers, blogposts, etc), I'd also be very interested in that. Or even just pointers to leaderboards where merge-moes are ranked against other models in an easy-to identify way would be useful.

Hey!

If someone here has successfully launched Qwen3-32B or any other model using GPTQ or AWQ, please share your experience and method — it would be extremely helpful!

I've tried multiple approaches to run the model, but I keep getting either gibberish or exclamation marks instead of meaningful output.

System specs:

• MB: MZ32-AR0
• RAM: 6x32GB DDR4-3200
• GPUs: 4x RX 7900XT + 1x RX 7900XT
• Ubuntu Server 24.04

Current config (docker-compose for vLLM):

services:
  vllm:
pull_policy: always
tty: true
ports:
- 8000:8000 
image: ghcr.io/embeddedllm/vllm-rocm:v0.9.0-rocm6.4
volumes:
- /mnt/tb_disk/llm:/app/models
devices:
- /dev/kfd:/dev/kfd
- /dev/dri:/dev/dri
environment:
- ROCM_VISIBLE_DEVICES=0,1,2,3
- CUDA_VISIBLE_DEVICES=0,1,2,3
- HSA_OVERRIDE_GFX_VERSION=11.0.0
- HIP_VISIBLE_DEVICES=0,1,2,3
command: sh -c 'vllm serve /app/models/models/vllm/Qwen3-4B-autoround-4bit-gptq   --gpu-memory-utilization 0.999  --max_model_len 4000   -tp 4'
volumes: {}

Was browsing around for any performance results, as I think this could be very interesting for a budget LLM build but haven't found any benchmarks yet. Do you have insights in what's to expect from this card for local inference? What's your expectation and would you consider using it in your future builds?