KalamAI generated its first real article yesterday. End to end, in production, no hand-holding. I read it top to bottom and my stomach sank a little -- it was fine. Just fine. For a tool whose entire point is good writing, fine is a failure.
For weeks KalamAI has been a pipeline in my head: give it a topic, it searches, extracts the pages, ranks the terms that matter, and writes. Phase 3 landed the writing engine a few days ago -- brief to scored article. The modules before it wired up the search and extract steps. So when the first article came out, the plumbing was clearly working. The words just did not land.
The problem was upstream of the writing. The ranker was choosing the wrong terms to build the piece around. It leaned on raw frequency -- how often a word shows up across the pages it crawled. Raw frequency loves boring words. Guide, best, important, tips: they are everywhere, so they float to the top, and the article ends up hugging generic phrasing instead of the specific stuff that makes a piece read like it was written by someone who actually knows the subject.
Log-odds and a background corpus
So I spent the day rebuilding the ranking as v2. Two changes matter most. First, log-odds instead of raw counts -- it does not ask how often a term appears, it asks how much more often it appears here than you would expect by chance. That surfaces the terms genuinely distinctive to the topic. Second, a background corpus: I wrote a script to build a term-frequency prior from a big pile of general text, so the ranker has a baseline of what normal English looks like and can measure the topic against it.
On top of that, embeddings. I moved to gemini-embedding-001 at 768 dimensions, L2-normalized, so terms cluster by meaning and not just spelling. A coverage gate makes sure the article touches the important clusters instead of over-indexing on one. And a weekly SERP cache with a fail-closed budget, so I am not burning DataForSEO calls on every run.
The messy part was the third-party APIs, as always. Gemini kept throwing 429s and I had been ignoring the retryDelay it hands back -- now I honour it. DataForSEO was swallowing task-level errors silently, so a failed search looked like an empty one. jsdom was spamming CSS parse warnings into my logs. Small papercuts, but each one hides the real signal when you are trying to work out why an article came out flat.
Shipping the pipeline was the easy part. Making it write something worth reading is the actual work.
— -- me, staring at a mediocre first draft
I have not regenerated an article with v2 yet -- that is today. It might still be mediocre, and then I will have to go deeper into the writing prompts themselves. But at least the ranker is now pointing at the right words. Build in public means showing the fine drafts too, not just the wins.