RLMs

I have been an observer of the RLM field since ~february when it got really hyped. If you don't know what an RLM is it's basically a LLM agent but instead of just appending it's code output to history (and it grows huge hence ppl care about context windows) it kind of stores the steps it does in chunks and stores these to variables inside what is called a REPL (Read Eval Print Loop), the one that pops up if you write python3 in the terminal.

This is huge because just like you are able to compute something and stores it iteratively (effectively wiping your brain context with each variable creation, like you no longer care about the code to the right, you care what the variable holds.)

# the quick brown fox — coding tutorial royalty
sentence = "the quick brown fox"

words = sentence.split()
lengths = [len(w) for w in words]

longest = words[lengths.index(max(lengths))]

The agent can now do the same! but imagine each step here being a chunk of huuuge output that would otherwise fill the context. Basically it could program what it should have in it's context right now, not confined to what you asked at the start of the session. And it also allows for recursion, you can imagine that it just calls itself to do something else and report the status when it's done. Complete wipe of the context, and the result is just a variable so it doesn't have to impact the root agent's context at all (it could read it in full ofc and use some context).

prime-agent

yesterday prime intellect released their RLM harness, and it's benchmarks are bonkers. So i tried it out.

I have been mainlining deepseek's V4 flash with one issue: it has no eyes.

Come to the rescue: prime-agent.

I just asked the agent (deepseek) to add a subagent that ran with agents supporting vision.

I told it that I already had a codex subscription (20$/m), it asked me to log in.

After some tinkering, it found a bug where subagents wouldn't list the openai models in the harness, it fixed the bug (or the constraint I am not really sure) and deepseek had vision!

At first it was soo slow (over 60s for high reasoning sol) so i asked it to make a benchmark on speed for the gpt5.6 line sol/terra/luna and compare which was the fastest and most accurate on some vision tasks. We started out from this screenshot while working with it

We landed on a process of using luna-none, if that wouldn't yield anything useful it will continue with sol-none. It was honestly interesting that the benchmark landed on the thinking levels none. The images it evaluated on was a screenshot on my screen with two benchmarks: full desciption of the scene and a yes/no pass which looked something like this

   
   Strict visual verification task. Look at the screenshot. For EACH item below
   answer exactly YES or NO on its own line — YES only if it is clearly visible
   in the screenshot. Never guess; if uncertain or hidden, answer NO.

   Q1 the word Grok
   Q2 a Spotify window

Quality across configs

Notes:

• single-trial, noisy; none<->low never showed a consistent quality difference, high costs 3-9x latency
• not very scientific, no manual annotation so ofc sol scores well on the sol defined true output, but good to see the diff between models :)

And that is how far i got, an ~hour in and i got my dear whale some eyes with a process speed of ~2s on the yes/no task. Full reasoning was still up to 10s of seconds