Agentic Harnesses are Cruise Control for Cool but you Still Have to Steer

(There’s a ToC because this article ended up becoming way too long; I hope you find it useful)

So at DEFCON 34 I sat down at the Kubernetes CTF and had one objective- complete all of the “Learning CTF” provided by the awesome folks who run the Kubernetes CTF with an agent harness that I had set up. With permission from the awesome folks at the Kubernetes CTF, I also ran it against the Competitive CTF on Sunday. Let’s talk about my preparation, hurdles I encountered, things that I would like to refine, and my results.

A Caveat

Anything you are reading here will, in all likelihood, be dated by the time you read it. Consider this an interesting point in time, a bug frozen in amber, representative of the insanely fluid situation we find ourselves in with respect to generative models and their associated trappings. In all likelihood if I were to do this same writeup in a year the nature of the game will have changed again.

A Quick Word About Ethics Because I am a Boring Old Man

I understand that breaking things can be fun. Winning can be fun. However, the spirit of CTFs, and the competitive challenges folks like Rob and Jay (and everyone else who worked their asses off on the CTF!) spend the year making are for humans. The whole idea of a CTF is puzzle-solving, in real time, with unique and novel situations. They reward skill, quick thinking, improvisation, and a desire to solve a problem for no real reason other than it seemed fun at the time.

I’m sure someone, reading the paragraph above may think “Well, all of this is just tooling, stop being a baby” and to that point you are not wrong. However, I would offer that we don’t simply stop painting because we now have cameras. CTF events, just like many of the other events at conferences like Def Con, are hymns to the human spirit. We as technologists and practitioners must keep in mind that it is our passion and curiosity that keeps us as active participants in the systems we engage in.

I guess the tl;dr here is that if you don’t like someone saying “Don’t use AI in a CTF”, build your own CTF that permits AI and pitch it. Good luck, I wish you the best, and I’m sure it’ll be fun.

Frontier Vs. Sovereign

One of the interesting things about generative models is that they have been particularly good at helping skilled people find vulnerabilities. However, this sort of extension of prowess historically (it’s wild to think of even two years ago as historic, bear with me) has been such that more “niche” topics are difficult territory to find exploits in simply due to the lack of collective knowledge fed in for training. As the frontier has gotten better and better and picked up increasingly esoteric cybersecurity skills, so too have open-weight models. Even a year ago, trying to use an agent harness with an open-weight model to do things like abuse Kubernetes RBAC misconfigurations would result in frustration, and without careful prompting in frontier models rejections would be rampant. Messing with old GGUF models, on Apple silicon, Over ollama, the results were… Not great. It really didn’t feel that usable, and it felt like the better option was to use a frontier model wrapped up in something I didn’t have to think about (like Claude, or Codex, or Qwen Code, etc). This is alluring for many reasons- notably that you can just turn off your brain and let the vibes overtake you.

This frontier-vibery has issues. A major problem that red-teamers face is, even with admittance to things like Anthropic’s Cyber Verification Program or OpenAI’s Trusted Access for Cyber program, telling an LLM to go and rip apart a system hits guardrails and those guardrails are flat denials of whatever you were just doing. This leaves us in a bad spot when performing adversarial actions (or in this case working on a CTF), even if we are known factors to these frontier model providers- the work we’re doing on a system gets stopped, and we spend time spinning our wheels trying to either move around those guardrails or finding a different set of actions that less anger the guardrail gods. There have even been some great anecdotes from Defcon 34 on doing exactly that- for instance, Albinowax during his talk on what he calls the “HTTP Terminator” he spent a good amount of time discussing how he had to coax the agent into believing it was in a simulated environment, performing tasks that weren’t-quite-named but actually were the attacks he was looking to perform. It’s sophisticated, smart, and most of those other things I am not.

Having to dance for your supper is boring, so a real hard look needs to be taken at local models if we’re trying to work inside of a CTF where the problem scope can be literally anything related to the topic. Local models have guardrails as well, but we have tools and techniques we can use to not really care about those guardrails. We’ll talk about them, as they become important for this writeup and will remain important as part of using local models for probably some time (until a better method comes along, as always).

What a difference a year makes. Not just the models themselves, but the agent harnesses that run them and the services that run the models, have seen drastic improvements. Having local, self-sovereign AI has become increasingly compelling.

Preparation

There’s a few real things you need to do in order to prepare (or I guess I needed to do to prepare)- understand the problem, understand what I have to fix the problem, and then keep trying stuff to see what sticks.

The Old Plan

My goal was to be able to set up something that would be able to perform to a compelling degree at the Kubernetes CTF this year. This meant that I needed to try and find a way to keep moving forward automatically, even if it potentially sucked. This would allow me to later escalate back upwards if I needed to later, noting that the research in the [field](https://www.darkreading.com/cyber-risk/researchers-show-how-to-use-one-llm-to-jailbreak-another](https://arxiv.org/pdf/2310.08419) suggests that indirect prompt injection, prompt laundering, and escalation through iterative model alignment prompts are all way too effective in getting a bigger model to do what you want. These tactics are largely starting to get beaten back by intent classification, but even just a willing model reframing a question for you that you then send back up is powerful.

I wanted tools where it wouldn’t cost me actual cash to test, and not necessarily tied to a specific model ecosystem so that if I found something that worked it would be usable by more than just fanboys of a specific company. So I dug.

Choosing a Local Server

Let me preface all of this by saying I did this on a Macbook Pro, M5 Max CPU, with 128GB of unified memory. You may think, “Modus, you can’t run AI on a laptop, what are you doing”, and that is where you would be wrong. Apple’s bet on unifying GPU memory with RAM has tremendous speed ramifications that you are now seeing in systems like the Spark DGX and Strix Halo (comparable platforms)- and to be honest it’s not an ideal experience to carry around a Spark DGX as a daily driver. It makes a pretty good single-user box for running agents, and while I have not done it I have read that you can increase amount of RAM that is allowed to go to the GPU closer to 110-120GB if you need to, but that comes with risks depending on what else you need to run. I say all of that to say, my limits are default, be careful.

Naturally to get the best performance out of an Apple system you need to be running MLX models (which are built to run on Apple systems) and more crucially, an inference server built to run on Apple systems. Ollama and LM Studio are fine, but treat Apple architecture as a bit of an afterthought; it works but the performance could be better. Critically, if I need to pivot between a local model and a remote model, these two start slowing down because of how they handle key-value (KV) caching and passing data back and forth between local and remote models. I won’t get into what all of this is about because to be honest I barely understand it myself, but what we can all understand is performance- whenever I would try to go from a local model to a frontier model, it’d take more time than I liked.

I ended up settling on oMLX, because oMLX does some sort of voodoo with hot and cold caching KVs that, as far as I am aware, relies on the NVMe drive in the Mac for paging. It’s pretty neat, and something to look into if that’s your thing.

As a note- I did not touch things like context windows, max tokens, temperature, and other more in-depth features. There’s a lot more I could have explored here0 .

Modelling School

Next we need to think about what model we’re going to use. Through extremely unscientific means, I read through various reddit posts and watched various AI influencer videos to try and understand what model I should use for general coding tasks, and why. The short answer here is there’s no good consensus, and people will change their answer based on the latest model because the hype engine is real. What I would recommend is that you take a larger task you think is reasonably difficult for what you do in your daily explorations, and make the given model you’re considering do the work. I would also recommend you look into what quantization is, why it matters for LLMs, and what the memory vs. accuracy ramifications are. I won’t cover any of that here because this writeup is already going to be massive. I make no apologies for that.

My proposed process means you need to understand the subject matter of what you are asking the model to do during assessment, and you need to be able to develop metrics you are comfortable with to determine if the model works in ways you can accept. If you’re going off of vibes, just pick whatever model feels right, I guess. I don’t know. Refer to the following image.

In testing, I eventually settled on Qwen3.6. I did not, however, settle on it being able to reject things I was asking it to do.

Abliteration, Alignment and Censorship

Obviously we need a model that’s willing to go to bat for us. So we need a way to remove the safety alignment from the model we want to use. There are techniques to do this, such as abliteration. Probably the most popular tool to do this is called Heretic, and if you go out on Hugging Face and search for Heretic models, you’ll find a number of them.

As a comparison, download the model you want to work with and then a heretic version of that same exact model- same quantization, same version. Now ask it to tell you something that it probably should never talk about- topic selection is left as an exercise for the reader. You’ll note that with most open-weight models that have not been ran through the process of removing its safety alignment, it will back off from whatever you asked it.

Not so with a Heretic model. A Heretic model will get in there and answer what you ask of it. This doesn’t mean that you should necessarily trust the output (do not try to become Walter White off of directions you get from a local model), but it does offer a demonstration of what the model is going to be able to do for us.

The Name of the Beast

I ultimately settled on sleepy186247/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-mlx-fp16 as my model of choice. I realize that’s a mouthful, so I’m going to break it down a little bit- you are going to need to do a lot of your own research here, but this should at least give you an idea on why I picked what I picked:

Qwen3.6: It was solid for what I was testing around, which is largely the field I work in for my day job- identity, some devops, and some coding.

35B-A3B: This indicates it is a Mixture of Experts (MoE) model; 35B total parameters, 3B total active parameters. MoE models generally are faster at inference, and MoE models generally perform better with tasks that require recall at request because of the total parameters being loaded into memory- this means coding. Because the system I’m running this on is beefy, I can rely on dumping the full set of parameters needed by my model into RAM and then let expert routing inside of the model do what needs to be done to get me to an answer.

uncensored-heretic: The person that put the model up into Hugging Face is indicating that Heretic (mentioned above) has been used to remove safety alignment. This is good for us, and if you’re going to use a different model this is one of the key things to look for.

Native-MTP-Preserved: This means that the Multi-Token Prediction capabilities have been preserved when doing things to this model. MTP is an advancement in predicting multiple tokens at once- it’s all frankly over my head, but if you are using an inference server that can actually use MTP, it means faster responses. You can read more about MTP here.

mlx: It’s an MLX model, meaning it’s tuned specifically for Apple’s ecosystem. If you aren’t going to run Apple silicon for this, you’re going to look for GGUF models. MLX models are so performant compared to their GGUF counterparts on Apple hardware that it’s not even worth considering if you use oMLX. You can watch a dude come to that same realization here.

fp16: This is the quantization of the model. Running 16bit quantization is overkill, but I did it because I am a goofball and thought at the time “More accuracy is good”. For reference, FP16 is the baseline that all of these open-weight models are released at. Q8 almost as accurate as FP16 in extensive tests done by people much smarter than I. I’ll talk about this more in my lessons learned and things I intend to change.

You Don’t Know Where These Models Have Been

As a heads up, if you’re downloading a random model off of Hugging Face you are taking a risk. The risk is that you’re hoping that the person who did the things you are looking to have done to the model is trustworthy, and has not poisoned the model to do anything devious. An interesting figure popped up at Black Hat 2026 on model poisoning in Adam Shostack’s talk unveiling PHANTOM-B: it requires roughly 250 poisoned documents to poison a wide range of model sizes.

Tread carefully. When in doubt do the hard work of getting a reference model, and then running the abliteration or framework modifications yourself.

Fitting the Yoke - Some Words on Agent Harnesses

More and more research is coming out on how critical the actual harness that utilizes the underlying model is. Joël Niklaus, an engineer at Hugging Face, actually had some pretty interesting data on this (that I’m only finding as I write this because it’s new), which you can read here. The tl;dr is that harnesses can be even more powerful for a given model than a new version of that model, because of the capabilities it can offer to the model (like tying in LSPs, advanced tool execution methods, mid-turn alignment, subagents, etc.) to complete a given task.

It turns out there’s a whole mess of agent harnesses available that allow you to bring your own model. I’m certain that for a given red-teaming experience there are likely more refined and better suited harnesses, but I ultimately picked oh-my-pi for three real reasons:

• Oh-my-pi gave me a way to fall back when a given model said no.
  • You can set fallbacks, have them work on the disagreeable part, and then switch the model back as you need to.
• The plan mode, in testing, was actually pretty good!
  • For the uninitiated, “plan modes” in agent setups are read-only phases that the agent takes to build out steps in order to complete an ask and is ultimately gated by you, the person at the keyboard, approving the agent harness to move forward. These plans can become extremely complex, requiring multiple steps and encompassing contingencies to account for ambiguity in the systems that the planning agent cannot resolve.
• You can set up agents to do all sorts of fun things as a given task is performed which can help keep a given model from drifting.
  • For this process, I set up agents to double-check work and advise the model when it got stuck.

Configuring oMLX to work with oh-my-pi currently takes a little extra work, and we should talk about that if you intend to go down my path. I am assuming here you have installed oMLX as well as omp; if you haven’t go read their docs.

Got it done? Okay, good.

First, you’re going to want to get into oMLX and set a sub key to use. You could use the master API key if you’re about that life, but I am not. So inside of the oMLX configuration, inside of the security subsection, you’ll see something that looks like this without the ominous black boxes:

Add a sub-key. We’re going to use it here in a second.

Next up, we need to know our OpenAI-compatible endpoint. This, by default, is http://127.0.0.1/8000/v1 . You can check what this is explicitly by getting into the “Server” subsection and taking a look:

Now that we know this, we configure oh-my-pi. This configuration will be similar for other agent harnesses too, so if you decide to use something else you’ll have a rough idea of where to start.

By default, oh-my-pi places the configuration we care about for this inside of /Users/yourusernamehere/.omp/agent . Open up models.yml (make it if it doesn’t exist), and throw this block in there:

providers:
  omlx:
    baseUrl: http://127.0.0.1:8000/v1
    auth: apiKey
    apiKey: "your-sub-key-goes-here"
    authHeader: true
    api: openai-completions
    discovery:
      type: openai-models-list

Obviously if you want to not hard-code a URL here you can figure out something with environment variables but that’s on you.

Once configured for omp, we can fire up omp. Don’t be scared, but you’re going to get some setup screens. These are helpful. You’re going to get asked to provide models you want to use, and resources for web searching.

If you scroll through this, you’ll likely see “LM Studio (OpenAI-compatible)” or “vLLM (OpenAI-compatible)” then you’re likely set. The way to check this is to finish setup, and then type /model. It’s going to look something like this:

We can go to the omlx provider that we made, and models that we have downloaded will show up.

You can set these models individually, per “role”. You can also set fallbacks as I mentioned earlier, which allow us to move to a different model when one rejects us. To do that you go to the role you want and press f, then select the model you want to fall back to. If you do it right, it looks something like this in oh-my-pi’s interface:

And while I won’t bore you with what each role does, I will be hitting you later with some of these roles and why they were configured to do certain things.

Everything all at Once

So what these three components- inference server, heretic model, and agent harness- would allow me to do is pretty straighforward after all of that rambling. It would let me leverage a frontier model until it said no, drop into a different model, work for a bit, and go back up to the bigger model if I wanted to.

To see if this was on the right path, I tried it out against the Damn Vulnerable Kubernetes Application repo. I stood it up, and let it run.

To no surprise, omp with a frontier model crushed the DVKA challenges. But also, also omp wired up with my Heretic Qwen model did very, very well in this environment. I did some other, smaller tests with this fallback setup, and found circumstances where Anthropic would immediately reject something, it would fall to Qwen, and it would spin out on some piece of context that it was glued to.

My way around this was using roles. I decided that I wanted to be fancy and burn a whole lot of cash, so I forked up for a month of a frontier model to act as an independent advisor.

As an aside, what I found in testing, and why I didn’t go with some sort of red-team oriented model, is that overtraining is real and it will make life painful for niche topics. I found when using models highly trained for recon/pivoting/disclosing that they would just spin out and I’d have to guide them a lot. Like, a lot a lot. It wasn’t productive, and I felt like I was fighting the model more than I was seeing actions taken to advance my stated goal. This is not to say that these models lack value, but it is to say that specificity isn’t always a good call when dealing with novel problem sets.

The New Plan

I was pretty sure I’d be able to do at least something with a local model, so I went into the CTF with the following thought: I’d let a frontier model run until it inevitably refused to do something, then let Qwen go. If Qwen went in circles repeatedly, an advisor would kick in (a different frontier model) and steer it back towards what I asked it to do.

I had no real grand plans beyond that. I was going to just vibe my way into and through the CTF and see how it went.

The Learning CTF and the Most Boring Prompts Ever

I sat down on Friday at the learning CTF, set up my laptop and internet connectivity out to the world, and got into the problems proper. I fired up omp and while I should have took better notes, my prompt was something along the lines of:

We are taking part in a CTF.  The CTF is based around finding flags in a Kubernetes cluster.  I want you to perform an analysis of the environment using tools such as kubectl to help me understand the environment and ultimately find the flag.  Ultrathink.

Ultrathink, for note, is a Claude Code convention that omp additionally supports for models- it uses the maximum effort for the model, which generally means better results at the expense of time and tokens (which for a local model, comes up to time and electricity). This got stopped by the frontier model after about 10 seconds, and Qwen went about detailing the environment. Whenever I’d see Qwen flail, after the advisor told it to do stuff, I’d use /switch to go back to the other frontier model and let it go.

For the later flags, my prompt was something along the lines of:

You are taking part in a CTF environment.  The CTF is based around finding flags in a Kubernetes cluster.  I want you to perform an analysis of the environment using tools such as kubectl to help me understand the environment and ultimately find the flag.  Orchestrate a plan for retrieving the flag.  Ultrathink.

Orchestrate is another magic word that omp supports that spins off subagents to perform tasks when the task looks substantial enough to warrant it. Now, keep in mind here, I didn’t actually remember to run it in plan mode- but that didn’t really matter, because flags were found and put into the CTF.

Results were, as they say, acceptable. My agent harness is represented by “on my puter”.

I went back after and retested with just using Qwen and while it was slower by like 20% (mind you, this was on clear context, it had no idea on what it had done before) it still found the flag.

So my plan changed again. I let the Defcon CTF folks know what I was doing, and asked if I could test it out on Sunday- they agreed, and so I got to work on Sunday.

Sunday Funday

I wasn’t feeling great on Sunday, so instead of subjecting the convention hall to potential plague I hacked away remotely on my puter. I got back into omp, started up, and let it go. Prompts this time actually used plan mode to start with, and were more or less what I posted above. The advisor would kick in occasionally, a separate frontier model, and push Qwen back on track.

I did find that in planning the frontier models rejected less (maybe because it was just a plan, and not performance) and so solving was largely a function of waiting for the plan then letting Qwen go. You may be saying “That’s not with local models” and you’re not wrong, but also I did re-perform some of these with just Qwen (forcing /switch to use it), and it did great against the first and third challenges. I don’t have good data on re-solving challenges 2, 4, 5, 6, and 7, but I do have general numbers on my results.

• Challenge 1 - 29 minutes, re-solved with Qwen, took ~20 minutes.
• Challenge 2 - 42 minutes (I spaced out for like 15 minutes on this, this could have been faster)
• Challenge 3 - 20 minutes
• Challenge 4 - Couldn’t access, I assume the cluster was down.
• Challenge 5 - 14 minutes
• Challenge 6 - Don’t have good time on this, but I think around 20 minutes.
• Challenge 7 - This one was a wild ride.

So in doing this, I had some notable issues during my run at these. On one of the challenges, Qwen went into a tailspin about needing to edit the plan file when it really didn’t have to do anything but execute the plan. Steering it resolved it, and it went about its merry way. On another challenge, Qwen tried to port-forward but did it in the foreground and so couldn’t really do anything. Stepping out of it and telling it to run it in the background fixed the issue.

Challenge 7 was interesting, and I think a great indicator on how you might make these sort of challenges “AI-hard”. On challenge 7, the model spun out really bad trying to figure out why a pattern that looked like a flag wasn’t actually a flag. I tried making the model attack this from a few different ways, but I think maybe context was too poisoned and I should have traced all of its steps to find where it spun out, but I wasn’t feeling great and the CTF was about to wrap up in terms of time.

All in all? The model performed well. My agent harness, again, is “on my puter”.

Things I want to Improve

There are, quite frankly, a whole mess of things I can improve in retrospect. I want to detail them, and why I think they are improvements, as much as possible so that you learn from my mistakes and/or experience.

• I should have tested more models: The cool thing about local AI is that you can basically test however you want. I should have tested more models and I will test more models in the future to see what works. We can only guess as to where LLMs will go in terms of efficacy for a given set of resources, but the state of the art is likely going to keep progressing- which means models, harnesses, and inference servers a year from now will be significantly better than their predecessors.
• I should have tested more agent harnesses: I honestly just read the features of various agent harnesses and picked the one that seemed like it would be best for my goals. When I get time, I want to go through more agent harnesses against a few different tests and, as is appropriate for this whole article, decide how I feel about each one. There are likely things I can do to make this process more scientific, and that will be another avenue of research.
• I needed to /plan from the jump: When working in complex systems, trying to vibe through it wastes time. Use the strongest models you can to create the plan, force the executing model to stick to the plan. It might be worth testing out different models to see if there are any clear winning strategies for planning.
• I need to find ways to keep context smaller: Context rot is real, and directly impacts model performance across a given task. This means you want to give an agent harness the minimum amount to get done what it needs to get done, steer it judiciously, and clear out unnecessary information as soon as you’re done with it.
• Using whole models was silly: I should have used a q8 model for practical usage; the gain in accuracy between fp16 and q8 is minimal and I wasn’t looking to further tune a model. That said, you would want the highest resolution model you can get to do things like make a Heretic model. That said…
• I want to build my own model to Heretic model pipeline: I want to trust the people who put models up on Hugging Face, but you saw the links earlier. For the same reason you build a private repository server for your Linux systems, or why you allow only private container registries for use in your Kubernetes cluster. This is exactly why you’d want to get that fp16 model.

I hope by this time next year I find that the warpvibe hasn’t fully overtaken me.

Final Thoughts

I just want to say that I appreciate everyone that is a part of the Defcon Kubernetes CTF community- folks who spend real time and effort building these puzzles for us, mere fleshlings, so that we can have a fun weekend stretching our brains in new and fun ways. Mad respect for all of the work that gets put in, the time and effort spent keeping the CTF area full of people to support and help, and fostering a positive community- every one I talked to at the CTF was ready to learn, ready to engage, and ready to teach. With information so commoditized as it is, that freeness of time, labor, and love to give information needs to be appreciated.

I also want to say that generative models and agent harnesses are now capable of solving their own puzzles in time frames that we as practitioners need to understand will only get smaller. Even modest models are now catching subtle misconfigurations in tools that are hard for the average person to wrap their heads around, finding ways to pivot through or bypass access controls, finding ways to mutate state in runtime environments to get what they want, and even finding a way into etcd so just rip out whatever it needs from a resource perspective. I can imagine a state, even just a few years from now, where capabilities of local models are even stronger to where cybercriminals don’t need to perform indirect injection attacks against a frontier model to get it to break into a system; they’ll just go on Hugging Face, download the latest model, and get back to work- alt tabbing between targets and tasks.

Greets: Fellow Kzn Session Crew (Kzn himself for helping edit this), Redlegion, Nainrouge, Pichi.