“It wasn't by eliminating the impossible that you got at the truth, however improbable; it was by the much harder process of eliminating the possibilities. Terry pratchett (feet of clay)
I have spent a lot of time integrating AI into my practice and this is a summary of my approach.
First, let's state the ontology:
Agents are not magic see-all, know-all beings. They are just like us in one very important way from a security and safety point of view: partially trusted workers.
A lot of vibe-coding today is done the way a non-expert deals with a contractor: trust them.
With a contractor it makes sense, there are malpractice laws, you can sue them. With an agent we get exactly what we asked for, with fun side effects like prod going down because little Bobby Tables threw an automated tantrum. The LLM is transient, it has no skin in the game, no memory of the past, no stake in the future and no persistent identity except what's explicitly provided as context.
We work every day with partially trusted workers. If we are honest we even partially trust ourselves, which is why we use alarm clocks, reminders and take notes. What we do is create for ourselves a reproducibility substrate. If I make a decision regarding a project and send out orders to be carried out over several weeks, next time I get an update I should be able to continue working on it without starting fresh as if I had never seen that project in my life (we have context windows of some kind too).
A project can deliver value to the customer but still be a failure if it went completely overbudget and half the crew got killed in stupid, repeated accidents.
And now, the final piece of the ontology: incidents. From aviation I use the swiss-cheese incident model and Just culture. I consider not reaching goals as an incident.
All the measures we take, alarm clocks, reminders, safety helmets and gloves are layers between us and an incident. Those layers aren't perfect, they have holes (like swiss cheese). When the holes line up, we have an incident.
I might have forgotten to set my alarm clock, rushed through my morning checklist and left my safety helmet at home.
If I hit my head later on the job it should be systematically analyzed: what went wrong? Did I act recklessly? Should there be another layer like a gate check on the job site to verify that I wear a helmet?
Just culture says: there were individual failures (missed alarm, forgotten helmet), but there must be a systemic response. Those errors can happen to anyone else on the crew so we need to set up safeguards that will catch them. "Checklists are written in blood".
We have been handed powertools. We can refuse to use them, we can accept regularly losing limbs or build the right infrastructure around the risks to make them as safe as flying commercial while still being profitable.