Testset Update

It's been a couple of month since I last touched on the burgeoning field of AI assisted application development (AKA 'vibe coding'), it seems like this would be a good time for a quick update. As previously mentioned, I have spent the last several months working on re-deploying a legacy WordPress-based online magazine site as a modern React-driven Single Page Application (SPA) website, using a streamlined version of the legacy site as a CMS layer. The complete technology stack integrates Google Cloud Services (that's where the CMS is hosted), AWS (that's what production hosting runs on), Vercel (the aforementioned production hosting), GitHub, and Builder.io (the AI-assisted development platform).

In addition to the AI-assisted code generation, I had to conduct a fair amount of legacy content audit/cleanup: 700+ articles (and related digital assets) originally published during and immediately after the pandemic. This was fairly labor-intensive work that could not be automated or reliably outsourced. I had to do it myself.

Nor could I automate or outsource basic architecture / platform documentation. Even though the AI boom has resulted in an explosion of such tools, I neither trust nor need them. If I don't have an accurate high-level understanding of how a tech stack is put together, I have neither the knowledge or competency to direct the work of managing it -- irrespective of whether that work is being done by humans or robots.

An early decision I had to make was whether or not to force content management completely into the CMS layer or let business users work with the agentic interface to manage high-level content like landing pages, category pages, etc. The decision I made was to develop custom post types and custom API endpoints that would keep content management where it's supposed to be.

I've seen industry content that suggests that 'classic' CMS systems have effectively been replaced by vibe coding, to which I strongly disagree. Business users talking to AIs as a means of managing content is, in my view, a disaster waiting for one bad prompt to take down an entire website. I'll pass.

Another challenge I had was developing a workflow and (I really hate to use this word) a relationship with 'BuilderBot', Builder.io's agentic interface. BuilderBot has all the problems all LLM-based systems have in common: Verbosity, Sycophancy, Hallucination, and Amnesia.  In each case, I had to come up with a solution.

Verbosity

No AI ever answers a question with six words if it can find a way to use sixty. Telling a bot to be succinct can help, but after a while the word ceases to have meaning or impact. My solution is to make sure I am being succinct, generating prompts that exactly define my requirements and leave as little to interpretation as possible.  The concept of GIGO applies more than ever.

Sycophancy

There's not much you can do about this, other than ignore it. LLMs have been largely trained by people who love to be flattered and tend to get high on their own supply. Of course the AI thinks your analysis is 'insightful' -- just like the stripper really likes you.

Hallucination

Here's where it gets interesting. I base a lot of my strategy for working with AI on decades of experience working with offshore teams with wildly varying language skills. But I could be reasonably certain they weren't getting high on the job. AI is always high as a freakin' kite. The only solution to this is to essentially trust nothing you can't independently verify. And don't succumb to the illusion you are having a conversation with something that actually understands a damned thing.

Amnesia

AI's memory is only as good as your token budget. Depending on how busy a particular work session is, the bot may or may not remember any of it the following day. The solution I came up with may not apply in all cases, but it is working well for me. Every time BuilderBot successfully solves a problem, I tell it to document the solution in a markdown file and stick it in the 'docs' folder in the GitHub repo BuilderBot can directly access. Every time BuilderBot subsequently pulls a blank on its own previous work, I tell it to scan for specific keywords in the docs folder. So far, it has worked every time.

I could share some more tips, doubtless I will share more as I continue to work with this technology. But as any competent drug dealer will tell you, only the first taste is free. I'm happy to add 'AI training/management' as a keyword in my resume, but I would be a lot happier to apply what I've learned as a contractor/consultant on someone else's project. If I could possibly be of use on your project... Please let me know.

Cheers as always,
--MM