I was never a coder, per se.
I just grew up in an era where if you wanted anything interesting to happen on the internet, you had to have just enough overconfidence in your own ability to pop the bonnet and hit some code with a hammer in different ways.
And I don’t have much call to code now, but I like to try and keep my hand in.
For instance, I have built all of the subsequent versions of the Smithery website since Tom very kindly set up the first one on WordPress. And most versions of the Artefact Shop too, besides a beautiful version Joe built a while back.
Diving back in every few years to see what has changed is good practice.
Especially with the Agentification of everything. “Don’t worry, help is always on hand”.
Recently, I noticed that the integration of our Buttondown newsletter had stopped picking up people who subscribed to it from the Artefact site.
You can literally hear me cracking my knuckles, can’t you?
I decided to use it as an opportunity to test out Anthropic’s Claude as an assistant, to see what it would suggest as an approach to fix this problem.
The trials of trial and error
Me, to Claude: “I am trying to fix the integration between Shopify email sign-up and dropping those customers into Buttondown. Where do I need to look?“
This prompt resulted in a fairly useful to-and-fro between Claude and I, as it suggested ways in which I could tell it more about what I was seeing – “how did this work before?” etc etc
It then suggests commonly-used platform Zapier (for which I had a login anyway).
We merrily spend an hour or so putting together a little automated workflow that’s going to do what I need, based on the plugging in of various API keys and the like.
At that stage, everything is looking good.
Until it becomes clear that Zapier wants a monthly pro subscription for £15 to make the thing work.
Without any context on budget, existing platforms etc, Claude was just pulling from the part of its training data that sits inside those considerations. People have been writing lots of articles and guides on how to use the platform to achieve tasks like this, but they likely won’t start each piece with ‘this is for people with a pro account…’
Me: Zapier will cost a monthly subscription, which I don’t have
Claude: Ah, good to know! Let’s use a free option instead. I recommend Pipedream – it’s free for up to 10,000 invocations per month and perfect for this.
So therefore, we start again, this time in Pipedream.
Now, this is a platform that’s new to me, though similar to Zapier, maybe IFTTT, and of course Yahoo Pipes (shout out to the OGs…)
This time, it takes ages in the to-and-fro to get Claude to describe to me the things I should be looking for, copy & pasting, integrating, before finally we get to something workable.
I also don’t really feel that it’s been any quicker than following the typical written descriptions that you would used to rely on for this – either platform documentation or a community page.
It’s a fairly miserable experience.
Looking back, what becomes really interesting is (again) the context under which Claude was operating in this instance, and the limitations this might suggest for the broader future of augmented assistance from LLMs.
Retrograde Context Bubbles
I took me a while to work out why Claude would lead me in this merry dance.
It would continually ask me to do things in the Shopify Dev platform that just weren’t possible.
Claude: Now click reveal the API code once…
Me: There’s nothing there, nothing to click…?!?
It is because everything Claude knows about the Shopify development experience is based on information taken from before they had a major redesign and relaunch.
Claude is trapped in a context bubble of (I’m guessing) 2018-2023-ish, when lots of people are writing lots of things online for each other about how to solve problems. It exists in a world where the echoes of how a platform used to work dominate the discourse on which it has been trained.
Underpinning everything is still the logic of the programming language. But the human wrappers designed to help people interact and work with this language makes know sense to a retrograde LLM.
Perhaps programme languages are like simple grammar rules, do this, in this order, and this will be the result. They are the building blocks of text that should always behave in a particular way.
But programming platforms – like the Shopify developer interface – are contexts. Teams have decided that the thing that helps their community most is using these programming languages in particular ways.
And LLMs are really good at text (enough repeatable examples where doing x then y results in z), and really quote awful at context (where some people have designed a funky new interface for the code to sit in, usually to make it ‘easier’ for users).
Learn to code, all over again
This reminds me a bit of the Learn to Code moment.
Everyone was promised the power of computation to achieve amazing things if only they learned to code. I think it probably helped people learn what code was, how it worked, and why they should be interested in it.
Maybe you went on a ‘Learn To Code’ course? Was it useful? Are you doing any coding now?
My gut feeling is that not many people stuck at it.
In that way, it has parallel in the ‘prompting’ moment we are in now.
LLMs can help you do amazing things, if you know how to prompt.
When people say ‘oh, you’re not prompting properly what you need to doing is…’ well, that is still a form of coding.
It is looser, of course, and the end results are much less consistent. But that just makes it harder to learn in a consistent fashion. And most people don’t want to code anyway.
And the people who’re good at it anyway are better at it that LLMs.
Cognitive Debt, etc etc.
You must be logged in to post a comment.