Listening to another pitch about how AI can empower workers at various jobs across my industry, I was striken by the comparison in the title
3d printing, just like generative models, have it’s actual niche uses, where it’s obvious downsides are irrelevant and they come handy, e.g. prototyping, replacements, small-series production
Where it comes to the top-down AI promotion trend, it feels not unlike the idea of printing the whole product - a car, or a house, from the smallest details - applying the least effective method, doomed to have a worse than average outcome due to technological limitations
And screws, the thing that we nailed down long before, and that is completely incompatible with that mode of production, is a screaming, growling, shrieking example of how helpful tech can be mispurposed in the most stupid way
So there are a few very specific tasks that LLMs are good at from the perspective of a software developer:
And that’s… pretty much it. I’ve experimented with building applications with “prompt engineering,” and to be blunt, I think the concept is fundamentally flawed. The problem is that once the application exceeds the LLM’s context window size, which is necessarily small, you’re going to see it make a lot more mistakes than it already does, because - just as an example - by the time you’re having it write the frontend for a new API endpoint, it’s already forgotten how that endpoint works.
As the application approaches production size in features and functions, the number of lines of code becomes an insurmountable bottleneck for Copilot. It simply can’t maintain a comprehensive understanding of what’s already there.
I use it to generate unit tests, it’ll get the bulk of the code writing done and does a pretty good job at coverage, usually hitting 100%. All I have to do for the most part is review the tests to make sure they’re doing the right thing, and mock out some stuff that it missed.
You’re right, unit tests are another area where they can be helpful, as long as you’re very careful to check them over.
one other use case where they’re helpful is ‘translation’. Like i have a docker compose file and want a helm chart/kubernetes yaml files for the same thing. It can get you like 80% there, and save you a lot of yaml typing.
Wont work well if it’s mo than like 5 services or if you wanted to translate a whole code base from one language to another. But converting one kind of file to another one with a different language or technology can work ok. Anything to write less yaml…
They are getting faster, having larger context windows, and becoming more accurate. It is only a matter of time until AI simply copy-cats 99.9% of the things humans do.
Actually, there’s growing evidence that beyond a certain point, more context drastically reduces their performance and accuracy.
I’m of the opinion that LLMs will need a drastic rethink before they can reach the point you describe.
We have 100M context AI, we just need better attention mechamisms.