Prompt Engineering is a Job of the Past

A lot has been said about the jobs of the future. In fact, I wrote much of what has been said. From Robot Butler to Climate Refugees, I’ve covered it all. Mostly, I’ve talked about the jobs AI will create; those are the ones people are most curious about. One such AI job of the present and future has had many of us stumped for the past year or so: “prompt engineer.”

For a long time, my thinking on prompt engineering was as follows: 

• Productized LLMs are a brand-new technology that people have not yet fully tamed

• Twitter reply guys have just been burnt by crypto and need a new grift

• They see AI emerge as the “next big thing”, but have no engineering or coding talent; they do, however, speak English (barely)

• They rebrand themselves as “prompt engineers” (sounds fancy! Looks good on LinkedIn!) and share obvious advice to “help” people get “the most out of ChatGPT”

If it doesn’t sound like a real job that someone might be paid to do, trust your instinct. The above is correct for a majority of the “prompt engineers” you see on social media. However, having spoken to actual experts, my thinking has evolved. The truth is closer to:

• Productized LLMs are a brand-new technology that people have not yet fully tamed

• Sometimes LLMs behave in unexpected ways, and we need to understand why

• Because AI is a black box, we also need to understand what it can and cannot do so it can be better marketed

• Doing so involves more data analysis than spending 8 hours a day making wild guesses into a text box… but that’s also part of the job

Though this role is indeed important for today’s tech company, it was more so two years ago. Prompt engineering is doomed to disappear in the coming months.

The Role and Importance of Prompt Engineers

But first, a little background. Unlike what some would have you believe, actual prompt engineers do not solely experiment with already-built products like ChatGPT. They instead play a critical role in developing and maintaining the datasets used by the model to create a final product.

Furthermore, once datasets are created, tested and “used”, they are often the employees who will spend the most time with the products’ first demos. That’s because models that are not designed to be user-friendly will not be so unless they’re very carefully sanitized / tweaked by an army of prompt engineers. Welcome to the internet… and to the fact that big tech is scrapping it mindlessly.

AI models also need to have a sense of how “bad text” “works” to avoid reproducing it. For example, it needs to know about Nazi ideology to avoid telling people about Nazi ideology. But it still knows about it, and the right prompt might make that info come out — which must be avoided. Put another way, to avoid being “bad”, a bot needs to know what “bad” looks like. Only then can it be “good.” This creates another risk to manage. If (when) models become deceptive, we will need to know about it quickly. When asking an LLM “do you know how to build a bomb”, we will need to know if it does know, if it’s been told not to tell, or if it’s decided not to tell. This is not science-fiction: it’s already happening.

Prompt engineers do all that very important fine-tuning and risk management. In fact, a good Prompt engineer will go as far as to engage in “red-teaming” (gross) to test the AI against adversarial inputs (aka “pretending to be an evil user”). This work continues both before and after the product is released to the public.

Throughout the entire process, they’ll also figure out what their AI product is good at and bad at. This work is crucial to fine-tune the final product, making them invaluable assets to understand how to best market a new Gen. AI product.

How Prompt Engineering got Weird

Prompt engineering today involves a lot of experimentation; following hunches, and probing the AI’s responses to various inputs. This exploration sometimes leads to ethical and philosophical considerations, especially when AI responses border on the seemingly sentient.

For those into that sort of thing, there is a wonderful strangeness to AI models. For instance, certain prompts can unexpectedly improve an AI’s performance in specific tasks. If you ask a model to “take a deep breath”, it performs better in math tests. Like, that’s weird? It’s also odd that it took hundreds of iterations to figure out this was the best way to get the model to work out!

And if you ask a model to do something, and it keeps doing it wrong, you can tell it that another wrong answer “will lead to the death of an innocent man.” Answers then usually improve. We can’t read too much into it… but that’s also weird? Does it believe that a man’s life is actually at stake? Are we cool with this? And why does a model seemingly behave better if offered a tip? And why does it perform worse during the holiday season? Those are all types of questions prompt engineers have to grapple with… but AI’s quick humanisation is also a reason why the job may soon disappear.

The Changing Nature of the Job

Prompt engineering in 2022 was a lot harder than it is today. You had to have domain knowledge. For example, you were supposed to know that prompts couldn’t end with a space. This is why most good prompt engineers are ex-data scientists.

Today, productised LLMs make life much simpler. Dall-e 3 (an image generator) is a great example of this. When you give it a prompt, it rewrites it to create a better one for you. When I asked it to make a picture fitting the description “When AI goes Rogue”, it massively expanded and improved it (in mere seconds), until it became “A high-quality, landscape-oriented image depicting the concept ‘When AI Goes Rogue’. The scene includes a futuristic cityscape with advanced technology and AI elements, like holograms and digital interfaces, showing signs of malfunction and chaos. The city is engulfed in a surreal, slightly dystopian atmosphere, with robotic figures and digital screens displaying error messages. The sky is dark and ominous, reflecting the theme of AI going rogue, with subtle hints of digital distortion in the air, symbolizing the loss of control over AI systems. The overall mood is dramatic and thought-provoking, capturing the essence of AI’s potential to deviate from its intended purpose.” 

That’s a pretty good understanding of my intent, compared to what was asked! The software uses its knowledge of natural language to expand on prompts, making them more elaborate and improving the pictures. By doing so, it also fixes flaws in the prompt, and teaches the user what better prompts look like, and what they could yield. 

This suggests a future where AI itself guides users in optimizing prompts. Put another way, as AI models become more sophisticated and user-friendly, the role of prompt engineers is evolving / disappearing. What was once a specialized skill is gradually becoming more commonplace.

This transformation is far from complete. For example, ChatGPT today flat out refuses to answer some prompts rather than proposing alternatives; but this will inevitably change (GPTs are a first effort in that direction). And so, the careers many had envisioned for the past few months will not pan out. Less than a year ago, as MidJourney was taking off, there were talks of people making a living by selling elaborate prompt. How quickly this became laughable! It’s like calling yourself a typist in 2020. Technology has evolved in such a way that everyone can do it; it’s becoming less of a distinct profession and more a skill integrated into broader roles.

---

The irony is not lost on me that the first job created by AI might also be the first to vanish. Although the role will remain lucrative for a select few data scientists ($300k+)… the whole profession is an outlier. In fact, over the next few months, we will see “real” prompt engineers pivot towards becoming solutions engineer and work closely to clients.

The journey of the AI prompt engineer is a testament to the rapid evolution and impact of AI on the job market. As AI continues to advance, it’s fascinating to witness how roles adapt, evolve, or even become redundant. In this ever-changing landscape, the most constant aspect is change itself, challenging us to stay adaptable and forward-thinking.

Good luck out there.