Interested in writing your first TDS post? We always welcome submissions from new authors.
When prompt engineering first emerged as a mainstream workflow for data and machine learning professionals, two prevalent (and somewhat opposing) views seemed to emerge.
Following ChatGPT’s meteoric rise, some commentators declared that ChatGPT would soon become a critical task that would take over entire product and ML teams, and job ads for prompt engineers reached six figures. At the same time, skeptics argued that this was only an intermediate approach to fill the current competency gap in LLMs, and that as models improve in performance, the need for specialized prompting knowledge would disappear.
Nearly two years later, it seems like both camps have valid arguments. Prompt engineering is still deeply rooted in our lives. It continues to evolve as a practice, with more tools and techniques to support practitioners' interaction with powerful models. But it's also clear that as the ecosystem matures, prompt optimization may become less of a specialized skill and more of a mode of thinking and problem-solving integrated into a broader range of professional activities.
To get a sense of the current state of prompt engineering, understand the latest approaches, and predict the future of the field, we've collected some of the best recent articles on the topic. Read on!
- Introduction to Domain Adaptation: Motivations, Options, and Tradeoffs
For those just getting started with LLM practice, the three-part series is a great starting point to explore different approaches to producing reliable results from these large, unwieldy, and sometimes unpredictable models. The first part in particular provides a clear introduction to prompt engineering, including why it is needed, how it works, and what trade-offs need to be considered. - I'm AI certified. I'd like to share what I learned about Prompt Engineering.
“Prompt engineering is a simple concept – it's just a way of giving instructions to LLMs to complete tasks.” He writes from the perspective of a seasoned software developer who wants to stay up to date on the latest trends in the industry, and shares his experiences of humans and models stepping into sometimes counterintuitive paths. - Automating Prompt Engineering with DSPy and Haystack
Many ML professionals who have already played around with prompts quickly realize that: Many There is still room for streamlining and optimization when it comes to designing and running prompts, and we recently published a great, easy-to-follow, step-by-step tutorial focused on the open source DSPy framework for those who want to automate large parts of this workflow.
- Understand techniques to solve GenAI challenges
We tend to focus on the granular implementation aspects of prompt engineering, but like other LLM optimization techniques, it also poses a set of questions for product and business stakeholders. A new article from is a useful overview, and nicely offers guidance on when to consider different approaches, and how to combine them for best results. - Streamline prompts to reduce LLM costs and latency
Once you have a functional prompt engineering system in place, you can start to focus on how to make it more efficient and resource-conscious. For practical advice to get you started in that direction, don't miss our 5 tips for optimizing token use in your prompts (but without sacrificing accuracy). - From Prompt Engineering to Agent Engineering
For a sharp look at where the field is headed in the near future, check out 's high-level analysis: “It seems like we need to start moving from prompt engineering to something much broader: agent engineering, and establish the right frameworks, methodologies, and mental models for designing them effectively.”
