Managing AI Products: What Changes and What Doesn’t

AI is powerful, unpredictable, and evolving fast. For product managers, building AI-native products requires some new knowledge, but the fundamentals stay the same. This article will guide you through the similarities and differences between managing a product that’s AI-native compared to one that isn't.

Start with the Fundamentals: Learn the Tech, Not Just the Buzzwords

A PM doesn’t need to be a data scientist, but understanding LLMs, neural networks, and core ML concepts is essential.

I used to believe PMs didn’t need to be technical. But after working on an AI-native product, I’ve changed my mind. PMs don’t need to write code, but they need to understand how the different parts of an AI product fit together. How neural networks and LLMs work, what ‘temperature’ is, the differences between vector and SQL databases, and how prompts get from your system to a LLM and back.

"AI is just a technology. A powerful and extremely wide-reaching technology, but a technology nonetheless."

Will Chan, Product Leader | AI Innovation, Product Strategy, and Scalable Delivery

I highly recommend 3Blue1Brown's "What is a neural network?" series to learn the basics of neural networks and LLMs.

The better you understand the technology, the more impact you will have when deciding which problem to solve and how to solve them. Being able to speak on the same level as an AI engineer is invaluable to generating trust as a PM. This also enables you to translate the technical aspects for user-facing content and copy.

As a PM you should be able to confidently discuss the pros and cons of fine-tuning vs RAG, how the Vector DB is managed, and whether you should pursue an Agentic workflow for your AI solution or not.

Get Hands-On with AI Tools and Prototypes

Play with tools like v0, Lovable, or your own experiments using Cursor and OpenAI APIs to deepen your understanding.

The current slew of “1-shot” code editors out there has made it easier than ever to prototype, test, and ideate as a product manager. It’s imperative that PMs learn how to utilise these products. Quickly developing prototypes to show to either external or internal stakeholders is invaluable. They say a picture is worth a thousand words, a working demo is worth even more.

It feels like a superpower to be able to go from concept to a clickable and working prototype in minutes that you can then show to a user to get feedback. You can even prototype with the user live on a call to get immediate feedback and insights.

Appreciate the Limits of the Tech

AI can be magical, but also dumb in unpredictable ways. Knowing what AI can't do is just as important as knowing what it can.

It’s important to remember that LLMs are inherently probabilistic. Historically, it has always been the opposite: software is deterministic - you give the same input, you get the same output. This isn’t true with LLMs, which means that there is always going to be some variation in how the product responds. There are methods to reduce this variance, but there’s no way to eliminate it.

One of the most surprising things I learned while working on an AI-native product came during a conversation with one of our AI engineers. I’d suggest a feature and hear, "That’s easy, we can build that in an hour," then suggest something that seemed simpler and get, "That’s basically impossible." What stuck with me was how unpredictable LLM capabilities can be. It’s not always intuitive which tasks are easy for the model and which are incredibly hard. Here are some examples:

Things that seem easy but are hard for LLMs:

Accurately counting words or characters

Why it's hard: LLMs don’t have a built-in calculator or memory buffer; they predict text one token at a time, and a token isn’t the same as a word.

Real-world truth-checking (e.g. "Is this factually correct?")

Why it's hard: LLMs don’t have built-in access to a truth database, just patterns from training data, which may be outdated or wrong.

Things that seem hard but are easy for an LLM:

Translating between dozens of human languages

Why it’s easy: LLMs have been trained on lots of text in many languages, so they’re surprisingly good at translation, even for languages with less data.

Tagging or classifying content into predefined categories (e.g. spam/not spam)

Why it’s easy: LLMs are good at following instructions, especially if you show them examples. Which means they are great at tasks like labelling or sorting things.

At the end of the day, an LLM is just predicting the next most likely token in a sequence, nothing more, nothing less.

Level Up Your Prompt Engineering and Communication Skills

Writing great prompts is part art, part UX design. You’ll also need to explain AI behaviour to users and help them build the right mental model.

For all the noise around it, remember that AI is still new. There are still many industries where users are unfamiliar with how AI works and how to interact with it to get the best results.

The difference between a good and a bad prompt can be enormous, so take the time to learn how to prompt effectively. Use that knowledge when giving suggestions or creating guides for your users. The anatomy of a prompt can be the difference between an incredibly insightful and useful output and gibberish.

Here are some examples of good and bad prompts:

Summarisation

Bad: "Summarise this article."

Good: "Summarise the key points of this article in 3 to 4 bullet points using plain English, focusing on the author's main arguments and conclusions."

Why it’s better: The good prompt adds structure, target output format (bullets), tone ("plain English"), and focus.

Contract Review

Bad: "Check this contract."

Good: "Review this contract and identify any clauses that may present financial risk or liability to our company. List them in a table with clause number, risk level, and a short explanation."

Why it’s better: The improved version is specific about the task (risk identification), expected format (table), and how to analyse the text (financial risk).

Product Announcement Email

Bad: "Write a product update email."

Good: "Write a friendly product update email announcing our new AI search feature to existing users. Keep it under 200 words, highlight 2 key benefits, and end with a CTA to try it out."

Why it’s better: It defines the audience, tone, word limit, focus areas, and the goal of the message (engagement via CTA).

Google recently released a 68-page whitepaper on prompt engineering that you can read to get more information. If you don’t feel like reading, you can always upload it into NotebookLM to summarise, get a podcast, or even create a mindmap of the topics inside the paper.

AI Product Management Is Still Product Management

The core remains the same: solve real problems, validate with users, and ship iteratively. But be prepared to educate users and stakeholders along the way.

It’s easy to get distracted by the hundreds of news articles about new technologies that seem to be coming out every week, but remember that even though the technology changes, user problems don’t.

As Clayton Christensen argues in Competing Against Luck, user problems are surprisingly consistent over time.

AI is just a technology. A powerful and extremely wide-reaching technology, but a technology nonetheless. The core parts of product management don’t change when working on an AI product.

You still need to effectively:

• Document business objectives
• Define user tasks (JTBD)
• Understand pain points
• Internalise customer problems and share with your team
• Develop innovative solutions that really solve the problem (with or without AI)

and all the core practices that make product management effective, regardless of the tech stack.

Accelerate Your Learning by Sharing What You Know

Teaching others (even non-technical friends) is a great way to clarify your own understanding and reinforce what matters most.

I always find that I learn and retain much better when I have to teach someone else. Even though it might be hard, make it a point to try and explain some of these concepts to your friends and family. As well as organise round table discussions about the technology with your colleagues.

Remember, technology is always evolving. If you stay curious, stay close to your users, and sharpen your technical skills, you’ll be well-equipped to manage whatever comes next.