The Hidden Truth about AI Product Metrics (Part 2)

Hello Product Folks!

Today, we will go deeper into AI product development metrics and AI risks.

In Part 1, ‘AI is eating Product quickly’ I covered the paradigm shift for traditional vs AI product management.

As AI is new for most of us, I wanted to share the most common key AI product metrics and ways to handle AI risks to help you succeed as an AI Product Manager.

Let’s dive in…

ICYMI…

👉️ Sam Altman was at the AI for Good Global Summit and talked about GPT-5, the Scarlett Johannson controversy, and Helen Toner’s commentary.

👉️ Maven AGI raised $28 million and claims its customer support AI can solve over 93% of problems independently!

👉️ Perplexity Pages - Turn your research into shareable articles with AI
👉️ ElevenLabs launched Text to Sound tool to generate sound effects
👉️ Navigating Your 3 Career Levels (Link)
👉️ Step-by-step guide to get started with AI for Product Managers (Link)
👉️ AI is Eating Product Management Quickly (Link)

1- AI Risks and Mitigation

When it comes to new tech, like AI, there are always risks involved. It reminds me of the cloud computing wave in the early 2000s.

Senior execs are likely asking you for your take on AI risks in your organizations. It's super important to spot these risks early on and expose them to the team as you start exploring and possibly launching AI features.

Let's break down the key areas you need to keep an eye on:

• Data Security and Compliance: First things first, let's talk about keeping your data safe and following the rules. Make sure your data is handled properly and any (PII) personal info is removed to protect your customers. Avoid using data that's biased or collected illegally. Make sure you're following all the legal stuff and that your systems can handle the data safely.

  
  Example: Google's got strict rules for handling data to train its AI models. They make sure they're using top-notch, fair data and have super secure systems to handle it all. Even though, some embarrassing Gemini AI comments made the headlines once again.
  

• Risk Management Gates: Next up, let's talk about putting up some checkpoints along the way. You want to have systems in place to check things as you're building, testing, and launching your AI features. This helps spot any issues early on, like if your AI starts acting wonky or if you're relying too much on outside vendors.

  
  Example: Facebook's got systems in place to shut down certain AI features if they spot anything weird happening. It's all about keeping things stable and safely.
  

• Feedback Loop: Now, let's talk about staying connected with your users. You want to keep gathering feedback and monitoring how they're using your AI features. This helps you see what's working and what needs tweaking.

  
  Example: LinkedIn keeps an eye on how people interact with their job recommendation AI. They use this feedback to keep improving the system.

  
  By keeping these tips in mind and making the most of AI in your product management journey, you can handle AI risks like a pro and drive some great AI projects in your organization.

2- Basics of machine learning for PMs

Now, we will discuss the difference between Supervised Learning and Unsupervised Learning which are part of the main discussion you will have with your data science or ML engineering teams.

a- Supervised Learning

• Imagine you have a bunch of ‘labeled’ data' like pictures of elephants, cows and camels above.
  
  In short, you teach a machine to recognize these animals (Label it). You feed the machine images and tell (label) it which ones are elephants and which ones are cows.
  
  The machine learns to map features (like size, body length, ear shape) to the correct label (elephant or cow).
  
  This process is called supervised learning because the data is labeled. This is the main difference with unsupervised learning.

  
  e.g. Google Photos uses supervised learning to identify animals in your photos.

• Classification vs. Regression
  Under supervised learning, there are two main methods you’ll most likely discuss in your organization.

• In classification, you're drawing a line to separate different groups. For example, if you have data about two types of patients, classification helps you figure out which patient is sick based on their vitals.

• In regression, you're predicting a continuous numeric value. For instance, if you know patients' vitals, you can predict their sickness rate. Metrics like root mean squared error or R-squared measure the difference between our predictions and actual values.
  
  This is plotted on a line to show the relationship.

b- Unsupervised Learning

In unsupervised learning, you don't have labeled data. You just feed a lot of data into the system, and it finds patterns or clusters like the example above.

Three clusters are detected. e.g. It might group people based on their shopping habits without being told what those groups are.