Pioneering Your Own AI Product Management Journey: Choosing the Perfect First Project

The lively discussions ignited by my recent article, “We Need to Raise the Bar for AI Product Managers,” emphasized a shared enthusiasm for enhancing the realm of AI product management. Numerous current and aspiring PMs have since reached out, inquiring how they can expand their AI knowledge on their journey to becoming an AI product manager.

From my perspective, the most efficient AI PMs excel in two vital domains: identifying where AI can add value, and collaborating with model developers to deploy the technology effectively. This demands a robust comprehension of how different models are likely to perform when operational — a reality that often catches beginners off guard. The disparity between flashy demos or early-stage prototypes and actual product performance can be substantial, whether dealing with customer-facing applications or backend data pipelines that empower products.

The optimal method to develop this insight is by integrating various models into products and making numerous errors along the way. The next best option is to investigate what other teams at your organization are doing and learn from their mistakes (and victories). Unearth any documentation available and, where feasible, listen in on product reviews or team updates. Often, individuals who directly worked on the projects will be eager to chat, address your questions, and provide more context, especially if your team might be considering something similar.

But what if you aren’t employed at a company engaged in AI? Or your company is concentrated on a very narrow set of technologies? Or perhaps you are in the midst of a job hunt?

Besides exploring resources to acquaint yourself with terminology and best practices, I advocate developing your own AI projects. I particularly recommend side projects even if you can learn a lot from your day job. Every AI use case has its own subtleties, and the more examples you can closely examine, the faster you’ll develop a knack for what does and doesn’t work.

For a beginner project, I propose starting with LLMs like Claude or ChatGPT. You should be able to get something substantial operational within hours (minutes if you already know how to code and compose effective prompts). While not all AI projects at a real company will use LLMs, they are gaining considerable traction. More importantly, it’s much simpler to create your own working model with only basic data science or coding knowledge. If your coding abilities are rusty, utilizing the developer APIs will give you a chance to refresh, and if you get stuck the LLM is an excellent resource to assist with both code generation and problem-solving. If you’re new to both coding and LLMs, then using the online chat interface is a great way to warm up.

But what’s the distinction between using the ChatGPT website or app to make you more productive (with requests like summarizing an article or drafting an email) versus an actual project?

A project should strive to resolve a genuine issue in a replicable manner. It’s these subtleties that will help you refine some of the most crucial skills for AI product management work at a company, particularly model evaluation. Check out my article “What Exactly is an Eval and Why Should Product Managers Care” for an overview of model evaluation basics.

To ensure that what you’re working on is a genuine project that can undergo its own mini eval, make certain you have:

1. Multiple test samples: Aim for projects where you can analyze the model on at least 20 different examples or data points.
2. Diverse data: Ensure your dataset includes a range of scenarios to test what causes the model to fail (thus offering more opportunities to rectify it).
3. Clear evaluation criteria: Be clear from the outset how a successful model or product behaves. You should have 20 ideal responses for your 20 examples to evaluate the model.
4. Real-world relevance: Choose a problem that mirrors actual use cases in your work, personal life, or for someone close to you. You need to be well-versed to judge the model’s effectiveness.

Please avoid these specific projects unless one of them truly resonates with you. These are merely for illustrative purposes to help explain what constitutes a real project, as opposed to a one-off query:

Gift Recommendation Classifier

• Objective: Decide if a given product would be a suitable gift for a discerning friend or family member.
• Method: Use text generation to analyze product titles and descriptions with a prompt detailing the recipient’s taste profile. If you want to go a bit more intricate, you could employ vision capabilities to analyze the product description and title AND a product image.
• Test instances: 50 various product images and descriptions. To introduce complexity, your examples should feature some items that are clearly excellent, some that are obviously poor, many that are marginal, and a few that are entirely random.
• Assessment: Have the intended gift recipient appraise the product list, rating each on a scale (e.g., “no way”, “meh”, and “absolutely”) according to how well it aligns with their tastes. Compare these scores to the model’s predictions. Additional insights can be gained by asking the model to explain why it believes each item would or wouldn’t be a suitable match, aiding in diagnosing errors and informing prompt refinements, while also providing deeper understanding of LLM reasoning.

Recipe Book Digitization

• Objective: Transform your grandmother’s cherished out-of-print cookbook into a mobile app for your family.
• Procedure: Utilize vision functionalities to extract recipe details from photographs of the book’s pages.
• Test cases: 20 images depicting various sorts of recipes. For simplicity, you might start with desserts, including examples like 4 types of cakes, 3 types of cookies, etc.
• Review: Ensure that all essential ingredients and instructions from each recipe appear in the final output. Meticulously compare the LLM output with the original, verifying the accuracy of components, quantities, and preparation steps. Extra points for organizing the data into a structured format (like JSON or CSV) for easier app integration.

Public Figure Quote Extractor

• Objective: Assist a public figure’s publicity team in pinpointing any statement or fact attributed to them for the verification team to authenticate.
• Procedure: Employ text generation to analyze article texts and retrieve a list of quotes and facts concerning your public figure mentioned in each piece.
• Test instances: 20 contemporary articles about the public figure, covering a minimum of 3 distinct events from at least 4 disparate sources (consider one tabloid, a major outlet like the New York Times, and an intermediate source like Politico).
• Assessment: Thoroughly read each piece to check if any quotes or facts about the public figure were missed. Your employment might be at stake if the summarizer fabricates (e.g., attributing a statement not made) or overlooks critical misinformation. Confirm that all quotes and facts the summarizer identified indeed pertain to your public figure and are present in the article.

You can use any LLM for these endeavors, though from my experience, the ChatGPT API is the most user-friendly if you have limited coding skills. Upon finishing one project, evaluating another LLM on the same data becomes relatively straightforward.

The objective of beginner projects is not flawless execution but to engage with an intriguing project that presents some complexity to ensure you face challenges. Learning to troubleshoot, iterate, and occasionally realizing that something may not work will help sharpen your intuition on what is feasible and the effort required.

Cultivating strong intuition in AI capabilities and limitations is essential for effective AI product management. By participating in practical projects, you’ll gain vital experience in model evaluation, problem-solving, and iteration. This practical expertise will enhance your collaboration with model developers, enabling you to:

1. Pinpoint areas where AI can deliver true value
2. Create realistic projections for AI project timelines and resource needs
3. Meaningfully contribute to troubleshooting and evaluation workflows

Through these projects, you’ll gain a nuanced comprehension of AI’s practical applications and hurdles. This knowledge will set you apart in the fast-evolving domain of AI product management, equipping you to lead innovative initiatives and make informed decisions that drive product success.

Keep in mind, the path to becoming an expert AI PM is continuous. Embrace the learning process, maintain curiosity, and consistently seek out new challenges to improve your skills. With commitment and hands-on experience, you’ll be well-prepared to navigate the impactful landscape of AI product development.

Should you have any inquiries about your AI project or this piece? Reach out to me on LinkedIn to continue the discussion.