This article talks about the considerations and the change in mindset for a Product Manager when thinking about, building and rolling out GenAI based Products & Features.
In our previous blog post “Product Strategy for Generative AI and rise of Open LLMs”, we read about the possibilities that GenAI has opened up and the practical considerations there in. Though the possibilities seem limitless, the fundamental principle of building something valuable that users are willing to pay for very much remains relevant. For all the things that GenAI can do, it cannot sell itself (maybe one day it will!) and the Product Managers need to think deeply about the relevance and practicality of introducing GenAI in their products.
Building sustained value that customers are ready to pay for
From software eating the world to AI eating software, we are now talking about GenAI eating AI. But at its core, its still the classic phenomenon of a newer technology subsuming a relatively older technology and in the process creating new product avenues. The fundamental question of whether GenAI is a nail in search of a hammer still very much applies but based on the initial signs, GenAI definitely seems to have found the Technology-Product Fit for use cases such as near-human chatbots, content generation, document summarization and so on. When we are considering use cases beyond these in our respective domains, the first principles of Product Management need to be critically looked at. Am i building something that is generating value at a manageable cost? And is someone willing to pay for this value? Its common knowledge that GenAI applications have significant development costs spanning costly hardware and top quality engineers amongst other things. The tough question that all PMs need to be clear about is “How much will it cost me to build a GenAI driven feature vis-a-vis how much sustained incremental revenue will I be able to earn?”
Native GenAI workflows or GenAI on top of existing workflows
While the first thought for a PM would be to think about how can GenAI enhance existing Product Workflows, there is a golden opportunity to think about Native GenAI workflows. This is analogous to the transition from the web era to the mobile era. The initial product efforts during this period tried to fit web workflows into the mobile form factor. It took a while before PMs and Designers ditched web altogether and started imagining mobile-first workflows. GenAI is capable of introducing hitherto improbable product experiences but it will take some time for Product folks to discount existing flows and go GenAI-first.
Imagine a GenAI rendered avatar as your sales assistant joining you in all your customer calls. You introduce the assistant to your customers as like any other team member. In the meeting, this assistant is able to quote past conversations for context, share details about the product / service that you are struggling to remember or maybe even highlight any contractual nuances during a tough negotiation. Imagine the efficiency this would bring in sales meetings and with it hopefully conversions!
I expect startups to be the catalyst here for such experiences. GenAI-first startups with the ability to experiment and fail fast will quickly iterate and pave the way for this sort of native GenAI thinking to become commonplace.
Regulations, Privacy and the growing scrutiny for GenAI
The Future of Life institute released an open letter in March 2023 calling for a pause on GenAI related development work and ensure that the necessary safety protocols are put in place for GenAI activities. Amongst the 33,712 signatories are technology stalwarts Elon Musk and Steve Wozniak. Now these people definitely know a thing or two about mainstreaming cutting edge technologies and here they are acknowledging the risk elements in GenAI. The counterpoint to this letter is that the GenAI is being mistaken for Artificial General Intelligence which could one day surpass the cognitive ability of humans and end up taking over the world. Product Managers don’t have the time to wait for this debate to play out and are under pressure to get started with GenAI quickly. Now, its the PM’s responsibility to put in the necessary guardrails while leveraging GenAI. Coverage of data privacy norms, local government regulations, compliance requirements will form a critical section of a PRD (Product Requirements Document) which will have to vetted by the company’s Risk and Legal teams. Given the nascency of this technology, chances are that Legal teams may want to play safe and turn down some of the coolest GenAI use cases a PM is thinking of. Regulations around GenAI is an additional area that a PM will have to spend a significant time getting familiar with.
Impact on the Product of a potential of a genAI mistake
Majority, if not all, AI systems are probabalistic in nature and prone to error or inaccuracies. In such error scenarios, B2C applications can be comparatively more forgiving as compared to B2B ones. But irrespective, accountability needs to be fixed in case of any errors. Predictability of performance needs to be introduced so that PMs know that Production systems will work as expected day in day out. But how do we introduce a deterministic aspect into something that is probabilistic in nature?
Is it possible to have a conventional Product sign-off when its time for a Product Manager to allow a GenAI feature to be released into production? Sure, there are performance benchmarks that convey the “accuracy” or “correctness” of the deployed model. But what happens once it goes to production? Its definitely not a fire and forget. Just like a conventional production ML model can degrade over time and needs to be strictly monitored, GenAI based LLMs also need to be monitored for any sort of “degradation” or “incorrectness” or “hallucinations”. Sam Altman himself has acknowledged that it will be at least 18 to 24 months before we learn to deal with hallucinations in LLMs. The Product Manager now needs to make a provision for this and ensure that the hallucinations are caught at the earliest possible stage. The consequent disruption to the product usage or user experience should be minimised and corrective measures should be triggered immediately. While the PM has the thrill of introducing a cutting edge GenAI based feature in the product, the reality promptly sets in with the extra care that needs to be taken during the actual usage. I would envision this extra care to take a non-trivial amount of thought on the Product Manager’s part.
Tolerance for ambiguity
Another factor for PMs to contend with while using GenAI is the need to have a certain tolerance for ambiguity. When it comes to LLMs there is a balance to be struck between creativity and perfect accuracy and the models need to learn when we want one or the other. Naturally, its the Product Manager who needs to make the model learn the preference here (creativity or accuracy) based on the business case. Google is experimenting with a news writing product for which accuracy is critical. At the same time, its the creation ability of GenAI that is going to be key enabler for this product. The balance will dictate the success of this product. The bias towards accuracy or creativity needs to be a key requirements specification so that Data Scientists can tune the model accordingly. This will now go into the long list of trade-offs that a PM is supposed to do in their daily lives.
As Product Managers, its truly an exciting time to be thinking about new possibilities, business ideas and product experiences based on GenAI. But it does require some changes in thinking and mindset for Product folks. It will be interesing to see how the Product Management tribe copes with these additional dimensions!
