The integration of Generative AI, especially with Large Language Models (LLMs), into product strategy presents both an opportunity and a challenge.
What are the opportunities in a few bullet points?
To say that Generative AI has taken the world by a storm will be an understatement. Ever since ChatGPT, arguably the most famous application of Generative AI, was released at the end of 2022, every Product Head across the globe has been scrambling to include Generative AI in their Product Roadmap. And with good reason as well. GenAI has opened up a wide variety of user experiences and feature possibilities. With Open Source joining the party recently, GenAI is no longer the realm of BigTech only and has spurred a large number of startups looking to move quickly in this space. But what exactly are the opportunities that we are talking about here.
While one could envision leveraging GenAI for fields as diverse as Media, Entertainment, Education, Commerce, Healthcare and Security, let’s look at some examples of business impact from Gen AI :
- Automated and relevant content generation - Ecommerce companies are looking at highly personalized customer experiences across millions of users. Brand managers will no longer be struggling with incomplete catalogs via GenAI driven product images and descriptions
- Marketing & Ads - GenAI models finetuned on a proprietary dataset of product images can create newer images relevant to a given target audience (across geographies, demographies, seasonality) further driving the ad efficacy. Well, with text to video gaining ground, GenAI may even generate the entire advertisement for you!
- Improved customer support - Enterprise help chat bots have always been a hit or a miss before GenAI. Customer support is a melting pot of customer emotions (anger, frustration, concern) and GenAI has the ability to identify these emotions and respond accordingly. More importantly, the quality of response is vastly better given the ability of GenAI to develop near-human responses using a wide variety of historical data.
- Improved sales & marketing efficiency - GenAI can be the able assistant that every Sales rep in the world wished to have. Analysing leads, digesting marketing brochures, summarizing customer interaction history is all taken care of by this assistant enabling the Sales rep to focus on their core job of selling and closing deals.
- Compliance & Governance - Compliance rules across some of the sensitive sectors such as Healthcare, Finance, etc. are always buried deep in documents and rulebooks that run into hundreds of pages and are ever- changing. GenAI can make short work of this given its ability to consume large content and then use it further for interactions and analysis.
While the possibilities can be limitless, the landscape is filled with the allure of powerful LLMs, but also the hurdles of cost, scale, and risks.
Cost Implications of LLMs
Before integrating Generative AI or a custom LLM, one needs to understand the compute and production costs :
- Computational Costs: Adopting the LLM framework demands a hefty GPU commitment. The training of vast models consumes considerable computational power, escalating financial challenges, especially for burgeoning companies.
- Cloud Expenses: The boon of modern cloud architectures for LLM inferencing comes with its own financial woes. While Generative AI searches promise a superior user experience, they also caution with a potential tenfold rise in cost per query compared to conventional search avenues.
- Infrastructure and Energy Overheads: Embracing LLMs means investing in advanced GPU and TPU servers, their associated infrastructure, and the energy to power them. It's a substantial commitment, with some speculations suggesting AI infrastructure costs might cross the $76 billion mark by 2028.
For those deciding between in-house and commercial platforms like OpenAI, Palm, or Anthropic, resources such as [GPTforWork](https://gptforwork.com/tools/openai-chatgpt-api-pricing-calculator) can provide insights. Still, these don't encompass the nuanced costs of system scaling and tailored engineering.
The tradeoff in harnessing LLMs - LLMOps
LLMOps is an emerging area that integrates custom or cloud LLMs into production workflows that includes traditional data engineering and future curation from MLOps and specific turning and prompting unique to LLMs.
There are several techniques to improve and fine tune the LLMOps workflow that has both cost and quality implications.
| Approach | Description | Impact |
|---|---|---|
| Prompting | Engage a pre-trained LLM with concise instructions | Simple, yet occasionally imprecise and error-prone including hallucinations |
| Zero and Few shot prompting | Examples based prompting is a step up that helps further context and intent to LLMs | Improvement in quality and helps targeted responses with prompt templates |
| Fine-tuning | Augment pre-trained LLMs with targeted dataset training | Leads to precise responses and adapt for specialized tasks to include with the above two techniques. Also provides the right balance of power of Generic LLM and privacy (ChatGPT from GPT models and CodeLlama from Llama2 would be examples of this technique) |
| Pretraining and custom LLM development | Craft an LLM from scratch completely built with in house data | While this is the most customized approach for with highest quality for production use cases approach requires significant compute resources and expertise, massive corpus of internal data to train the model. This also needs to be augmented with fine tuning for further precision in specific internal use cases |
In the context of vendor dependencies, combining prompting with fine-tuning, particularly with open-source LLMs like Llama2, emerges as a compelling avenue.
Potential Issues with Cloud LLMs and the Open-Source Road Ahead
Venturing into the vast realms of Generative AI andLLMs has some pitfalls. Foremost among these is the specter of data privacy. These models, richly fed with diverse datasets, may inadvertently leak proprietary knowledge. Here, the emergence of open-source models like Llama2 signals a an approach akin to the rise of open source software over the past few decades With its robust open- source following, Llama2 not only assures transparency but also brings adaptability for fine-tuning, reducing the extensive data demands and the potential risks of their bulkier counterparts.
