It’s crucial to understand that generative AI is continually improving, sometimes even on a weekly basis. The AI tools available today will continue to evolve over time. However, the principles for responsible evaluation and implementation remain largely consistent. In this installment, I’ll use my experience developing integrated solutions using generative AI to walk you through the process of evaluating the latest tools in the market, a process that may differ from your usual software evaluation. The goal with this blog series is to help you navigate this new landscape with confidence and make informed decisions that can drive your business forward.
What Can Generative AI Do for You?
Generative AI employs machine learning models to generate new content from data, such as text, images, or video. Think of generative AI as a diligent high school student who is a great writer who can only remember about 10 pages of text. It can generate personalized content at scale, but it should not be tasked with creating things outside of the information you input. In other words, for most business purposes, you shouldn’t rely on it to come up with creative solutions, discover new ideas, or cite new references. And you need to think about if all the instructions for the output can fit into 10 pages of text. Generative AI works best when strategically deployed for specific, repetitive tasks, rather than given too much autonomy.
Future Generative AI Tools: A Preview
Here are some examples of what you can expect to see in the new wave of AI software and how it will be used to increase business productivity:
- Generating personalized employee survey results for every manager. Generative AI can sift through the results for a manager’s group, identify trends over time, compare to company results, and explain how these all fit together. It can then repeat this process for thousands of other managers, completing everything in just minutes. This not only saves time but also ensures that each manager receives insights tailored to their team.
- Drafting a patent application using a couple of pages of plain language description. Inventors are often adept at describing what their invention does and how it works, but may struggle to translate these descriptions into legal language. Generative AI can bridge this gap in seconds.
- Emailing daily customer survey summaries and comparing key metrics to historical averages to alert you to areas that need attention. AI-generated reports deliver key insights as soon as they are available, eliminating delayed analysis. Using AI to do a first pass of a large unstructured data set and output key highlights can save incredible amounts of time, resulting in happier customers served by more productive employees.
- Creating sales call summaries with next-step recommendations to assist sales managers in guiding their teams. Generative AI can also create summaries of customer sales communications, including recommendations for next steps and alerts for any missteps by your salesperson. This can help you streamline your sales process and enhance customer relationships.
Evaluating Generative AI Tools: What to Look For
Evaluating a new generative AI tool differs significantly from evaluating traditional software. Instead of a lengthy assessment of specific features, the focus should be on the AI itself - how it was developed and how well it achieves its goals. Here are some key things to consider:
- Quality of output: How accurate, coherent, and useful is the content the AI generates? Look for examples of actual output to determine if it meets your needs. Ask if your own company’s data can be used to generate tens or hundreds of samples for evaluation.
- Customizability: Can the AI be customized or fine-tuned for your specific needs and use cases? This is a double-edged sword. While customization can yield tailored results, requiring customization for optimal results may lead to real-world output not matching the value you expect from a demo.
- Ease of use: What do employees need to input to generate content? The less input required, the easier the AI will be to use. Complex interfaces and the need to provide clear instructions can hinder adoption, lead to poor results, and frustrate your employees.
- Integration with editing: The output should be designed to work in tandem with your workforce. A great first draft that saves time but still needs refinement is better than your employees thinking they’re getting a fully finished product. Most people prefer editing and refining to starting from scratch. Ensure AI tools have simple editing capabilities and can ingest feedback to improve over time.
- Data and security: Ask about what data the AI uses, how it ensures privacy and security, and whether it is SOC 2 certified. More data and stricter security policies are better. Be cautious of any AI company that can’t explain how data is kept private, secure, and used responsibly.
- Pricing model: Consider how the tool is charged - per generation or as a bundled add-on service? Can you pay extra for higher quality or more sensitive outputs? Some AI vendors charge by volume, tier, or additional capabilities. Choose an option that fits your budget and needs.
- Development and testing process: A rigorous development and testing process, including simulated and real-world data, leads to better quality AI. It is essential that the development process includes ongoing monitoring, as AI models continue to evolve. Ask about how the AI is built and improved to assess its ability to generate valuable, unbiased results.
Creating a Roll-Out Plan
Even during the evaluation phase, it’s important to consider how you would roll out the generative AI tool to your employees. This involves considering training needs, time efficiency, stakeholder buy-in, and potential ROI. Use these tips when evaluating your adoption plan:
- Employee training: How much training is required? Can employees grasp it after a few uses or is robust training necessary? Is it worth conducting a pilot with a small group of employees to identify any issues? For a tool to be truly transformative, it ideally will require little to no training, and instantly result in higher productivity. If this isn’t the case, you’re not looking at a high-end AI solution.
- Time savings vs quality improvements: Will the AI save employees time or increase output quality? Ideally, a balanced mix of both time-savings and quality improvement can significantly propel your business performance.
- Stakeholder buy-in: Engage with stakeholders in related roles or departments. Address any concerns and ensure the concept and use cases are fully understood and supported before rolling out the AI. Generating excitement and alignment around an AI initiative leads to faster, more widespread adoption.
- Return on investment (ROI): Calculate the potential ROI to determine if implementing the AI will have a meaningful impact. Hard numbers speak loudest when making a business case. While soft benefits like improved experience or productivity are important, estimating cost savings and revenue potential is key to building the excitement that you’re about to make a transformative change.
Real World Example: Macorva’s Radiant AI Manager Experience
At Macorva, we take pride in creating top-tier AI products. We provide a comprehensive AI solution for HR and management teams to streamline their work and deliver data-driven insights. Our new generative AI performance management solution is a testament to this. Here’s some examples of how we’re putting manager experience data to work:
- Performance management documents: Our tool automatically generates performance management documents using employee engagement surveys, 360 feedback, customer surveys, and manager notes. This saves managers time and ensures that all relevant data is considered.
- Data used to summarize, not create: We design our performance documentation to utilize the wealth of data you already have - surveys, notes, etc. Our AI sifts through this information and creates documents that annotate and reference this data. We believe that AI should show its work. Thus, your managers start off with a comprehensive document, and employees understand where everything came from.
- Create first drafts: Our tool outputs first drafts of performance reviews, OKRs, SMART goals, employee development plans, 1:1 meeting agendas, etc. This provides managers with a solid starting point and reduces the time spent on these tasks.
- Smart output: The output goes directly into an easy-to-use text editor, with a clear approval step before results are visible to the employee being managed. This ensures that managers have full control over the final output.
- Simulated data: We use simulated data and typically try hundreds of different formats of inputting simulated data to achieve the best results. This rigorous testing process ensures that our tool is highly effective and reliable.
- Testing and validation: We spend months programming and testing a function, and then months validating what it looks like when we use customer data. This ensures that our tool is ready for real-world use and can deliver accurate, useful results.
- Security: Everything is SOC 2 certified, and everything is done with full privacy. We take security and privacy very seriously and have implemented robust measures to protect your data.
- Pricing: Our tool is included as an integrated add-on for higher-level plans. Customers can upgrade to use outside data or for higher quality compute power, giving you flexibility and control over your costs.
The Final Word
Generative AI is more than just a passing trend–it’s the future of business operations. As it becomes more accessible and applicable, it is reshaping business processes, enhancing productivity, improving output quality, and enabling businesses to thrive in competitive environments. Operating a business without Generative AI will soon be akin to running an enterprise without the internet. It’s time to take the leap and embrace the future. We're here to guide you - in our next installment, we'll dive deeper into creating a customized rollout plan for your business to implement a generative AI tool.