Under the Hood: AI-powered Case Notes Summarization
At Peach, we continue to invest in cutting-edge technologies that push the boundaries of loan servicing. One example of this is our focus on generative AI-powered servicing enhancements to improve agent productivity and the borrower experience. Because we’ve built our servicing tech in-house and in an API-first manner—and because our platform fully integrates loan management, servicing, collections and compliance—Peach is uniquely positioned to help lenders quickly realize the benefits of generative AI.
Our newest AI-powered innovation is Case Notes Summarization, a capability designed to reduce the time agents spend reviewing case histories in the Peach CRM. By employing generative AI to systematically and succinctly summarize a case’s full history of notes, we’re now able to meaningfully streamline agents’ case management workflows. In this post, we’ll take a closer look at how Case Notes Summarization works as well as the technology behind it.
The business objective: Increasing agent productivity
The core business objective of Case Notes Summarization is simple: To increase the productivity of agents by reducing their time spent reviewing case histories.
This is an especially important pain point in the context of collections, where it’s typical for multiple agents to work a single case and where cases may contain dozens or even hundreds of notes. Historically, agents have always reviewed each note to piece together a clear and accurate chronological picture of the case history and associated loans. This is a time-consuming and tedious process that’s susceptible to human error.
However, this challenge fits squarely within the wheelhouse of generative AI. Which is why we’ve invested in a tool that will, with the click of a button, generate a concise summary of all notes attached to a given case.
WATCH CASE NOTES SUMMARIZATION IN ACTION
With this tool, agents gain quick access to a concise yet comprehensive overview of the case history. This empowers them to quickly comprehend the current loan status and see the chronological sequence of any relevant events associated with the case. As a result, agents can take the necessary actions more quickly and with greater context. Agents can in turn handle more cases in a given time period.
While the primary benefit of Case Notes Summarization is increased agent productivity, there’s an additional positive impact on the borrower experience. Any reduction in the time borrowers spend waiting on an agent is a net positive, as are improvements in agents’ understanding of case histories. By quickly gleaning the essential points of the case, agents can engage with borrowers more thoughtfully, reinforcing trust and providing more personalized solutions.
An example summary for a case with extensive historical notes.
Translating business objectives into machine learning goals
To assess the effectiveness of Case Notes Summarization, we needed a consistent testing framework. We determined that the utility of Case Notes Summarization could be understood through a proxy metric: the normalized number of queries made to our case note endpoint. As Case Notes Summarization gains traction within servicing teams, agents consult the case notes themselves less often, leading to a reduction in queries to the case note endpoint. This change is a positive indicator of increased efficiency in the handling of cases, of a smoother workflow that reduces the bottleneck in information retrieval, and of more time being redirected toward proactive customer engagement.
Concurrently, for offline evaluation, we employ the standard ROUGE-L metric for measuring the preciseness of our summary-generation algorithm.
Machine learning tools and methods
A crucial part of the engineering process in developing this feature was selecting the appropriate tools. Our collaboration with Google's Vertex AI proved fruitful, giving us a diverse set of offerings to explore. We decided to leverage a blend of self-hosted open-source models and other advanced models available on Vertex AI, ensuring the best fit for our requirements.
To refine our models, we conducted a series of detailed experiments, exploring methodologies such as fine-tuning, distillation and prompt engineering. We also leveraged techniques like chain-of-thought and self-consistency sampling for prompts, coupled with frequent tests using various large language model (LLM) parameters and temperatures.
Our experiments were carefully monitored and recorded, enabling us to meticulously manage and version all experimental models. This systematic tracking allowed us to effectively compare experimental iterations and determine optimal models.
A/B testing, monitoring and improvisations
At Peach, we value consistent performance monitoring and enhancements. And so we’ve ensured steady tracking of system performance, swiftly detecting and handling anomalies. A/B testing is also pivotal to our approach, especially when it comes to testing various models and prompt variations. The data insights obtained in this process enable us to make necessary adjustments and improvements.
Integrating LLMs into our production environment presented unique challenges, particularly in managing the variability in output format inherent to natural language processing. To tackle this challenge, we invested heavily in precise prompt engineering, meticulously crafting our queries to guide the LLMs toward generating more predictable and structured responses. Concurrently, we fortified our downstream processing code, enhancing its ability to handle a range of input variations.
Where we go from here
Our journey in developing our Case Notes Summarization tool reinforced our conviction around the vast potential of LLMs in solving real-world challenges for lenders.
We’re still near the start of our generative IT journey, and we already have an extensive roadmap in place. We will continue to navigate the rapidly advancing AI landscape to identify the platform enhancements that offer the highest return for our clients. As ever, we are committed to delivering unmatched efficiency and value in disruptive ways, continually orienting the lending technology industry toward the areas of greatest opportunity. Stay tuned for more updates as our roadmap comes to fruition.
If you'd like to learn more about how our AI-powered servicing tools can increase your operational efficiency, please reach out.