Mandy Gu on Generative AI (GenAI) Implementation, User Profiles and Adoption of LLMs

Transcript

Introductions [00:27]

Srini Penchikala: My name is Srini Penchikala. I am the lead editor for AI, ML, and the data engineering community at the InfoQ website, and I’m also a podcast host. Thank you for tuning in today to this podcast. In today’s episode, I’ll be speaking with Mandy Gu, who is currently a senior software development manager at Wealthsimple. She leads the machine learning and data engineering initiatives at the organization.

We will discuss the topics of generative AI and the large language models, or LLMs, especially how these technologies are used in real-world projects. We’ll also talk about how different user profiles in the organizations influence the adoption of LLMs. Hi, Mandy. Thank you for joining me today. Can you introduce yourself and tell our listeners about your career and what areas you’ve been focusing on recently?

Mandy Gu: Thanks for the introduction. Excited to be here. So a little bit about myself, as Srini mentioned, I’m currently at Wealthsimple, where I’m supporting our data engineering and machine learning engineering teams. One of the things that we’ve been focused on for the past few months was building the infrastructure and internal tools to support gen AI adoption and usage throughout the company.

Implementation of AI Programs [01:36]

Srini Penchikala: Yes, thanks, Mandy. So let’s talk about that. Can you talk about your experience in establishing AI programs in organizations? What typically the challenges are: organizational, technical, and sometimes people-related. So, skill sets. Can you talk about what your experience has been and what you can share with our listeners?

Mandy Gu: Yes, so the past year and a half to two years has been this really interesting time ever since ChatGPT blew up and ever since we pushed the boundaries of accessibility and just democratizing AI access. Things have changed very rapidly in the industry, and our AI strategy has really been centered around three things.

So the first is building up a lot of the LLM foundations and platform to support future work. And then from there we find use cases within the company to apply this technology to boost employee productivity. And in doing so, we are also developing our confidence and guardrails and skill sets to bring this technology to our clients, where we can create a more delightful client experience by optimizing workflows using LLM. Most of my focus has been on the platform side and building up the reusable building blocks and then supporting the teams in accelerating AI adoption for employee productivity and also to find those use cases to optimize the client experience.

I think one fairly meta challenge in rolling out these AI programs just has been how quickly the industry has been moving. We have pretty much a new state-of-the-art foundational model every two weeks or so, and OpenAI and all of these other companies are constantly releasing new things. So it’s been a fairly intense process of just keeping up with these new changes in the industry and finding the best ways to apply it internally while dealing with a lot of different user profiles and toeing the right balance between moving quickly and ensuring that we are making the safe way the default.

Srini Penchikala: Can you talk more about those reusable building blocks? Are they common solutions that other teams across the organization can reuse or?

Mandy Gu: Yes, so on a high level we have two sets of the reusable building blocks: the first are the foundational models or access to the foundational models, and then the second is some of the components we use to facilitate other teams to build multi-stage retrieval systems. So for the foundational models, we’re hosting a mixture of both open-source models and some of the foundational models from fully managed services such as AWS Bedrock.

So the way that we kind of tie this together is we’ve adopted a lightweight serving framework, LightLLM, and we’ve used this as a proxy server in front of all of these models. And then on top of this, we’ve been able to provide programmatic access that’s integrated with the rest of our stack so that any developer or any of our other services can just seamlessly integrate and communicate with these models for whatever they need to do.

We’ve also built a custom front end to recreate the consumer-like ChatGPT and cloud desktop experience so that we’re able to ensure that our employees are using these technologies but in a safe and contained way. So that’s the first set of the reusable building blocks that we offer. And then the second is that we have a lot of the platform pieces for both facilitating orchestration to update and read from our vector database and integrating that with our various knowledge bases so that we can build internal reg tools on top of that for both employee productivity and, in some cases, this has also been leveraged for back-office workflows.

Business and Technical Use Cases [05:05]

Srini Penchikala: Sounds good. Thank you, Mandy. So for these solutions, can you talk about what are some of the business and technical use cases where the team leveraged machine learning solutions?

Mandy Gu: Most of the use cases so far have been internal, and a lot of them have been used just by various people within the business for their day-to-day tasks. We actually have fairly impressive adoption statistics; close to two-thirds of the entire company use LLMs in their day-to-day, and most of their interaction is through our LLM gateway. So this is the front end that we maintain that talks to all of these foundational models and just lets our end users pick and choose which models they want to talk to and base their chat experience there. When we look deeper into what some of these use cases are, a lot of the times it is for content generation. So people would write something and then ask the LLM to either proofread it, augment it, or to continue or to generate content based on a prompt.

We also have a lot of use cases for information retrieval, just people asking questions, and we’ve been able to integrate a lot of these models with our internal knowledge bases, and that way people have been able to leverage it to ask questions about our company documentation and various of our internal knowledge bases as well. In terms of some of the workflows that we’ve been able to optimize, the most prominent one is for our client experience agents. So we’ve built a workflow where we’re taking the Whisper model to transcribe a lot of our client calls, and in doing so we’re able to take this transcription and augment a lot of our current machine learning use cases in this space to provide classifications that we then use to triage these client calls to the appropriate teams.

Generative AI Program and LLM Gateway [06:48]

Srini Penchikala: And also you mentioned about LLMs and the models. Can you talk about the Generative AI programs implemented internally to improve operational efficiency and streamline modeling tasks? Because gen AI brings more power to the business requirements, right?

Mandy Gu: Yes, I mean the biggest program that we’ve rolled out–this is something we’ve been working on for quite a long time now–is our own LLM gateway. So the motivation of having our own LLM gateway as opposed to leveraging one of the consumer products out there is so that we can make sure that all of this interaction with the LLMs stay within our own cloud tenant and that we’re leveraging the foundational models that’s hosted as a part of our VPC. So what this LLM gateway offers is, it’s an internal web URL that all of our employees can go onto, and we have been refining this front end to resemble as closely as possible the consumer products that OpenAI and Anthropic offers, and there’s a chat interface.

And then we also provide a prompt library and tools and ways of integrating, like, just getting knowledge from several of our integrations, including with our internal knowledge bases. And the way that we’ve positioned this program is we want this to be everyone’s first touchpoint with LLMs. We discourage the use of ChatGPT and these other products internally, or at least for work-related purposes, because of some of the security risks that come with it and the need for a lot of our employees to work with sensitive information. So that’s been one of the programs that we are working on.

Srini Penchikala: That’s good. So this LLM gateway looks like it provides an entry point to different teams who could be using different large language models and maybe different vector databases, right? So is that one of the responsibilities of this LLM gateway?

Mandy Gu: Yes, that’s basically it. The motivation is really getting as much exposure to these technologies as possible, but doing so with the right guardrails in place.

Typical Gen AI application development Lifecycle [08:44]

Srini Penchikala: Okay. And also, what are some best practices in terms of the life cycle of a typical gen AI application development? We don’t want teams to go off in their own direction and sometimes either duplicate the efforts or deviate from the standards. So do you have any recommendations on what are the best practices?

Mandy Gu: I think the investment in the platform and in these reusable building blocks will ensure that we can bake in the best practices and how we want different teams to work with these technologies as a default. So for us, internally, the way that we leverage our LLM gateway and the proxy server in front of our APIs, this is how we point people to interact with the models.

And this will avoid, for instance, somebody directly interacting with the OpenAI SDK and potentially exposing our sensitive information that way. We’ve also been able to configure and choose a lot of the optimal configurations and ways of interacting with these models as well as just prompt templates, and we offer them as configurations for the end users. Those are some of the ways that we’ve been able to ensure that we’re operating from a high standard when using these technologies.

Srini Penchikala: Does the gateway also help with any–what are they called? Hallucinations or the accuracy of the responses, or is it more of just a privacy and security type of checkpoint?

Mandy Gu: The main value proposition is more from a privacy standpoint, but we do offer some ways of dealing with the risk. We file these under specific models, but they’re actually just leveraging some of the models that we provide. But for all of the models integrated with our internal knowledge base, one thing that we’ll do is actually return where it gets the information from.

So this provides a way for the end user to kind of fact-check the answer and build confidence that way. And by offering the ability to just ground the answers against context that we’ve curated and verified internally, that does allow us to ensure that the LLM is at least reading this information from the right place. Outside of that, we’ve experimented and evaluated different prompt templates, and we’ve been able to find some effective ways of just instructing the LLM to return more reliable answers, although that’s not a safe proof measure.

Srini Penchikala: So do you run all the components of the AI solutions on-prem? Because this LLM gateway, would it also work against some API in the cloud or?

Mandy Gu: When we first started, everything was models that we served completely in-house. A few months ago, or actually closer to a year ago, we adopted AWS Bedrock. And since then we’ve actually been migrating more of our models from this internal self-hosted stack onto AWS Bedrock. Bedrock does give us the ability to serve these models within our VPC. So in that aspect it’s pretty much the same thing, but we have been migrating things to Bedrock just so that it’s easier to manage.

Srini Penchikala: You mentioned that one of the best practices is to invest in a platform, right? So you mentioned AWS Bedrock. Can you talk about any other specific technology or infrastructure changes that were put in place for this gen AI application stack? What are the technologies and tools that you used?

Mandy Gu: I mean, I’ll say of the foundational models, there was quite a bit that we had to stand up for our retrieval systems. So we adopted a vector database, and then we also built a lot of integrations between that and our orchestration platform so that we could just ensure that the indexes were being kept up to date, that all of our knowledge base was being indexed on a periodic basis.

So from an infrastructure perspective, those were the two main things that we focused on. And then outside of that, we’ve been building just a lot of integrations between the various knowledge bases and different systems that this may have to talk to. One new development that’s really taking shape this year is the MCP servers, having the ability to talk to an API or an SDK through like a language server. And one thing we’ve been working on over the past few months is building up some of the infrastructure to support that.

AI Agents [12:41]

Srini Penchikala: Yes, MCP has been getting a lot of attention. Also, the trend that’s been getting a lot of attention are the AI agents. Mandy, do you have any comments on AI agents? What should they be used for, and when the teams or developers should not use that? It could be overkill or not the best solution.

Mandy Gu: That’s a good question. I think it’s very hard to say, and this is a space that’s also changing very rapidly. There’s definitely been certain spaces where AI agents have proven to be more effective than others. I think this is definitely an area that’s worth keeping an eye on because with a lot of the advancements in integrating reinforcement learning with the way that we interact with LLMs, we’ve seen some massive leaps in terms of ability and relevant generations.

So there’s been a lot of developments in this space. I think the best thing we could do is to just make sure that we’re evaluating and we have a well-defined success criterion in mind because regardless of the AI technology we’re working with, there’s always going to be use cases where it’s really good and use cases where it’s not so good. And that’s something that’s been quite in flux with gen AI.

User Profiles and Adoption of LLMs [13:48]

Srini Penchikala: Yes, so we have to wait to see how it goes. In terms of the adoption of LLMs and the AI programs. The different organizations will definitely experience different levels of maturity and commitment from the employees and the leadership. So can you talk about, I know you mentioned about this earlier, the different user profiles and how different user types can influence the usage of insight, the LLMs? And what insights you can share, our listeners will be interested in who may have some similar situations in their own projects, like how do they need to manage the users and user profiles to get the best out of LLM adoption?

Mandy Gu: So I think with all new technologies, there’s about three to four different user profiles. And certainly this has been true for LLMs and generative AI.

From my experience, LLMs have been a very divisive topic. There’s those who love it, there’s those who hate it, and there’s those who just absolutely cannot stand it, especially in a workplace setting. I mean, starting from one end of the spectrum, you have the advocates; you have the people who are really excited about these technologies.

They’re the ones who are proactively experimenting with the new foundational models, the new tools that come out. And they’ll be really proactive in just finding ways of applying these technologies, like, “Hey, maybe I can use this as a part of this work that my team does”. I mean, I think the risks of working with these people is that they may set unrealistic expectations about the benefits of LLMs or downplay the other risks, such as security or privacy, but these are the people who are going to be very engaged, who’s going to be very eager to adopt new tools and try new things.

And then on the other hand, there’s a lot of people who are detractors of generative AI. They’re very distrustful or critical of this technology, and they’re going to be very focused on the negative sides, whether it’s the ethical concerns, the security concerns, or the environmental concerns of training these models. And they’re not going to be as receptive to org-wide mandates to adopt LLMs.

I find with these people, giving them the space to address their anxieties, being very transparent about your expectations of them, I think that’s going to work well in getting them to a place where they’re still skeptical but not as much of a detractor. And I think for most of the organization, they’re really going to fall somewhere in the middle.

There’s going to be people who are very curious but may not be completely sold on this new technology. And then there’s going to be people who are more skeptical, and I think for this group, they’re going to need some faster feedback loops to see more value. And again, that transparency from leadership to address any AI anxieties.

Srini Penchikala: Definitely. Also, can you talk from your experience, any specific metrics or indicators to bring all these different users together? Like you said, some team members may want to see some metrics before they can fully embrace the AI program side, so do you have any recommendations on that?

Mandy Gu: Specifically for these users? I think it’s going to be fairly specific. If you’re a developer, the metrics that you care about are going to be very different than if you’re someone who works in operations. I think if they’re able to see the metrics that are related to their work, so maybe for a developer how much faster they’re completing tickets or minimizing the number of touch points to work with other teams, for instance, I think those are some of the metrics that they would like to see.

And then these metrics will likely be quite different for other teams as well. I actually found that… Well, one, it’s actually really hard to put together these metrics and to really show the value this way, but I’ve found with a lot of people who are a little bit more skeptical that, and this doesn’t always work, but it’s really making sure that they try the technology at least once and giving it a chance. Taking something that they do day to day and then applying it with some of these new tools, I find that’s usually a good way of showing people, like, “Hey, this actually works”, or “This may actually help with my day-to-day”.

AI Education and Training [17:35]

Srini Penchikala: It definitely will be different for each different type of user. Also, can you talk about any education or training programs you may have developed to improve the skill sets of developers who are interested in using AI right away or even to make the others aware of the value of AI solutions compared to the traditional software development solutions? Do you have any recommendations on what kind of education and training programs the other companies should consider?

Mandy Gu: Yes, so there’s a few different programs that we’re running right now just across the organization, and some of them have definitely been very successful. So we’ve been having, within different pockets of the organization, just weekly demos, and they can be something really small. This is a workflow that I’m now using an LLM to do, or maybe I’m trying a new tool, or maybe I’m trying just a new way of doing things.

And that has been a really good way of getting people to see the value of some of these new technologies. And it’s been a great source of inspiration as well, seeing where your teammates are getting value with and on the other hand, where they may be struggling. So one of the rituals is having these regular demos in place. We’ve also been able to leverage asynchronous communication via Slack and other forums quite effectively.

So again, very much focused on just sharing some of the work that’s come out, but not just the good things. Even just sharing the AI fails and the WTF moments with these new tools. I think that brings a certain human side to it. We’ve also had a lot of our leadership team, including our CTO, kind of just come in and do “ask me anything” and provide that open space for people to ask questions. And that I find has just been a really effective way in addressing a lot of the anxieties around AI because for people who are distrustful of this technology, without the necessary context, they will think the worst-case scenario, but once they actually hear about the vision, the strategy, it’s a lot easier to address those anxieties.

Lessons Learned [19:31]

Srini Penchikala: You mentioned about WTF moments, right? So can you talk about some of the lessons learned or any other insights you can share on some of the roadblocks the companies typically run into when they’re trying to embrace machine learning and AI solutions and when they’re trying to establish the programs? What kind of lessons learned you can share?

Mandy Gu: I think one thing that comes to mind is just given how fast this space moves, it’s going to be really hard to keep up with the latest and greatest technologies. And this was something we struggled with for a long time because we would be building something fairly cool or bringing a technology internally, and then two weeks later, OpenAI would release a new model or a new way of doing things, and then at that point we would ask ourselves, “Do we continue what we’re doing, or do we just pivot to the next new thing?” Oftentimes it was hard to fight back the urge of pivoting, and this created a lot of work in progress, and it created a lot of times where we were putting in a lot of effort, but we weren’t really seeing the results or delivering the value that we had hoped.

So I think the insight that shows up here is bet on the racetrack, not the horse. And in a lot of the cases, it’s better to get started with something than to spend the next three to six months trying to chase the latest thing. So I think that’s one thing that shows up. The other one is that whether or not you like it, AI is happening; people internally are going to be adopting it. The people we work with externally are also going to be adopting it. And in some organizations, in a lot of cases, this is going to be on your client’s minds as well. So every single organization, regardless of which stage they are in their AI journey, they need to think about how to deal with the inevitable AI. You can call it a revolution if you will. But this thing that’s happening, and this is kind of showing up in a lot of smaller ways too.

For instance, there’s a lot more softwares right now for job seekers. There’s a lot of AI assistants that help them with interviews or that helps them do coding assistance, and this is something that your hiring team will need to tackle to get on top of this. But also making sure that we’re still evaluating our candidates the way that we intend to as these new tools are surfacing. So I think those are two of the lessons that kind of show up. And then maybe just the third that’s kind of relevant to the intersection of development and AI is that as we’re getting really excited about all of the advancements in this space, a lot of what we need to do to be a successful R&D organization will still be relevant.

So, for instance, fostering a culture of writing clean code, that’s going to be something that helps both the traditional way of doing things, but also if you want to apply AI, like code assistance or other tools, to your code base one day, that’s also going to be what you need from a foundational point of view.

Srini Penchikala: Yes, that’s kind of the main thing, right? So a lot of the tasks that are more boilerplate code-kind of thing. So I mean, as developers, we should never have been working on those anyway. So now we can use the AI tools and agents to help us with automating those tasks so we can focus on the creative side of the software development, the design, the technology additions, and the customer interaction. So more of those things.

Thanks, Mandy. So there were a lot of interesting topics we talked about. Thank you very much for joining this podcast. It’s been great to discuss machine learning and gen AI, especially from the perspective of adoption of these technologies in real-world projects and what has worked for you and what didn’t work.

To our listeners, thank you for listening to this podcast. If you would like to learn more about AI, ML topics, check out the AI, ML, and data engineering community page on the infoq.com website. I encourage you to listen to the recent podcasts, especially the trend reports that we publish on different topics like architecture, AI/ML, culture and methods, and also the various articles we publish on all of these different topics. Thanks, everybody. Thank you, Mandy.

Mandy Gu: Thank you.

Mentioned:

• AWS Bedrock
• LightLLM
• LLM Gateway
• Whisper
• Model Context Protocol (MCP) servers