Your Users Don't Care If It's AI - They Just Want Results

Hi folks, welcome to another Engineering Evolved. My name is Tom, the host. Uh, I know it's been a while since we've done the last one, so apologies for that. Uh, we are back, uh, b- bigger and better than before. Uh, today we're gonna talk about the fact that your users really don't care if it's AI, AI or not.

They're more interested in getting stuff done. Uh, so without further ado, uh, as we dig into this, like, concept of, like, the concern about AI or whether something's vibe coded or is it like, you know, nicely curated by a human, we're gonna talk about various different points. But, like, AI, machine learning, all that type of stuff, of course, has been around for many decades. Um, if we dig into some of the more obvious stuff, the fact that you have a junk filter and the fact that y- you know, various bits of mail you never get to see because the junk fil- the junk filter exists is exactly that. This is how machine learning aids us on a daily basis.

Now, things like spam filtering, fraud scoring, map rerouting, all that type of stuff, it is all machine learning. It is all AI, if you wanna use a, you know, broader term for it. I've been around for quite a while now, and you know, back in the day, we called it machine learning, which lived on in, you know, in the data science and data modeling ecosystem. These days, AI is the term, and that largely also encompasses LLMs. But I bet, without putting much thought into it, you could name a whole bunch of different places where ML, maybe not obviously, but gets used under the hood to, uh, offer up a more cohesive experience to the user, learning from data that's gone before so that you can have a better experience today.

We're gonna dig into this. Welcome to Engineering Evolved. Let's get going. So of course, when we're talking about AI in the current ecosphere, what we're talking about, for all intents and purposes these days, is powered in some way or driven by or curated by an LLM. AI-powered means not much more than that to the average user.

Now, of course, in reality, there's lots of different bits of data science, and a lot of it hasn't changed much in the past 20 years. Like, an awful lot of, like, modeling, linear regression stuff, all those types of things, they've been around since the dawn of time. They're mathematical models at the end of the day. But of course, these days, with the additional compute power that all the, you know... Well, not even larger companies.

All the medium to large [chuckles] size companies have access to, everything can be AI-powered and AI-driven because compute is cheap, relatively speaking. But it does mean that when you see everything is being powered by AI, it's a bit of a misnomer in terms of how people are trying to sell their software. Like, everything these days becomes AI-powered, AI-driven, because teams are selling the architecture instead of the outcome. So, like, if you're, if you're trying to get onto the hype cycle, then naming your thing AI-powered, you know, will make a large difference, whereas leveraging AI to drive a specific part of your platform or even to drive the development of your platform is vastly different, you know, as a, as a consumer to the person doing the building. And I think a lot of companies are losing sight of the fact that, to a degree, no one really cares if your thing is AI-powered or not, as long as they get the answer that they want out of the far end.

And if they don't get the answer that they want, it doesn't matter if it's AI-powered or not. They will go a different direction. And that's, you know, that's really the crux of everything. It's, like, on the market today. Like, companies feel concerned that they are not, uh, jumping on the hype train.

They wanna get their AI-driven, AI-powered stuff out there. And I'm not for a second saying that they shouldn't leverage, you know, some elements of LLMs or machine learning, you know, statistical modeling processes to get, um, additional functionality and support into their platform. But not everything needs to have a big, fat AI-powered label stuck on it, and that, I think, is my take at the moment. Let's hear from our first sponsor. Why hire when you can partner?

Concept Cloud's leading engineers build your startup's prototype without the overhead. Launch faster. Conceptcloud. com. Okay, back at it.

So of course, from an engineering perspective, where do LLMs come in useful, and how can you leverage them properly? Now, LLMs, of course, are guessing machines. They guess the next word in a phrase of stuff, assuming that you're doing language stuff. It is a large language model, so they're gonna guess the next word, and they're gonna pick out common phrases and come up with something that feels like it's right. And of course, for things like repetitive things like coding, they are excellent at doing so.

Now- The real reason that we're talking about this is that your users grade you on whether it works the same way twice and not on the impressiveness of the model. So you might have a super expensive, uh, Opus model running somewhere, but if you can't get the same answer out of it twice for the same question you're sticking into it, then users are gonna lose trust in your platform. Now, of course, there are many different models, and different models cost a different amount of money depending on what you're trying to achieve. And so finding that, um, [lip smack] area where you can tune it up and pick out a, uh, a model that works well for the exact question you're going to ask it is important from a software development and engineering perspective. If you just chuck every question at an Opus or a Sonnet or a ChatGPT 4, or you know, there are so many different models [lip smack] that depending on what you're trying to do, you really need to figure out the correct model for that question.

And there's a trade-off between cost, speed, inference, you know. And when you stick them all together, what gives you that ability to be able to be confident in the answer that the LM- LLM is giving you so that you can then hand that back to your user and have your user trust the output of the product that you're selling them. Because they don't care whether or not it's an LLM, they care whether the output that you are feeding them is the correct one. So like, you know, as you're doing, um, your software development, uh, evaluations, fallback paths, latency budget, and then like things like graceful degradation when ChatGPT or Anthropic for the 600th time this month decide they're just going to drop offline, what happens to the product at that point? And then, you know, also from that, what happens, as we have seen last week, for example, if a frontier model that your pipeline is built on decides to get rug pulled and then suddenly you don't have access to it.

All of this needs to be built in. And so whether you're using a frontier model or not, you have to be able to deal with the ability for it to not be there, because otherwise your customers are not gonna be impressed if suddenly you're not respo- returning responses because the API is down, or if you're hosting it locally and it's fallen over. But to that point, the requirement for not running... Or not the requirement, but the desire to run a frontier model might land you in greater jeopardy than picking a model off the shelf, tuning it up, tweaking it, and then hosting it internally if it is core to the criticality... Sorry, if it is critical to the operation of your system.

If it-- If, if you can't deal without it, and then you're depending on a third-party service for an API, for an LLM, good luck, because you're gonna have to deal with outages and your customers getting rather pissed off. So, you know, this is the type of stuff where you have to be able to factor all this type of thing in when you're engineering your solution, and that is, you know, day one stuff. Like, if you cannot engineer yourself around the idiosyncrasies, the model upgrades, downgrades, rug pulls, different services, different providers, you're gonna have problems down the line when you've got customers on the system and no ability to be able to deal with, uh, outages or that type of stuff. [microphone thuds] So I'm gonna leave that one there, but y- you know what I mean. Like, you've gotta think about all this stuff upfront, because I don't know if you've seen Anthropic's uptime, but it isn't great.

And if they also... Like, you know, if, if, um, [lip smack] Fable had been around a couple more weeks and people had built stuff on top of it, and then it got pulled, there'd be absolute uproar, and that's all the type of stuff that you have to be able to deal with. Of course, the cost of the label can also be an, an issue. AI can be a trust liability and, you know, people brace for hallucination and for being trapped away from a human, and you can't charge an AI tax because pricing follows outcomes and not transformers. So a few things to sort of unpack here.

AI, as we said at the start, can definitely cause a few interesting issues when people know about the fact that you're using AI inside of your platform. Now, some people will not care. Most people will not care. But you also have to brace for the fact that occasionally your application is going to hallucinate or potentially return information that is incorrect and have it displayed to a human. So what can you do about that?

What can you do to ensure that, A, you give the AI the most information for it to be able to return a valid answer? And what happens when it seems to be erroneous? So of course, you know, you can put guardrails in to try and track erroneous responses or things that don't conform to the output that you're expecting. You can also, with, um, hallucinations by feeding it more accurate data. Now, inside of organizations, there's a mad rush to be able to get everything AI ready, AI centric, you know, good to go for an AI perspective.

Now, that For, you know, is great, but at the same time, if you are feeding it junk data, then you're gonna get, you know, garbage in, garbage out to a degree. I mean, hopefully your LLM can work its way through it. But if you can guide it using APIs or data entry that give it a better chance of being able to find the data and focus in on the data that is important, it will be cheaper from a token spend perspective, and it will be more accurate for your end users. And so if you can deal with that, that will obviously get you further down the line to being able to ship a product that has AI inside of it. Now, when it comes to actually shipping, people do not care whether or not you've got AI in it.

Just because you've got an AI bot inside of your, like, application these days doesn't mean that people are suddenly gonna pay twice the price because you've now got a whole bunch of tokens that you need to be able to, like, you know, uh, price for when it comes to actually shipping your product. So you can't charge an AI tax on top of the application. But what you can do is use LLMs to give a more effective and efficient outcome that will allow for users to be happier and generally, you know, more comfortable using your platform. And that at the end of the day is what we're trying to drive with AI usage, is to either give them more insight or give them a way of being interact with-- being, being able to interact with your platform without having as much, um, oversight, overhead, and things they have to do. And so if you can turn that LLM integration into an outcome, they are gonna be more, uh, comfortable paying for whatever they have to pay for rather than you trying to force an AI, um, integration down their throats that, you know, looks like a tax.

Because you're suddenly gonna have to charge them more because people are gonna be burning tokens. [lip smack] So I hope that sort of makes sense. Like, you know, you've gotta be able to deal with these types of things from within, uh, a product ecosystem and figure out how you're going to integrate this stuff into a platform, and then also make it available to the users. Because all of that type of stuff, um, is not just as simple as like [snap] "Yes, I'm gonna integrate this platform, like integrate this LLM into my platform right now. " You need to be able to figure out how you're gonna monetize it, optimize it, and also tune it up for the operation that you're trying to give it, and that is not as simple as like, "Oh, I'm just gonna stitch Claude into my workflow and then magically it's gonna start working.

" So bear that in mind, and now a second message from our sponsors. [upbeat music] The same minds that worked on the last Mars Rover now work on yours. Concepta Cloud builds startup MVPs that perform. ConceptaCloud. com.

Okay. So now going back, just taking a step back and thinking about this sort of, you know, coherently. There is of course a line where users do care that AI is being used in a platform because data provenance, data usage, you know, is a human seeing the data, um, [lip smack] and pretending that AI isn't in your platform when it's making the consequenc- consequential decisions that a user, a consumer of your product, um, is then gonna act upon means that you have to be honest about the fact that when you're using AI inside of a platform, you have to tell 'em and you have to be, uh, upfront about where it's being used and how it's being used to be able to offer them, uh, ability to challenge the output that comes with your application. And pretending it's invisible when it's making consequential decisions is also a failure in itself I believe. You know, and so making sure that there is, um, a real, you know, a real definition about and upfront honesty about this is where we use AI in our application to be able to offer you more features, functionality and flexibility in what we do is something that you have to be able to do and have to be able to tell, um, you know, your customers about because that type of stuff is gonna be important to them down the line.

So hopefully this has been a sort of whirlwind episode. But like [laughs] hopefully this has been, um, you know, a bit of a thought-provoking episode. It's like how do we leverage LLMs inside of an environment today? And there are so many different ways of doing it. How do we add I- like but instead of asking how do we add AI, start asking what's now instant that wasn't?

Because of course the difference between a ML model and an LLM is worlds apart. Like you don't have to do all the data prep. You should do a lot of it, but you don't have to do as much of the data prep as you did before. You can ask questions that before would take absolutely ages to get right and have the answers in front of you, you know, within seconds. And so how can we leverage all the knowledge that these LLMs contain and have been taught, and then make the products that we are building more efficient, more effective and more available to our customers and to our end users so that they get- bang for their buck rather than an organization just adding a powered by AI tagline into something because they don't want to miss the train.

That is the question that everybody should ask it should be asking not anything else but like how do we add value how do we make this available to our customers in a way that they would benefit because everybody looks at chatbots but of course chatbots are the possibly one of the most inefficient ways of interacting with an LLM if you have to sit there and type everything to a chatbot you're going to waste years of your life just like asking a chatbot and then correcting it and so from a product development and product design perspective the question of course now is how can you implement an LLM and add value to your product without having to sit there and have a user type everything in and then read the response back and those who really excel in the AI space will figure out how to square that circle and you know get to a position where they are capable of integrating an LLM into a platform without having a user having to look at yet another chatbot take that one away with you have a think about it from a product development perspective an engineering perspective how would you integrate an LLM into what you're trying to do but without creating a chatbot to have to deal with it when you think when you figure that one out go build it I hope you found this useful thank you very much for tuning in and listening to the latest engineering evolve my name is Tom we will be back soon thank you very much again bye for now Thanks for listening to Engineering Evolved with Tom Barber, where ideas meet innovation and leadership drives change. If you enjoyed today's episode, please leave a rating and review wherever you listen. It helps more leaders discover the show and keeps the evolution moving forward. From idea to investor demo in weeks, not months. Concept to cloud.

World-class engineers accelerating startup success. Concept to cloud. com.