Speaker: I previously worked at LinkedIn, Pinterest, Nextdoor. That’s where I met Shane and, uh, and Travia. Uh, Travi or Jean, do you guys want to give a little bit of intro about you guys? Go for it, Saravia. Okay, I can go for it. I’m Shavya Madipali. Hello, everyone. Um, I’m Ajrali from India, South India. I speak Telugu. I see Attendee asking this question. That’s my native language. I’ve been here in the US for a long time. I have a… Uh, I did my master’s here, and I worked at Microsoft, I worked at, uh, Nextdoor, that’s where I work. I met Shane and Hi. And we kicked off our Data Enable podcast from there, and now I lead Growth Data Science at Grammarly, which is now called Superhuman.
So yeah, that’s about me. Hey everyone, I’m Shane. My principal data scientist at the same company. Hi, works at Entre, legal tech company. been in tech and data science for about a little over 10 years. And… yeah, teach a course on AI evals, that’s kind of my, like, primary focus area. Um, and then I also teach this course with high and strawberry around AI analytics for builders, so… Excited to meet you all here. Um, yeah, if you want to connect, I’m gonna drop a link to our… Slack, actually, in the chat as well, so we keep the conversation going after. the chat here, and if, as we kind of, like, go through, feel free to drop your, like, LinkedIn in the chat as well.
We’ll get in touch with you. Cool. Awesome. Um, yeah, so, um, we, as Shane mentioned, and we’ll get into a little bit later, uh, in the, in the presentation, um, we run a couple courses. One is on. helping, or I guess it’s called AI analytics for builders, and the whole idea is, uh, sort of like, if we zoom out from this lesson, we wanted to help people to become analytically independent, being able to ask really great questions, get really great insights with the help of AI, and delegate the execution to AI instead of. you having to run all the… Uh, all the queries in an analysis. And then there’s also an AI evils.
for product development that Shane runs, and then we also have really great, uh, free lightning lessons coming up. almost one a week for the next, uh… many, many weeks, so, uh, check them out on… dataneighbor.com. Cool. Alright, so let’s see. So, let’s get into… why we’re here today. And, uh, by the way, we’re probably gonna run this for 45 minutes, and then, uh, can stay over for questions. So, um, if you have any… Any questions you have, feel free to drop it in chat at any time, and Shane. Uh, feel free to interrupt if, uh, if there’s something that we should cover. Alright, cool. So, let’s get into why we’re all here today. So today, uh, you know, I think the premise is pretty simple.
So, the idea is, some people… can extract gold from their questions, while others. get garbage. So, probably, I’m hope… I’m guessing people here have seen. I’ve seen that. Some people ask really great questions. That always seems to be consequential. Some people always get into the running around the circle kind of… kind of questions. So, what exactly is the difference between these type of questions? And uh… Uh, and how do we actually get better at it? So, the idea is, uh, your question quality really determines your output quality. This is especially true in the age of AI, because. the more generic things that you ask of your LLMs or your chatbot. the less precise of an answer.
Or probably an answer that does not align with what you’re actually thinking about comes out. So we wanted to, uh, given our collective experience in the tech industry of working with. stakeholders to really ask questions, really ask thoughtful, sort of like, um… things. We wanted to just, uh, kind of, like, bring this. lesson together to make sure, um, you know, like, we understand what are the elements of those. Um, and, uh, you know, like, one thing that we can sort of. sort of assume is that given the same set of data, um. Uh, and we’re gonna run a little bit of a… kind of like a poll here. Uh, two different questions. Which one do you think.
is a better question that actually would lead to… better answers, better outcome. Drop in the chat. One or two. Come on, I’m expecting some ones, no ones? Oh, Attendee wants one. Attendee’s S1. Nice, cool. Uh, I think we have almost… consensus here. Um, so, uh, 2 is certainly better. It’s definitely longer, a lot more context. And in this lesson, we’re just going to go over, sort of, like, breaking it down into what makes it better. For, uh, you know, like, versus number one. But I’m glad that, uh, everyone… almost everyone, chooses picks number 2. Alright, cool.
So, uh, what you’re gonna get out of today, uh, by the end of this session, you will know how to frame questions that drive decisions, so we’re going to go through a simple 3-item question quality checklist. So, that’s a very simple framework to evaluate any analytical questions you may come across. Number two, we’ll show you how to use AI as a coaching partner, so moving beyond just. answering your questions, but using it to sort of, like, give you feedback to, uh, help you refine questions. And then number 3 is, uh, you’ll take away the prompt that we actually would use here afterwards.
And, uh, that’ll be in your follow-up email, so that you can, uh, whenever you want to ask, whenever you… You want to refine a question, feel free to sort of, like, play around with it. Alright, cool. So, question quality checklist. This is the most important sort of framework that we, uh. Uh, that we should all take away. So, uh, it’s very simple. It’s three steps. So, almost like three elements that really make a question. That is analytically sound, and that’s answering it would actually give you something, give you a direction, or give you an answer that you can actually act on immediately. And those components are, uh, number one, decision tied. So, when a question is asked.
Does this inform a decision? Like, what decision is it? So, almost like, think of it up front. An anti-pattern to what this is, and we’ll go through some examples and also evolutions of how to think about, you know, like, from something that is not to something that is, uh, in all three elements of this. Um, you can imagine. something like, hey, it would be interesting to know, and then, you know, like, some question, uh, is probably not a decision tied, uh. doesn’t have the decision-tied component of it, because, you know, like, what are we gonna do about it? Or what are we going to do with that information? Um, so that’s what, uh, the number one check is. The number 2 is data-grounded.
So, can I actually answer this with data? So, sometimes there could be a lot of questions where, hey, it’s just impossible to find the data, or the data is just. Uh, like, in the, like, uh, does not exist. And so, even if the question is well-formed. it’s impossible to answer. So, you know, I think… I hope this is, uh, this is also obvious. And then number 3 is, uh, is the question. specific enough, um, meaning, like. Uh, you know, like, it’s bound it somehow, so it’s not like a boil the ocean, look for everything under the sun kind of thing, but it’s very narrowly targeted at something that would. be sort of like, hey, if I can get to this answer, I’m good to go.
And so, we’ll break down what that looks like, um, in the next few slides with examples. All right, cool. Um… I don’t know how to animate, so that’s why, uh… I would have done this with, uh, with sort of, like, animations of, uh, 1, 2, 3 in sequence, but I don’t know how to do that, so the answer is already. here, but let’s walk… let’s… let’s walk through it. Uh, so example progression of a question, um, from vague to sharp. In this example is… The very first version of this is why are users churning? Um, and, uh, what’s… what’s the, uh, I guess, what’s the, you know, is that a good question? Is that not… not a good question?
I guess with chat, it’s a little bit… it’s a little bit hard, but it’s not a good question, um, because, number one. The decision is unclear, so what are we going to do with this? Like, is this just interesting? Did an executive. ask me why users are turning, or what am I trying to do here? So, the decision is unclear. Uh, Attendee. Great… great response, too vague. So, like, you know, like, which users? What time frame? There’s a lot of stuff that’s missing here. Nothing is specific. Nothing is grounded. Uh, so, if we progress down to number 2. refined us a little bit, we can ask. what’s our churn rate by cohort? That’s a valid question. Right?
The… The question itself is getting a little better. The… the, sort of, like, the elements are still missing, some of them. Um, so the data does exist, meaning, like, the measurement is much clearer, like, we’re looking for churn rates, uh, by cohort. But what do we do with this information? Like, is this a good-to-know? Are we trying to figure out what to invest in, or are we just trying to be, like, uh, you know, like, uh, I don’t know, like, insights of the month and good to, you know, like, something to… something to impress others. So, um, you know, like, the… sort of, like, the decision is, uh, is… is what’s really missing here.
And then, number 3, like, another refinement, another attempt at refining it could be which user behaviors in the 30 days, in the first 30 days correlate with churn. So we can trigger interventions before they leave. Now, it’s a very long question. But it does hit all the components where the decision is clear, like, we’re actually trying to intervene. like, something triggers us to think that churn is something that we should be investigating, and that we should. want to actually take action to intervene, but we need a direction for how to intervene. And so, uh, like, decision is clear, the data we have. again, like, how to measure churn for first 30 days, that kind of stuff.
And then it’s very specific. It’s like, hey, this cohort is what I’m looking for, and first 30 days is what I’m trying to correlate that against. Do you guys see, kind of, like, the evolution here? From super vague to something much more refined? And you can imagine, like… Answer number 3… is a lot less work, but also a lot more targeted than number one, right? Cool. Alright, so… I like the comment in the chat, too, around, like, what does churn even mean? Like, churn… I mean, I’ve had… I’ve been in… in… conversations where there’s two execs talking about churn, and one’s like. Did the customer leave? And then everyone’s like.
Is the customer not use the product, and they’ll pass the quarter, or are they using the product and not using paid features? Like, churn also in itself. is nondescript. That’s a… that’s a really good point. So, you know, if your organization, or if you. Uh, already have, sort of, like, something well-defined, then that’s certainly much. better, and you guys are in a much better place than. If the company doesn’t even know what churn even means, or hasn’t agreed to it. So, yeah, it’s a great… that’s a great point. Uh, the previous example assumes churn is universally aligned in your company, but if not, we have a lesson next week to actually chat about how to. Uh, how to make that happen.
Alright, cool. So, let’s, uh, let’s move on. All right, cool. Example 2. Progression 2. Uh, from left to right. So… first iteration of a question. How are users behaving? Is that a good question? Did that give away? All right, I see some… oh, well, no, I see… not, sorry, I see, uh, uh, uh, head shaking, so that’s good. Um, cool. So, yeah, way too big, right? Not a great question, terrible, um… What does behaving even mean? How do you even measure it? what time frame, what are we trying to do, that kind of stuff, right? Uh, number two. this is probably better, right? What features do power users use most?
like, at least the behavior part is a little bit more guided now, like, uh, what features they’re already using. So, uh, that’s… You know, like, that’s better than how are they behaving, behaving in what way. So, this is at least. you know, within the confine of a product, what are actually… what are people actually doing there? Um, uh, and specifically for power users. Again, the decision here is missing. Like, what are we doing with this information? So that, uh, you know, like, there’s many flavors of answering this question, or crafting an answer to it.
But if the decision is unclear, the answer is also extremely open-ended, and also the process of executing that for that answer is going to be not a fun… not a fun thing to do. Alright, the final iteration of this guy, which three features correlate with 30-day retention above 50% for users acquired in the last quarter? And should we surface them in onboarding? you guys see, like, I mean, again, like, more words? much more of a verbose. I’m not… I’m not advocating for more words or more verbals, but I’m advocating for more targeted way of framing your question.
And it doesn’t have to be spelled out specifically, but you should always keep in mind that these are the elements, even if I don’t say it out loud. that’s where I’m going with it, to get the best, sort of, like, outputs, either from. Uh, a data team, if you work with them, or, uh, from AI, like… Just like everybody has access to. So, you know, like, I hope this is clear. Number 3 is way better. Three features, very much, uh, like, decision… the decision is laid out here, what you’re gonna do with it. The data is. Again, like, we can… we can debate what… how retention is defined, assuming it’s already defined.
Then, you know, like, the data hopefully is already there, and uh… and very specific, and, you know, also with a threshold that you… that we know defines what good looks like. So, um, you know… The, uh, again, like. The more that we can get closer to number 3 or to the right. the better, I can guarantee you, your answers is going to be, and the less back and forth you’re gonna have to. work with, whether it be AI or humans to get that answer. Let us move on. Alright, example 3. I put this in last minute, so, uh, let’s see, let’s see how this goes. Cool. So, assume we have a marketing performance question, um, around, uh, return on investment, so… Love to write again.
How’s our marketing performing? If you’ve worked with marketing teams, you’ve probably got something very similar in the past. I see Attendee nodding vehemently, so I’m guessing you work with marketing quite a bit. Yeah, so, um, very, you know, like, very classic question that’s like, hey, how’s our X performing? That’s, that’s sort of, like, uh, a lot of the… the questions that either the high ups or yourself, or your stakeholders. would ask, because that’s the easiest to answer. You don’t… it doesn’t require any brain power, doesn’t require any thinking, you just. blurb it out and, like, let the data guide you, right?
Like, that’s the… that’s… that’s sort of, like, the thing that most people like to do. Um, but again, if we run it through the checklist, uh, or the framework, decision is unclear, performance against what, what would you change if you know this information? Like, always be asking yourself that. Alright, moving on to the middle, which… This is another refined question from the very beginning. So, which channels have the best ROI? Now, this is getting better. getting much better, actually. But again, decision, unclear. That’s compared to what threshold, and what do you do with this list?
Um, again, the thing that’s missing in a lot of questions that we’ve come across in our experience has always been. What decision are you trying to drive? And so. The more that you can think ahead of time why you’re asking this, and what are you going to do with the information, the better the outcome. For the output you’re gonna get, um, to… to actually get at the real question behind the question. Uh, that you’re actually looking to… looking to ask. All right, and the third… refined a final state is which channels exceeded our 1.5x return on. ad spend target last quarter, and should we shift budget from underperformers to scale them?
So, return on ad spend is just, uh, uh, every dollar that I put in in advertisement, how many dollars do I get back? So, like, I get back. $1.50 every dollar that I put in. So, uh, this… Question gets to… understanding, um… understanding, sort of, like, uh, hey, should we, like, should we be allocating, rebalancing our budgets for… For marketing purposes. Do you guys see how this evolution is… makes things much more clear, and doing this exercise? around you into, sort of, like. a decision-forcing answer at the end of the day. Like, you cannot not get a… Answer that doesn’t drive decisions. If you think of the decision up front. Cool. Alright, um, let’s… get to a common pushback.
Uh… I’m guessing some of you might have encountered this. I certainly have many, many times. In fact, I encountered it last week, so, uh… Uh, what about if I’m just exploring, or what about just letting the data. guide us, or I’m just looking at the data. Nothing, like, nothing truly… I’m not deciding on anything. But I’m just looking, I’m just curious what, you know, like, that seems like a pretty common thing, right? Like, like, why can’t I just. be exploring. Um… The rebuttal to that, why that is ineffective, is that even exploration needs a direction. Very… in very few circumstances. Would you be like, I’m okay with.
the entire map canvas, being the sort of, like, all possibilities of exploration. Like, that’s just not effective, right? Like, if you’re… If you’re setting sail to some destination, and you’re like, I don’t know where it is, I have no hypothesis, I’m just gonna go. You know, like, whichever way. That’s probably not, not, not, not, uh, not, not the way that you would, uh, spend your resources and stuff. So, reframing. the word exploration as a decision is actually. itself, getting you a lot closer to an effective, sort of, like, uh, effective question in itself.
So, some examples that I listed here… Uh, instead of saying, I’m just exploring user behavior, or I’m exploring X, uh, instead, you can just reframe that into what patterns would change our next sprint priorities. Like, you’re trying to look for something that does something for you, so… say that something out loud, or that latter something. Correct, I’m already mixing them onto some things, but, like, uh, you get what I’m saying. Uh, so, uh, you see how, like, one is extremely vague, the other one is. forcing some guardrails on, or I guess at least forcing some direction to the very vague thing, and answering that itself. forces you into… Uh, looking at the right things.
The second one is, uh, I’m just looking at the data. Um, so, you know, common pushback, again. Uh, another way to reframe it is, uh, what would I need to see to recommend we invest in X? Or what do I need to see? To, uh, to do… to do Y. And like, you know, like, that, that sort of framing. gets you to be sort of, like, refining your questions to be like, hey, I’m actually interested in this direction, or I’m actually looking at… wanted to see this and that. Um, and then the final one that’s listed out here, I don’t have a decision yet, so I’m just looking. Uh, the… the rebuttal to that is, what’s worth investigating further?
is your decision, and do not overlook that, like… Uh, if you have some hunch of, like, I don’t have a decision yet, but I’m interested in this area. interested in that area itself is a decision, because you have some sort of. Knowledge, subconsciously, consciously, that. There might be a smoking gun in this area that I really wanted to dive in. So, list that out, and that becomes sort of your decision. So, undirected exploration. It was like a fishing expedition. You’re just trying to… you’re just trying to see where the fishes are. Uh, but directed exploration is discovery with purpose. So that’s the way to really think about it. So next time, if you’re like.
hey, I’m just exploring, reframe it. If you hear your stakeholders or your people that you work with. telling you to, you know, like, hey, go dig up this data, I’m really curious. Uh, force them to, like, hey, let’s rework that a little bit up front so that we can get a lot better answers down the line. Cool. Alright. So, let’s see… Oh, okay, cool. Uh, yes, um… Uh, we brought up this, uh, or previously, uh, one of you brought up this point, that, uh, how do you even define a metric? Like, what, what does churn mean? uh, you know, like, how do we get to a place where we can actually ask some of these questions, or if the actual measurement is not clear, then how do we even ask the questions?
We, you know, next week around this time, or I guess next week, Wednesday, exactly a week from now. Um, uh, I’m also running another lightning lesson around, uh, the title for it is Design Metrics That Don’t Lie. Uh, it’s certainly a click-baity one, but the idea is. what it says. It’s, uh, how do you actually come up with metrics that drive decisions? So, not the qual… not the question, because we covered today. But the actual measurements, how do we actually come up with non-vanity metrics? That actually gets you to measuring the stuff that really matters.
We see this struggle a lot in many, many companies, and it’s not a straightforward one, and so we’ll provide very similarly, like, some framework and some, uh. some not-quite checklist, but, like, uh, like, ways of thinking about it. So that, uh, so that we can all… so that you can all, like, take away and go through… and go through them, uh, uh, in, you know, like, as you, as you embark on any exercises like that. So, you can… Feel free to scan this QR code. That gets to the registration page. or go to bit.ly slash metricsdesign, sign up there, and you’ll get the recording, even if you can’t attend. But we’ll follow up with links to these as well.
Oh, and Shane… put the links… QR doesn’t work, sorry, Attendee. my QR code’s still… Apparently, it’s not great. But we’ll… we’ll follow up. Cool. Alright, uh, let’s see, I think we can skip this practice moment. Yeah, originally I was gonna have. everyone do a practice, but let me… let me skip this. Okay, so, uh, key insights, humans or AI will execute whatever questions you give it. The checklist that we just went through ensures that you’re asking the right questions. So, uh… You’ve probably heard of some variations of this quality input leads to quality output. In other words, garbage in, garbage out, so… Uh, spend time up front to make sure whatever the input is, is not garbage.
then you can… you can basically ensure that, uh, whatever output it is, is also not garbage. Alright, cool. So, um… Let’s see, so… yeah, so… we’ll get into this segment of, uh, how to actually use AI as your coaching partner, uh, to sort of, like, go through the things that we, uh, that we went through. Um, so I’ll do, like, uh, I’ll do, like, a quick demo here. Um, but assume the question is, like, uh, either you asked. Or you’ve gotten this question, or you’ve heard of someone. brought up a very similar question. Hey, can you look into our support tickets? They’ve spiked recently, and the support team is overwhelmed. Not sure what’s driving it.
So, well, I guess, uh… drop in chat, yes or no? Is this a good question? feel free to say yes, it’s cool. I think it could be a good question. Attendee says no. If DT says yes. Shane says, I hate support tickets. Okay. No, Attendee, cool. Okay, well, we’ll ask AI if this is a good question or not. Uh, so, let’s see… All right, so I’m going to switch window. So, this is a prompt that you all are gonna get. Uh, and I’m just gonna go through… go through it live with y’all. So this is ChatGPT, hopefully everyone has, uh, has used it. Uh, in the past, and know what it is. So, let me see. You guys can see my screen, right? This ChatGPT canvas? Alright, so I’m just gonna copy and paste. My question in?
And don’t worry, you don’t need to read the whole thing here. And I’m gonna give it a question. All right, so this is a question quality quote, blah blah blah, what question do I want to. give it, so I am copy and pasting the exact same thing here. Uh, far right. Cool. So, I pasted in the exact question that, uh, that was on the slide, and uh… Uh, and then this is what it’s telling me, so… Uh, thanks for a raw question, this is exactly a kind of starting point that benefits from sharpening. So basically, this is not a good question, so let’s work on it. Uh, in a variable-like way.
So… Um, uh, you will see that, uh, it’s gonna walk through, kind of, like, one by one, um, on… sort of, like, the things that we talked about in the checklist. So, what decision would you make differently? What would you do? if you know this information, and I saw Attendee says, we want to focus on fixing them. Um, uh, fix… big stuff, let’s call it. well, I guess fixed stuff is not great either. So, okay, so bug fixes. bug fixes for high priority stuff. Right, getting sharper, so I’m getting the validation from AI, uh, and then the next one is, uh, data-grounded. Actually, what data do I have to be able to look at that? So I’m going to say ticket volume. By day, right?
And then, so now I get two checklists, um, and is it specific enough? So how would you know when you’re fully answered this? And it’s giving me some examples, so I can say. Um, you know, like, normally. Uh, I know anything above, uh… 10% week. over a week spike means. Uh, news, so let’s fuse that as a threshold. So I do… what I did there? Cool. So now I passed all the checklists, and uh… and things like that, and it helps me refine them into, uh, into… uh, like a, like a… like, a question that we saw previously. So it gave me some… some examples of it, and uh… Uh, and then these are, uh, these are certainly much better ones that, uh, that we can ask.
So, this is almost like a… kind of like a… like an async coach, that you ask AI to, uh, to, to help. Uh, it could either be a… question that you already have, or it could be… I have this vague idea, I wanted to formulate it into a good question, kind of, uh, kind of use. So, um, either way works. And so, we’re gonna switch back. to here, so this is… this was the prompt, and so I’ll be giving everybody this guy. Uh, okay, cool. So, uh, this is what we did. We did a demo walkthrough of a conversation with an AI coach, so… exactly what we… what we saw. If you found an answer, what would you do? What data do you have access to, and how will you know when you’re done?
So, all the three elements that we talked about. Cool, and then, uh, the refined questions, these were some of the… some of the candidates, not quite exactly, um… Uh, from the outputs of, of, uh, of… of ChatGPT, but, like, the flavor is the same. So, for example, like, which ticket. Categories drove the biggest increase in volume past two weeks. Should we prioritize a block fix? blah blah, or certain user segments driving disproportionate ticket volume, and should we prioritize fixes for those segments? Both of these are really, really great questions, extremely targeted, and we know. exactly what we’re gonna do when we get the answer to it.
So that’s what… that’s where we want to get you all to… Alright, cool. Um… So, uh, I could have very well given everybody here, just like a, did this question pass this, or, you know, like, just give the answer directly of, uh, you know, like, uh, every time when you feed it something, it’ll give you the answer. Uh, but, you know, like, our philosophy is that coaching is better than grading, so… Uh, if you notice what just happened. we… you didn’t just get a, yes, no, this is good, this is bad. But you had a conversation with that AI thing, uh, through kind of, like, this prompt, and it helped you discover the gaps in your thinking.
Uh, and by the end, you understand why a question was weak, not just that it is weak. Because, uh, let’s be honest, it’s, uh… Uh, you know, like, especially in the world of AI, where getting an answer is so easy. the process of… the fundamentals, like acquiring the fundamentals, knowing how to actually do it yourself first before delegating. is extremely important, and we really want to help everybody to sort of, like, develop that muscle. And so, the checklist is a skill. The prompt helps you internalize it faster. That’s… that’s… that’s all the philosophy that we have with, uh, with this. Alright, cool. Um, wow, that was a very quick, almost 40 minutes.
Um, check your email after this session around a PDF guide on question quality. It will have the coaching prompt that we just went through. And then we’ll also send over this deck for your reference, or if you missed any of the slides, or wanna refer them in the future, uh, definitely, you know, feel free to leverage them, and you’ll get them in your inbox today. Um, cool. And, uh, we also have a free AI Builders community. This is a Slack channel, uh, that, uh, we recently. uh, stood up to, uh, to just kind of gather people who are interested in the whole, uh, you know, like, AI building community. Um, you know, regardless of what your, uh, sort of, like, what your focus is.
So we have, uh, members, uh, uh, this is all three. We have folks from all over the world, um, who are. PMs who are, um… engineers, uh, who’s trying to build with AI, who is trying to learn more about AI. Uh, and so, definitely join us. We’ll share a lot of free resources and, uh. Um, we would love to have you join us. So, the link to that is bit.ly slash AIconnect. Cool, and uh… as I mentioned at the very beginning, um, we also run this course on Maven that is called AI Analytics for Builders. The idea that we’ve all… that Shane, Travier, and myself feel extremely passionate about.
Is to help people become analytically independent, meaning you ask really great questions, you have a very analytical mindset, we teach you all the framework that we’ve learned over the years in our own profession. in order to be able to, sort of, like, uh, kind of, like, um, you know, ask great questions, make really informed decisions with data. But you can delegate the execution of getting those answers, of getting to, uh, you know, like, quality results. through the use of AI, instead of, you know, like, learning all the technical skills and stuff like that. So… Um, the thing that we sort of, like, uh, put an analogy on is, uh, in today’s world.
There is… you’ve probably heard of vibe coding, right? Like, everyone can… anyone can, uh, vibe code stuff. even if you’re not a software engineer, to build applications or write software code. You can use AI to help you become a product manager, um, through, you know, like. doing, you know, like the product requirement, uh, documents and things like that. Or you can use AI to help you be a designer, even if you have no idea how to design stuff. Uh, what we’re seeing missing in the market is really around the data piece. Like, how do you become analytically sufficient, self-sufficient, such that you don’t need to rely on.
a data analyst or a data scientist, or if you’re a data professional, you can. be better at being more effective in how you do, sort of, like, data work. and offload a lot of that to AI, so you can focus on the most important things. So we run a 5-week course that starts in April. Um, it’s gonna have comprehensive training around all the things, uh. Uh, all the things that, uh, that we learn around analytic workflows and mindsets, and how to think about it, and, uh, and then we also teach how to. how to do those execution with AI. Uh, it will have a capstone project by the end of the… by the end of the course. Uh, we’re running a limited time discount, um.
to everyone who’s, uh, who’s, uh, who’s joining this lightning lesson, 25% off. valid through February 14th. promo code AIBuilderLL at checkout. If you scan the, um, the… the thing here, and I forgot the… Forgot to put in the link, but we’ll… we’ll have all that information. Uh, uh, in the follow-up email.
One thing I’d like to add is we collectively… Hi, um, Shane and I have a 40 years of experience, like, if you look through multiple companies, this is our first attempt to try to get all the knowledge that we gather over the years into a single course, So, uh, we are basically trying to get everything to you guys in a way of like, with AI now, like, uh, try to use our experience and how do we get… to use AI and get to these. So… Yeah. Just wanted to reiterate on that. Yeah, um, uh, AI is not quite replacing us quite yet, and we know where and how, uh, where it’s going, uh, through the podcast that we do on the side as well, so, um, certainly we’ll share. Uh, a lot of those to you all.
Um, so, final thought, and we’ll get into Q&A, is, uh, the question isn’t whether you use AI for analysis or AI for anything, really. Everyone is going to. The question is, are you going to be the question… are you going to be the person asking the right questions, or the person generating garbage? Don’t be the latter, because it’s very easy to fall into that. Uh, and uh… uh, you know, like, just as a mantra in the AI world, we wanted to emphasize this. Uh, cool, okay, that’s the… that’s the end of my repaired, uh… programming, so I’m gonna go back to here and just leave it up, and uh, if you have any. If you have any questions, the floor is yours, and I’ll stay on for however long.
Folks want to hang out? Yeah, I think Shane and I can also answer any questions. So, one thing I’ll probably like to add is that Through all this experience in data and data science, What I’ve observed is… Critical thinking and curiosity, and going in the right direction is literally what like, leads to impact. It saves you a ton of time. Imagine… I can… I can see myself, like, you know, 10 years ago when I started off, I went in directions that were took me a long time, but lead to me in places that didn’t lead to any impact.
But today, if the same question comes to me, How do I handle it is like… so much, so different and so much better, and that is purely the experience teaching me which angle do I pursue? How do I get my curiosity to, you know, uh, tackle me, and what questions to ask? The question that you get, and things like that. So, that’s a reason, I think, today’s session by Hyde really resonates. Um, and it’s something that we literally use it in our day-to-day jobs as well. And we’re running this as a first cohort, so we’re new to, sort of, like, uh, teaching in this way, so you can, if you decide to join us. you’re going to be helping us shape where this goes.
Has anyone spent a bunch of time on an analysis or something? Just because, like. I don’t know. A question was phrased in a way. that they interpret it. I’ve definitely, I feel like early in my career, I’ve spent, like, probably months sometimes on. analysis, or building a dashboard, or something, or some readout, because… VP of product had some question that came to me in a Slack or in the hallway, and then… It was, like, months later, like, I actually didn’t really care about that. I was, like, telling everyone else, like, oh, I’m doing this thing for the… VP of so-and-so, um… I wonder if anyone else has experienced that.
I just find, like, asking good questions is also just extremely important for your career. Like, it could be, like, a career maker if you can… Kai, if you can get your workflow and your mindset. Where, like, as much of your work. turns into something happening at the company, a decision, an action, something changing because you were there. Like, because you’re basically the key. a lot of that just comes from… making sure the question is framed from the very beginning. Attendee’s asking, can you show me examples of weak versus strong AI questions for my use case and help me write mine?
Uh, yeah, I mean, we’ll send you the prompt, and uh, certainly… You can, uh, you can iterate with it, and it will tell you why, um, you know, some questions are better than others. Yeah, we can talk… we can talk about that in that Slack community, too, Attendee. Um, if we want to go… a little deeper. I know we don’t have, like, a whole bunch of time here, but definitely down to talk through your use case. I mean, to add, Attendee, basically, the more context you gather from the people who’s asking the question, The more context you give to AI, the better the direction of, you know, you solving it comes from. And, uh, exactly the framework that Hai shared as well. is going to, like, literally help you.
in, like, getting there faster. I think to your point as well. I’ve been experiencing. Patient possible feedback is. It’s important, but it’s difficult sometimes, in that I’ve definitely had an example where. was asked by a higher-up. to do some analysis that they thought was important, that was important to them, they wanted this dashboard, this kind of metric. And then we spent some time. taking it from Excel prototype to the Tableau dashboard, but um… Actually, with the frontline leadership. Below, executive. Um, it was really meaningful. didn’t have the right liberation context included to really make sense of some of the higher-level summaries, and so.
You know, could have saved a lot of time. getting that beforehand, before, you know, the effort to put up a high-level summary that wasn’t actually useful. Yeah, that’s a great… that’s a great point. One secret trick that’s, uh, I guess not so secret, because we talked about it quite a bit, is, uh. Uh, you know, like, if you’re the one that asks questions. Uh, always ask yourself what decision I’m trying to make with this information, and if you’re the one that’s being passed a question to go solve. ask the person that gave you that question what decision are you trying to drive with this question? I can tell you, at least in my experience, probably 60% of the time, they would be like.
Actually, that’s not a really good question, so let me just, uh… you know, it’s cool, you don’t need to work on it. And so, you know, like, just asking one follow-up question could, uh, potentially save 60% of the time, so… Uh, that’s a very high ROI. thing to do. if this is a predictor model that we are trying to look for the time saved, I think the first feature that you’d get is, did you ask the second question? Or did it just take up and just go work on it, you know? Uh, I would say, as a new manager, when I was starting out as a leader, the first thing is, like, oh my god, this… CPO has this thing, let’s just, you know, get my team to work on it, right?
Versus… Uh, irrespective of who the person is, hey, uh, why are you asking this? Because you could get it from here already, and would just… you know, trying to be curious. And then they were like, oh, you’re right! And then… stop, like, the entire… Whirlpool that starts when a C-suite person asks it just gets… died down right after. Yeah, totally. Uh, I see a question from Attendee. Hey, how’s it going, Shane? Long time you’ll see. Hey, Shane. Nice. That’s cool. Uh, the question is, are there differences in structure and types of questions you would ask to a human versus AI? Um, I mean, structurally, no.
I guess, practically, yes, because humans have emotions, and you need to meet them where they’re at. Uh, I would say AI is certainly a lot more. Um, if this is recorded, like, in the future state where AI understands everything. Like, you can sort of, like, you know, you can stretch it to its limits, right? And it’s not going to be like, hey, I’m hurt by whatever you’re asking me. So, uh, human has the emotional aspect, so you need to meet them where they’re at. You need to make sure of, sort of, like, uh, you take empathy and all that kind of stuff in mind. I think AI is much more free-flowing, and the sort of, like, common denominator of both.
is the more, to Shabi’s point, the more context you can kind of build them up. Uh, whether human or AI. the better that they’re… that they’re gonna be at in… kind of, like, giving you the answer, and then, you know, the context could come from the questions. the questions themselves as well. That’s why that’s almost like, um, uh, sort of like the, uh, the P0 there. To double-click on what Hai shared about to humans, you should meet them where they’re at, it’s actually the most important thing, and it’s actually a well-developed skill as well. Knowing where the person’s coming from.
In fact, you could… the way you converse with them, and the way you understand the question and talk with them might just make your relationship with that person in a way that you’re not, like, against them asking the question, or against working on something. You want to definitely tell that, or, you know, show that attitude when you ask a question back. Uh, because that would help them, basically, before another question comes, they know how to use the self-serve tools that are already available, right?
And it’s not that data doesn’t want to support it, but it’s more that Um, we are here to support, but your question could be something that’s already Uh, you know, being able… can be answered with the existing stuff, or might not be even important enough. And the other question behind the question is more important. Yeah, I feel like sometimes you have, like… like you said, like, explain… Depends on the persona you’re working with. Sometimes you have to explain why you’re asking a question, because some people can… be taken aback when you question their questions, like, I’m not trying to question you, dude, like… Uh, I actually want to really help you.
And so, if you can work through this with me, then, like, we’ll make sure that. whatever you need gets done, and you look really good, and everyone wins. Um, so, like, always trying to, like, reframe that way. You don’t have to do it with AI. I mean, maybe we should be doing that with AI, I don’t know. Cool. Um… Cool, and uh, we don’t have to hang out here all day. Uh, we’ll, again, we’ll send out the deck, we’ll send out all their information, uh, probably send out the recording tomorrow. Uh, as well, so that you can always watch back or, uh, you know, like, if there’s any segment that you have more questions about, feel free to reach us on, uh, in, in this, uh.
Uh, in the Slack community, we’re very responsive, and, uh, you know, we… to be able to sort of, like, connect every like-minded person in today’s world. Uh, so that’s, uh, so that we can, you know, like, we can share information, share knowledge, and help everyone succeed. Awesome. Thank you so much, everyone. Thank you, everybody. Thanks for joining. See you all next week, uh, design metrics, if you want to join us. See youYeah