Speaker: they don’t like to be touched, so not… no expert in that, so… but we do get hourly update from Sravya as she’s embarking on her wonderful retreat. Yeah, I’ll see if she joins us. Yeah, cool. Alright, well, um, so, yeah, let’s get back to this. Uh, we… just a little bit of a quick orientation, we probably have about, uh, let’s call, like, 30-minute to 40 minutes of content. First 20, We’ll walk you through a framework for turning analysis findings into action plans. That actually get done. And then the next 20, um, I will switch to my screen, and we’ll watch Cloud Code do its thing. in, uh, actually taking a real finding and building a plan live.
Um, for those of you who have joined some of these Lightning lessons in the past, um, the format is going to be pretty similar, uh, and we’ll cover as much as possible. Um, but we can only zoom in on, you know, like, this one very specific topic today. Um, so, you know, like, I think, um, the… the real main idea here is that if you’ve ever presented an insight or a recommendation, or a finding from data that, uh, what’s called died in a Slack channel or some presentation, Um, this is a good session for you. All right, let’s proceed. Okay, cool. So… Um, so let’s consider this scenario.
Uh, you’ve been in this meeting, you presented an analysis, or you probably watched someone else presented one. And then the recommendation was pretty solid. people nodded, someone said, great, let’s circle back next week. Uh, and then nothing happened. Uh, if you’ve seen a scenario or have come across this exact scenario before, um… Drop a comment, or drop it in chat and see if this resonates. Yeah, I hate it when someone is like, oh, that’s very interesting. It’s very interesting. And then it’s like, man, yeah, it’s so interesting. What are you going to do with it? Like, I remember one of my first jobs I joined this company and then two months later, my manager Data science quit.
I was reporting too. And so then I reported suddenly to the VP of product and It’s just way too many. And I was like 24 or something, and it’s like one of my first data jobs. There’s too many layers between there, and, like, I didn’t have direction on what to do, and so I was like spending all this time on these analyses. And then the VP is like super nice, but He was just like, yeah, cool, that center is like, keep, go do more of that. But I was like, what are you going to do with this? Like Yeah, I don’t know, it kind of sucks like When you spend all that time on analysis, even if you get some positive feedback, if nothing happens. So I don’t know if that’s happened to other people here.
At any stage in their career. Yep. Anybody? Any reaction to this, folks? No? Everybody’s like, hey, everything turns into action, no worries here. Cool. Well, maybe, maybe that’s the case. That would be… that would be amazing, uh, actually. Well, so going back to this hypothetical example, or Shane’s, uh, example here, is that… The thing is, generally not with the analysis itself. The analysis is generally fine. The problem is often what comes after, so most recommendations, if a report or analysis even has one. Typically end with something like, we should look into this. you know, like, that’s the action item.
And, uh, no owner, no metric, no deadline, and then let’s call it 2 weeks later, nobody remembers what was decided. And the work that you all did, or someone else did, um, just stops and just dies. So, this is sort of, like, the posture that we’re looking at, and hopefully, you know, like, it certainly resonates with me. I’ve seen many of that. I’ve done it myself. very often, so it’s not just a, uh… so it’s quite common, uh, and if that’s… if you’ve never heard of this, uh, scenario, I would love to talk to you. Cool. Okay, so typically, um… kind of, like, four failure modes, um, and uh, you know, like, again, like, I’ve done all of them myself. Um, you probably have as well.
Uh, number one here is the action item. Trump, where someone types, let’s say, like, uh, you know, hey, investigate X, uh, into a Google Doc, there is no owner, Again, no success metric, no date, and you think, The action item is captured, but it’s so unspecific that actually nothing… nothing gets done. Like, nobody picks it up. So that’s pretty common. That’s the… You know, like, a bunch of stuff listed out in the action item, just unclaimed. And then, number 2 is… the we should trap. This one is That actually is super common, right? Um, you know, for example, like, uh, we should look into the checkout flow based on the recommendation, or we should run an experiment based on some findings.
Um, so the main… symptom there is, uh, shared ownership. So, which often ends up meaning no real ownership, so… You know, like, even if you can name a team to own it, a team is not a person, so it’s easy for everyone to assume that someone else has the ball. And then, number 3 is the Let’s Monitor It sort of trap. Where the recommendation says, hey, monitor this for a few weeks, um, there is no baseline, there is no target, so the monitoring would just run indefinitely, um, because there’s no point at which you’re defining, hey, this is done, or this is, um, a point in time where we’re good enough to make a decision either way. So, no clear point when it stops, and then, uh, and a decision.
And finally, number 4 is the, uh, let’s call it, like, the buy Q3 trap. Which is, when the deadline is a quarter, so if the deadline is, like, something like a Q3, or next quarter, or, you know, whatever, when we have time, uh, you know, Q3 spans a very big range, and nobody can put Q3 on a calendar. That’s just not possible. Um, so the… the date for which you would circle back is going to slip. So, these are just, you know, some common patterns for when a action item, or just a recommendation does not actually get implemented, because it’s just not specific enough.
Um, and so, through this lesson, we’ll, you know, like, once we identify these things, there’s actually ways to, uh, um, to, uh, to make sure these things are, uh, are, are… are included. Okay, cool. Um, so, let’s see. So, quick orientation before the framework. Um, as many of you might have already, um, you know, like, played around with the AI analyst system, uh, everything that I’m about to show you lives in this repo, this, um, GitHub repo here. It’s called the AI Analyst. Many of you are familiar in this lesson. So, you know, I could say Agentic analytic system that acts and behaves like a senior product data scientist, solving for analytics workflows.
So… Uh, it’s free, clonable, you can, uh, you can just, you can just clone it, you can use it, you can run it on your own data, Um, and uh, we’re gonna zoom in over the next… let’s call it, like, 15-20 minutes, on just one piece of it, which is this skill in the repository called Close the Loop. And, uh, this is one skill that auto-triggers any time when an analysis in your Cloud Code session ends with a recommendation, It will make sure that it’s tightened to a spot where it bakes in the framework that we’re just gonna… that we’re gonna go over in just a little bit.
Um, you don’t need the repo open to follow along, but if you want to clone it and follow in real time, like, please feel free to do it as well. Uh, it should be… should be pretty interactive after a few more slides. All right, let’s see… Okay, cool. So, uh, the framework is pretty simple, um, where, uh, We call it the close the loop checklist, and here’s the whole… the whole, uh, 6 steps here. Where, you know, we’re gonna just walk down the table here. Number one is the decision owner, so who who’s actually accountable, so just being very specific on that, and each of these are basically a question related to the different fields that we’re gonna just walk through quickly.
The second one is, what’s the success metric, obviously? Third one is, what is the baseline? So, where the metric is today, um, that we’re looking to move. Based on the recommendations that we are presenting. Fourth one is the target, so where it needs to get to. And then a checkback date, so when do we actually come back and look at the outcome and look at the results? Uh, and, uh, number 6 is a fallback. So, what… what would we do if, uh, if… you know, like, whatever recommended does not work out. Like, you know, what’s next, kind of thing.
So… These are just ways to pre-think ahead of time, so that, um, you know, like, all the scenarios, like, are mapped out, and they don’t really take that long of a, uh, to… to do. But it’s… it’s gonna have massive impact, because you don’t need to… you’re being very specific, and it’s almost like an implementation plan that you can just go and follow along. Um, once you’ve had it defined, versus it would be, you know, like, hey, action item of, uh, someone’s gonna pick someone’s gonna pick it up, and hope for the best. Yes, just something to interject in there too is like this system and anything it’s like some of this stuff Some people are already doing it already.
Maybe all of us, some people are doing already. Maybe some of it you’re doing consciously, but you’re just, like, you’re not writing it down, you’re just talking about the meeting. Maybe somebody you’re doing kind of subconsciously. I think, like, our… what we’ve seen very successful is, like. Being Very purposeful and conscious about applying the framework, documenting it all. And then the great thing with agentic systems and documentation is that it’s so easy to automate and set up. So these kind of balls don’t get dropped because it’s really easy to have a great discussion And be excited.
And then kind of just like forget This, which is kind of like paperwork, but it’s really important paperwork. I mean, some of the stuff that I’ve talked to People I’ve worked with, whether they’re ICs or whether they’re managers or whether they’re directors when like they complain about nothing’s getting actioned on this for so long or like I’m not getting headcount for X, Y, and Z. And it’s like Oh, can you share, like, the document around it and like all these things? And it’s like I don’t have a document around it’s like, okay, so what I was like, oh, I’ve been complaining about it to these people in this meeting for years.
And it’s like Doesn’t mean shit if it’s not written down like if it’s not written down, it didn’t happen. It’s kind of like my thing. And then this system just like just makes sure that happens every single time. Yeah, good point. Cool, yep. Documentation is very important, especially nowadays with AI’s help. And also, letting AI know what’s going on. Okay, cool. So, next few slides, um, I am going to walk through some of these in a little bit more detail with examples, and then we’ll get into Cloud Code and see what… what it does with it. And then hi, I think you’ll probably go into this as you think, but there is a question in the chat.
And as Attendee, if you want to ask your question, actually Yeah, go for it. To the group and then we can make sure we hit it. Yeah, I can’t find my chat button here. Sorry. Uh, yeah, hi. Uh, hi, hello. Uh, yeah, my question was that, like, in a lot of, uh, these scenarios. Um, sometimes… like, I’m assuming, let’s say, we are seeing, like, the leads number has dropped below, uh, 10%, like, which is kind of an acceptable threshold. But then how we want to investigate is not very straightforward, and it can take a lot of, like, possibilities. to, like, you know, follow up.
So, is that something that you’ll be also, like, covering in this, uh, um… discussion today, like, um, I’m assuming that when, like, in a simpler segmentations and all could be, like, predefined. And the context could be fed, uh, into the AI system. But, like, How do we deal about scenarios where, uh, We don’t have enough context, and we are not… sure, like, what parts of investigation we should explore. Yeah, yeah, that’s a great question.
Actually, 2 days ago, um, we did a lesson on root cause analysis in Clock Code, and uh… we can actually… probably share a link with you, or if you go on Maven, um, you can probably find it, or DM us, and we’ll send you the link to the previous lesson, where we walk through exactly how to do… how to, you know, like, leverage AI to decompose. into the specific drivers of certain movements, in the top-line level metric that you’re looking at, and uh, you know, like, it’s… the system can actually help you go pretty deep, and then, you know, obviously you won’t have to understand the data yourself, you won’t have to understand the business context of whether that makes sense.
Um, but the execution part, and how deep it can go, is now completely… different, versus if you were to do it by hand, where you’re hamstrung by how fast you can do it yourself. For example, hopefully that makes sense. Got it, yeah, okay. So in, uh, in this discussion, basically, we have assumed that we have figured out how to, like, do the… Yeah. root cause analysis, and it is all about… more focused on the next steps, like, Correct. going from the next steps to, like, closing back the loop. Okay, thank you. Correct. Yep.
Doing the, um, yeah, doing the actual investigation is in the previous lesson, and, um, this one is Yeah, given you have the recommendations, how do you tighten it up such that it gets action done? Got it. Thank you. Uh, okay, so, let’s see. So, this is the first pair here. the what and the who. So, if you remember from the previous chart, um… Decision owner and success metric. So, decision owner is one single human. Um, not a team, and not a function, a name. So, if you can’t name a person, then the work tends to drift, uh, and nobody clearly owns it moving forward.
Um, so this is pretty important as precise as possible to, you know, find a person, name it, and like, you know, hey, based on this recommendation, we should do X, Y, and Z, and this person is going to work on it. Um, this is… not super comfortable if you were to have to, like, assign it to people, but once you get through the comfort level of that, um, that actually is a very powerful way to drive accountability. And then, obviously, the second one is a success metric. So, you know, this one… You want to make sure that it’s actually observable, so, um, you know, you know what you’re measuring against and what success looks like.
And, um, uh, the metric itself should be as close to the change that you are promoting or recommending as possible. So, you don’t want to do something like, you know, NPS score when you’re trying to, you know, put in a hotfix for something that’s broken, for example. You want to measure something that you’re actually trying to drive. Uh, and not, you know, like, something that’s completely unrelated. Alright. And then the next set are these three things, which is… the how much and by when. So, 3 fields here. baseline, target, and the checkback date, if you remember from that table just a couple slides ago.
Baseline, obviously, is what the metric is right now, so captured… today, or whenever you make the recommendation, for example, with an actual date on it. So, um, I think folks tend to skip this one. Because it feels obvious, like, you know, of course we know what it is right now, like… That’s the… that’s the analysis. Um… But then in 2 weeks, someone could ask, um, was it actually that bad, uh, before? And if… If it’s not clearly documented, then it’s… you know, you might have to… or someone may re-litigate, for example.
So, just make sure that this one is clear, and you know, it generally takes very quickly, uh, it doesn’t the second one here is the target, so what is the specific number that as precise and as specific as possible, that you want it to move. So, something like, you know, like, hey, less than $200 in this case. We’re talking about payment tickets, and we’ll get into that same example with cloud code, um, versus… You know, like, oh, just reduce significantly, like, that’s not, uh, that’s not very specific, um, and you can’t really action on it. If you don’t know what good looks like. Um, cool.
And the third one is the checkback date, so when do you actually come back and look at what’s happened, uh, in the future? So, a real calendar date, not the next quarter or by Q3 thing that we just talked about. Um, you know, like, be as specific as possible. Like, on June 22nd, we’re gonna take a look again, and uh, we’re gonna make a decision. If it hits this, or if it doesn’t hit this. So, just a little bit more paperwork, as Shane puts it.
I really actually like the… the analogy there, um, you know, you just have to write it down and move forward, and that guarantees you to be a lot more… you’ll have a lot higher probability that someone gets actually acted on than, uh, if something is left completely in a very vague format. So that’s the baseline target and date, and then this is the final one, which is the fallback. Um, this is probably what everybody wants to skip. Um, you know, like, it may seem, again, uncomfortable, like, this is the question about what do we do if whatever we recommend don’t work. Like, it’s not fun to sit with, right?
The whole reason that we’re even… trying to implement the recommendation is such that we think it’s gonna work. So, talking about failure can feel a little bit like jinxing it. But it’s still pretty helpful, because, uh, answering this fallback question. Probably takes a couple minutes to think through. Um, but skipping it tends to cost more later. So, when the check back date… comes, and, you know, like, you don’t hit the target, or whatever, now you’re deciding to do you know, what to do under time pressure with not a plan in place, um, sorting it out ahead of time actually makes that moment a lot calmer.
So, it’s always good to, you know, pre-think different scenarios, um, while you’re… while it’s still fresh in your mind. And typically, a fallback could be one of three things, right? Um, you know, revert, so, like, you know, let’s, you know, we tried, and it doesn’t work, and that’s fine. Um, you can escalate, so bring in more people to be like, hey, this is still worthwhile to pursue, or this still needs to be fixed. And so, you know, if we need another We need another, uh, plan or roadmap for it, or just pivot, um, you know, try a different, complete, different, uh, approach, and, uh, and see if you can solve the exact same problem. Um, knowing that the first approach didn’t work.
So, just ways to, again, like, um… strategize and think of the different scenarios that things might play out. Um, the more that you can have something like this, the more people will take it seriously, in terms of, hey, yeah, we’re gonna… we’re gonna go try this out now. Because everything is so well thought out. Yeah, the answer we also think about life The way I think about it is like Life is like a series of bets. It’s like where we’re going to invest our time. What’s gonna happen in terms of like what we get out of that time, whether it’s energy or happiness or money or whatever. Same thing with like business or building a product.
It’s especially a series of bets in this case when you’re making decisions based off statistics that are based off some sample of a population that’s A distribution that can contain noise in the metrics that you’re looking at. So like Everything’s not gonna succeed, everything’s not gonna go right. Probably things are gonna fail more than they succeed. That’s… if everything was just, like, successful all the time, and all the ideas happened perfectly as you did it, then most of us wouldn’t need jobs. Because you wouldn’t need people to, like. be trying to make These bets as strong as possible and reduce uncertainty as much as possible.
And so, things will inevitably not work out, and that’s totally fine, but it’s You don’t want to be in some annoying situation with some person who makes it all political and is like, Shane, you said this is going to happen. Why is it not happening? You want to be in a position to be like Yeah, I thought it was going to happen with, like, this amount of confidence and like, unfortunately, it didn’t. Luckily, like, we did have we knew there was some chance it wasn’t going to work.
And here’s, like, our backup plan And that backup plan could literally be like, oh, this is actually a learning that we’re getting and we should try these things instead because we’re like kind of like disproved some, like, highly confident hypotheses we had That’s how you want to work. And it’s just like it removes a lot of that kind of like negative energy also, too, if everyone just gets aligned up front, and then everyone’s kind of rowing in the same direction, move the finger pointing and all that kind of stuff Yeah, that’s actually a really good point.
Remove the finger-pointing, because… you know, if things get contentious, or if people are like, hey, you know, almost like, once you outcome, and then it’s favoring your side, then you would not want to give up your position. If you’re not pre-aligned on, hey, that’s not the best for whatever, like, the company or the entire organization, for example. Uh, okay, cool. So, let me see… Okay, we’re gonna get into a demo. Um, so just really quick setup. Uh, for folks who are not familiar with the scenario, um, we are going to use the AI Analyst system. Again, we’re gonna try the close-the-loop skill.
In that, the setup is, in the system, we also have a fictional dataset for a fictional company called Nova Art. Many people on this call is probably familiar with that. Think of it as an e-commerce company, just like Amazon, so you buy stuff, you put it in a cart, you check out, and you go through payment flows and things like that. So, um, you know, all of us have bought something on an e-commerce website, so whatever your favorite is, it works very similarly.
And so, um, the setup here is that, um, Uh, we’re looking at a payment ticket spike, so people are complaining to our customer success team, or our customer support team, about um, you know, like, uh, issues with iOS payments, uh, and… And, uh, we’re able to isolate the cause, the root cause, to be a very specific app version that caused the entire spike. And, you know, We now have some recommendations, and we want to close the loop on this, and make sure the… the recommendations actually get acted on. So, what does that look like in plot code once we, uh, once we invoke the skill? So, that’s what we’re gonna do. hopefully that makes sense, and I’m gonna switch over to… my terminal here.
All right, let’s see, can you guys see this? Black screen, I… Yeah, okay, cool. So, um… Uh, this is VS Code, for those who are not familiar. And this is the AI Analyst System, just very quickly walk… walking you through. This is the AI Analyst System. Pretty much, um, the one, very similar to the one that, uh, that you can clone, that we open source. So, um, it’s got a bunch of skills, got a bunch of agents that, um, that that perform… that are instruction files for Claude to act like a senior data product scientist. Senior Product Data Scientist, and what we’re gonna do here, and here is my terminal with Claude Code open, um, for those who are not aware, how this whole thing works.
Basically, you install Cloud Code in your terminal, and you can start talking to it and do analysis off of this, uh, these sets of instruction files, uh, in the system to to, um, to… to do your analytics work. So, I will have a series of prompts that I can 2, which is… The first one I’m going to paste in is, uh, again, um, many of you are familiar, but for those who are not, you’re basically just talking to it once you have Clot Code set up. So my first question to it is, can you describe what the NovaMart data is about? What should I have What should I know about this company? That kind of stuff.
So, for anyone who is new, generally, to a… either a new company, new dataset, or new… product features or… or a new domain, um, you know, first thing you would… want to do is figure out as much… with as much information as possible, what does it look like? Like, what do you know about it? So, this is exactly what it’s doing. Um, just giving a rundown of what Nova Mart is, uh, similarly to e-commerce company, and… here’s the business, uh, the different channels that it acquires.
Uh, or people have access to, uh, the acquisition, uh, channel, how people Or how the company acquires traffic, monetization layers, margin, um, different, uh… uh… different… questions that we can ask it, the different themes and all the data tables that it has, so on and so forth. So we’ve got all these almost, like, summary statistics, being described by Claude on the whole… the full dataset about Nova Mart, so we get a better sense of, you know, kind of, like, what this… what this data that we’re working with is all about.
So, once we understand that, let’s do… Uh, so I want to do the… I just want to… I just want to recreate the environment of the setup that we just walked through in terms of the payment ticket spike. So, I’m just gonna have it… do it for us. Um, for folks who were with us a couple days ago, and the root cause analysis lighting lesson, some of these steps are almost like, uh, like, you know, very similar to that. Um, what I want to do here is to you know, number one, show again for folks who weren’t there that, uh, you know, you can have Cloudco do root cause analysis, And then the second part is, once you have the analysis, then how to tighten it up such that the recommendations get acted on.
So, the payment ticket volume per order by month, this is the prompt, and it goes into… looking through the database, starts to aggregate the data, doing the summaries, doing the calculations. I can… you know, like, I just… I just pressed Ctrl-O to essentially look at what it’s doing in detail, so, you know, I can check those SQL queries here as it’s running it, uh, and then I can see, sort of like, hey, what are you looking at in terms of numbers and stuff? Um, it says clear pattern here with a striking June 2024 spike, a little dramatic, uh, but you can see sort of, like, some of the thinking that Claude is doing. Data’s ready, let me collapse this… Cool. Okay.
So… It’s done doing the thing. It’s pretty quick, uh, the data’s not super vague, and so here’s the chart that it came back with, so… We can take a look at that. Uh, so this is the chart that it looks at, and uh… This is payment tickets in terms of, on a monthly basis, exactly what we asked it to do, and then it starts to flag that, hey, in June, there’s… is a spike here that looks very anomalous. Uh, and then, you know, draws me this chart. It has It has instructions for the style and also, you know, like, titles and stuff not being generic and things. Um, we’ve done that in a previous Lightning lesson around data visualizations. You can, you know, also check that out as well.
So, on this one, um, the payment support ticket, uh, it’s observing and pointing out that the June thing is real, that it’s, uh, spiking, and so on and so forth. So, the next thing I’m gonna have it do is… Uh, can you do an analysis on the payment ticket spike in June? So I’m referring to that, and it would… it would understand what I’m referring to.
And again, it’s gonna go into with the context of, you know, because we’re in the same session, with the context of the previous steps, it understands, okay, like, I flagged this, uh, this… this anomaly here, and you’re referring to, you know, that spike in June, like, I understand what you’re looking at, so I’m just gonna do a, uh, the 2x spike, uh, root cause investigation here. the, you know, AI is really good at debugging itself and fixing things, so, you know, it’s encountering errors and stuff like that, um, but that’s cool. Like, most of the time, I don’t need to be involved, because it knows how to self-heal, so self-correct and… try different things until it needs my attention.
So, it’s dug into different, uh, different dimensions, and now it’s figured out, kind of like the… like a… Uh, like a high-fidelity root cause signal. So, it’s able to isolate to version 2.3.0, the one that we saw at the setup, concentrated in a very confined time range, and then it’s gonna… do some, uh, charting and looking at tables to make sure there’s actual evidence for it. So, we’re gonna let it finish. And see what it comes back with. Uh, 2.3.0 is flooded with payment complaints without actually blocking transactions. So, yep, it’s able to do all that, and if we expand, Ctrl-O again for folks who are interested, you can, again, check the queries.
look at what it’s doing… And you see that it’s actually doing multiple steps on this. In terms of trying to triangulate different, uh, different ways to figure out if something is actually an anomaly versus, you know, data issues and stuff like that. Okay, um… I don’t know why it’s taking so long. But… I guess it’s doing. very thorough analysis right now. Alright, so it’s done some, again, some triangulation on itself. No double-charged bug, no damage to the transaction, support got flooded. Uh, so on and so forth.
for your data seals it, so… It’s done multiple rounds of, uh… of back and forth, so the whole root cause analysis, sort of, um… steps is, you know, it drills into one, take a look as far down as possible, and if it can’t find anything that could explain, then it comes back up to take some… picks another dimension, go deep, and then, you know, so on and so forth. So, that’s effectively what it’s doing here. One of the triangulation things that he was just saying is when we built in around like a sort of validation layer. So there’s You know, there’s different types of validation.
There is, like, the validation of someone manually checking the code that’s being written, which is what, you know, you really do want to do before you share stuff out to a stakeholder to see if it’s, you know, writing the right SQL, writing the right Pyth There’s validation around the numbers themselves. So like, if you have ground truth or past analyses or other sources of truth that you can tie out numbers against That is a very like concrete validation, but not always available. So if you do a new analysis that was never done before, you’re not going to have that.
And then another form of validation is more around Well, then there’s validation around like, is the data just like really wonky? Like it’s like this just like logically doesn’t make sense as something a business would do. There’s actually stuff around here Around that, like, I don’t know, if certain numbers are negative, that should never be negative, etc. Or do things to be true. What the triangulation thing is, is that if we run similar types of analysis to test the hypotheses in different ways, because you can have a hypotheses about what’s going on But there’s many different approaches to test the hypotheses.
So I’ll try and test it in different ways to basically see, like, are you getting the answer from different methodologies? So you can think of like a one approach for this like this is totally not what we’re doing here, but say you’re building like causal inference models or something to understand like the relationship between two things. If you build like a regression model, parentheses score matching model And like a I don’t know, I did a different diff or something, or before after.
If, like, the majority of those outputs of different methods at trying to analyze a problem, come up with the same directional, like, answer of, like, what you should do, what decision you should make Then that’s like a pretty nice validation. But if you have different methods of analyses recommending different actions out from your hypothesis, then it’s like, okay, something is going wrong here. And the really cool thing is before, as a single person That would take a lot of work, because you have to deal with them all separately on your own, or you’d have to have multiple analysts or data scientists run those analyses in parallel, which is total overkill.
But now you can be like, all right, I’m going to send off You know, 4 sub-agents to go try and test out this hypothesis with different methods and we’ll see if they all come around with like the same direction of what the business should do. So it’s not like a validation in terms of, like, the number itself, but in the action you should take. Yeah. So cool. Alright, so… came back with a bunch of analysis, recommended next steps, and things like that, so… here, I’m just gonna say, can you close the loop on the analysis recommendations? And it should just pick up the skill that we’ve been Talking about.
You know, I’ll apply the close-to-loop skill, turn recommendations into trackable commitments. So, something very tangible, very specific. Uh, so this should just come back in just a little bit. Um, for those who are… Wondering, I’m on Opus 4.7, tends to be a little slower than the model before. Opus 4.6. But, uh, you know, that’s cool. Still works really well. So, um… Yep, so it’s gonna build the close-to-loop plan based on what we talked about. And it writes it out in an empty file, so a Markdown file, think of it as just bunch of texts, um, that agents are really good at parsing. And… let’s see, written the thing, the MD file, and let’s see what it comes back with.
Oh, and now it asks me, uh, because it needs all these information, who should own the decision, so let me just do engineering lead. Uh, so this is about fixing this spike here, so who should be the one o’clock? Because it knows nothing about the composition of people in the company. Um, by when should a decision be made? So, let’s call it, um, this week. Uh, who follows… who follows up to confirm the action worked. Uh, let’s do… same as decision owner, and then I submit, and so now it grabs all these unclear parameters, and then it will bake into the… to the, uh, to the checklist and to the plan.
And effectively, it’s doing the exact same thing that, uh, that we just walked through in terms of the slides, and, you know, take a look. Action number one, confirm the defect, dig into the changelog, and the owner is, uh, or success metric is, uh, Uh, the… the, uh… The code change is found, um, the target or I guess there, like, a window here, um, corroborate. So, owner is engineering lead, decide by is that thing, check in, the checkback date is here, and here are the other action items. And then it’s got the guardrail thresholds, fallback, if this is not confirmed, then do X. Um, confidence even gives a confidence score, so, which is… which is cool.
Uh, so… Uh, you know, gave me some caveats to flag, and it’s all cool. So, if you have The domain business… context and knowledge. Certainly, this is a plan that you can iterate on. And so it’s not just a, you know, whatever Claude tells me, then I’m good with it. No, uh, for the most part, you know, at the end of the day, the human is still accountable, so this helps you narrow things down much quicker, But at the end, still you, who has the best domain expertise around what it’s looking at. So, very quickly, I’ll just show the last one, which we like to show people quite a bit.
Um, you know, like, we invoked a skill, we used it to craft a plan, so now, why don’t we ask Claude to just Give us the diagram of how this even works. So, I’m looking… I’m asking it to give me ASCII diagram, basically just, like, a flow chart of, uh, what the workflow looks like, and so… Um, the whole idea is, hey, the very top is, is there recommendations? Yes, no?
And then it flows into the decision, recommendations, um, who’s the decision maker, what’s the deadline, that kind of stuff, and what’s the success tracking, guardrails, follow-ups, when’s the check-in date, who is the owner of that, and then, uh… And then the different scenarios around, hey, if it’s not successful, then what’s the fallback plan? And if it’s inconclusive, what happens? Uh, and then it just goes through this loop of recommendations, decisions, success tracking, follow-up, all the different things that we’ve talked about, and then… Um, and then, like, when to check in, stuff like that.
So, um, you know, like, this is one way to help visualize what it’s actually doing behind the scene. And to understand, sort of, like, the workflow of where decision points are, where the… the different, um, uh, different checkpoints for the… for Claude to… to, uh, to make certain actions, and what happens if… the reverse is true, for example. Alright, so that’s all I wanted to… talk about in the demo. Let me go back to the thing here. Uh, my computer’s spinning a little bit, so… Hang on one second… am I still on, actually? Yep. Okay, cool. Everything looked really frozen at first. Okay, cool. So, um… Yeah, we have 15 minutes or so. So, we’re done with the content.
And, um, a lot of folks in this, uh, in this lesson today is going to be joining us tomorrow. Um, for a 2-day boot camp, so we have a Cloud Code Analytics Bootcamp, where we’re gonna go through how to build and… and… And, uh, agentic analytics system, just like the AI Analyst, so understanding how to how to actually craft something that tailors to your use case, and opening up, kind of like, you know, like, how this thing is made, how the sausage… how the sausage is made, such that you can use it, not just using it, but, like, you can improve upon it, build on top of it, or, you know, spin up something completely, completely new.
for attending this Lightning lesson, we have a discount code for you, 20% off, ACTION20. I think we can drop in the course link for register, so this expires tonight, because we’re hosting the bootcamp tomorrow. It’s gonna be 2 days, 4 hours each day, um, May 23rd, so Saturday, Sunday. Would love to… would love to have you there. the way sausage is made is very disgusting. Agentic analytic systems is is much more approachable and enjoyable. So I don’t want to hide or scare you away. Sorry, we’re all bad… bad analogy. Um, the spirit is there. Attendee had a question around Attendee, were you asking if the root cause analysis stuff is covered in the weekend?
uh… the root cause analysis… I think that’d be like more the five week. I don’t know if you want to tell Attendee a bit about that versus like the boot camp, if you want to explain the difference between Oh, I see, like a concept of… does, too. root cause analysis. Yeah, he was just responding to me and Attendee’s thread around the roof cause analysis workshop the other day, and he was wondering if that was covered in the boot camp this weekend. Got it. Um, yeah, so this weekend, we’re… talking about, uh, we’re going over, um, building in Cloudco, building agentic systems, just like the AI analyst.
in, uh, in the two-day bootcamp, we have a separate course that we didn’t talk about in this, uh… In this lesson here, where it is a 5-week course around the foundational concepts in analytics. So, um… think of it as, um, the idea is to help everybody become analytically independent. Um, what we’ve done is, uh, Shane, Sravya, and myself have pretty much distilled all our experience of what good looks like in terms of doing analysis and analytics workflows. Um, and, um, uh, and all of that stuff.
boiled down into different… in the five-week course, where we would… where we systematically teach how you… how you ask better questions, how do you define metrics, how do you then… Uh, do root cause analysis? How do we do causal stuff? Uh, how do you present and secure maximum impact off of the things that you’ve already done? The idea is for… and then how to ex… how to offset… offload the execution to AI. So the idea is, um, uh, anybody could be analytically sufficient in that they can do their own analysis, they can ask their better question, and then with the help of AI, or, like, an AI analyst system, uh, you can actually do the analyzing, uh, without having to involve the data team.
So, um, so, you know, no matter if you’re a data professional, you can… get the best practices in various different steps, in different lessons. Uh, and different topics in the 5-week course, uh, and if you’re, you know, like other builders, like product manager, engineer, and stuff like that, you can get the, uh, also get the same value out of, uh, out of that. So that’s a… that’s a different offering that we have. I’m happy to… get into it more, uh, if anyone’s interested. Yeah, Attendee, you just asked for that link. I dropped it in there, but Attendee, since I think you’re in the boot camp this weekend, don’t buy this course for the current prices on there. We do like a two for one.
So if you’re in the boot camp And then you want to go do the five week builders course, then we just deduct what you’ve already paid from the boot camp from the price of the builders course. So we need to make you a special promo code for that. Or if you do the builders course first and you want to do the boot camp, I just manually add you to for free to a future bootcamp. Because there’s a little bit of overlap in week two. They’re different, but there’s just like a little bit of overlap to the point where I’m like, I don’t want people paying for this thing. kind of twice. But they are different. Yeah, like I said, this weekend is about building a system.
and then the 5 week is about doing data science by using Claude code or other AI tools instead of writing SQL and Python. Yeah. And Attendee, I think you have a question around… Uh, do we already cover something around opportunity sizing, prioritization, friction identification? Yes, we have those in the 5-week. Uh, as well, and I’ve… I don’t remember. We had a workshop on one. I’ll find it. Yeah, I think we might have a lightning lesson. We’ve done 15 of these, at least, if not 20. I think you’ve done like 20. Yeah, let me… I’m gonna find the opportunity sizing Lightning lesson I’ll drop it in here Okay. Yeah, so we’ve… we’ve done it, um, in a very similar format as this.
If you’re ever curious about the whole… the full… the full thing, um, our five-week definitely covers… covers that as one of the lessons, one out of, like, 80 or something. Yeah. Yeah, the five week is like Yeah, it’s almost 100 lessons and it’s like a combination of it’s async content that’s self-paced. But then we have three hours of Live like kickoffs and office hours throughout the week. So it’s like you watch the videos and learn on your own, and then do like the practice sets, and then We discuss live, so you’re not like Spent like, you know how like right now it’s like pretty one-sided where me and I are speaking to you guys.
It’s like not like that in the in the five week or builders, really. It’s a lot more interactive. This is fun, but it’s so much more fun when we’re all interactive. It’s like friends You know? And you have a continuous group of, uh, community of people who are taking the course at the same time, so we tend to see… Yeah. a lot of interactions between, and knowledge sharing and stuff between the students. Something cool that we didn’t really get into here in the Insights to Action is that Obviously, like, we’re… we ended the insights to Action where it’s like In Cloud Code still giving us our plan and our kind of like framework results.
But You can also really easily like connect to like slack or notion or Google workspace and create a skill where this Automatically is syncing to that, right? Where it’s like it creates a Notion page, it tags everyone in it, and then automatically posts it to, like, a Slack channel. So you can automate it even further. I think that’s You know, as much of that kind of administrative work You can do as possible things that you used to have to copy and paste to do Claude code’s pretty good at automating that. Because you’re not going to like take a screenshot of that or copy and paste it somewhere and reformat it. You can have Claude do that for you. Yeah. And no, it’ll be the wiser.
They don’t even need to know you’re using it. Although in this day and age, you want people to know you’re using it. You gotta… you gotta, like, signal people, like, I’m using AI, I swear more impressive is its clock mode and terminal. Yeah. Cool. Andy? Any other questions can be about this, can be about other stuff. Cool. No questions is fine too. Yeah. Yeah, hopefully, well, we’ll see some of you tomorrow. That’ll be fun. And then Yeah, I know it’s also, for those of us in the US, it’s a holiday weekend, so if you’re doing something else this weekend, but you do want to take the bootcamp, we run it monthly.
So we’ll do it again June 13th to 14th, and then July 18 to 19, I think we haven’t scheduled, but it’s my wife’s birthday on the 19th, so I might have to change that one. I’m not sure. I told her that the other day. She was like, you what? I was like, I’ll talk to Heinz Ravi and maybe we’ll move the July I might not… maybe I’ll dip out early on that one. But July is TBD. Yes. But June 13, 14, that’s locked in. We’ll definitely do on that. Then we’ll do something in July. Thank you. The duration of the bootcamp, yeah, it’s 7am to 11am Pacific. So it’s 4 hours each morning.
We do a break for about 10-15 minutes in the middle of it after hour 2, although we’ll be we’ll be around during the break for questions, and then You know. If stuff’s running over or people have questions over, then we stay a little bit over for sure. And then the recording, it takes Zoom about 2 h to process like that 4 h recording. So then I dropped the recording up out to the Slack channel or email About 2 hours later, so around 1:00 PM, 1:30 Pacific, in case you had to leave early or you’re on a different time zone. So we had a bunch of people join from, like, Australia. It’s extremely late in Australia at that time, so some folks joined for the beginning Watch the rest I think later on.
Thanks, Shane. One quick question, if I can ask. Yeah. Um, I’m just curious to know, like, one way to, like, follow through all the, uh, how to set the AI analysis, following the AI… analyst repo. Uh, I’m curious to know, like, any suggestion, like, why you think the bootcamp could be, like, more helpful in that way, to, like, you know, directly join and, like, attend the course, rather than, like, doing it DIY. So you can definitely do it. DIY, like, I don’t want you, like, pressure to join us, like, immediately tomorrow, like, feel free to, you know, do that first. Hopefully people have done it.
We’ve had people who’ve done a DIY and then come to the boot camp later on, but it’s mostly because, like, You’ll have to extend this system to your company. And so you kind of want to have an understanding of how everything’s worked, because this is going to become your tool. This isn’t like a software product where you know, you’re buying some software product, and it’s just like you hit a button and it’s going to do exactly what you want. It’s like a non-deterministic system. Stuff’s going to go off the rails. It’s going to like break.
And so you kind of have to own it and turn it and nurture it, just like you would, like, a junior analyst counterpart or something, and you have to customize it to kind of your company’s use case and data. You’ll also just have, like A lot of really good ideas that like we don’t have. So we’ve this thing’s been forked, like, over 100 times now, and is at all these other companies, and all of, like, our, like, my friends who’ve used it, our colleagues at our company, but also our students, like, they’ve just built it out to stuff that I wouldn’t have ever thought of and made it a lot better for for their use case.
So that’s actually we’re doing a thing next week with our five week course where we’re just like having like a kind of post course wrap-up with our cohort of, like, sharing out of, like, hey, how have you developed this further in your company? So, like, we can take those ideas and do them at ours. So I would say if you’re familiar with agentic systems, or you’re like ready to just like learn on your own, then you can totally do that. But if you want some help, then That’s what that bootcamp’s for Got it. Thank you. Yeah. No worries. And with that, we’re at time. Thank you, everybody, for joining, and we’ll see some of you tomorrow. Cool. See, everyone. Thanks again.