From Local Verification to Cloud Agents

Isolate the work

(26:06) Uh are you doing that in like a work tree kind of isolated with like the sub agents and and then the reviewer agent and and all that? Is it happening like in some type of cloud hosted environment? Like what's the the more the practical steps? If I wanted to go do this uh and like set up a verification system for something, what would I what would I do or where would I start?

Um I think that uh the best place to start is local because you can observe you can definitely observe what your agents are doing. (26:35) So, uh, if you're building a verification skill for yourself, uh, I would definitely start local and just have your agent bring up the application, whether it's like a CLI or, uh, desktop app or whatever.

And so, you can actually observe, right? You can see how the agent is interacting with the the application. (26:54) You can see it, you know, how it calls like the different APIs that that allow it to interact with the uh the application. Um but uh for me personally uh I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful.

Uh and the really powerful thing about cursor is the the cloud agents actually where if you spend a little bit of time setting up your environment these control skills these verification skills pay a huge amount of dividends because it's not just something that makes you as a single engineer better.

Share verification across the team

(27:32) It actually levels up your whole team uh and even your whole company because uh you can actually start thinking about cloud agents. start thinking about automations that automatically do things like uh I I get I I kind of talk about this a bit later, but I'll just kind of get into it. (27:51) Uh where where you know, for example, like I talk a lot about this agent we have called Benny, right?

who who uh you know takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh it's you know its desktop. It runs cursor in its own computer and it uses the same control skills to interact with the application and try to reproduce the bug uh or the user report, right?

And this is so so powerful because at once I can immediately I I get so much information from this automatically like here in this example you can see that uh (28:27) the Benny actually reproduced the bug uh but it's already fixed on main. So it actually confirms that we fixed this problem already and all I need to do is just release another build of of cursor.

Uh, so that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure like is this fixed, is this not fixed.

Scale gradually with trust

(28:50) So you you you gain back so much time. Uh, but you know, everybody on my team benefits from this. Everybody at the company benefits from this. Uh, so definitely think that uh, you know, keeping these uh, using cloud agents is super powerful. Uh, but yeah, it's like a journey. You have to trust it first, right?

before you you get to this point. (29:12) And that's it goes back to what I was saying here where, you know, it's very hard. It's almost impossible. And I would definitely encourage you not to try to jump from, you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn a hundred of thousand or thousands of cloud agents right now because you're just going to waste a lot of tokens.

The local-to-cloud journey

(29:34) Um, and it's going to be extremely expensive. >> Yeah. So just to kind of recap so far, basically the if we wanted to go on the journey that you've kind of gone on, it would be to start with verification, building some some skills and some some ways of determining that the agents are producing at least like correct code.

Whether like you said whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at you know stack traces, looking at you know the the actual behavior in the app and (30:00) so on. Um, and then once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals, I guess, on their own, right?

So whether that's like a bug report that comes in or something, they can go and pick it up and solve the problem and and give us back a PR. (30:16) And then maybe the last step is like automerging the PRs, which uh is where you're at, maybe not where everyone is at. >> Um, and then and reviewing them on main, but um, is that is that about right?

>> Yeah, exactly. I think yeah, that's why I drew this this uh this this curve, right? because that this this basically describes my journey of you know when I started barely could use a couple agents and I was just observing every single thing. (30:40) I think there's really no shortcut for going from here to there because this is really about your personal level of trust in agents, right?

Um obviously, you know, as a as engineer, you don't want to just slop code into production. So, how do you actually build up that trust? Takes um a lot of uh I guess taste and judgment. Um but, uh you know, like I think plugins like Pstack definitely kind of help you uh get up to speed much quicker.

(31:09) Uh, and so I guess it's like if you trust me and you trust Pstack, then in by extension you can maybe trust your agents. But if you don't trust me and I I definitely would not encourage people to blindly trust me. Uh, uh, you know, if you build up your own set of skills that you can obviously, you know, take a look at PAC and kind of fork it, make it your own, improve the skills.

Verify the application locally
                │
                ▼
Observe and build trust
                │
                ▼
Investigate reports in the cloud
                │
                ▼
Increase autonomy gradually