Pstack: Reusable Skills and Evals

Where Pstack came from

(15:45) So, P stack is pretty interesting because uh well, first of all, the name is kind of goofy. like the P the P and P sack is like potato potato snack because I um so uh uh there is a pretty uh famous person Gary Tan who is the CEO of Y Combinator and he's come up with this plugin called Gstack uh Gary Stack and uh funnily enough we share the last name we have no relations uh but I thought it would be funny to kind of you know poke fun at Gary and make peace stack my version of of of of [laughter] his plugin.

(16:25) Uh but kind of tailor it to my own set of p engineering practices. Uh but I honestly actually never set out to build PAC. Uh it just started with a bunch of skills, right? Like I started with that control glass skill and then I started with another skill like called how which I also noticed through like observing agents.

(16:46) Um, so like you know in the early days of me, you know, trying to climb this ladder, I was like super in the loop and I was basically nitpicking my agents to an extreme degree. I was like I would tell it um, you know, this feature has stopped working. Here's a bug report. Like why isn't it working?

And very often the agent would just like confidently state like, "Oh, it has to be this, right? It has to be this thing. (17:13) " And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading the code that I thought should be affected. And that made me just extremely suspicious.

And at that point, I was like, I'm not going to I can't trust any this agent anymore cuz it's just it's just completely hallucinating. And I think I think it's very easy to just you know like build up that distrust and not and kind of feel helpless like you know you don't know how to help your agents succeed.

Turn observations into skills

(17:43) But like again I think the the the management analogy is super helpful because like imagine if you were a manager of an engineering team and you had an engineer on your team who was a really good coder. No business context whatsoever. you know they they just you just hired them and they they onboarded you know like five seconds ago.

(18:02) Uh and so how do you actually teach that person to be effective? So how you do that is through a skill. uh a skill being just you know it's just markdown right but you know it encodes a lot of information instructions a lot of uh you can really draw out a lot of intelligence from an agent by well some people on Twitter call it like you know pull the agent to a different latent space which is kind of like a fancy way of just saying like since uh you LLMs are sort of like they predict the next token uh when you give (18:35) it some high quality tokens uh to to begin with then you know it it can kind of pattern match on like a higher space that's you know smarter um so that's like a very interesting model there but yeah I built Pstack very very incrementally uh so uh started with just really observing how agents you know all the fail different failure modes of of that agents were having and every time I saw that I just okay I'm just going to make that a skill right like stop hallucinating actually go and search up, look up the code, use a lot (19:10) of sub agents, uh, and yeah, stop guessing.

>> Yeah, that makes sense. One, one kind of followup question here, uh, both from myself and from a bunch of people in the chat. So, >> I guess it's two two parts. So, one is like how do you maintain these skills? So, like the product changes over time. Obviously, there's a lot of people who are shipping against the codebase.

(19:30) So, how do these skills get maintained? Uh and then second to that is like how do you know when your verification is is good enough? Uh like and you know you can trust that the ver verification loops that you've built are going to I guess you trust that the outputs uh when they're done. >> Uh yeah maybe I'll talk about um I think there are some related maybe I'll start with this one first.

Treat skills as something to evaluate

(19:56) So like how do I maintain these skills? So, um, if you're not familiar with this concept, an eval [clears throat] is essentially like a way to, uh, well, I the mental model I have is like it's like a unit test for an agent. Um, and, uh, you can actually make your own evals. You don't need like a special framework for them.

(20:18) You can build you can you can build one depending on like, you know, how scientific and how rigorous you want to be. Uh, my screen is red. >> Yeah, there's a little button. Um, sorry. Do >> you mind like disabling the drawing or something? I I can't see my screen. >> Yeah, sorry. If you guys could not draw on the screen, that'd be great.

(20:37) But, um, there's a little button in the >> troll. >> Yeah, the little drop down. >> How do I clear? >> Yeah, you got it. Perfect. >> Yeah. So, eval test your skills basically. And actually in Pstack we ship uh under potato mode there's a playbook if you search for it called eval playbook. Um and it's uh uh it's like not it's actually pretty pretty rigorous the way it's done.

(21:10) Uh but um essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh what I want the skill to do. Um, and then it spawns all these sub aents and it's it creates individual directories for them, uh, which are cleverly named to not let the sub agent know that it's being evaluated because, uh, agents can actually tell and when they do, they change their behavior.

Use rubrics and independent agents

(21:44) Uh, but it does a bunch of stuff like that to um essentially yeah like test whether or not the skill I'm making or changing is actually doing what I think it does. Um, and one of the really nice things about cursor is that we are we we support so many different models. So you can actually eval your skill across all sorts of different models um and you know get a sense of how well it performs across that different matrix.

(22:13) um especially for the models that you use. Uh so I do this a lot. Every time I I modify a skill, I will run one of these uh like the EBAL playbook uh and make sure that you know it's actually leading to a result I want. Uh but I will say like maintaining skills is actually pretty hard. Uh it requires I think a lot of taste and observation.

(22:38) So, you kind of need to be very good at being a backseat driver, you know what I mean? Like if you do pair, if you ever done pair programming, for example, uh, and you watch a co-orker code and you just like you, you could probably do this better, you know, you could do, you know, like why did you not do this, right?

You you ask a lot of questions to your coworker. (22:56) And it's kind of a similar thing here. You like you don't want to just be a passive observer of your agent. You want to be very in the driver seat in the initial stages when you're building up your own set of skills. uh you know obviously you can use something like PAC but if you're building your own set of skills it's very I think you know opening up the all the tool calls and like reading the code and reading all the uh the agent behavior and their thinking blocks is a really great way to see where they they fail right like what (23:26) what you know where are they being done and then you can go and build a skill for that and then with verification how you trust it is it's I think it's also a very similar iteration loop uh where you know like I actually did the same process for verifying the verification skill where I actually get um so one thing that's interesting about eval is that you can sort of hill climb them meaning that uh your eval can produce a score right uh a score that you can get your coordinator to produce uh but also comp uh you can have a judge agent of a (24:03) different model to uh kind cross reference and make sure that the first model is not being biased, right?

The model that's judging all of the sub aents that are running the thing. Uh but you can also like hill climb. So meaning that you can you can use like /loop in cursor and you can say okay keep looping on this eval right until everything is 10 out of 10 as an example.

Define the task and rubric
              │
              ▼
Run agents in isolation
              │
              ▼
Judge the results
              │
              ▼
Improve the skill and repeat

Iterate across skills and models

(24:28) Uh and I did the same the basically the same approach with the control skill. And so I kind of it was very it was very hands-off actually. Uh so you know I uh I kind of built I built that skill that way like the CLI in that skill. Um and over time it's gotten really good. Uh but yeah it was definitely not super smooth at the beginning.

(24:50) It required a lot of iteration and I think there's an analogy here for me which is um well I make this analogy later in a different slide on my drawing here. Uh but I think of it like uh you know as a as a engineer now you're sort of more like you uh like maybe a manager or the analogy I like is like you're like a a chef in a restaurant.

Uh you you're the head chef. (25:17) Uh you're not cooking all the food yourself anymore. You have a team of cooks, right? You have line cooks, you have a sue chef, you have, you know, all these different stations. Um and it's your job to really design the environment. you know, you you're in charge of setting up the kitchen.

You're in charge of, you know, like giving tasks to different people. (25:39) So, um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I've basically built uh these verification skills. >> Yeah, just just one followup there on like to go try to go one layer deeper. So, are you let's say we wanted to build um an eval or a skill for for something and we wanted to kind of get better on its own, which is is what I think you're suggesting.