Grilling on a token usage requirement

I’ve just done a grilling session on a requirement to log token usage. Now that I have my blog up and running I can record my thoughts immediately. Firstly, while the grilling is powerful, if you deviate with a question, midstream it can get confusing as to what the AI has decided on your answers. Sometimes it needs to be told that you didn’t actually answer the question so that you can go back and review. Secondly, I was able to change my mind or introduce new or changed concepts, but I’m not fully convinced that this is a free flowing conversation.

As you watch the agents reasoning output you can sometimes see that it has made, or possibly made, a decision that you don’t agree with. For example in this case I could see that the batching of token usage records for a chat turn was not being considered. In the end the agent and I agreed this was a phase 2 concern. A pretty good agile outcome.

I think a full spec iteration would require multiple grilling sessions over the same spec, each with a limited scope for changes.

I wonder how a team would cope with the verification of a spec. Who would decide when the spec is complete and correct?

Maybe if you get so far down the wrong road it’s better to throw away and start again.