Quotes -- When Tests Lie: Using Observability to Keep AI Honest

Short excerpts selected from the transcript for grounding answers. Preserve transcript artifacts when quoting.

Tests as incomplete signals

Um but again, that gives you grounded tests on what real behavior is and real edge cases. Edge cases are really interesting. Um

Source: L0288-L0292

Observability

production code or not. So, please welcome on stage Justin Cormack. >> Cheers. >> [applause] >> Nice to be here. Great event.

Source: L0005-L0009

Instrumentation

you know, and so I was like, well, I'm going to build lots and lots of tests. Um and just using testing, I think, you know,

Source: L0127-L0131

AI behavior evidence

of 100% of test coverage from test coverage from integration tests. Um and I it didn't really um

Source: L0160-L0164

False confidence

Um but I still have a lot of tests. I mean, 75% of my code base is tests. Um test coverage is between 75 and 100% for each file. Like, it's like

Source: L0192-L0196

Operational feedback

thinking, "Hm, is that really true? Have we got tests for this?" Asking the AI to write tests for them. And finding other weird things that were approximately equal to the documentation

Source: L0343-L0347

Tests as incomplete signals

have weird errors. Um flaky tests were really interesting. I said that um like AWS was AWS converges to uh truth over time, which is really

Source: L0374-L0378

Observability

it like this is the opportunity to fix flaky tests. AI is good at it. Um, and it will and it will really help your tests if you if you just get rid of all

Source: L0412-L0416