The ROI on unit tests, as well as the answer to "Can we test it?" is changing fast in the age of AI.
1. AI is making unit tests nearly free. It's a no-brainer to ask Copilot/Cursor/insert-your-tool-here to include tests with your code. The bonus is that it forces better habits like dependency injection just to make the AI's job possible. This craters the "cost" side of the equation for basic coverage.
2. At the same time, software is increasingly complex: a system of a frontend, backend, 3rd-party APIs, mobile clients, etc. A million passing unit tests and 100% test coverage mean nothing in a world when a tiny contract change breaks the whole app. In our experience the thing that gives us the most confidence is black-box, end-to-end testing that tests things exactly as a real user would see them.
I haven't looked at much at the quality of unit tests generated by AI but I would imagine it would suffer from the same general problems as humans. That is that most of these unit tests would be bad tests. They would include assumptions about implementation and make changes to the code harder.
Yes- black box and end-to-end testing. Better forced by the nature of the black box to test the behavior of the system. It's still possible to test the wrong things (behaviors you don't care about) but that tends to stand out more. Also still difficult to make these tests reliable.
I've talked about this in a few places, but there is a simple way to get to higher testability and trust in the automated tests being done with little to no extra effort for complex systems — this slightly tunes how you define the testing pyramid, but it's important to always test exactly enough, or you hit maintenance burden that kills you down the line.
All it takes is for a component developer to build a fully functioning component by having proper fakes for all the external dependencies. If this is religiously applied through the org, then all components in the org will have all of their real business logic and integration points sufficiently tested.
Then, your end-to-end tests can work against a very complex system that is extremely fast, but which uses the actual business logic, other than the very extreme boundaries to external systems not under your control.
However, AI is usually not much of a help here, because most software is not written in the way to easily construct either a fake or real production component from the same component — and AI is trained on existing source code.
1. AI is making unit tests nearly free. It's a no-brainer to ask Copilot/Cursor/insert-your-tool-here to include tests with your code. The bonus is that it forces better habits like dependency injection just to make the AI's job possible. This craters the "cost" side of the equation for basic coverage.
2. At the same time, software is increasingly complex: a system of a frontend, backend, 3rd-party APIs, mobile clients, etc. A million passing unit tests and 100% test coverage mean nothing in a world when a tiny contract change breaks the whole app. In our experience the thing that gives us the most confidence is black-box, end-to-end testing that tests things exactly as a real user would see them.