If the executed test scripts emit TAP-formatted output to STDOUT, individual passes and failures are counted and displayed in the test run’s UI.
Heroku indicates that the individual tests are displayed on the UI if the output is TAP compliant.
They don't mention what this UI looks like, or how to know if the output is correctly being accepted as valid TAP output to get the enhanced UI.
I’ve taken some screen grabs of the before and after to save you some commits.
If your failure output looks like the second image on Heroku, your test output is TAP compliant.