Blackbox Test Failures
Stories of blackbox test failure.
Wrong build with dev branch
A build was released to production which is based on an active dev branch (v2.6+dev) instead of the release branch (v2.6) by mistake.
The dev branch was ahead of the expected release branch for a new feature development.
All blackbox tests passed the build as the new feature development luckily did not break any existing features, and there was no impact on production.
It was not noticed until it caused upgrade failure to the next version 2.8.
The upgrade from release branch v2.6 to v2.8 works, but failed for production devices from v2.6+dev to v2.8.
The reason is that v2.6+dev has a new database migration script to add a column named mqttTransportEnabled which was later renamed as logMqttTransportEnabled in v2.8, but the nodejs sequelize db migration script filename remained the same. So while upgrading from v2.6+dev to v2.8, db migration is considered as done already and won't apply the change for the new column name logMqttTransportEnabled, and app fails to start.