My Test Stories
Lessons learned in my test pratices.
Push for 3rd party regression resolution
Our linux app ran on a 1GB Raspberry Pi hardware.
Then we added a new 3rd party gateway, which are container-based, and there were just enough to CPU/Mem resources to accommendate these new containers.
And we were in the transition to upgrade our hardware from 1GB to 4GB with a little extra cost but much better performance.
Later 3rd party gateway released an update with performance regression and it failed to run on 1GB hardware.
Management team was okay with the regression and planned to replace existing 1GB HW that we had shipped with the new 4GB version as not too many were shipped to the customers.
I reported the regression to the 3rd party anyway.
Initially the 3rd party could not replicate the issue with their hardwares which have more the 1GB RAM.
Following their measuring methods, along with our own metrics, I showed the evidence of regression.
They accepted the regression, and made a new release by removing unneccesory containers,
and the final performance is even better than the initial version, and we can run it in 1GB HW with room, the best ever.
No HW replacement, and we can run it on old HW comfortably.
Common Software Quality Mistakes
This lists a few common software quality mistakes where dev and QA should pay attention to.
Two features are developed in parallel
When two teams or people are developing separate features, there may be bugs when the two features are used together though there is no problem for code merge.
Because they are not aware of the other features during their development.
Blackbox Test Failures
Stories of blackbox test failure.
Wrong build with dev branch
A build was released to production which is based on an active dev branch (v2.6+dev) instead of the release branch (v2.6) by mistake.
The dev branch was ahead of the expected release branch for a new feature development.
All blackbox tests passed the build as the new feature development luckily did not break any existing features, and there was no impact on production.
It was not noticed until it caused upgrade failure to the next version 2.8.
The upgrade from release branch v2.6 to v2.8 works, but failed for production devices from v2.6+dev to v2.8.
The reason is that v2.6+dev has a new database migration script to add a column named mqttTransportEnabled which was later renamed as logMqttTransportEnabled in v2.8, but the nodejs sequelize db migration script filename remained the same. So while upgrading from v2.6+dev to v2.8, db migration is considered as done already and won't apply the change for the new column name logMqttTransportEnabled, and app fails to start.