A little over a year ago Xojo switched to using a new issue tracker
IMHO this was a good move.
Not only does it seem to be more capable it lets them quit spending time on something they can buy and just use the heck out of. It lets them focus on their core product rather than ancillary things like bug trackers.
That said, a little after they did this switch I started watching some of the numbers that were publicly visible. Things like open case count, open bug count, etc.
A GIANT caveat. The numbers I see are NOT the same as what you see. From what I can figure out this is because it will depend on what cases are confidential and I might see a different set of those than you do. So the numbers should be close – but unless you’re me you will likely see different values.
That said here’s what I found :
The day I started tracking things there were 6313 open reports of all types (bugs, feature requests, etc), 29787 closed and a total of 36100 reports
As of this writing there are 6960 open reports, 31924 closed and 38884 in total.
2784 new reports (note not necessarily BUGS – just reports)
Thats an average of 7.6 new reports per day
Also the day I started there were 2018 open bug reports and 24769 closed.
Today there are 2276 open and 26054 closed.
Xojo has closed a lot of bug reports in the interim.
But, as you can see, they are slowly falling behind.
Perhaps why archiving reports was even started.
The day I started recording numbers the # of archived bug reports was 4154.
Today that number is 4286
Note that it did creep up for some time but has halted since Xojo is no longer having their bot auto-archive old reports.
So even after closing many reports, and archiving many, the # of open reports has still crept up quite steadily from release to release,
The other thing I have watched is the first public case #. This is the first report that I can see on any given day. It seems to be a purely sequential number applied to any report.
That day the first # was 69400.
Today its 73040.
This suggests in total over the one year time frame there have been 3640 reports (feature requests, bugs, and whatever other categories) It also suggests there are many confidential reports that cant be seen. What type they are or what those might constitute I wont speculate about.
But it does suggest the average number of reports per day is some where between 10 and 11.
Having a years worth of data lends itself to some analysis.
The average # of new reports (not specifically bugs) is about 11 per day.
Of those, just under 50% end up being bug reports.
As well since we know the release date of every release during that period we can see how many reports, and bug reports there are from the last release to the next.
Release 2022r2 to 2022r3 (78 days) there were 868 reports (381 bug reports)
2022r3 to 2022r3.1 (6 days) there were 97 new reports (48 bug reports)
2022r3.1 to 2022r3.2 (19 days) there were 307 reports (130 bug reports)
2022r3.2 to 2022r4 (34 days) there were 475 reports (181 bug reports)
2022r4 to 2022r4.1 (5 days) there were 45 reports (28 bug reports)
2022r4.1 to 2023r1 (98 days) there were 983 new reports (318 bug reports)
2023r1 to 2023r1.1 (13 days) there were 196 new reports (85 bug reports)
As of today 2023r2 has not shipped.
From the last release to now there have been 963 new reports & 334 bug reports.
Every release Xojo fixes many bugs, adds some new features, and makes other changes.
What their own issue tracking system says though is the the # of bugs fixed is quickly outpaced by the number of new bugs. You’ll notice on the graph that FIXED never exceeds BUGS which means there are more new bug reports than are being fixed.
So the number of open bug reports continues to grow over time. To the point they fall so far behind in fixing bugs that they archive bugs that they haven’t analyzed, reviewed, or fixed.
It seems clear from their own numbers that staying the course and doing the same things they’ve always done and in the same way they’ve always done is a losing proposition and eventually there will be so many open bugs that archiving simply has to start again. Or bug reports will simply be left to languish forever. Neither of these are great prospects.
A change in process that focuses on creating quality software from the outset, well before ANY code has been written, and extending to the point it’s released that includes extensive unit testing, REQUIRING unit tests for every piece of code, and expanding the unit tests as bugs are reported & fixed would be a great start.
Expanding the team so developers have the time to create all this is necessary. Not just nice, but necessary.
And someone needs to be “the code god” and have few development duties & more oversight and review every last check in and make sure they do meet all standards of code (format, unit tests, integration tests, etc)
And an expansion of the internal QA team is needed
The other thing I followed was the beta releases as far as possible.
What I observed is that, for every beta release the # of participants starts out relatively high. If thread reads is an indicator of the actual # of downloads of a beta release. Since Xojo posts expiring download links it seems thread reads is a reasonable proxy for actual download. It should at least trend in the same direction as actual downloads.
Most betas seem to start with somewhere between 200 and 300 reads (downloads?) of the initial beta release. The one outlier here is Android which started with > 500 reads. However they all trail off and end up with somewhere around 150 or so.
There are supposedly hundreds of Testers yet the actual number that seem to test is a fraction of the total (about 1/6th)
Many testers haven’t even logged in during the latest betas. About 50% of all members of the testers group haven’t logged in since the beta for 2023r2 started.
The same trend was true for Android testing where about 20% of the entire testers group hadn’t logged in for the entire time Android was in beta (560 days)
Now you might ask yourself “if Norm can see this from outside surely someone at Xojo is watching these trends as well ?”
To be honest I have no idea if they are. Or arent.
Either way the #’s done paint a picture of a process that is getting better, or of a very involved & engaged testing process.
However, there are things I am sure of.
I’m sure this will get read.
I’m sure I’ll get an email about it and how it violates some rule Xojo has never posted or has just invented for me.
And nothing will change 🙁