Observations on the Beta Testing Process

I have been reading all the messages regarding issues and problems with build 114800. I am running Trainz TR2019 plus. I guess I am fortunate in that all the routes I run work without a hitch. I was a beta tester so several issues along the way, so I submitted a bug report and they got fixed.

However I probably use only 30% of the functionality of Trainz so some of the issues identified would not even cross my radar when testing. It may be the same for other beta testers. As an example there were several apparent problems with in-cab controls. I have never used them so I didn't think about testing these.

There apparently were problems with reversed engines, but that apparently only affected people not using Trainz plus.

I don't know how many people are involved in beta testing. I do know that several weeks before build 114800 went public there were no major issues discovered and reported (show stoppers), hence the decision to go public. So one conclusion is that build 114800 is performing well for the beta testers.

Another conclusion may be that the beta testing process needs to be broadened to get a more meaningful sample of users who have different versions of Trainz (Plus, Platinum, Basic). Also there may be a need to include people who are really pushing the game to the limits and would be the most likely to find errors in scripting etc. One of the big issues identified and fixed was the portal issue which exist in build 111951. Along with this issue was a problem with the Central Portal Control, which was identified and fixed.

Trainz is a very complex simulation program and N3V has done a great job in moving it along. I doubt whether the programmers spend a lot of time building and running routes. They need more help in identifying these issues so that they can be fixed. The question really is how does that happen?

I agree with most everything you say. But I have to point out that release candidate versions are made public in the forum for anyone to test. I suspect that few people bother to do so because they don't want to mess up their working install. That is certainly the way that I feel and I don't bother to patch until others have reported their experiences. I believe that proper testing is hard work and I thank all the users that are willing to pay for the privilege of being able to test. I'd rather spend my time enjoying the game. Now I understand that N3V can't fix what they can't reproduce and I imagine they get a lot of bug reports that fail to provide enough detail for them to reproduce the problems. That is always going to be a problem in an open beta system.

William

I was also a Trainz Plus SP4 beta tester and before I upgraded to Plus I was a TRS19 Platinum Edition SP beta tester.

You have pointed out several drawbacks of the beta testing system. Obviously the more beta testers you have the more likely you are to discover problems. If potential testers see it as "risky" i.e. it will interfere with my route development, or time consuming then fewer users will apply.

One issue I found is the advice that beta testers often get is not to undertake any development work during beta testing. Yet that is an area which has the most potential to discover problems - when you are "pushing the envelope", so to speak. When I was beta testing TANE SPs I ignored that advice and continued with development work in the beta version. I did find bugs that no-one else seemed to have reported but I also lost a lot of work and time and had to go to my backups - some of which were corrupted by one of the early SP betas. A lot of swearing (at my own stupidity) and frustration, but I knew the risks when I became a beta tester. I did not blame N3V for that.

With TRS19/Trainz Plus I decided to play it safe. I did all my development work in the stable (non-beta) TRS19/Trainz Plus on one computer and copied the work (via .cdps) to the beta on another computer and tested what I had done there. I discovered a few minor bugs. Only when the final SP4 release came out which made me continue my development work in SP4 did I discover more serious (or annoying) problems.

Another issue is reproducibility by others. There have been a few bugs I have reported in TANE and TRS19 where it seems that my reports were the only ones. Most were acknowledged as bugs by N3V QA but it is highly likely that because no-one else had experienced them they were given a lower priority - and I can fully understand this. You prioritize your scarce resources to fixing those issues that are seen as more common or urgent. Plus they were bugs encountered in very obscure and probably seldom (if ever) used situations. I am pleased to report that at least one of them has since been fixed. It takes patience!

One issue I discovered early in TRS19 beta testing I did not report as a "bug", it seemed to be more of a "curiosity" than a "bug". I have since reported it and provided more documentation to N3V QA but I did document it on the Trainz Wiki. Only recently (today) did I get feedback from other users that prompted me to report it. Perhaps if I had acted sooner it may have been fixed in SP4, or not!

I tend to be timid, even when beta testing, and I don't always "push the envelope" as much as it should be pushed. But now that I have two computers with enough oomph to run Trainz Plus at somewhere near its potential, more or less, that may change in future.

My thoughts and experiences.
Peter

I don't have Plus, just a basic install.
On one of my 3 working installs (each on its own SSD) I updated to SP4 because it had been Beta tested by those that could and I put my trust in N3V that it was good enough for release to the public.
I now find myself with a faulty SP4 and reporting those faults that I have found, so I am not a Beta tester I am now a 'conscripted' Public Release tester.
Don't get me wrong, I am happy to report what bugs that I do find, but, maybe if those members with a basic install (there are many heavily involved users of TRS19 that are) were allowed to beta test, we could have helped N3V before SP4 was released.
I tried plus and did not find the UDS useful (in my opinion) and I very rarely purchase DLC.
I do buy a 3 Month FDC to help the upkeep.

Beta testers know the risk factor and need to take measures to minimise the risk, but now it's risky for the public to install a Service Pack and that's not good for N3V's reputation.

cheers,
Graham

Beta testers have no access to the source code. Many have little or no idea as to graphic or physics engines, databases or compilers. They cannot be sure which routines affect other (sub)routines. They are expected to operate as a consumer, not a coder and therein lies the issue. When bugs are discovered, unless they are in specific coder lingo, and meticulously described debugger outputs, it falls on deaf ears. I was quick to report the reversed assets bug and got flipped off as having a bad system and emotional issues by fanbois that love to hear themselves talk.

I won't make that mistake again.

Moreover, how this slipped past non-Plus users is negligence. It's disquieting to know dev didn't see something so blatantly obvious the instant Surveyor is loaded.

Coder v bean counter. Who's to blame? My guess, it's almost always the latter.

Many things should have been caught even before it is in a beta
-Test-track, this takes 2 minutes to find/test for anyone
-Sideways moving sliders in Driver list that overlaps text
-the shock movement in camera mode 3
-the F2 bugs, it requires 1 pc at n3v with standard/regional (or a script/account to limit rights)
-New water although nice is still under development, needs serious warnings so users don't ruin years of work


I never blame beta testers, but thank them for all hard work
It would help if there was a standard quick check list, that every beta tester could do in lets say 2 hours
and besides that detailed info what has changed, so a specific build beta test can concentrate there


I prefer to be at the begin of the line (generate/propose ideas) and at the end (Omega tester)
greetings GM

I tested as well, although, not as the furiously as I normally do due to outside obligations this time around. For my testing, I have a duplicate of my working setup that I update to the beta version du jour. This gives me a chance to vet out any faulty content that may appear on the horizon later, known issues not counted of course. Having a duplicate setup means I can test to my heart's content without worrying about messing up my working installation. This setup also allows me to compare operations so I can see if an issue occurred in a previous version, or is a new one in the current test.

As Paul pointed out in the original post, there are many things reported that we never saw, and likewise we never can catch everything. This is why we need as many eyes on the beta and problems reported to get them on the repair list.

I agree that there should be more internal testing prior to release to beta. If the code that affects a subsystem has been changed, test it before moving on. This is a fault of N3V's internal process and that needs to be addressed. As I said before, this is pretty obvious that the coder(s) do not use the product in any fashion. Back in the Brew Crew heyday, the developers were required to build routes and put the product through its paces. This didn't have to be an elaborate setup, but only a way to test the functions such as splines, engine interiors, the Test-track, and other early pre-beta anomalies. I know N3V is short-staffed, but that's no excuse for a poor product release. That's like bringing a car to a repair shop and finding the oil-cap missing. When asking the mechanic about that, he says he's busy and doesn't have enough employees.

dleblanc said:

Moreover, how this slipped past non-Plus users is negligence. It's disquieting to know dev didn't see something so blatantly obvious the instant Surveyor is loaded.

Coder v bean counter. Who's to blame? My guess, it's almost always the latter.

Click to expand...

IMO we need more Non Plus users to actually test rather than suddenly appear on the forum complaining about things not being picked up by beta testers, those who think they could do a better job need to step up and participate in testing so these things get dealt with quicker. It's easy enough to set up a second install for testing so you don't break anything in existing routes etc, if you have the disk space of course.

Need to remember that N3V does not have a large army of coders such as the major software houses have at their disposal, heck even Microsoft can't release anything without bugs, first Win11 cumulative update had 74 bug fixes! including major issues with Ryzen CPUs causing a performance hit of up to 15%, thankfully now fixed however wasn't on the released build.

I adopt the principle that just because something doesn't affect me, doesn't mean there isn't a problem for others, we all have a vast multitude of different assets, routes, various scripts, preferred methods of doing things and even hardware so one should not dismiss any reports of problems out of hand, just because they are not affected.

IMHO, all versions of Trainz (regional, standard, plus, platinum) are basicly the same code
the difference is scripted with Product rights.
You start the game, you get checked what product rights you have according to your account
and the game selects a subset of features enabled/diabled.


If there would be an option to select a lower product right
a plus user could emulate how a regional/standard would be (no UDS but F2 Quickdrive)
limiting beta to plus only, in the end gives us the current situation


Many bugs/problems were detected during beta FI:
-swithes in cabins, Rbrooks
-wheel rotation p_dehnert


if not taken serious at that time, it leads to what we have now
Something does not go, by winking it away, saying this is current design, or say content issue or local issue
it lingers on and in the end make users leave or frustrated.

It was mentioned that it's important to log the issues in the bug-report, and if that's not done then the issue won't be caught. This might have been part of the problem with catching these errors, or bugs in general, because people only post them here and not the official bug report. The problem here is the developers don't read the forums because they're busy chasing bugs.

I tried to give an example of developing something in the Thread Set-Driver-Condition-Rule
Not only share everything that has changed, but also give insight in the thoughts behind it
then read and take all feedback in, not only from the thread on this forum, but also on other forum
and mail and PM's,


ofc. for a much more complex development as Trainz this is harder.
A bug report in this case is good, but other users can't see IF it is reported, HOW it is reported
and what has been done with it or the reply it has gotten. hope this makes sense.

If you report a bug already reported that's being dealt with you get a reply to that effect and it's reference number, leastwise I have. However IMO the more that report the same bug the more likely it is taken as serious instead of a one off or operator error etc.

It would be interesting to know how many Plus beta users actually beta test. I suspect there is a large number who go beta just to get the "latest features" and who do no testing at all. I'm not the most "aggressive" tester there is and I have turned in reports. But it seems that it's always a select FEW people who report and/or mention "bugs" in reply to Tony here in the beta forums. It seems to me that Plus (beta) is leveraged more to help with revenue than to help find more beta testers. But I could be very wrong. I have no idea how many different names appear on Bug Reports vs how many users of the Plus Beta stream there are.

wreeder said:

I agree with most everything you say. But I have to point out that release candidate versions are made public in the forum for anyone to test. I suspect that few people bother to do so because they don't want to mess up their working install.... That is always going to be a problem in an open beta system.

William

Click to expand...

Hi William - I didn't know that release candidate versions were made public in the forum for anyone to test. Where does one find it or was it only available to plus users/subscribers?
I would say that the current beta testing arrangement is a 'closed' beta system, available to subscribers only.

euromodeller said:

I don't have Plus, just a basic install.
On one of my 3 working installs (each on its own SSD) I updated to SP4 because it had been Beta tested by those that could and I put my trust in N3V that it was good enough for release to the public.
I now find myself with a faulty SP4 and reporting those faults that I have found, so I am not a Beta tester I am now a 'conscripted' Public Release tester.
Don't get me wrong, I am happy to report what bugs that I do find, but, maybe if those members with a basic install (there are many heavily involved users of TRS19 that are) were allowed to beta test, we could have helped N3V before SP4 was released.
I tried plus and did not find the UDS useful (in my opinion) and I very rarely purchase DLC.
I do buy a 3 Month FDC to help the upkeep.

Beta testers know the risk factor and need to take measures to minimise the risk, but now it's risky for the public to install a Service Pack and that's not good for N3V's reputation.

cheers,
Graham

Click to expand...

Hi Graham - I am also just a basic install user. The subscription model is not for me. Tried it once and found no benefit. I don't fully understand N3Vs reasoning behind their beta testing arrangement. It's a bit of a cheek to expect customers to pay a monthly fee to be able to test their product. I am in limbo at the moment in that my copy of Trainz 19 was purchased via Steam. Steam does not give you the option to reject an update and does not permit you to backup the program until the update is complete. So at the moment I am stuck and have not played the game since SP4 was released for fear of becoming a 'conscripted public release tester' like yourself. I have manually backed up my Trainz install and Data folders just in case.

clam1952 said:

IMO we need more Non Plus users to actually test rather than suddenly appear on the forum complaining about things not being picked up by beta testers, those who think they could do a better job need to step up and participate in testing so these things get dealt with quicker. It's easy enough to set up a second install for testing so you don't break anything in existing routes etc, if you have the disk space of course.

Click to expand...

Hi Malc - Can non-Plus users beta test then? If so, how do we go about getting added to the beta test team?


To dleblanc - you were quick to report the reversed assets bug. If this forum engine supported thumbs down it would have got one from me for the person's reply who flipped you off.


We know now that Tony Hilliam has provided a link for the SP3 installer. This indicates to me the serious defects in the SP4 release. I cannot remember having a link to a previous installer before. Why not make the installer available from the MyTrainz dashboard instead of tucking it away in a forum? In fact all versions, base and service packs should be available to the purchaser. It should be for the end user to decide which version works best for them.

To Tony - thank you for the SP3 link, but will this work for Steam users? If not then what do we do? I'm stuck in limbo at the moment.

Best regards to everyone,

John

@johnarron

The Beta testing was open to all TRS19 users this time not just Plus, it's normally announced in the TRS19 Beta forum https://forums.auran.com/trainz/forumdisplay.php?112-TRS19-amp-Trainz-Plus-Beta and the Newsletters.

Not so long ago there was a beta test for TRS19 that didn't include Plus users, all depends on what N3V are doing as to who is required to test what.

And on rare occasions one has to apply to test, that is often hardware related from what I remember, again usually announced in the Newsletter.

Plus there was a test for Android and such going on that I think has now finished.

You don't have to have the full Gold Plus version for access to the Plus new feature testing as there is the cheaper Plus only version that doesn't have all the "free" DLC with it.

G.M. said:

It would help if there was a standard quick check list, that every beta tester could do in lets say 2 hours

Click to expand...

A good idea and one that was implemented for an early T:ANE SP beta but not since. There was a long list of steps that they recommended beta testers perform - basically going through all the Surveyor tab tools one by one. It took some time to complete but it did yield the cause of at least one bug that had eluded all the bug hunters.

I probably test up to 10% of everything in the betas. Just too much going on. I don’t think people really understand how complex software is, then to add all the parsing of the files, models etc needed for the game and make is backwards compatible with old content. There just isn’t that much trains enthusiasts to have a large company. They tried to ensure there is not any show stopping bugs.

so much software has bugs and yet much of that software is by large companies.

I tend to prioritise what I'm testing to what I use / need to be working plus I check any problem posted on the forums to see if I'm affected by it. Testing this time around, not helped as I was also on the Windows 11 insider testing, underwent a hardware upgrade and reinstall of everything and was also testing the TRS19 beta and SP3 in both Win10 and 11 plus trying to get my head around Blender and PBR.
Good news excluding the SP4 bugs, is no problems with Win11 and Trainz, pretty much the same as in Win10 albeit slightly improved frame rates and quicker loading time etc here.

I have Plus primarily to support N3V, while I can afford it, not for the free Payware you get while subscribing. I also have or rather had the normal non Plus version, unfortunately It updated to Plus due to me having Plus permissions.